Hadoop pseudo-cluster installation

Install Java and cloudera yum repo

yum install java-1.6.0-openjdk.x86_64
curl -O http://archive.cloudera.com/redhat/cdh/cloudera-cdh3.repo
mv cloudera-cdh3.repo /etc/yum.repos.d/

Ensure that you have hostname and localhost entries in /etc/hosts

comment out ipv6 entry

Create hadoop user and group manually

Create "hdfs" and "mapred" user with group "hadoop"
groupadd hadoop
useradd -G hadoop hdfs
useradd -G hadoop mapred
passwd hdfs 
passwd mapred 

Install hadoop packages

yum install hadoop-0.20
yum install hadoop-0.20-conf-pseudo

Create directories for hdfs files and mapred temporary files as root

mkdir -p /data/hadoop
chown -R hdfs:hadoop /data/hadoop

as hdfs

chmod -R 755 /data/hadoop
mkdir -p /data/hadoop/cache
chmod 777 /data/hadoop/cache
chmod +t /data/hadoop/cache

mkdir -p /data/hadoop/tmp
chown hdfs:hadoop /data/hadoop/tmp
chmod 777 /data/hadoop/tmp

mkdir -p /data/hadoop/nn
chown hdfs:hadoop /data/hadoop/nn

mkdir -p /data/hadoop/dn
chown hdfs:hadoop /data/hadoop/dn

mkdir -p /data/hadoop/snn
chown hdfs:hadoop /data/hadoop/snn

As mapred:

mkdir /data/hadoop/cache/mapred-tmp
chown mapred:hadoop /data/hadoop/cache/mapred-tmp

mkdir /data/hadoop/cache/mapred-local
chown mapred:hadoop /data/hadoop/cache/mapred-local

mkdir -p /data/hadoop/mapred-system
chmod 777 /data/hadoop/mapred-system
chown -R mapred:hadoop /data/hadoop/mapred-system

Move and store default configuration to another directory as root.

mkdir -p /etc/hadoop/conf.pseudo.copy
cp /etc/hadoop/conf.pseudo/* /etc/hadoop/conf.pseudo.copy/
cd /etc/hadoop/conf.pseudo/

Edit various configuration files with entries for the directories made above
hdfs-site.xml:

    <property>
        <name>dfs.replication</name>
        <value>1</value> 
    </property>
    <property>
        <name>dfs.name.dir</name>
        <value>/data/hadoop/nn</value> 
    </property>
    <property>
        <name>dfs.data.dir</name>
        <value>/data/hadoop/dn</value> 
    </property>
    <property>
        <name>dfs.permissions.supergroup</name>
        <value>hadoop</value> 
    </property>

core-site.xml:

    
    <property>
     <name>hadoop.tmp.dir</name>
        <value>/data/hadoop/tmp/${user.name}</value> 
    </property>
    <property>
        <name>fs.checkpoint.dir</name>
        <value>/data/hadoop/snn</value> 
    </property>

mapred-site.xml:

   
    <property>
        <name>mapred.local.dir</name>
        <value>/data/hadoop/cache/mapred-local</value> 
    </property>
    <property>
        <name>mapred.temp.dir</name>
        <value>/data/hadoop/cache/mapred-tmp</value> 
    </property>
    <property>
       <name>mapred.system.dir</name>
       <value>/data/hadoop/mapred-system</value>
    </property>

You may have to set the JAVA_HOME sometimes. Mostly the scripts should be able to figure that out.

vi /etc/profile and vi /usr/bin/hadoop
export JAVA_HOME=/usr/lib/jvm/java-1.6.0-openjdk.x86_64

You can also add it to : /usr/lib/hadoop-0.20/bin/hadoop-config.sh if "sudo service hadoop-0.20-namenode start" cribs about java_home

Format Hadoop file system with command:

sudo -u hdfs hadoop namenode -format

Now you can start services with the following commands:

/etc/init.d/hadoop-0.20-namenode start 
/etc/init.d/hadoop-0.20-secondarynamenode start
/etc/init.d/hadoop-0.20-datanode star
/etc/init.d/hadoop-0.20-jobtracker start
/etc/init.d/hadoop-0.20-tasktracker start

or you can configure services to startup during boot time:

sudo chkconfig hadoop-0.20-namenode on
sudo chkconfig hadoop-0.20-jobtracker on
sudo chkconfig hadoop-0.20-secondarynamenode on
sudo chkconfig hadoop-0.20-tasktracker on
sudo chkconfig hadoop-0.20-datanode on

Thats it and try testing your installation with a few simple hadoop commands.