>>>>>>>>>>>>>>>MULTI NODE CLUSTER SETUP <<<<<<<<<<<<<<<<<<
Note: All Nodes should be in the same n/w. HW to set the network.
STEPS For Multi Node Cluster setup :
1.1 Remove the known hosts to the systems:
sudo mv .ssh/known_hosts .ssh/known_hosts_old
1. First check the IP address of all the machines:
$ifconfig
2. Ping the Slave machine from the Masters and from the Masters to the
Slaves to check the connections.
$ping -b IP address (192.168.1.1)
3. Register the following in the MASTER MACHINE ONLY.
sudo gedit /etc/hostname
master
3.1. Register the following in the SLAVE 1 MACHINE ONLY
sudo gedit /etc/hostname
slave1
3.2 Register the following in the SLAVE 2 MACHINE ONLY
sudo gedit /etc/hostname
slave2
4.1 Make the following entry in the MASTER MACHINE ONLY.
sudo gedit /etc/hosts
192.168.1.6 master
192.168.1.7 slave1
192.168.1.5 slave2
4.2 Make the following registry in the SLAVE 1 NODE ONLY.
sudo gedit /etc/hosts
192.168.1.6 master
192.168.1.7 slave1
4.3 Make the following registry in the SLAVE 2 NODE ONLY.
sudo gedit /etc/hosts
192.168.1.6 master
192.168.1.5 slave2
7.Configuring ssh
[SSH Should be configured in MASTER AND ALL THE SLAVE MACHINE]
$sudo apt-get install ssh
$sudo apt-get install rsync
ssh-keygen -t rsa -P ""
ssh-copy-id -i /home/user/.ssh/id_rsa.pub userame@master
[Only For Master Node]
ssh-copy-id -i /home/user/.ssh/id_rsa.pub username@slave1
[Only For Master Node]
ssh-copy-id -i /home/user/.ssh/id_rsa.pub username@slave2
[Only For Master Node]
$whoami (cmd to find the username)
8.Check whether the ssh has been configured successfully or not : [from
MASTER MACHINE ONLY]
ssh username@master
ssh username@slave1
ssh username@slave2
9. Configure the masters file with the name of the master node
(from MASTER NODE ONLY)
cd /user/home/downloads/hadoop-1.0.3/conf
sudo gedit masters
master
10. Configure the slaves file with the name of all the slaves node name
(From MASTER NODE ONLY)
cd /user/home/Downloads/hadoop-1.0.3/conf
sudo gedit slaves
master
slave1
slave2
11. Configure the following files
( From MASTERS AND ALL THE SLAVE NODE)
cd /user/home/Downloads/hadoop-1.0.3/conf
sudo gedit core-site.xml
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://master:54310</value>
</property>
</configuration>
sudo gedit hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
</configuration>
sudo gedit mapred-site.xml
<configuration>
<property>
<name>mapred.job.tracker</name>
<value>hdfs://master:54311</value>
</property>
</configuration>
12. Format the Namenode
(FROM MASTER NODE ONLY, MASTER NODE AND ALL THE SLAVE NODES)
cd /user/home/downloads/hadoop-1.0.3/
bin/hadoop namenode -format
ssh username@slave1
cd /user/home/downloads/hadoop-1.0.3/bin/
bin/hadoop namenode -format [From master node only]
exit
ssh user@slave2
cd /user/home/downloads/hadoop-1.0.3/bin/
bin/hadoop namenode -format [From master node only]
exit
cd /user/home/downloads/hadoop-1.0.3/bin
bin/start-all.sh
14.For checking daemons are working fine :
jps
