Simple Daemons running scripts for a Hadoop cluster as authored by me

Imagine you have 100 nodes in a cluster and you have to start specific daemons I mean tasktracker or datanode in those machines. You  can manually start those daemons in the nodes or if you have password-less ssh setup form the master to all slaves, (as shown in my post http://sunayansaikia.blogspot.com/2011/06/apache-hadoop-clouderas-distribution.html), how about starting those daemons from the master alone! Interesting, I suppose. Note: if you have not done the password-less part, you have to enter password manually for every machine when you try to ssh to them.
    If you have installed the Apache hadoop manually from the tar, you get scripts named start-all.sh and stop-all.sh, start-dfs.sh and start-mapred.sh to start all the required daemons in their respective machines.

What if you have installed hadoop using  the cloudera's distribution. You have to start the daemons with like service hadoop-0.20-daemon_name {start/stop/restart} in all the slaves. This would mean you have to start the daemons individually in each machine. I haven't yet gone into much detail to enable or use the {start/stop}-all.sh scripts in Cloudera's distribution but I tried to develop my own scripts to do the trick for me, starting necessary daemons in all the node in a cluster. So, here is what I have done.

Save the below mentioned text in a filenames slavesstart.sh or whatever you like with an .sh extension to start required daemons in slave nodes, slave1...slaveN where slave1...slaveN are the slave ip aliases you setup in /etc/host in master (namenode running) machine. After saving make sure you give the execute permission for execution for the script execution. You can try this for giving the execute permissin  chmod +x slavesstart.sh
.Then to execute the script try ./slavesstart.sh. Done!

#!/bin/bash
#Script to start tasktracker and datanode daemons on slave machines Cisco POC

#author Sunayan Saikia
#edit the "slave" part with the slave names you configured in /etc/host of the master machine

for S in {1...10}; do
ssh -l "root" "slave$S" "service hadoop-0.20-tasktracker start";
ssh -l "root" "slave$S" "service hadoop-0.20-datanode start";
done

Similarly, to stop daemons in slaves use this scripts containing text as below and save it as
slavesstop.sh,


#!/bin/bash
#Script to stop tasktracker and datanode daemons on slave machines Cisco POC

#author Sunayan Saikia
#edit the "slave" part with the slave names you got writing below lines for #every slave machine

for S in {1...10}; do
ssh -l "root" "slave1" "service hadoop-0.20-tasktracker stop";
ssh -l "root" "slave1" "service hadoop-0.20-datanode stop";
done
     

For master node daemons start and stop use these text below and save it as

masterstart.sh and masterstop.sh

#!/bin/bash
#Script to start namenode, jobtracker and secondarynamenode daemons on master
#machine Cisco POC

#author Sunayan Saikia

service hadoop-0.20-namenode start
service hadoop-0.20-jobtracker start
service hadoop-0.20-secondarynamenode start

And,

#!/bin/bash
#Script to stop namenode, jobtracker and secondarynamenode daemons on master
#machine Cisco POC

#author Sunayan Saikia

service hadoop-0.20-namenode stop
service hadoop-0.20-jobtracker stop
service hadoop-0.20-secondarynamenode stop


Update: To setup password-less ssh from a single master machine/PC to multiple machines/PCs you can use my script mentioned here http://sunayansaikia.blogspot.com/2011/06/script-to-set-password-less-ssh-to.html