I’ve been playing with more clustering as I prepare for a RedHat class in August and figured I would write about ClusterIt. I was looking to run a few commands on about six servers and went looking for a simple solution. I believe ClusterIt provides an elegant solution for very little work.

Commands
Here is a list of commands and their description from their respective manpages.
dsh – Run a command on a cluster of machines as defined in the CLUSTER environmental variable.
dshbak – Takes input from the dsh command and formats it to look nicer for the user.
run – Run a command on a machine at random.
rseq ñ Run a command on a sequence of machines or cluster.
pcp – Copy a file to a number of machines.
pdf – Display free disk space across a number of machines, can be for a single filesystem or the entire machine.
prm – Delete a file, directory or list of files on a number of machines.
rvt – Remote terminal emulator.
clustersed ñ Quickly dissect cluster files, used to cut individual groups out of a cluster file.
dtop – Used to remotely monitor and display top information, this program segfaulted on my system.

There are also some more involved commands, the daemons for these must be set up on the remote machines.
barrier ñ Used to synchronize execution of commands on slower and faster machines. When a barrier is set, the process is not released until all of the nodes or processes have met the barrier condition.
barrierd ñ The daemon portion of barrier that accepts connections from the client program barrier.
jsh ñ Run scheduled commands on remote machines.
jsd ñ A simple command scheduling daemon for remote execution.

Installation
The first thing you need to do is make sure you have ssh password-less login set up. I went to our network management server and added a couple of the servers that needed to be able to run commands remotely.

In case you are doing this from scratch, here is the sequence of commands. Generate private/public keys on your management server A.

ssh-keygen -t dsa
press enter when it asks for the filename
press enter when it asks for the passphrase (yes, a blank passphrase)

This will generate two files: ~/.ssh/id_dsa and ~/.ssh/id_dsa.pub. You now want to allow access from this server (A) to the remote server (B) by putting the contents of ~/.ssh/id_dsa.pub from A into ~/.ssh/authorized_keys2 on B.

cat ~/.ssh/id_dsa.pub | ssh B 'cat >> ~/.ssh/authorized_keys2'

Make sure permissions are correct and are not writable or readable except by the owner. Do this on both server A and B.

chmod a-x,go-w,o-r ~/.ssh/*

And to verify it works.

ssh B ls -la

Now it’s time to install ClusterIt. I like to have a suite of programs installed in a common directory but don’t want to modify my MANPATH or worry about other nonsense. This is how I installed ClusterIt.

./configure --bindir=/usr/local/clusterit
make 
make install
cd /usr/local/clusterit/
ls

If you read the manpage for dsh or one of the other program in ClusterIt you can see a number of environmental variables and how to set up the ClusterIt environmental variables and files. A snippet of the manpage for dsh.

ENVIRONMENT
dsh utilizes the following environment variables.

CLUSTER            Contains a filename, which is a newline separated 
list of nodes in the cluster.

RCMD_CMD           Command to use to connect to remote machines.  
The command chosen must be able to connect with no password to 
the remote host.  Defaults to rsh

 ...removed for brevity...

FILES
The file pointed to by the CLUSTER environment variable has the 
following format:
           pollux
           castor
           GROUP:alpha
           rigel
           kent
           GROUP:sparc
           alshain
           altair
           LUMP:alphasparc
           alpha
           sparc

This example would have pollux and castor a member of no groups, 
rigel and kent a member of group 'alpha', and alshain and altair a 
member of group 'sparc'.  Note the format of the GROUP command, 
it is in all capital letters, followed by a colon, and the group name.
There can be no spaces following the GROUP command, or in the 
name of the group.

As a result I set up my .bashrc with the following options for ClusterIt.

CLUSTER=/etc/clusterit/servers
export CLUSTER

RCMD_CMD=/usr/bin/ssh
export RCMD_CMD

PATH=$PATH:/usr/local/clusterit
export PATH

Make sure you re-source your .bashrc.

source ~/.bashrc

And I have a simple /etc/clusterit/servers file:

cat /etc/clusterit/servers
B
C
D

Now to test.

dsh uptime
B:  17:44:26 up 24 days,  6:32,  5 users,  load average: 0.02, 0.01, 0.00
C:  17:46:56 up 443 days,  9:53,  2 users,  load average: 0.00, 0.00, 0.00
D:  17:46:56 up 443 days,  9:52,  1 user,  load average: 0.00, 0.01, 0.00

Testing
And finally run some commands.

man pcp
pcp /usr/local/bin/script.sh /usr/local/bin/script.sh 
dsh /usr/local/bin/script.sh -d /tmp
dsh scp /tmp/output.txt user@A:/tmp/

That last command you must have password-less login from the ClusterIt servers back to your management server.