Slavery 101 – The Hudson Edition

Hudson is excellent at setting up distributed clients to help with the build, however there are a couple of gotchas that people should be aware of before attempting to do it.

Windows client issues

Shared path problems

When Hudson launches a build process on the slave, it also passes through the paths to certain executables from the server to the slave. This can be a major problem. If you have homogenous environments, you have to make sure you have the set up on master and slave IDENTICAL in terms of paths and program locations.
If, however, you have a heterogenous setup (say a Linux master and one or more Windows slaves), there is more of a problem. When you try and run, you get the following in the job’s console:

ERROR: No classworlds*.jar found in usrmavenapache-maven-2.0.9 -- Is this a valid maven2 directory?

Obviously, this directory is not valid in Windows. There are two solutions:

  1. Reinstall maven (and, as you will see later, Java) in the paths the server expects
  2. Use junctions, the Windows symlink equivalent supplied by sysinternals.

Creating junctions

Junction is a Sysinternals utility to allow Windows systems to create symbolic links (completely different from shortcuts), created by Mark Russinovich. You can get it (and more detail about what it does) from Microsoft TechNet

On the same drive as the slave’s Hudson FS Root (the default we use for slaves is d:slave, so I shall use this as the example), make the linux parent paths to match the master thus:

mkdir usrmaven
mkdir usrjava

Now create the junctions (symlinks) to the real locations of Maven (and Java)

junction usrmavenapache-maven-2.0.9 "D:Program FilesApache Groupapache-maven-2.0.9"
junction usrjavajdk1.6.0_06 D:javajdk1.5.0_10

Obviously you have to place your locations of Maven and Java here

Yes, I know this sucks. Yes, I know that I’m fooling Hudson into using Java 1.5 instead of just installing Java 1.6 but we have had issues with JNLP and installs of multiple versions of Java. As such, since we’re not compiling any 1.6 specific functionality, this will work.

Starting the client manually

This is not a problem; simply log in to the windows box using remote desktop and browse to the Hudson slave’s page.
Now launch the JNLP file (provided you have already set up any necessary junctions on the client)

Starting the client automatically

This needs to be done through ssh. We tried other options, but they all buffer the i/o on stdout and stdin whilst ssh does not, and Hudson communicates with it’s slaves using the stdout/stderr and expects immediate responses. As such, sshd must be installed on the Windows slaves. Since most have cygwin installed already, it is a fairly trivial matter to install and configure the sshd package.

Installing and configuring the Cygwin sshd on Windows

Two additional packages have to be added by re-running setup.exe from the cygwin site and selecting the following options:

Admin --> cygrunsrv
Net --> openssh

This will install a few other required packages too. That’s fine.

Now open a Cygwin bash command shell and run the following command:

ssh-host-config

Answer ‘yes’ to everything except the following question, to which you must answer no:

*** Query: Should privilege separation be used? (yes/no)

When the process is complete, you can start sshd as a Windows service. You can do this through the Control Panel or the command line with

net start sshd

It should be automatically added as startup type ‘Automatic’, and so will come up with Windows at normal boot time.

Setting up the Windows users in Cygwin

Because we use domain logins, the regular cygwin install does not set them all up by default. Normally this is not an issue. Here, however, we need them and their groups to be defined correctly. Fortunately, cygwin can do this for us.

Start a cygwin bash shell and issue the following (obviously replace username with the domain user you want to use to run the slave commands):

mkpasswd -l > /etc/passwd
mkpasswd -d | grep ^username >> /etc/passwd
mkgroup -l -d > /etc/group
  • Note: if you’re not on a domain, you can not worry about the ‘-d’ switch on mkgroup and the entire middle command in the above code; all of your users will be picked up by the ‘-l’ versions.

Back to linux for a minute or two…

We now need to register the newly set up slave machine in the known_hosts file on the Hudson master box.

Log in to the Hudson master with ssh, as the user under which Hudson runs.

Now connect to the new slave via ssh, as user username like this:

hudson@hudson.master:~$ ssh username@slave
Warning: Permanently added the RSA host key for IP address 'xxx.xxx.xxx.xxx' to the list of known hosts.
username@slave's password:

You can now log out again. CTRL-D is the fastest way to log off!

.ssh and authorized_keys

Now on the Windows slave we need to add the Hudson Master’s ssh public key to allow it(master) to issue commands without being prompted for a password. The key is plain text and is both shown here and attached as an ‘authorized_keys’ file.

Change into the username user’s cygwin home directory (C:cygwinhomeusername on most machines; YMMV) and make a .ssh folder.

Now create a text file called authorized_keys, all lower case, just like that, and paste in the key:

ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAn4qSzsbrVdW4HB/ap/4I6ncmuQKsTWpHmur4EsP1SCsXq7SWzd8eoupiW/c9gCXw2q1opNgAkxpApjNfuKBfcsEDlqwiDeylyZqBhafwHNDrRpqcggQNldz154mooQyM3xs/rQCsZ49AygyL9cASu70X8+incw3m0XvhaVstl3ZcLh8/1WQueq+1EM6RJv6J5LusPHR+/eEfev2Y7YX3Mws+Qyr0du+LjvhHbBgajy+0Ew4jr7j62F3zsIAk5cMWRVRf0espuHMnmkjbMrckMcKJi9aM+0GILrhunr4Y7nk+ZVvA5rzlcmnC5qjgXZgjvjAs5A64b67PpF8oVa912w== hudson@hudson.master

Save and close the file.

If you want to, you can test this by repeating the above Back to linux for a minute or two… step. You should not get asked for a password.

The startSlave.cmd file

In the username user’s cygwin home directory (remember: C:cygwinhomeusername on most machines), either save the attached startSlave.cmd file or create a Windows command file called startSlave.cmd with the following contents:

cd /d d:slave
java -version:1.5 -jar slave.jar
  • NOTE: the ‘-version:1.5’ command let’s javaws know that a minimum java version of 1.5 is acceptable; it will pick the highest number above that, but won’t run if you only have Java 1.4 installed.

Defining the slave on the Hudson master

Fill in the fields as appropriate. A good standard would be to name the slave the same as the pod it is on.

Note the Launch method and Launch command fields. The launch command is the ssh command that will be issued on the slave;

ssh username@slave cmd /c "startSlave.cmd"

Starting the Hudson slave

As soon as you save the definition, Hudson will attempt to start the new slave’s listener. If it stays idle, you’ll have to check the log and find out why!

WARNING! DANGER, WILL ROBINSON!

If you are using Subversion or CVS, please make sure you have the appropriate command line clients installed and pathed in, otherwise the build may fail at the last step.

It's only fair to share...
Share on FacebookGoogle+Tweet about this on TwitterShare on LinkedIn

Leave a Reply