Continuing my plans to set up an IPVS high-availability LAMP stack on EC2, I needed to add the kernel modules for IPVS. I have been using the CentOS machine images provided by RightScale, which have unneeded services disabled and, although they are set up to work with RightScale's software, work very well for general use. Unfortunately, the IPVS kernel modules are not among those pre-installed on the AMI.

I might have expected a simple kernel build followed by a make modules install, especially since the RightScale AMI description says that it has kernel headers that are "useful to compile kernel modules." Actually, the default EC2 kernel was compiled with gcc 3.0.2, and the CentOS AMI has 3.1, so the version magic doesn't match. I spent awhile trying to get around this, puzzled by a broken link to a RightScale blog entry that was supposed to say how to do this. I eventually found this link (no longer available on the web) which outlines the basic steps, including the important fact that you have to compile these kernels from the Amazon Developer AMI (which has the right version of gcc).

The above-linked blog post is based on either compiling all the modules or compiling a packaged kernel module like fuse. In order to save time, I wanted to only compile the modules I needed, which is the process I'll outline here.

First, start of an instance of the Amazon Developer AMI, ami-26b6534f. cd to the kernel source and run make menuconfig, turning on (as modules) the components you need.

# gcc -v | tail -n1
gcc version 4.0.2 20051125 (Red Hat 4.0.2-8)
# cd /usr/src/linux-2.6.16-xenU/
# make menuconfig
[trim]
*** End of Linux kernel configuration.
*** Execute 'make' to build the kernel or try 'make help'.

Now, instead of building the kernel with make, you will prep the kernel to have external modules compile against it. Do this with:

# make modules_prepare

Now, cd into the directory that contains the modules you want to build. Basically, you will use a feature of the kernel's Makefile that allows external modules to build against the kernel, but instead of compiling an external module, you are compiling a subdirectory. (Note: the old syntax for doing this was with the SUBDIRS= parameter, and make was run from the top-level directory. This is the new way.) Here's an example:

# cd net/ipv4/ipvs/
# make -C /usr/src/linux-2.6.16-xenU/ M=`pwd` modules
make: Entering directory `/usr/src/linux-2.6.16-xenU'
CC [M] /usr/src/linux-2.6.16-xenU/net/ipv4/ipvs/ip_vs_conn.o
CC [M] /usr/src/linux-2.6.16-xenU/net/ipv4/ipvs/ip_vs_core.o
CC [M] /usr/src/linux-2.6.16-xenU/net/ipv4/ipvs/ip_vs_ctl.o
CC [M] /usr/src/linux-2.6.16-xenU/net/ipv4/ipvs/ip_vs_sched.o
CC [M] /usr/src/linux-2.6.16-xenU/net/ipv4/ipvs/ip_vs_xmit.o
CC [M] /usr/src/linux-2.6.16-xenU/net/ipv4/ipvs/ip_vs_app.o
CC [M] /usr/src/linux-2.6.16-xenU/net/ipv4/ipvs/ip_vs_sync.o
CC [M] /usr/src/linux-2.6.16-xenU/net/ipv4/ipvs/ip_vs_est.o
CC [M] /usr/src/linux-2.6.16-xenU/net/ipv4/ipvs/ip_vs_proto.o
CC [M] /usr/src/linux-2.6.16-xenU/net/ipv4/ipvs/ip_vs_proto_tcp.o
CC [M] /usr/src/linux-2.6.16-xenU/net/ipv4/ipvs/ip_vs_proto_udp.o
LD [M] /usr/src/linux-2.6.16-xenU/net/ipv4/ipvs/ip_vs.o
CC [M] /usr/src/linux-2.6.16-xenU/net/ipv4/ipvs/ip_vs_wrr.o
CC [M] /usr/src/linux-2.6.16-xenU/net/ipv4/ipvs/ip_vs_wlc.o
Building modules, stage 2.
MODPOST
CC /usr/src/linux-2.6.16-xenU/net/ipv4/ipvs/ip_vs.mod.o
LD [M] /usr/src/linux-2.6.16-xenU/net/ipv4/ipvs/ip_vs.ko
CC /usr/src/linux-2.6.16-xenU/net/ipv4/ipvs/ip_vs_wlc.mod.o
LD [M] /usr/src/linux-2.6.16-xenU/net/ipv4/ipvs/ip_vs_wlc.ko
CC /usr/src/linux-2.6.16-xenU/net/ipv4/ipvs/ip_vs_wrr.mod.o
LD [M] /usr/src/linux-2.6.16-xenU/net/ipv4/ipvs/ip_vs_wrr.ko
make: Leaving directory `/usr/src/linux-2.6.16-xenU'

Next, you'll want to install the modules. One peculiarity of external builds is that the modules that are built are built without dependancy to the rest of the kernel. They will have correct interdependencies, though (in this case, ip_vs_wrr and ip_vs_wlc happen to be dependent on ip_vs). Because these are being added as external modules, as though they were not part of the tree, they will be installed to /lib/modules/`uname -r`/extra/. This is fine by me, as I would prefer to have my manually-added modules distinct from those present on the AMI image.

# make -C /usr/src/linux-2.6.16-xenU/ M=`pwd` modules_install
make: Entering directory `/usr/src/linux-2.6.16-xenU'
INSTALL /usr/src/linux-2.6.16-xenU/net/ipv4/ipvs/ip_vs.ko
INSTALL /usr/src/linux-2.6.16-xenU/net/ipv4/ipvs/ip_vs_wlc.ko
INSTALL /usr/src/linux-2.6.16-xenU/net/ipv4/ipvs/ip_vs_wrr.ko
make: Leaving directory `/usr/src/linux-2.6.16-xenU'
# ls /lib/modules/2.6.16-xenU/extra/
ip_vs.ko ip_vs_wlc.ko ip_vs_wrr.ko

Copy these off this running image so that you can shut it down. For this example, I'm using the latest RightScale CentOS image for the small x86 EC2 instance size, ami-5c08ed35. Copy the modules onto it and try to insert them into the kernel. The "extra" directory does not exist; you'll have to create it.

# mkdir /lib/modules/2.6.16-xenU/extra
# scp foo@bar:/path/to/modules/*.ko .
The authenticity of host... [trim]
ip_vs.ko 100% 95KB 95.3KB/s 00:01
ip_vs_wlc.ko 100% 4628 4.5KB/s 00:00
ip_vs_wrr.ko 100% 5677 5.5KB/s 00:00
# depmod
# modprobe ip_vs_wrr
# lsmod
Module Size Used by
ip_vs_wrr 3968 0
ip_vs 84800 2 ip_vs_wrr
dm_mirror 19892 0
dm_mod 49044 1 dm_mirror

The modules appear to be correctly installed. To double check, I went ahead and installed the ipvsadm program and verified that it was able to read and write the IPVS table in the kernel. Next step: actually load-balancing some traffic!