Howto make your private VM Cluster, Part I

I wanted to make some experiments with DRBD, pacemaker and others. The goal is to test a few configurations for an high availability scenario. Since I don't want to make changes to my machine the solution is to use Virtual Machines. I decided to go all open source and use KVM. Since I want more that a single VM I decided to setup a private bridge to which all the VMs would connect. The bridge would provide DHCP and DNS services and NAT + Firewall to the internet.

I use Gentoo. If you use another distribution or firewall tool you should adapt these instructions and scripts to fit your needs.

First I created a script that sets up my bridge. The script is as follows (I called it setupBridge):

#!/bin/bash
set -x

# Setup the bridge
sudo brctl addbr br0
sudo ifconfig br0 192.168.100.1 netmask 255.255.255.0 up

# Launch DNS and DHCP Server on br0
sudo dnsmasq -q -a 192.168.100.1 --dhcp-range=192.168.100.50,192.168.100.150,forever --pid-file=/tmp/br0-dnsmasq.pid

# Launch updated firewall configuration
sudo firehol /home/nsousa/KVM/firehol-br0.conf start

The bridge is created and I give it the 192.168.100.1 ip address. I use dnsmasq to provide DNS and DHCP. The DHCP provided addresses will be from 192.168.100.50 to 150. Finally I adapt the firewall using firehol. Here is the firehold configuration file (firehol-br0.conf):

version 5

# Allow all traffic in the Bridge
interface br0 bridge
    server all accept
    client all accept

# Accept all client traffic on any interface
interface any world
    client all accept

# NAT to the internet
router bridge2internet inface eth0 outface br0
    masquerade reverse
    client all accept

# Bridge Routing
router bridge-routing inface br0 outface br0
    server all accept
    client all accept

This basically allows the VMs to connect to any service running on br0 (the host). It allows any one to connect to anyone as client (so the host can go to the internet). Finally it sets up NAT using masquerade so the VMs can use the bridge to access the internet, but as clients only.

To undo all these changes I have a script that tears all this setup down. I called it teardownBridge and here are its contents:

#!/bin/bash
set -x

# Stop DHCP and DNS Server
sudo kill -15 `cat /tmp/br0-dnsmasq.pid`
sudo rm /tmp/br0-dnsmasq.pid

# Stop the bridge
sudo ifconfig br0 down
sudo brctl delbr br0

# Reset the firewall
sudo firehol /etc/firehol/firehol.conf start

Before moving to the script that starts one VM I first need to show the script that sets up the interface for qemu to use (I called it qemu-ifup):

#!/bin/bash
set -x

if test $(/sbin/ifconfig | grep -c $1) -gt 0; then
    sudo /sbin/brctl delif br0 $1
    sudo /sbin/ifconfig $1 down
fi

sudo /sbin/ifconfig $1 0.0.0.0 promisc up
sudo /sbin/brctl addif br0 $1

The goal here is to add the tap device created to the bridge. I first remove it if it is already there.

Last but not least the script that starts a VM. This script sets up a tap device for the current user, configures KVM start-up parameters and starts KVM. When KVM ends it removes the tap device to keep the system clean. Here is the contents of the script adapted to use a hard disk and the minimal install cd for gentoo (I called it startVM):

#!/bin/bash
set -x

# Create tap interface so that the script /etc/qemu-ifup can bridge it
# before qemu starts
USERID=`whoami`
IFACE=`sudo tunctl -b -u $USERID`

# Setup KVM parameters
MEMORY="-m 512"
IMGPATH="/home/nsousa/KVM"
IMAGE=gentoo.qcow2
CDROM="-cdrom /home/nsousa/Downloads/install-x86-minimal-20101123.iso"
MACADDRESS="DE:AD:BE:EF:5D:A3"
NET="-net nic -net tap,script=/etc/qemu-ifup"

# Start kvm
kvm $MEMORY -hda $IMGPATH/$IMAGE $CDROM -net nic,macaddr=$MACADDRESS -net tap,ifname=$IFACE,script=/home/nsousa/KVM/qemu-ifup

# kvm has stopped - remove tap tap interface
sudo tunctl -d $IFACE &> /dev/null

Next step is to make a minimal gentoo installation on a VM to use as base for all the VMs in the cluster. I'll probably refactor the startVM script to separate the common parts (the private parts will be the disk image to use and MAC Address).

Stay tuned for more updates.