1. Albert Hopkins
  2. hemp


hemp / README

What is hemp?

Hemp is a tool that can be used to create a private "cloud" of virtual
servers.  It utilizes such tools as libvirt, kvm/qemu and fabric for much
of its power.


The basic idea behind hemp is the creation of a "cloud" of virtual machines
based on a previously-created base machine.  One creates/installs a base
virtual machine using libvirt.  You may use command-line tools, Virtual
Machine Manager (virt-manager) or whatever libvirt-based tool you prefer to
create the base VM.  Once the base VM is created, hemp is used to create
the private cloud.  Everything for creating the private cloud is included
in a single directory.  This include the base VM image, a libvirt XML
template for creating the VMs, and a skeleton fabric file and helper
module.  Once this directory structure is created by hemp, you never need
the hemp tool again to manipulate your private cloud.  Moreover, your
private cloud can be move or copied within the host or even to a new host
machine and still stay intact.  So you can create your cloud and
fabric-based infrastructure and share it with others.


The first thing you do is create your base VM in libvirt.  You allocate
storage, network, and other hardware the same way you usually create a
virtual machine.  Install the OS, shut down the VM and you're done.
Currently hemp only support Linux-based VM's and has only been tested with
Gentoo, Ubuntu, and Red Hat-based guests.

It is recommended that the base VM uses a QCOW2 image for it's disk,
preferably a compressed QCOW2 image, though any format recognized by
libvirt/kvm/qemu will work.  Also, currently the skeleton fabfile created
by hemp only knows how to work with base VM's that have exactly one drive,
and where the root filesystem is on the first partition of the drive.  LVM,
software, RAID, etc should not be used on the VM's disk image.  If you want
to use a fancier storage layout on your VMs, you will have to alter the
skeleton fabfile's create() function accordingly. But the easiest thing to
do is to stick to one drive and root on partition 1.

For example, if we have created a base virtual machine called "lilpenguin"
in libvirt, then we simply run:

    # hemp lilpenguin [directory]

On most systems you will want to do this as root.  Technically it's not
required that hemp be run as root, but hemp needs to be able to connect to
the libvirt hypervisor and also requires read access to the virtual
machine's disk image.

This command creates the directory provided or, if not provided, a
directory with the same name as the virtual machine in the current
directory.  Once run the directory structure will look like the following:

    |-- fabfile.py
    |-- helpers.py
    |-- images
    |   `-- base.qcow2
    |-- settings.py
    `-- template.xml

At the root of the directory are: the skeleton fabfile.py which you will
use and edit, a helpers.py module which is used by the fabfile, a
settings.py file which is used to hold configuration variables, and
template.xml, which is the XML template used to create the virtual machines
in libvirt.  The images subdirectory contains copied base VM's drive image
and will also contain the images of the cloud VMs, which themselves will be
snapshots of the base image.

NOTE: Although hemp copies the base disk image to a file called
"base.qcow2" it is not necessary that the base disk image be a QCOW2 file.
The "base.qcow2" file is an exact copy of the base VM's disk image so the
.qcow2 extension can be ignored.  If you want to be sure what format the
image is you can simply run "qemu-img info base.qcow2".

After this structure is created you need not use the hemp tool again.
Everything you need for manipulationg your cloud is contained in the

Look at the settings.py module.  It contains configuration data for your
cloud.  The most important setting is the NUM_SERVERS variable.  By default
it tells your fabfile how many virtual machines to boot up.  Also by
default your virtual machine names will be based on the name of the
directory your private cloud is contained in.  In our example, the
directory is named "lilpenguin" so the virtual machines will be named
"lilpengiun01", "lilpenguin02", etc.

To create and start the virtual machines simply change to the directory and
run "fab create":

    $ cd lilpenguin
    $ fab create


You can see that your virtual machines are created by running "fab status"
or "virsh list".

The create() fabric function does several things, for each virtual machine

    1. Creates a QCOW2 snapshot based on the base image. This file exists
       in the images directory.
    2. Mounts the QCOW2 image using the qemu-nbd tool.
    3. Edits files in the image to set the hostname and other chores.
    4. Unmounts the image.
    5. Reads the template.xml file, creates a new XML string based on the
       newly created image file and the vm's hostname, and feeds that to
       libvirt, tellint it to create and start the new vm.

The create() method does this all for you.  Your cloud VMs will use the
same (virtual) network that the base VM used.  This network should have a
DHCP server (such as the dnsmasq-ed virtual bridge libvirt uses by default)
or at the very least use zeroconf/MDNS. Otherwise the defautl fabric set up
will not be able to reach the VMs by name.  If your setup is different
(e.g. you want to give your VMs static IPs and put their names in your
/etc/hosts) then you will need to edit the fabfile.py accordingly.  Also,
since we are using fabric, it is assumed that your VMs will all be running
an ssh server and that you have created a user that you can ssh as (you'll
likely do this on the base virtual machine).

Continue changing the fabfile as you please. You may want to add more
functions, have the guests install various software, put/configure various
files, and run certain jobs and services.

When you are finished with your cloud, it's easy to tear it down.  Simply

    # fab destroy

This performs a "hard" shutdown of the VMs (pulls the plug). It then
removes the snapshot images for the VMs and basically puts the cloud
directory in it's original state (except for whatever changes you've made).
After this is done you can run "fab create" again to re-create and start
your cloud from scratch.

This in effect creates "throw-away" VMs.  You create them. Use fabric and
other tools to manipulate them, and then "destroy" them.  The main
advantage of this is that it's easy to create and test various
configuration and deployment schemes.