Overview

EasterBunny

EasterBunny is a simple commandline application that collects all installed eggs in a Python-installation to create a package index out of them.

To do so, it either uploads them to an EggBasket instance, or places them into a destination directory, possibly with HTML files to be served statically.

For simple usage instructions, use:

$ easterbunny --help

Goals

The goal of EasterBunny is to provide a integral part of a deployment-architecture that allows python packages and their dependencies to be installed with a determined set of versions.

Additionally, basing a deployment on EasterBunny with one of the later on described strategies guarantees (backups non-withstanding) that packages are available, regardless of their actual availability on the central PyPI. You can't suffer from pulled versions, or even fully removed packages.

Now we will describe the various methods of index-generation and how to use them to install packages.

Installing from a local directory

After collecting the eggs with:

$ easterbunny --dest=<destdir>

you can install whatever packages are in there (including their dependencies, which are fetched automatically) using:

$ easy_install -H None -f <destdir> <Packagename>

For more information, see here.

Installing from a hosted package index

When additionally to a --dest-parameter the flag --index is given, EasterBunny will arrange the eggs into subdirectories based on the package-name, and additionally create a top-level index-file that points to the individual packages.

This whole directory can be served from a HTTP-server to act as Python package index. To use that home-grown index, use easy_install as follows:

$ easy_install -H <hostname>[:<port>] -i http://<hostname>[:<port>]/path_to_index

A quick way to test or use this is the static WSGI-middleware. It comes with a simple build-in webserver, and you can serve a generated package index with a command like this:

$ static <path-to-package-index> localhost:<port>

ATTENTION

Depending on your webserver, you might get problems if you don't specify dependencies in the exact case they are spelled in the package's name itself. This is good practice anyway, but sometimes not done properly and might lead to problems.

Installing from EggBasket

EggBasket is a PyPY clone, originally written by Chris Arndt. Ableton uses a patched version that has a database-driven VersionSet-mechanism to capture concrete version-sets.

This version might not be released, so we won't further describe using it.

Deployment strategies

There are two major ways of deploying your application and ensuring it lives in a controlled environment:

# create a versioned egg and install this using easy_install.

# distribute a file-structure, preferrably by checking out and
updating a version control system repository.

While EasterBunny can be of use in both of them, the latter is much easier to the developer as well as the deployment process itself. So we will only shortly describe the former, and concentrate on the latter.

Deployment using an egg

When using this method, one needs to place the current development state as egg into a virtualenv, together with all it's dependencies. Most practically, this is done using:

$ python setup.py install

This virtualenv is then processed by EasterBunny using one of the above mentioned ways. Then on the production system, the egg is installed in the described way.

The disadvantage of this methodology is that neither the sources nor the resulting egg have ways of specifying a concrete versionset. So the deployer has to remember to use the right versionset via passed commandline-options to easy_install.

This is even more important in case of rolling back a deployed version.

Additionally, this method creates more pain while developing. Usually, one develops using:

$ python setup.py develop

inside the working-copy. However, the need to create an actual egg disrupts this - the egg is placed into the virtualenv as described above, but then the developer has to remember to re-enable local development using the python setup.py develop.

Instead, basing the deployment on a working-copy of the project's sources is much more powerful and easier to use.

Deployment using a working-copy

The central piece that makes this method preferrable is the possibility to use a setup.cfg alongside the setup.py in the working-copy's root.

In this INI-style config-file one can specify - amongst other things - the following section:

[easy_install]
find_links = <path-to-versionset>
allow_hosts = <restriction-of-hosts>

Whenever easy_install or python setup.py <something> is invoked inside the working-copy's root (which is the normal way to do it), the setup.cfg is found & applied to the installation of whatever dependent package is needed.

Using this mechanism, setting up a deployed version of the system consists of four simple steps:

  • checking out or updating a working-copy to the latest or a specific version.
  • purging the virtualenv to be used for this working-copy, simplest way to do so is to remove & re-create it.
  • activate the virtualenv
  • making the sources available for running (development or production) using the simple python setup.py develop inside the working-copy.

The great thing is that this simple steps apply to all scenarios - production system deployment, setting up a new worker's environment, a feature branch, updating the worker's virtualenv after another developer introduced new dependencies, deploying the system for integration-testing - all rely on the same, simple, basic steps.

Installing or upgrading packages not in the version-set

The one thing that is a bit more elaborated due to the constraining of all setuptools-operations through setup.cfg is the installation or upgrade of packages not yet or in outdated versions inside the version-set.

But "elaborated" is a strong word for - assuming an activated virtualenv of course - simply

  • changing the current directory to some other place without a setup.cfg, most times simple the parent of the working-copy.
  • invoking easy_install there.

Once this has been done, the virtualenv is of course in a different state than the one in the captured versionset. So we need to update that.

This works like this:

  • collect the new versionset using EasterBunny.
  • adapt setup.cfg to reflect the changes.
  • commit to the repository.

Concrete usage scenarios