Overview

CKAN packaging HOWTO

A. Packaging a CKAN site

In order to package CKAN and dependencies as Debian files you'll need the following tools:

sudo apt-get install -y python wget dh-make devscripts build-essential fakeroot cdbs

And depending on the source repositories of the things you are packaging you'll probably need:

sudo apt-get install -y mercurial git-core subversion

1. Package your Python Library

The easiest way to do this is with a tool called BuildKit:

pip install buildkit

For each Python package you wish to build a .deb file for you run the buildkit.deb command. Here's an example:

python deb.py /path/to/dir ckanext-dgu 1.3~01 http://ckan.org python-ckanext-csw python-ckanext-harvest

Let's break this down.

python deb.py
This is just how you invoke the command from the command line
/path/to/dir
The directory containing the checked out module you wish to package
python-ckanext-dgu
The name of the .deb file to be created. This should always begin python- for Python libraries and would usually follow ckanext- for all CKAN extensions. If you are packaging a Python dependency you would use its name instead.
1.3~01
The version number of the package. The ~01 postfix is the packaging version, so that you can package the same actual version of the code multiple times
http://ckan.org
The homepage for the package, usually ckan.org for ckan extensions.
python-ckanext-csw python-ckanext-harvest
Any extra arguments are treated as the Debain names of dependencies

When you run the command you will get your .deb file created.

Note

All packages that CKAN itself depends on are already packaged using a slightly different process, based on the exact requirments specified in the ckan source distribution's requires directory. Speak to James Gardner if you need any of these re-packaged.

To release an upgrade of a package it must have a higher version number. There is a chance you may want to release a more recent version of a package despite the fact the underlying version number hasn't changed. For this reason, we always add a ~ character followed by a two digit number to the end of the actual version number as specified in setup.py for the package.

For example, if the version number for CKAN is 1.3~01, a package named python-ckan_1.3~01_amd64.deb would be produced.

2. Create an application package to represent your extension

Your application itself isn't a Python library, even though it uses one.

For packages that don't represent Python libraries it is actually easier to build the .deb manually rather than using Debian's tools.

Let's look at how you would package ckan-dgu.

Create a directory named ckan-dgu. Then within it create a DEBIAN directory with three files:

control

Package: ckan-dgu
Version: 0.2~06
Architecture: amd64
Maintainer: James Gardner <james.gardner@okfn.org>
Installed-Size: 0
Depends: python-ckan
Recommends: apache2, libapache2-mod-wsgi, python-apachemiddleware, ckan, python-ckanext-dgu, python-ckanext-harvest, python-ckanext-csw
Section: main/web
Priority: extra
Homepage: http://ckan.org
Description: ckan-dgu
 Extensions for CKAN to run the data.gov.uk site

Note the indentation after the description, you can add multiple lines if you like.

The installed size isn't technically correct but it seems to work.

postinst:

#!/bin/sh
set -e

if ! [ -f /etc/ckan/dgu/dgu.ini ] ; then
    echo "Creating the DGU instance..."
    ckan-create-instance dgu dgu-dev.okfn.org dgu-dev.okfn.org
    echo "done."
    PACKAGED_CRONJOB="/tmp/${instance}-cronjob"
    if ! [ -f ${PACKAGED_CRONJOB} ] ; then
        echo "Creating cron tab for ckan user"
        cat <<EOF > ${PACKAGED_CRONJOB}
# m  h dom mon dow  command
*/10 *  *   *   *   paster --plugin=ckan harvester run --config=/etc/ckan/dgu/dgu.ini
30  23  *   *   *   python /usr/lib/pymodules/python2.6/ckanext/dgu/bin/gov-daily.py /etc/ckan/dgu/dgu.ini
EOF
        crontab -u ckan ${PACKAGED_CRONJOB}
    fi
    a2ensite dgu
    /etc/init.d/apache2 reload
fi

postrm

#!/bin/sh
set -e
# Any commands that happen after removal or before upgrade go here

The postinst script gets run after install. The postrm file gets run after uninstall. In this case, ckan-dgu depends on CKAN anyway so by the time postinst is run, the usr/bin/ckan-create-instance script is available so can be used to set up a new instance. It is safe to call the script multiple times, because it will exit due to the set -e line if directories already exist.

Any additional files you want merged into the filesystem structure should be placed in the root directory too. apt-get will copy them to the correct location. Have a look at how the ckan package has the usr/bin/ckan-create-instance command added as an example.

Finally we want to package up the .deb file. From within the ckan-dgu directory run this:

dpkg-deb -b . ..

This will create the ../ckan-dgu_0.2~06_amd64.deb package (using the version number specified in the DEBIAN/control file) ready for you to upload to the repo.

If everything builds correctly, add your source to this repository so that others can see your configuration.

3. Import all dependencies into your repo

Once you have packages you'll want to put them in a repo. You can do that as described here:

First copy the .deb files to the server:

scp -i "new-keypair.pem" *.deb ubuntu@ec2-46-51-149-132.eu-west-1.compute.amazonaws.com:release/20110328

Then add them to the repo like this:

cd /var/packages/debian/
sudo reprepro includedeb lucid ~/release/20110328*.deb

You can remove them from the repo like this:

cd /var/packages/debian/
sudo reprepro remove lucid python-ckan

Here's the pool of packages after the import:

$ cd /var/packages/<instance>-dev
$ find . | grep ".deb"
./pool/universe/p/python-apachemiddleware/python-apachemiddleware_0.1.0-1_amd64.deb
./pool/universe/p/python-ckan/python-ckan_1.3.2~10-1_amd64.deb
./pool/universe/p/python-ckanext-dgu/python-ckanext-dgu_0.2~06-1_amd64.deb
./pool/universe/p/python-licenses/python-licenses_0.6-1_amd64.deb
./pool/universe/p/python-ckanclient/python-ckanclient_0.6-1_amd64.deb
./pool/universe/p/python-vdm/python-vdm_0.9-1_amd64.deb
./pool/universe/p/python-ckan-deps/python-ckan-deps_1.3.4-1_amd64.deb
./pool/universe/p/python-owslib/python-owslib_0.3.2beta~02-1_amd64.deb
./pool/universe/p/python-formalchemy/python-formalchemy_1.3.6-1_amd64.deb
./pool/universe/p/python-solrpy/python-solrpy_0.9.3-1_amd64.deb
./pool/universe/p/python-markupsafe/python-markupsafe_0.9.2-1_amd64.deb
./pool/universe/p/python-ckanext-qa/python-ckanext-qa_0.1~09-1_amd64.deb
./pool/universe/p/python-ckanext-csw/python-ckanext-csw_0.3~04-1_amd64.deb
./pool/universe/p/python-pyutilib.component.core/python-pyutilib.component.core_4.1-1_amd64.deb
./pool/universe/c/ckan/ckan_1.3.2~09_amd64.deb
./pool/universe/c/ckan-dgu/ckan-dgu_0.2~05_amd64.deb

B. Deploying a CKAN Site

First create the file /etc/apt/sources.list.d/okfn.list with this line, replacing debian with the correct repo you want to use:

echo "deb http://apt-alpha.ckan.org/debian lucid universe" | sudo tee /etc/apt/sources.list.d/okfn.list

Then add the package key to say you trust these packages:

sudo apt-get install wget
wget -qO-  http://apt-alpha.ckan.org/packages.okfn.key | sudo apt-key add -
sudo apt-get update

Now you can install CKAN, just like any other Debian package:

sudo apt-get install ckan-dgu

At this point you should have a running instance. You may need to copy across an existing database if you need it.

C. The Release Process

For any instance of CKAN, the following release process occurs:

  • Package up all the .deb files in a directory with the release date in the format yyyy-mm-dd_nn where nn is the release the number of the release on each day. eg 2011-03-13_01

  • Import them into the dev repositoy:

    cd /var/packages/<instance>-dev
    sudo reprepro includedeb lucid /home/ubuntu/release/2011-03-13_01/*.deb
    

    Here's the pool of packages after the import:

    $ cd /var/packages/<instance>-dev
    $ find . | grep ".deb"
    ./pool/universe/p/python-apachemiddleware/python-apachemiddleware_0.1.0-1_amd64.deb
    ./pool/universe/p/python-ckan/python-ckan_1.3.2~10-1_amd64.deb
    ./pool/universe/p/python-ckanext-dgu/python-ckanext-dgu_0.2~06-1_amd64.deb
    ./pool/universe/p/python-licenses/python-licenses_0.6-1_amd64.deb
    ./pool/universe/p/python-ckanclient/python-ckanclient_0.6-1_amd64.deb
    ./pool/universe/p/python-vdm/python-vdm_0.9-1_amd64.deb
    ./pool/universe/p/python-ckan-deps/python-ckan-deps_1.3.4-1_amd64.deb
    ./pool/universe/p/python-owslib/python-owslib_0.3.2beta~02-1_amd64.deb
    ./pool/universe/p/python-formalchemy/python-formalchemy_1.3.6-1_amd64.deb
    ./pool/universe/p/python-solrpy/python-solrpy_0.9.3-1_amd64.deb
    ./pool/universe/p/python-markupsafe/python-markupsafe_0.9.2-1_amd64.deb
    ./pool/universe/p/python-ckanext-qa/python-ckanext-qa_0.1~09-1_amd64.deb
    ./pool/universe/p/python-ckanext-csw/python-ckanext-csw_0.3~04-1_amd64.deb
    ./pool/universe/p/python-pyutilib.component.core/python-pyutilib.component.core_4.1-1_amd64.deb
    ./pool/universe/c/ckan/ckan_1.3.2~09_amd64.deb
    ./pool/universe/c/ckan-dgu/ckan-dgu_0.2~05_amd64.deb
    
  • Test on the dev server, if everything is OK, copy the dev repo to UAT:

    $ cd /var/packages/
    $ sudo rm -r <instance>-uat
    $ sudo cp -pr <instance>-dev <instance>-uat
    
  • You can now run this on UAT:

    sudo apt-get update
    sudo apt-get upgrade
    

    Because it is an exact copy of the dev repo at the point you tested you can be sure the software is the same

  • If all goes well, repeat this process with staging and live repos to deploy the release.

D. Maintenence

All CKAN

ckan/usr/bin/ckan-instance-maintenance

I've been using this for removing installs during testing. This should eventually form the basis of a postrm script:

sudo a2dissite dgu; sudo /etc/init.d/apache2 stop; sudo rm -r /etc/ckan/dgu; sudo -u postgres dropdb dgu; sudo -u postgres dropuser dgu; sudo rm -f /etc/apache2/sites-available/dgu*; sudo rm -f /etc/apache2/sites-enabled/dgu*;  sudo rm -r /var/lib/ckan; sudo update-rc.d -f dgu_harvest_gather remove;  sudo update-rc.d -f dgu_harvest_fetch remove;