Issues

Issue #31 new

Need ability to override deps on the command line

Monty Taylor
created an issue

One of us will add an issue I guess, but here's the general gist of what we're trying to solve.

First of all, here's an example tox.ini file:

https://github.com/openstack/python-glanceclient/blob/master/tox.ini

The OpenStack CI infrastructure is an automated gated trunk. That means that we push changes up to a gerrit instance code review, and then if they are approved, Jenkins grabs the changes, tests them, and if they pass tests it submits a message back to Gerrit which causes the change to be merged. We run all of these tests in clean environments, and for unittests and pep8 checks, we use tox to manage the creation of virtualenvs and the running of tests in those environments.

https://jenkins.openstack.org/view/Gate/job/gate-python-glanceclient-python26/configure https://jenkins.openstack.org/view/Gate/job/gate-python-glanceclient-python27/configure https://jenkins.openstack.org/view/Gate/job/gate-python-glanceclient-pep8/configure

https://github.com/openstack/openstack-ci-puppet/blob/master/modules/jenkins_slave/files/slave_scripts/run-tox.sh

Perfect, right?

Problem is, we test somewhere around 200 changes a day all hands off, so things like transient network problems between our build hosts and PyPI (which happen often) become a major impediment - even if all we're talking about is the index look up for the dep name.

So as an optimization, we have an out-of-band job which runs pip bundle on our deps lists:

https://jenkins.openstack.org/view/Gate/job/python-glanceclient-venv/configure https://github.com/openstack/openstack-ci-puppet/blob/master/modules/jenkins_slave/files/slave_scripts/build-bundle.sh

Installing deps into a virtualenv from a bundle file avoids all network accesses, as all of the index lookups are pre-done. Problem is, there isn't a great way to do conditional deps lists - I can't override deps in tox:jenkins. So we make a bunch of 'fake' environments in our tox.ini file that jenkins can use jenkins26, jenkins27, etc.

This is a bit ugly though, as it encodes specifics of our CI implementation and specifically the caching concerns we have on a system that runs gobs of tests per day that don't apply to any of the individual developers.

You may remember I took a stab at making tox virtualenvs relocatable a while back as an attempt to solve this systemically - turns out that virtualenvs just are not possible to relocate in a sensible manner.

What I REALLY want is a way for tox to understand bundle-based caching, so that I could run a tox command in build-bundle that would produce something I could later drop into a location on a different machine, so that when tox decides it needs to create a virtualenv it could use the discovered bundle file instead of the directly listed deps, at which point I could get rid of the jenkins* envs inside of tox.ini and treat the problem of caching as a step that the CI system needs to do inside of a pipeline of activity.

As a half-way step to that (which I figured would be a much more invasive patch) we wrote the patch to do dep override on the command line, because at least that way we can have jenkins job do things that are specific to jenkins and have nothing to do with a normal use of the repo, like::

tox -epy27 --deps=.cache.bundle

py27 can then simply describe how to run the job, and just because we've already pre-created the dep bundle, we can tell it to use that. That gets us 95% of the way there- we can clean up the tox.ini files in the close to 30 branches we're managing this way and then can treat the further solution to the problem on the CI side.

Does that make sense? Sorry for the novel here.

Comments (3)

  1. holger krekel repo owner

    Thanks for the write up - interesting to see how you are using things!

    Did you try using mirrors, i.e. patch tox to use the 'pip install -M' option?

    I haven't used bundles myself yet but i think it makes sense to consider them and for tox to be able to create them and later use them. Do you have a suggestion on the involved command line options and the behaviour? (Specifying how things should work from the user side is one important part - maybe the implementation is not that hard, not sure currently).

  2. Monty Taylor reporter

    My pleasure- and thanks for writing tox - we have a bunch of stuff going on and tox is letting us delete a bunch of code. :)

    We thought about mirrors, but then we still have cache-invalidation and pull-through failures ... as in, what happens if it's not on the mirror yet. (Have I mentioned that making things work on the internet consistently is hard?)

    Bundles are actually SUPER easy ... you just replace the install word with bundle and then the name of the output file ... "pip bundle $bundlename blah blah"

    As a possible UI - something like "tox -epy27 --bundle" - which potentially would create a bundle file in .tox/py27 or something? Then, when you run tox -epy27, if there is a bundle there, tox will use it, otherwise it will just use the deps in the ini file? (--bundle could be --cache or --download or something)

    I betcha you're right- figuring out workflow is probably going to be WAY more work than actually implementing it.

  3. Anonymous

    I solved the described problem for myself in a slightly different way. But I agree, you should have all packages locally without needing any network access.

    Each project has a downloads/ subdirectory that is used to cache all Python packages. I have a build-script that allows me to populate this downloads/ subdirectory via pip (and requirements files). Then I build a local package-index directory structure within it (via pip2pi or ...). Now, tox can now use and install the packages from the downloads/ subdirectory without requiring any network access.

    EXAMPLE:

    # -- file:tox.ini
    [tox]
    ...
    indexserver =
        local   = file:///{toxinidir}/downloads/simple
    
    [testenv]
    ...
    deps=
        :local:argparse >= 1.2
    
    

    It would be helpful if the downloads/ directory would be sufficient. pip can do this but I found no way to make tox work with it (without any network access).

  4. Log in to comment