easy_install doesn't pass its arguments down to setup.py bdist_egg

Issue #227 resolved
Erik Bray
created an issue

I discovered this the other day when PyPI was down and I wanted to tell a user to use our own package index.

If I run easy_install -i http://my/index foo, the 'foo' package will be downloaded from my index, and easy_install will run setup.py bdist_egg on it. However, the 'foo' package has requirements listed in setup_requires, and since none of the options from easy_install are passed down, it will still go to PyPI to try to get the setup_requires dists.

This is a slightly tricky issue, but I think I have a solution, albeit a slightly hackish one. Code forthcoming if it turns out to work.

Comments (12)

  1. Erik Bray reporter

    It should be noted that this solution does prevent someone from installing a distribution named simply '-'. But if someone comes complaining about that they have bigger problems.

  2. bbinet

    We also have our own package index and we always come across this bug when deploying an application behind a restrictive firewall. Even if our own package index is whitelisted, installing our application with:

    easy_install -i http://my.own.packageindex/pypi -H my.own.packageindex my_package
    

    will fail if one of my_package dependencies has a setup dependency declared in setup_requires, because our own package index is then not taken into account.

    I've tested the fix-issue-227 branch and I confirm that it solves the issue. What do you think about merging this branch into default?

    Note that this problem had already been reported on the distutils-sig mailing list:

    http://mail.python.org/pipermail/distutils-sig/2010-September/016904.html

    and on the buildout issue tracker:

    https://bugs.launchpad.net/zc.buildout/+bug/257555

  3. Erik Bray reporter

    Wow, I totally forgot that I fixed this. I even still get bitten by this every now and then, but in most cases I've been able to brush it off. I need to check that this still merges cleanly with the latest tip. Might not be a bad idea to add a test too, though I'm not entirely sure how that would work...

  4. Jason R. Coombs

    I'd really like to see this fix integrated. So I took a look at the branch (thanks for this work; it's a tricky problem), and it inspired me to dig deeper.

    In doing so, I found that the egg fetcher already had support for reading easy_install fetch parameters from the file system (setuptools/dist.py:274 dist.parse_config_files()). And the setopt module has support for setting those parameters in the config file.

    So I've adapted your branch with a different approach that's a little less invasive. Here's the updated diff.

    Instead of having the egg fetcher have to interpret args on the command line, easy_install makes them available in a .cfg file, which was already honored by the egg fetcher.

    Erik, what do you think about this approach instead? I've tested it against my use case and it's working.

    I agree it would be nice to have a test, but I'd be more inclined to get this merged and released sooner than later.

  5. Jason R. Coombs

    As you can see, I added a test, but it's currently not working. Somehow, when I try to invoke easy_install, it fails to install anything (and silently). I hope at another time to try to figure out why... but here's the traceback.

    ======================================================================
    FAIL: test_setup_requires_honors_fetch_params (setuptools.tests.test_easy_install.TestSetupRequires)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "C:\Users\jaraco\projects\public\distribute\setuptools\tests\test_easy_install.py", line 280, in test_setup_requires_honors_fetch_params
        self.assertTrue(os.listdir(temp_install_dir))
    AssertionError: [] is not true
    
    ----------------------------------------------------------------------
    
  6. Erik Bray reporter

    Thank you for taking a look at this. I had once again nearly forgotten about it after PyCon ;

    Yes. As you might have guessed the code in my original patch was largely copied out of fetch_build_egg() it looks like. But I like your approach. If I get a minute later today I'll take a look at the test and see if I can figure out why it's not working.

  7. Erik Bray reporter

    It appears that in the test, something has set

    distutils.core._setup_stop_after = "commandline"
    

    so the easy_install command never actually gets run. I have no idea how that's happening though. The makeSetup command in setuptools.tests.__init__ does this, but it should restore _setup_stop_after in a finally block. So I'm not sure if that explains it...

  8. Jason R. Coombs

    Success!

    After only five hours of tracing the code, I found there were several issues with the way the tests were constructed. I think I finally have a test implementation that captures the failure and then the failure is fixed by merging in the branch. If you're curious, read the changelog for a breakdown of the steps I took to get there.

    So the issue is resolved, but I suspect the tests now only run on Python 2.6 and later (due to the context managers I added). Next step is to get the tests running on Python 2.4 and 3.2. Then we can release.

  9. Log in to comment