distribute's sandboxing doesn't preserve working_set; leads to setup_requires problems

Create issue
Issue #205 resolved
Tarek Ziadé repo owner created an issue

I've got a tricky problem I'm trying to deal with. Here's the scenario:

I'm trying to build a package that has two requirements in setup_requires; let's say setup_requires = ['package_A', 'package_B']. The problem is that "package_B" also contains "package_A" in its own setup_requires.

What happens when I do an install/build is that package_B is downloaded and installed--in the process it also downloads and installs package_A in its sandbox. This wouldn't be a problem except that the sandboxed temp install of package_A is left in pkg_resources.working_set.entries, and so it's assumed package_A is already installed, and the main install doesn't try to fetch it. However, when it goes to import from package_A, I get an import error (because the sandbox it was installed in has since been deleted).

If I reverse the order it doesn't work either for a different, but related problem. package_A gets installed into the cwd and is added to pkg_resources.working_set. When package_B is installed it uses the existing package_A installation, so that's fine. But then, due to some complicated interplay, package_A never gets added to sys.path.

Just to confirm my theory about this, I modified setuptools.sandbox.run_setup to also save off and restore pkg_resources.working_set.{entries,entry_keys,by_key}. This solves the problem, and the build works regardless of how my setup_requires are ordered. Is there any reason this shouldn't be done? It seems reasonable to me...

It would also still be nice to find a workaround until and unless this gets patched. All I can think of at the moment is monkey-patching :/

Comments (13)

  1. Wichert Akkerman

    Extra context: PJE says he fixed this in setuptools in 2009, so it may be possible to pull a patch from there.

  2. E. Madison Bray

    Zooko: Yeah, that's the same issue. In setuptools it's fixed by saving and restoring all the globals in pkg_resources. I was able to fix it for myself just by saving/restoring all the data attributes of pkg_resources.working_set (.entries, .by_key, etc). I don't know if there are cases for preserving anything else, but there may be.

  3. E. Madison Bray

    I'm attaching an archive containing some example projects that exhibit the problem. Not sure how easy it would be to make into a unit test though.

    To explain, ProjectA has no dependencies, ProjectB has setup_requires=ProjectA, and ProjectC has setup_requires=ProjectA,ProjectB.

    Try running ProjectC/setup.py build. You'll find that it still doesn't work in the branch. The problem is that working_set.getstate() doesn't *copy* the relevant attributes.

  4. E. Madison Bray

    Here's a patch that adds the missing bits to this. With this patch, my example works.

    Though one other strange problem is cropping up: In my example I use find-links to point to source packages for PackageA and PackageB. When I install PackageC, it correctly uses the local source dist to install PackageB, but then when it tries to install PackageA it goes to PyPI first, and then falls back on the local file. This seems like a separate issue though.

  5. E. Madison Bray

    For the sake of it, here's a patch that fixes the aforementioned find-links issue. Though that issue's really a tangential one. Also, this fix feels a bit hackish, so I wonder if there's a better way.

  6. Tarek Ziadé reporter

    I'll see if I can make it a unittest. Thanks for the extra patches, I am giving you a write access so you can apply them to that branch (if you don't mind)

  7. Tarek Ziadé reporter

    Ok so the patch as a whole looks good,

    I am going to try to make a test for this, and even if not I want to merge this to default and release the next version asap

  8. Tarek Ziadé reporter

    ah you did add a test :)

    Are we good to merge in you opinion ? I don't really know either if there's a better way, and the rest seems not to be broke so../

  9. E. Madison Bray

    I did? The only 'test' I added was a couple sample projects laid out to illustrate the issue. I haven't made a proper unit test out of that. At any rate, the issue is still fixed...

  10. Log in to comment