distribute's sandboxing doesn't preserve working_set; leads to setup_requires problems

Tarek Ziadé avatarTarek Ziadé created an issue

I've got a tricky problem I'm trying to deal with. Here's the scenario:

I'm trying to build a package that has two requirements in setup_requires; let's say `setup_requires = ['package_A', 'package_B']`. The problem is that "package_B" also contains "package_A" in its own setup_requires.

What happens when I do an install/build is that package_B is downloaded and installed--in the process it also downloads and installs package_A in its sandbox. This wouldn't be a problem except that the sandboxed temp install of package_A is left in pkg_resources.working_set.entries, and so it's assumed package_A is already installed, and the main install doesn't try to fetch it. However, when it goes to import from package_A, I get an import error (because the sandbox it was installed in has since been deleted).

If I reverse the order it doesn't work either for a different, but related problem. package_A gets installed into the cwd and is added to pkg_resources.working_set. When package_B is installed it uses the existing package_A installation, so that's fine. But then, due to some complicated interplay, package_A never gets added to sys.path.


Just to confirm my theory about this, I modified setuptools.sandbox.run_setup to also save off and restore pkg_resources.working_set.{entries,entry_keys,by_key}. This solves the problem, and the build works regardless of how my setup_requires are ordered. Is there any reason this shouldn't be done? It seems reasonable to me...

It would also still be nice to find a workaround until and unless this gets patched. All I can think of at the moment is monkey-patching :/

Comments (13)

  1. Erik Bray

    Zooko: Yeah, that's the same issue. In setuptools it's fixed by saving and restoring all the globals in pkg_resources. I was able to fix it for myself just by saving/restoring all the data attributes of pkg_resources.working_set (.entries, .by_key, etc). I don't know if there are cases for preserving anything else, but there may be.

  2. Erik Bray

    I'm attaching an archive containing some example projects that exhibit the problem. Not sure how easy it would be to make into a unit test though.

    To explain, ProjectA has no dependencies, ProjectB has setup_requires=ProjectA, and ProjectC has setup_requires=ProjectA,ProjectB.

    Try running ProjectC/setup.py build. You'll find that it still doesn't work in the branch. The problem is that working_set.getstate() doesn't *copy* the relevant attributes.

  3. Erik Bray

    Here's a patch that adds the missing bits to this. With this patch, my example works.

    Though one other strange problem is cropping up: In my example I use find-links to point to source packages for PackageA and PackageB. When I install PackageC, it correctly uses the local source dist to install PackageB, but then when it tries to install PackageA it goes to PyPI first, and then falls back on the local file. This seems like a separate issue though.

  4. Tarek Ziadé

    I'll see if I can make it a unittest. Thanks for the extra patches, I am giving you a write access so you can apply them to that branch (if you don't mind)

  5. Tarek Ziadé

    Ok so the patch as a whole looks good,

    I am going to try to make a test for this, and even if not I want to merge this to default and release the next version asap

  6. Tarek Ziadé

    ah you did add a test :)

    Are we good to merge in you opinion ? I don't really know either if there's a better way, and the rest seems not to be broke so../

  7. Erik Bray

    I did? The only 'test' I added was a couple sample projects laid out to illustrate the issue. I haven't made a proper unit test out of that. At any rate, the issue is still fixed...

  8. Log in to comment
Tip: Filter by directory path e.g. /media app.js to search for public/media/app.js.
Tip: Use camelCasing e.g. ProjME to search for ProjectModifiedEvent.java.
Tip: Filter by extension type e.g. /repo .js to search for all .js files in the /repo directory.
Tip: Separate your search with spaces e.g. /ssh pom.xml to search for src/ssh/pom.xml.
Tip: Use ↑ and ↓ arrow keys to navigate and return to view the file.
Tip: You can also navigate files with Ctrl+j (next) and Ctrl+k (previous) and view the file with Ctrl+o.
Tip: You can also navigate files with Alt+j (next) and Alt+k (previous) and view the file with Alt+o.