`setup.py develop` doesn't support package_dir arg to setup()

Issue #177 resolved
slinkp
created an issue

I just ran into a problem that somebody originally reported as a bug against pip:

http://bitbucket.org/ianb/pip/issue/72/editable-install-doesnt-respect-package_dir

It was closed as invalid because setup.py develop has the same behavior. The suggestion was to file the issue upstream against distribute, but I don't see a relevant bug here. So, I'm filing it :)

Attached is a small distribution demonstrating this issue. It contains two packages and uses the package_dir argument to setup() to allow an unconventional source tree layout.

This doesn't work when you do setup.py develop, because all you get is an .egg-link file pointing to the root of the source tree, and python can't find the actual packages:

{{{ (td)pw@slinktopp2 TestDistribution $ python setup.py develop running develop install_dir /tmp/td/lib/python2.6/site-packages/ running egg_info writing TestDistribution.egg-info/PKG-INFO writing top-level names to TestDistribution.egg-info/top_level.txt writing dependency_links to TestDistribution.egg-info/dependency_links.txt reading manifest file 'TestDistribution.egg-info/SOURCES.txt' writing manifest file 'TestDistribution.egg-info/SOURCES.txt' running build_ext Creating /tmp/td/lib/python2.6/site-packages/TestDistribution.egg-link (link to .) Adding TestDistribution 0.1a to easy-install.pth file

Installed /tmp/td/src/TestDistribution Processing dependencies for TestDistribution==0.1a Finished processing dependencies for TestDistribution==0.1a

(td)pw@slinktopp2 TestDistribution $ python -c "import foo" Traceback (most recent call last): File "<string>", line 1, in <module> ImportError: No module named foo

(td)pw@slinktopp2 TestDistribution $ python -c "import bar" Traceback (most recent call last): File "<string>", line 1, in <module> ImportError: No module named bar }}}

But it works fine when running setup.py install: {{{ (td)pw@slinktopp2 TestDistribution $ rm /tmp/td/lib/python2.6/site-packages/TestDistribution.egg-link /bin/rm: remove regular file `/tmp/td/lib/python2.6/site-packages/TestDistribution.egg-link'? y

(td)pw@slinktopp2 TestDistribution $ python setup.py install

running install install_dir /tmp/td/lib/python2.6/site-packages/ running bdist_egg running egg_info writing TestDistribution.egg-info/PKG-INFO writing top-level names to TestDistribution.egg-info/top_level.txt writing dependency_links to TestDistribution.egg-info/dependency_links.txt reading manifest file 'TestDistribution.egg-info/SOURCES.txt' writing manifest file 'TestDistribution.egg-info/SOURCES.txt' installing library code to build/bdist.linux-x86_64/egg running install_lib running build_py creating build/bdist.linux-x86_64/egg creating build/bdist.linux-x86_64/egg/bar copying build/lib.linux-x86_64-2.6/bar/init.py -> build/bdist.linux-x86_64/egg/bar creating build/bdist.linux-x86_64/egg/foo copying build/lib.linux-x86_64-2.6/foo/init.py -> build/bdist.linux-x86_64/egg/foo byte-compiling build/bdist.linux-x86_64/egg/bar/init.py to init.pyc byte-compiling build/bdist.linux-x86_64/egg/foo/init.py to init.pyc creating build/bdist.linux-x86_64/egg/EGG-INFO copying TestDistribution.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO copying TestDistribution.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO copying TestDistribution.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO copying TestDistribution.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO zip_safe flag not set; analyzing archive contents... creating 'dist/TestDistribution-0.1a-py2.6.egg' and adding 'build/bdist.linux-x86_64/egg' to it removing 'build/bdist.linux-x86_64/egg' (and everything under it) Processing TestDistribution-0.1a-py2.6.egg creating /tmp/td/lib/python2.6/site-packages/TestDistribution-0.1a-py2.6.egg Extracting TestDistribution-0.1a-py2.6.egg to /tmp/td/lib/python2.6/site-packages Removing TestDistribution 0.1a from easy-install.pth file Adding TestDistribution 0.1a to easy-install.pth file

Installed /tmp/td/lib/python2.6/site-packages/TestDistribution-0.1a-py2.6.egg Processing dependencies for TestDistribution==0.1a Finished processing dependencies for TestDistribution==0.1a

(td)pw@slinktopp2 TestDistribution $ python -c "import foo" (td)pw@slinktopp2 TestDistribution $ python -c "import bar" }}}

Comments (21)

  1. Jason R. Coombs

    Alex and I had an interesting conversation (last November) about this problem and the challenges it entails. I don't see that our conversation was logged publically, but I'll check my local logs to see if I can find the conversation, which outlines some of the inherent challenges with this effort.

  2. Jason R. Coombs

    Here's the log of that conversation:

    (20:00:21) aclark: jaraco: i think i found a 'python setup.py develop' bug and it occurs to me with distribute i could actually fix and release it…
    (20:18:28) jaraco: aclark: tell me about it.
    (20:23:11) aclark: jaraco: django_pci_auth has multiple packages in a single distribution and "setup.py develop" appears to not handle them properly. "setup.py install" installs an egg (which has the packages in them) and works as expected: https://raw.github.com/aclark4life/django-pci-auth/master/wtf.png
    (20:24:04) aclark: jaraco: not sure what the fix is yet, but i can probably whip up something to show the error
    (20:24:26) jaraco: I'll look at django-pci-auth
    (20:24:29) jaraco: Is it open source?
    (20:25:08) aclark: jaraco: https://github.com/aclark4life/django-pci-auth
    (20:25:14) jaraco: ah, yes.
    (20:25:17) jaraco: I see setup.py
    (20:25:28) jaraco: It declares two packages 'django_pci_auth' and 'axes'
    (20:25:36) jaraco: Then it declares two package_dir.
    (20:25:49) jaraco: I never remember quite how those work.
    (20:26:36) aclark: jaraco: i do because i just spent three days figuring it out ;-)
    (20:26:57) aclark: jaraco: this sort of explains it: https://github.com/aclark4life/dist_test
    (20:30:27) jaraco: So that diff you sent - is that the sys.path using setup.py develop (wtf1) versus setup.py install (wtf2) ?
    (20:31:07) aclark: jaraco: as does: http://www.python.org/files/sigs/sc_submission.html just over half way down
    (20:31:16) aclark: jaraco: yep wtf1 is develop
    (20:32:11) aclark: hrm, i haven't checked out install_path yet, wonder what that does
    (20:32:43) jaraco: Is there a reason you didn't use package_dir = {'axes': 'django-axes/axes', 'django_pci_auth': 'django_pci_auth'}
    (20:33:36) aclark: jaraco: i don't think that's the right syntax…
    (20:34:12) jaraco: no? Isn't package_dir a mapping of package names to the folders where those packages exist?
    (20:34:24) aclark: it's certainly the syntax you'd like it to be :-)
    (20:36:53) aclark: jaraco: not exactly… i don't think it will let you specify "django-axes/axes".
    (20:37:15) aclark: jaraco: it's so convoluted it's actually hard to explain what it does other than "counter-intuitive"
    (20:37:37) aclark: In this example:
    (20:37:38) aclark:   setup (
    (20:37:38) aclark:     ...
    (20:37:38) aclark:     packages = ['foo', 'foo.bar']
    (20:37:40) aclark:     package_dir = { 'foo': 'src' }
    (20:37:43) aclark:   )
    (20:37:49) aclark: src/bar becomes foo.bar
    (20:37:56) aclark: that should be reason enough to jump out the window…
    (20:38:01) aclark: but that's how it works
    (20:38:10) jaraco: But that's consistent with my description.
    (20:38:36) aclark: Not really. you said "folders where those packages exist" and there is no foo package inside src
    (20:38:54) jaraco: When you write {'foo': anything}, anything _is_ the foo package (and should contain __init__. It's only confusing because the naming convention is fighting the purpose of package_dir.
    (20:39:13) jaraco: No, foo package _is_ source.
    (20:39:17) jaraco: *src*
    (20:39:22) aclark: Right
    (20:39:34) jaraco: So it would be unwise to name it that way.
    (20:39:49) aclark: Anyway, this isn't the bug :-)
    (20:39:52) jaraco: Have you tried 'axes': 'django-axes/axes' ?
    (20:40:20) jaraco: Which says that the directory 'django-axes/axes' is the package 'axes'.
    (20:40:27) aclark: Probably… but I can always try it again for fun. The module already loads though, so I'm not sure why i'd break it.
    (20:40:40) jaraco: Ah, right.
    (20:40:53) jaraco: Because 'django-axes' is being added to sys.path.
    (20:41:04) aclark: i.e. it works as expected (given the convoluted nature of the system)
    (20:41:11) aclark: Well
    (20:41:13) aclark: When it fails
    (20:41:14) aclark: in develop
    (20:41:23) aclark: it's django_pci_auth that is not found
    (20:41:45) aclark: When you "install", both axes/ and django_pci_auth/ are inside the django_pci_auth egg
    (20:42:07) aclark: Let me try what you suggested just to remind myself
    (20:42:11) jaraco: I can see why it might be failing.
    (20:42:38) jaraco: Consider this. What if you had named the directory django_pci_auth_pkg.
    (20:42:39) aclark: jaraco: Error: No module named axes
    (20:42:53) aclark: ok
    (20:43:13) jaraco: And then in the setup.py, package_dir={'django_pci_auth': 'django_pci_auth_pkg', ...}
    (20:43:31) aclark: This is all just dancing around that insane syntax which I've already figured out
    (20:43:44) aclark: I don't care about that anymore, as I'm already going to have nightmares :-)
    (20:43:59) aclark: Only care about what I think is "develop" doing the wrong thing
    (20:44:05) jaraco: In other words, you've named them the same, but setuptools can't know that you've named them the same.
    (20:44:14) jaraco: How would setup.py develop work in that case?
    (20:44:19) aclark: (and of course, this is ridiculous edge case, but it should still work)
    (20:44:35) jaraco: What could it add to sys.path to cause django_pci_auth to be importable?
    (20:44:40) aclark: setup.py develop should work like install does; i shouldn't get an import error
    (20:44:43) aclark: brb
    (20:44:48) jaraco: !
    (20:48:57) aclark: i mean
    (20:49:09) aclark: the egg is in sys.path and the egg has django_pci_auth and axes packages
    (20:49:31) aclark: is there any way to get develop to add both packages to sys.path? right now it adds axes only AFAICT
    (20:49:36) jaraco: That's right, because setup.py was able to resolve 'django_pci_auth' and 'axes' packages to actual directories.
    (20:49:48) aclark: and develop can't?
    (20:49:51) jaraco: But again, re-read what I wrote before.
    (20:50:01) aclark: ?
    (20:50:03) jaraco: How would develop do that if it doesn't have actual directories named the correct thing.
    (20:50:32) jaraco: What if you had named the directory django_pci_auth as django_pci_auth_pkg.
    (20:50:40) jaraco: How would setup.py develop add that to sys.path?
    (20:50:45) aclark: Presumably the same way the egg does it… or add some .pths or something
    (20:50:56) jaraco: The egg does it by creating a copy.
    (20:51:08) jaraco: But if you have a copy, then you won't be editing the live files.
    (20:51:26) aclark: So develop should create a .pth for each thing install would copy maybe
    (20:52:00) jaraco: And that's effectively what it does, except how can it do that if the package doesn't exist as a directory in the file system.
    (20:52:02) jaraco: ?
    (20:52:28) jaraco: So in your particular case, it would be possible, but only because 'django_pci_auth' maps to the same name.
    (20:53:06) jaraco: But if it didn't map to the exact same name, it would not be possible, even with a .pth file, to cause 'django_pci_auth_pkg' to be imported as 'django_pci_auth'
    (20:54:08) jaraco: So to make setup.py develop work when package_dir is anything other than {'': something}, it would require some tricky acrobatics.
    (20:54:44) jaraco: And would only work on systems that support symlinks.
    (20:54:58) jaraco: Here's what I suggest instead.
    (20:55:36) jaraco: Remove the package_dir directive altogether, and create a symbolic link so setup.py finds both packages in ''.
    (20:55:38) jaraco: i.e.
    (20:55:55) jaraco: ln -s django-axes/axes axes
    (20:56:17) jaraco: You're welcome to create a patch for distribute as well.
    (20:56:27) aclark: Ugh, maybe. OK.
    (20:57:08) jaraco: But just beware that the problem is probably more tricky than it seems at first blush.
    (21:01:03) aclark: No I'm already aware it's pretty tricky :-)
    (21:01:12) jaraco: okay, then!
    (21:01:56) jaraco: Send a patch or a pull request. It'd be nice if develop worked in more cases.
    (21:02:50) aclark: Well here's what I don't get
    (21:03:04) aclark: I can actually import django_pci_auth and its settings when i run python after running develop…
    (21:03:07) aclark: But: ImportError: Could not import settings 'django_pci_auth.settings' (Is it on sys.path?): No module named django_pci_auth.settings
    (21:03:36) aclark: happens
    (21:03:50) aclark: when i try to start django… and that's when i printed out sys.path
    (21:04:11) jaraco: I'm guessing that starting django removes '' from sys.path (your current directory).
    (21:06:00) jaraco: A related issue: https://bitbucket.org/tarek/distribute/issue/178/develop-with-use2to3-should-use-build-lib
    (21:06:45) jaraco: Oh, and I think this is your issue: https://bitbucket.org/tarek/distribute/issue/177/setuppy-develop-doesnt-support-package_dir
    (21:08:06) aclark: Looks like it
    (21:09:29) aclark: Although:
    (21:09:30) aclark: package_dir = { 'foo': 'foo/foo', 'bar': 'bar/bar',
    (21:09:31) aclark: }
    (21:09:44) aclark: Is suspect based on the way I've seen package_dir work
    (21:09:59) aclark: Wuuuut a mess.
    (21:10:33) aclark: Does distutils have any tests?
    (21:14:03) aclark: jaraco: here's the full tb FWIW: https://gist.github.com/4076214 I mistakenly though the imports failed in python but they don't…
    (21:14:31) aclark: So maybe the reported bug was false and develop is doing the right thing but something else fails (django in this case)
    (21:14:44) aclark: weird
    (21:15:18) jaraco: distutils does have tests (in the Python project) and distribute also has tests.
    (21:15:49) jaraco: I think the bug is valid and correctly reported.
    (21:16:13) aclark: jaraco: ah! so if django removes '' that's exactly my problem…
    (21:16:41) aclark: jaraco: well if you look at TestDistribution it has the wrong syntax for package_dir, you can't have path separators
    (21:16:58) aclark: you want to have them, but you can't. (you can call that a bug i guess)
    (21:18:07) jaraco: Are you sure you can't have path separators?
    (21:18:18) aclark: Pretty sure
    (21:18:28) aclark: I just tested it again when you asked
    (21:18:38) aclark: What i have works, path separators do not
    (21:18:48) jaraco: Whet if you add the path separators, but then run setup.py install?
    (21:18:59) aclark: And if you look at gwards docs there are none… i'll try
    (21:18:59) jaraco: My guess is the installed package will work.
    (21:19:14) jaraco: I think the path separators are valid.
    (21:19:28) aclark: pftsss beers at zpug in 2013 for the winner? :-)
    (21:19:46) aclark: You acredit way to much sanity to Greg Ward :-p
    (21:26:07) ***aclark owes jaraco a beer
    (21:26:09) aclark: !dm aclark
    (21:26:21) aclark: hah! and there is not even a pmxbot to demotivate me
    (21:26:34) jaraco: lucky day
    (21:27:20) aclark: in that case i don't understand why it worked with my wacky syntax
    (21:28:11) aclark: jaraco: so if the bug is valid it can be fixed? or is still too tricky like you said
    (21:29:09) jaraco: it worked because (a) your current directory was sufficient to find django_pci_auth and (b) package_dir[''] contained 'auth'. The two together made it work.
    (21:29:15) jaraco: Yes, it's still valid.
    (21:29:16) aclark: jaraco: i also wonder if path seps work on windows
    (21:29:30) jaraco: And it's still tricky. I don't think you can do it without adding a custom loader to distribute.
    (21:29:35) aclark: jaraco: ah ok thx
    (21:29:41) jaraco: Windows does accept '/' for path seps.
    (21:29:43) aclark: now i see why you made the _pkg point
    (21:29:47) aclark: cool
    
  3. Jason R. Coombs

    And a few more details the following day:

    (09:45:25) aclark: jaraco: how about having `develop` install the "wrong" dir into sys.path e.g. "src" thereby requiring the user to name the dir to match, i could live with that
    (09:45:36) aclark: jaraco: (and thanks for all the help yesterday!)
    (09:49:44) jaraco: aclark: if we're going to require the user to rename the dir to match, I feel like it's only half-solving the issue... there's nothing about package_dir which says that you must name the dir to match the package name.
    (09:50:16) jaraco: The other issue is that might expose even more packages than are desirable.
    (09:50:49) jaraco: For example, if 'src' contained two packages, both would now be on sys.path, even though setup.py would have only used one.
    (09:53:30) jaraco: As I think about the problem more and more, I wonder if there's a good way that setuptools could install a custom import finder that would find packages based on some metadata created by setup.py develop.
    (09:54:48) aclark: interesting
    (09:55:01) aclark: jaraco: in the meantime i just mangled sys.path in django-admin.py
    (09:58:05) jaraco: If we built a custom finder, setup.py develop could probably also be made to work on Python 3 with 2to3 projects (where the packages get built first in another location).
    (09:58:06) aclark: jaraco: also does package_dir work w/namespace packages by any chance? I want to do this: https://github.com/aclark4life/plock i.e. include a bunch of namespace packages in a single distribution
    (09:58:22) aclark: jaraco: nice
    (09:58:23) jaraco: Of course, that would still require that 2to3 be run after every edit, which isn't really the 'develop' mode.
    
  4. Jason R. Coombs

    I don't think there's a quick fix for this issue. It's very complicated and somewhat nuanced. The package_dir parameter essentially says "use directory A as package B". But the 'develop' command relies on the fact that A == B and adds the parent of A to sys.path, allowing the import of package B. When A != B, the default import mechanism can't be used. In fact, it's possible for the package name to not exist in the file system at all. In that case, how should Python import that package?

    I imagine symlinks could be used to fake the package directories... or maybe a custom finder could be created to locate packages based on package_dir metadata. I like the latter solution better because symlink creation is difficult on Windows (due to security constraints) and only available in Python 3.2+.

    Of course, this approach would need to be carefully developed and wouldn't be a quick fix. Sorry.

  5. slinkp reporter

    The link did indeed go to setuptools 230, whose title is " python setup.py upload_docs doesn't ask for login and password." What does that have to do with "setup.py develop"?

  6. Jason R. Coombs

    Sorry. When I first wrote the comment, I had written "Setuptools <hash>230", which even though I added the hyperlink, Bitbucket replaces <hash>230 with a link to issue 230 in this project, so Distribute 230. I noticed the error right away and corrected it, but not before you got to the link (or found it cached in your e-mail). It's since been corrected to point to Setuptools 230.

  7. Felipe

    Very interesting problem. I realize this doesn't solve all of the nuances for everyone, but what I've done on my project (package_dir={'minecart': 'src'}) is symlinked minecart to src, and then added minecart/ to my .gitignore. I realize there are other issues here, but for me the most important was to keep src named src and yet be able to import it when in the virtual env.

  8. Jason R. Coombs

    @Peter Ruibal In the 2014-07-11 change where I marked this issue as resolved, I linked to Setuptools 230, where this issue can proceed. The distribute project is defunct and will not be getting any maintenance. I observed just now that Bitbucket was no longer honoring the relative path to /pypa/setuptools/issues/230 so I added the scheme and host and the link now resolves to the proper issue.

  9. Log in to comment