respect the PYTHONPATH

Zooko O'Whielacronx avatarZooko O'Whielacronx created an issue

http://bugs.python.org/setuptools/issue53

This is currently causing a problem for developers of the Tahoe-LAFS project. The ticket for Tahoe-LAFS is: http://allmydata.org/trac/tahoe-lafs/ticket/709

There is a patch attached to http://bugs.python.org/setuptools/issue53 and some discussion about the patch.

Comments (6)

  1. Zooko O'Whielacronx

    Here is a demonstration of this issue on the pycryptopp buildbot. The buildsteps in question are:

    step 9: create-egg

    python setup.py bdist_egg
    

    step 10: install-to-egg

    python -c "import glob, os, subprocess, sys; os.mkdir('egginstalldir'); tahoe_egg = glob.glob(os.path.join('dist', '*.egg'))[0]; sys.exit(subprocess.call(['easy_install', '-d', 'egginstalldir', tahoe_egg]))"
    

    step 11: test-from-egg

    python -c "import glob,os,subprocess,sys;trial=os.path.join(os.getcwd(), 'misc', 'build_helpers', 'run_trial.py');os.chdir('.');testsuite=subprocess.Popen([sys.executable, 'setup.py', '--name'], stdout=subprocess.PIPE).communicate()[0].strip()+'.test';os.chdir('egginstalldir');os.environ['PATH']=os.getcwd()+os.pathsep+os.environ['PATH'];os.environ['PYTHONPATH']=os.pathsep.join(glob.glob('*.egg'))+os.pathsep+os.environ.get('PYTHONPATH','');sys.exit(subprocess.call([sys.executable, trial, testsuite], env=os.environ))"
    

    Here are three consecutive builds on the buildbot of the exact same pycryptopp source code but with different packaging tools installed. The first one is with my own private fork of setuptools, which I call "zetuptoolz":

    http://tahoe-lafs.org/buildbot-pycryptopp/builders/Zooko%20zomp%20Mac-amd64%2010.6%20py2.6/builds/28

    You can see the declaration of which version of setuptools it is running in the output of step 2 (show-tool-versions):

    setuptools: [setuptools 0.6c14devdev (/Library/Python/2.6/site-packages/setuptools-0.6c14devdev-py2.6.egg)]
    

    You can see that it correctly loads and runs the code which is in the .egg file which is at the beginning of the PYTHONPATH, in step 11:

    http://tahoe-lafs.org/buildbot-pycryptopp/builders/Zooko%20zomp%20Mac-amd64%2010.6%20py2.6/builds/28/steps/test-from-egg/logs/stdio

    The next build is with setuptools:

    http://tahoe-lafs.org/buildbot-pycryptopp/builders/Zooko%20zomp%20Mac-amd64%2010.6%20py2.6/builds/29

    Declaration of version of setuptools:

    setuptools: [setuptools 0.7a1dev-r78878 (/Library/Python/2.6/site-packages/setuptools-0.7a1dev_r78878-py2.6.egg)]
    

    You can see that it incorrectly loads the code from a different .egg which is installed in the system instead of the .egg which is passed at the front of the PYTHONPATH, thus contradicting the specification of the python search path in http://docs.python.org/install/index.html?highlight=pythonpath%20path#modifying-python-s-search-path

    http://tahoe-lafs.org/buildbot-pycryptopp/builders/Zooko%20zomp%20Mac-amd64%2010.6%20py2.6/builds/29/steps/test-from-egg/logs/stdio

    The test script notices that it is testing the wrong module and errors out with this message:

    AssertionError: We seem to be testing the code at '/Library/Python/2.6/site-packages'
    (according to the source filename '/Library/Python/2.6/site-packages/pycryptopp-0.5.17-py2.6-macosx-10.6-universal.egg/pycryptopp/test/__init__.pyc'),
    but expected to be testing the code at '/Users/pycryptoppbuildslave/Zooko zomp Mac-amd64 10.6 py2.6/build/egginstalldir'.
    

    The third build is with distribute:

    http://tahoe-lafs.org/buildbot-pycryptopp/builders/Zooko%20zomp%20Mac-amd64%2010.6%20py2.6/builds/30

    Declaration of version of setuptools:

    setuptools: [distribute 0.6.14 (/Library/Python/2.6/site-packages/distribute-0.6.14-py2.6.egg)]
    

    Results: same as setuptools:

    http://tahoe-lafs.org/buildbot-pycryptopp/builders/Zooko%20zomp%20Mac-amd64%2010.6%20py2.6/builds/30/steps/test-from-egg/logs/stdio

    AssertionError: We seem to be testing the code at '/Library/Python/2.6/site-packages'
    (according to the source filename '/Library/Python/2.6/site-packages/pycryptopp-0.5.17-py2.6-macosx-10.6-universal.egg/pycryptopp/test/__init__.pyc'),
    but expected to be testing the code at '/Users/pycryptoppbuildslave/Zooko zomp Mac-amd64 10.6 py2.6/build/egginstalldir'.
    
  2. Zooko O'Whielacronx

    Here is the relevant patch from zetuptoolz:

    http://tahoe-lafs.org/trac/zetuptoolz/changeset/20081115185932-92b7f-d451b10759897b0fa5d038f2c42cdcd3901289a8/trunk

    Index: setuptools/command/easy_install.py
    ===================================================================
    --- setuptools/command/easy_install.py  (revision 560)
    +++ setuptools/command/easy_install.py  (revision 562)
    @@ -1365,5 +1365,5 @@
                     "import sys; new=sys.path[sys.__plen:];"
                     " del sys.path[sys.__plen:];"
    -                " p=getattr(sys,'__egginsert',0); sys.path[p:p]=new;"
    +                " p=getattr(sys,'__egginsert',len(os.environ.get('PYTHONPATH','').split(os.pathsep))); sys.path[p:p]=new;"
                     " sys.__egginsert = p+len(new)\n"
                 ) % data
    
    
  3. Zooko O'Whielacronx

    Here is the same test with the current tip of the 0.6-maintenance branch of distribute:

    http://tahoe-lafs.org/buildbot-pycryptopp/builders/Zooko%20zomp%20Mac-amd64%2010.6%20py2.6/builds/31

    And here is the same test with the following patch applied to distribute:

    http://tahoe-lafs.org/buildbot-pycryptopp/builders/Zooko%20zomp%20Mac-amd64%2010.6%20py2.6/builds/33

    --- a/setuptools/command/easy_install.py        Tue Sep 21 17:35:08 2010 +0200
    +++ b/setuptools/command/easy_install.py        Tue Sep 21 22:28:34 2010 -0600
    @@ -1536,7 +1536,7 @@
                     "%s\n"
                     "import sys; new=sys.path[sys.__plen:];"
                     " del sys.path[sys.__plen:];"
    -                " p=getattr(sys,'__egginsert',0); sys.path[p:p]=new;"
    +                " p=getattr(sys,'__egginsert',len(os.environ.get('PYTHONPATH','').split(os.pathsep))); sys.path[p:p]=new;"
                     " sys.__egginsert = p+len(new)\n"
                 ) % data
     
    
  4. Anonymous
  5. Log in to comment
Tip: Filter by directory path e.g. /media app.js to search for public/media/app.js.
Tip: Use camelCasing e.g. ProjME to search for ProjectModifiedEvent.java.
Tip: Filter by extension type e.g. /repo .js to search for all .js files in the /repo directory.
Tip: Separate your search with spaces e.g. /ssh pom.xml to search for src/ssh/pom.xml.
Tip: Use ↑ and ↓ arrow keys to navigate and return to view the file.
Tip: You can also navigate files with Ctrl+j (next) and Ctrl+k (previous) and view the file with Ctrl+o.
Tip: You can also navigate files with Alt+j (next) and Alt+k (previous) and view the file with Alt+o.