easy_install or 'setup.py develop' with setup.cfg ignores dependency_links and allow_hosts

Issue #264 open
Michael Pedersen
created an issue

This issue is actually somewhat complex, and decidedly different behavior from the way that setuptools works. While it affects the TurboGears tools, I believe it will affect others at some point too.

We provide two packages for use with TurboGears: TurboGears2 and tg.devtools. We're only now deprecating support for Python 2.4 (we had to wait until CentOS stopped supplying it by default).

This means that we have to have special, old versions of the packages that we use. In order to accomplish this, we specified the dependency_links in our setup.py, and the allow_hosts directive in our setup.cfg. This means that when someone does "easy_install tg.devtools", everything just works: Our egg index is used, and only that index. People got the versions of the packages that they needed, and it worked.

Now, virtualenv has switched to using distribute by default. As a result, we are getting more and more people who have their virtualenv ignoring the directives we supply, resulting in them getting versions of the packages that are known to fail.

To make things even more confusing for people, if they do "easy_install tg.devtools", and hit Control-C immediately after that package is installed (before any others are downloaded), and then re-run the same command, then these directives will be honored and used.

Finally, when using "python setup.py develop", these directives are also ignored. The same problems occur there.

Comments (2)

  1. Jason R. Coombs
    • changed status to open

    I just ran easy_install tg.devtools using a virtualenv with distribute and setuptools, and both gave basically the same behavior. Both seemed to honor find_links and neither seemed to honor setup.cfg (specifically allow_hosts). Here's the output from the virtualenv with setuptools:

    PS C:\Users\jaraco\envs> virtualenv tgtest
    New python executable in tgtest\Scripts\python.exe
    Installing setuptools................done.
    Installing pip...................done.
    PS C:\Users\jaraco\envs> .\tgtest\Scripts\easy_install tg.devtools
    Searching for tg.devtools
    Reading http://pypi.python.org/simple/tg.devtools/
    Reading http://www.turbogears.org
    Reading http://www.turbogears.org/2.0/downloads/2.0.2/
    Reading http://www.turbogears.org/2.0/downloads/2.0.1/
    Reading http://pypi.python.org/simple/tg.devtools/www.turbogears.org
    Reading http://www.turbogears.org/2.1/downloads/2.1.4/
    Best match: tg.devtools 2.1.4
    Downloading http://www.turbogears.org/2.1/downloads/2.1.4/tg.devtools-2.1.4.tar.gz
    Processing tg.devtools-2.1.4.tar.gz
    Running tg.devtools-2.1.4\setup.py -q bdist_egg --dist-dir c:\users\jaraco\appdata\local\temp\easy_install-tlyqa3\tg.devtools-2.1.4\egg-dist-tmp-vskd05
    Adding tg.devtools 2.1.4 to easy-install.pth file
    Installed c:\users\jaraco\envs\tgtest\lib\site-packages\tg.devtools-2.1.4-py2.7.egg
    Reading http://www.turbogears.org/2.1/downloads/current/
    Processing dependencies for tg.devtools
    Searching for TurboGears2>=2.1.4
    Reading http://pypi.python.org/simple/TurboGears2/
    Reading http://www.turbogears.org/2.0/downloads/2.0.3/
    Reading http://www.turbogears.org/
    Best match: TurboGears2 2.1.4
    Downloading http://www.turbogears.org/2.1/downloads/2.1.4/TurboGears2-2.1.4.tar.gz
    Processing TurboGears2-2.1.4.tar.gz
    Running TurboGears2-2.1.4\setup.py -q bdist_egg --dist-dir c:\users\jaraco\appdata\local\temp\easy_install-uoed9b\TurboGears2-2.1.4\egg-dist-tmp-wemczl
    Adding turbogears2 2.1.4 to easy-install.pth file
    Installed c:\users\jaraco\envs\tgtest\lib\site-packages\turbogears2-2.1.4-py2.7.egg
    Searching for Babel
    Reading http://pypi.python.org/simple/Babel/
    Reading http://babel.edgewall.org/
    Reading http://babel.edgewall.org/wiki/Download
    Best match: Babel 0.9.6
    Downloading http://ftp.edgewall.com/pub/babel/Babel-0.9.6-py2.7.egg
    Processing Babel-0.9.6-py2.7.egg
    creating c:\users\jaraco\envs\tgtest\lib\site-packages\Babel-0.9.6-py2.7.egg
    Extracting Babel-0.9.6-py2.7.egg to c:\users\jaraco\envs\tgtest\lib\site-packages
    Adding Babel 0.9.6 to easy-install.pth file
    Installing pybabel-script.py script to C:\Users\jaraco\envs\tgtest\Scripts
    Installing pybabel.exe script to C:\Users\jaraco\envs\tgtest\Scripts
    Installing pybabel.exe.manifest script to C:\Users\jaraco\envs\tgtest\Scripts
    Installed c:\users\jaraco\envs\tgtest\lib\site-packages\babel-0.9.6-py2.7.egg
    Searching for WebError>=0.10.1
    Reading http://pypi.python.org/simple/WebError/
    Best match: WebError 0.10.3
    Downloading http://www.turbogears.org/2.1/downloads/2.1.4/WebError-0.10.3.tar.gz
    Processing WebError-0.10.3.tar.gz
    Running WebError-0.10.3\setup.py -q bdist_egg --dist-dir c:\users\jaraco\appdata\local\temp\easy_install-yhdu5i\WebError-0.10.3\egg-dist-tmp-r3m86q
    warning: no files found matching 'eval_template.html' under directory 'weberror'
    Adding weberror 0.10.3 to easy-install.pth file
    Installed c:\users\jaraco\envs\tgtest\lib\site-packages\weberror-0.10.3-py2.7.egg
    Searching for WebFlash>=0.1a8
    Reading http://pypi.python.org/simple/WebFlash/
    Reading http://python-rum.org/wiki/WebFlash
    Best match: WebFlash 0.1a9
    Downloading http://www.turbogears.org/2.1/downloads/2.1.4/WebFlash-0.1a9.tar.gz
    Processing WebFlash-0.1a9.tar.gz
    Running WebFlash-0.1a9\setup.py -q bdist_egg --dist-dir c:\users\jaraco\appdata\local\temp\easy_install-go3cb1\WebFlash-0.1a9\egg-dist-tmp-uqk2fj
    Adding webflash 0.1a9 to easy-install.pth file
    Installed c:\users\jaraco\envs\tgtest\lib\site-packages\webflash-0.1a9-py2.7.egg
    Searching for Pylons>=0.9.7
    Reading http://pypi.python.org/simple/Pylons/
    Reading http://www.pylonshq.com/
    Best match: Pylons 1.0.1rc1
    Downloading http://pypi.python.org/packages/source/P/Pylons/Pylons-1.0.1rc1.tar.gz#md5=02f218f879c5b71737423196dea6d227
    Processing Pylons-1.0.1rc1.tar.gz
    Running Pylons-1.0.1rc1\setup.py -q bdist_egg --dist-dir c:\users\jaraco\appdata\local\temp\easy_install-r_nzwv\Pylons-1.0.1rc1\egg-dist-tmp-wrrg9n
    warning: no previously-included files matching '*' found under directory 'tests\test_units\session'
    warning: no previously-included files matching '.DS_Store' found anywhere in distribution
    warning: no previously-included files matching '*.hgignore' found anywhere in distribution
    warning: no previously-included files matching '*.hgtags' found anywhere in distribution
    Adding pylons 1.0.1rc1 to easy-install.pth file
    Installed c:\users\jaraco\envs\tgtest\lib\site-packages\pylons-1.0.1rc1-py2.7.egg
    Reading http://www.pylonshq.com/download/1.0
    Searching for WebOb==1.0.8
    Reading http://pypi.python.org/simple/WebOb/
    Reading http://pythonpaste.org/webob/
    Reading http://webob.org/
    Best match: WebOb 1.0.8
    Downloading http://www.turbogears.org/2.1/downloads/2.1.4/WebOb-1.0.8.zip
    Processing WebOb-1.0.8.zip
    Running WebOb-1.0.8\setup.py -q bdist_egg --dist-dir c:\users\jaraco\appdata\local\temp\easy_install-1jfw2q\WebOb-1.0.8\egg-dist-tmp-0kuaoe
    no previously-included directories found matching '*.pyc'
    no previously-included directories found matching '*.pyo'
    Adding webob 1.0.8 to easy-install.pth file
    Installed c:\users\jaraco\envs\tgtest\lib\site-packages\webob-1.0.8-py2.7.egg
    Searching for Paste>=1.7.1
    Reading http://pypi.python.org/simple/Paste/
    Reading http://pythonpaste.org
    Best match: Paste
    Downloading http://www.turbogears.org/2.1/downloads/2.1.4/Paste-
    Processing Paste-
    Running Paste-\setup.py -q bdist_egg --dist-dir c:\users\jaraco\appdata\local\temp\easy_install-mswldk\Paste-\egg-dist-tmp-rtaxdy
    warning: no previously-included files matching '*' found under directory 'docs\_build\_sources'
    Adding paste to easy-install.pth file
    Installed c:\users\jaraco\envs\tgtest\lib\site-packages\paste-
    Searching for Pygments
    Reading http://pypi.python.org/simple/Pygments/
    Reading http://pygments.org/
    Reading http://pygments.pocoo.org/
    Best match: Pygments 1.5
    Downloading http://pypi.python.org/packages/2.7/P/Pygments/Pygments-1.5-py2.7.egg#md5=8ea02c2eafb2580336b7637be629f7ce
    Processing Pygments-1.5-py2.7.egg
    creating c:\users\jaraco\envs\tgtest\lib\site-packages\Pygments-1.5-py2.7.egg
    Extracting Pygments-1.5-py2.7.egg to c:\users\jaraco\envs\tgtest\lib\site-packages
    Adding Pygments 1.5 to easy-install.pth file
    Installing pygmentize-script.py script to C:\Users\jaraco\envs\tgtest\Scripts
    Installing pygmentize.exe script to C:\Users\jaraco\envs\tgtest\Scripts
    Installing pygmentize.exe.manifest script to C:\Users\jaraco\envs\tgtest\Scripts
    Installed c:\users\jaraco\envs\tgtest\lib\site-packages\pygments-1.5-py2.7.egg
    Searching for Tempita
    Reading http://pypi.python.org/simple/Tempita/
    Reading http://pythonpaste.org/tempita/
    Best match: Tempita 0.5.1
    Downloading http://www.turbogears.org/2.1/downloads/2.1.4/Tempita-0.5.1.tar.gz
    Processing Tempita-0.5.1.tar.gz
    Running Tempita-0.5.1\setup.py -q bdist_egg --dist-dir c:\users\jaraco\appdata\local\temp\easy_install-mpjbxs\Tempita-0.5.1\egg-dist-tmp-9rrzrk
    C:\Python\Lib\distutils\dist.py:267: UserWarning: Unknown distribution option: 'use_2to3'
    Adding tempita 0.5.1 to easy-install.pth file
    Installed c:\users\jaraco\envs\tgtest\lib\site-packages\tempita-0.5.1-py2.7.egg
    error: Installed distribution webob 1.0.8 conflicts with requirement WebOb>=1.1.1

    Can you provide an example where the install succeeds with setuptools but fails with distribute? Do you know why I'm not able to reproduce what you say happens with setuptools (your index is used and only your index)?

    I agree that the behavior you expect is desirable. If this behavior is already in setuptools, we should provide a similar implementation. If not, we will have to come up with one from scratch (and be more cautious about the implications).

  2. Michael Pedersen reporter

    Unfortunately, not anymore. I would *swear* that this worked when I tested it back in July, 2011. I know I tested it and re-tested, since we had so much controversy over it within TG. However, a couple of months ago, it was found not to be working, and I was not able to reproduce it at that time.

    All I can figure is that some unique confluence of circumstances that i was not aware of were making it work.

    As for how to fix this, I really don't know. The complexity of this problem grows extremely quickly. What happens if a user depends on TurboGears2, but then has their own dependency_links with packages that provide version conflicts? While this case sounds outrageous, I've seen signs that this is going to start happening sometime soon. Some of our packages that we rely on have introduced new versions that are incompatible, and some users are starting to look at them.

    We're working on fixing it, but we're not there yet. I don't have any answers, only difficult questions.

  3. Log in to comment