Issue #22 wontfix

Default HTTPS PyPI package download failing on Python 2.4.3

Jurko Gospodnetić
created an issue

Installed Python 2.4 on Windows. Installed setuptools 0.7.2 into the new Python 2.4 installation. And then running easy_install on any package fails to download the package from PyPI.

If you change the package's PyPI URL using the easy_install's -i option to use HTTP instead of HTTPS - then everything works fine.

Sniffing the network traffic with Wireshark shows that the client simply shuts down the network connection after receiving a 'hello' message from the server.

Reproduced using:

Python 2.4.3/x86 on Windows 7/SP1/x64 with setuptools 0.7.2

The problem DOES NOT occur when using Python 2.4.4:

Python 2.4.4/x86 on Windows 7/SP1/x64 with setuptools 0.7.2

Console session demonstrating the problem:

C:\>py24 -m easy_install pytest
Searching for pytest
Reading https://pypi.python.org/simple/pytest/
Download error on https://pypi.python.org/simple/pytest/: (2, 'The operation did not complete (read)') -- Some packages may not be found!
Couldn't find index page for 'pytest' (maybe misspelled?)
Scanning index of all packages (this may take a while)
Reading https://pypi.python.org/simple/
Download error on https://pypi.python.org/simple/: (2, 'The operation did not complete (read)') -- Some packages may not be found!
No local packages or download links found for pytest
error: Could not find suitable distribution for Requirement.parse('pytest')

C:\>py24 -m easy_install -i https://pypi.python.org/simple pytest
Searching for pytest
Reading https://pypi.python.org/simple/pytest/
Download error on https://pypi.python.org/simple/pytest/: (2, 'The operation did not complete (read)') -- Some packages may not be found!
Couldn't find index page for 'pytest' (maybe misspelled?)
Scanning index of all packages (this may take a while)
Reading https://pypi.python.org/simple/
Download error on https://pypi.python.org/simple/: (2, 'The operation did not complete (read)') -- Some packages may not be found!
No local packages or download links found for pytest
error: Could not find suitable distribution for Requirement.parse('pytest')

C:\>py24 -m easy_install -i http://pypi.python.org/simple pytest
Searching for pytest
Reading http://pypi.python.org/simple/pytest/
Best match: pytest 2.3.5
Downloading http://pypi.python.org/packages/source/p/pytest/pytest-2.3.5.tar.gz#md5=18f150e7be96b5fe3c388b0e817b8087
Processing pytest-2.3.5.tar.gz
Writing c:\users\jurko\appdata\local\temp\easy_install-obyruy\pytest-2.3.5\setup.cfg
...installation continues from here on...

C:\>py24 -m easy_install -i https://pypi.python.org/simple psutil
Searching for psutil
Reading https://pypi.python.org/simple/psutil/
Download error on https://pypi.python.org/simple/psutil/: (2, 'The operation did not complete (read)') -- Some packages may not be found!
Couldn't find index page for 'psutil' (maybe misspelled?)
Scanning index of all packages (this may take a while)
Reading https://pypi.python.org/simple/
Download error on https://pypi.python.org/simple/: (2, 'The operation did not complete (read)') -- Some packages may not be found!
No local packages or download links found for psutil
error: Could not find suitable distribution for Requirement.parse('psutil')

C:\>py24 -m easy_install -i http://pypi.python.org/simple psutil
Searching for psutil
Reading http://pypi.python.org/simple/psutil/
Reading http://code.google.com/p/psutil/
Best match: psutil 0.7.1
Downloading http://psutil.googlecode.com/files/psutil-0.7.1.tar.gz
Processing psutil-0.7.1.tar.gz
Writing c:\users\jurko\appdata\local\temp\easy_install-q6zmk_\psutil-0.7.1\setup.cfg
...installation continues from here on...

The problem seems to be caused somewhere in Python's HTTPS implementation. Should setuptools then revert to using HTTP on such platforms?

This causes different setup packages on such platforms to fail when they attempt to download their requirement packages. The only workaround I could find for this behaviour as a setup packager was to do something like this in my package.py script:

if sys.version_info < (2, 5):
    # Python 2.4 seems to have issues with setuptools collecting its
    # requirement packages from PyPI using HTTPS. This has been encountered
    # using Python 2.4.3/x86 on Windows 7/SP1/x64 with setuptools 0.7.2. As a
    # workaround we replace setuptools's PackageIndex class with one that
    # always uses the HTTP transfer protocol instead of HTTPS when dealing with
    # this Python version.
    #
    # Note that this workaround affects only setuptools's automated requirement
    # package downloading. Any requirement packages can still be installed
    # manually by the user, using a suitable package index source URL.
    import setuptools.package_index
    OriginalPackageIndex = setuptools.package_index.PackageIndex
    class NoHTTPSPackageIndex(OriginalPackageIndex):
        def __init__(self, *args, **kwargs):
            OriginalPackageIndex.__init__(self, *args, **kwargs)
            cue = "https:"
            if self.index_url.lower().startswith(cue):
                self.index_url = "http:" + self.index_url[len(cue):]
    setuptools.package_index.PackageIndex = NoHTTPSPackageIndex

but the solution seems rather hackish and I am not really sure how portable/stable it will be over different setuptools releases.

Best regards, Jurko Gospodnetić

Comments (5)

  1. Jason R. Coombs

    I don't think the problem is strictly with setuptools. It works for me in my environment. I suspect your issue lies in that Python 2.4 doesn't have built-in SSL support and something about the SSL support you do have is broken. Can you try this simple test and see if your environment is able to use https through Python exclusive of setuptools?

    Python 2.4.4 (#71, Oct 18 2006, 08:34:43) [MSC v.1310 32 bit (Intel)] on win32
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import urllib2
    >>> urllib2.urlopen('https://pypi.python.org/pypi/')
    <addinfourl at 36524360 whose fp = <socket._fileobject object at 0x022DC068>>
    

    I'd like to avoid adding fallback support (which may have security implications as well) if we can help it. That said, if we can demonstrate a good reason why HTTPS does not work in some environments, then we should definitely consider a fallback.

  2. Jurko Gospodnetić reporter

    Hi, thanks for checking the issue report. :-)

    I rechecked on my end and the situation is as follows:

    With Python 2.4.4:

    Everything works as expected - both from the interpreter and when using easy_install.

    I noticed in the console output you quoted that you're using Python 2.4.4. And your version information is exactly the same as the one reported in my working environment.

    With Python 2.4.3:

    1. urllib2 based code you suggested works fine and afterwards I can successfully use the returned object's readline() method to read the data from the network. I tested this using 'https://pypi.python.org/simple/pytest/' as well.

    2. I installed setuptools 0.7.2 into that Python installation, from sources, by running just 'py24 setup.py install' from the setuptools's top level project folder. Then I attempted to run 'py24 -m easy_install pytest' which again failed attempting to download data from 'https://pypi.python.org/simple/pytest/'.

    Could you give it a try using the following:

    C:\Users\Jurko>py24
    Python 2.4.3 (#69, Mar 29 2006, 17:35:34) [MSC v.1310 32 bit (Intel)] on win32
    Type "help", "copyright", "credits" or "license" for more information.
    

    Best regards, Jurko Gospodnetić

  3. Jason R. Coombs

    Thanks for the clarification. I hadn't noticed the 2.4.3 distinction. I read through the changelog for 2.4.4 and don't see anything that jumps out at me that would indicate the issue.

    I may at some point have more time to delve into the issue, but given that it works on 2.4.4, my first recommendation would be for the client systems to upgrade Python to 2.4.4.

    I will entertain pull requests that address this issue, although any workaround should be suitably encapsulated and only invoked when necessary.

  4. Log in to comment