1. Tarek Ziadé
  2. distribute
Issue #256 wontfix

python2.7 throws socket.timeout directly instead of wrapping it in urllib2.URLError

Florian Schulze
created an issue

this should be added before {{{except httplib.BadStatusLine, v:}}} in open_url in package_index.py:

{{{

!python

    except socket.timeout, v:
        if warning:
            self.warn(warning, v)
        else:
            raise DistutilsError("Download error for %s: %s"
                                 % (url, v))

}}}

Comments (6)

  1. Jason R. Coombs

    Your subject asserts "python2.7 throws socket.timeout directly instead of wrapping it in urllib2.URLError". The docs suggest that urllib2.urlopen always wraps errors in URLError, and indeed I find that's the case on my system.

    Python 2.7.2 (default, Jun 12 2011, 14:24:46) [MSC v.1500 64 bit (AMD64)] on win32
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import urllib2
    >>> urllib2.urlopen('http://99.99.99.99:446/foo', timeout=3)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "c:\python\lib\urllib2.py", line 126, in urlopen
        return _opener.open(url, data, timeout)
      File "c:\python\lib\urllib2.py", line 394, in open
        response = self._open(req, data)
      File "c:\python\lib\urllib2.py", line 412, in _open
        '_open', req)
      File "c:\python\lib\urllib2.py", line 372, in _call_chain
        result = func(*args)
      File "c:\python\lib\urllib2.py", line 1199, in http_open
        return self.do_open(httplib.HTTPConnection, req)
      File "c:\python\lib\urllib2.py", line 1174, in do_open
        raise URLError(err)
    urllib2.URLError: <urlopen error timed out>
    

    Can you reference more detail about where and why Python 2.7 might throw a socket.timeout?

  2. Florian Schulze reporter

    (Reply via flor...@gmx.net):

    It happens for me on Ubuntu 11.10 with system Python 2.7 when using distribute-0.6.24. It happened when installing buildout.dumppickedversions while it tried to access the then down http://svn.plone.org/svn/collective/buildout/buildout.dumppickedversions page referenced from the simple index page at http://pypi.python.org/simple/buildout.dumppickedversions/. My buildout was pretty simple:

    [buildout]
    extensions = buildout.dumppickedversions
    parts = supervisor
    versions = versions
    
    [supervisor]
    recipe = zc.recipe.egg
    eggs = supervisor
    
    [versions]
    
  3. Jason R. Coombs

    I have tried reproducing the problem on Ubuntu 11.10 with the system Python 2.7.2+ using distribute 0.6.24 and I cannot reproduce the problem:

    jaraco@livid:~$ python
    Python 2.7.2+ (default, Oct  4 2011, 20:06:09) 
    [GCC 4.6.1] on linux2
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import setuptools
    >>> setuptools.__file__
    '/home/jaraco/.local/lib/python2.7/site-packages/distribute-0.6.24-py2.7.egg/setuptools/__init__.pyc'
    >>> from setuptools import package_index
    >>> package_index.open_with_auth('http://99.99.99.99:446/foo')
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "/home/jaraco/.local/lib/python2.7/site-packages/distribute-0.6.24-py2.7.egg/setuptools/package_index.py", line 753, in _socket_timeout
        return func(*args, **kwargs)
      File "/home/jaraco/.local/lib/python2.7/site-packages/distribute-0.6.24-py2.7.egg/setuptools/package_index.py", line 779, in open_with_auth
        fp = urllib2.urlopen(request)
      File "/usr/lib/python2.7/urllib2.py", line 126, in urlopen
        return _opener.open(url, data, timeout)
      File "/usr/lib/python2.7/urllib2.py", line 394, in open
        response = self._open(req, data)
      File "/usr/lib/python2.7/urllib2.py", line 412, in _open
        '_open', req)
      File "/usr/lib/python2.7/urllib2.py", line 372, in _call_chain
        result = func(*args)
      File "/usr/lib/python2.7/urllib2.py", line 1201, in http_open
        return self.do_open(httplib.HTTPConnection, req)
      File "/usr/lib/python2.7/urllib2.py", line 1171, in do_open
        raise URLError(err)
    urllib2.URLError: <urlopen error timed out>
    

    Every time, a URLError is raised. This leads me to suspect there is something that has broken urllib2 in your environment or perhaps buildout.

    While we could add a fix for the uncaught exception, that exception should not be raised at all according to the spec and according to my experience.

    My inclination is to not introduce a fix for an error for which we cannot reproduce it and which shouldn't be occurring. If you can describe a technique that reproduces the issue, we can investigate further and track down the cause.

  4. Christophe Combelles
    • changed status to open

    I got the same timeout traceback during a moment when werkzeug.pocoo.org was not responding. I'm on Debian Squeeze with Python 2.7.3rc1 and distribute 2.6.26dev. At this time I could pdb a little bit and saw that the timeout was not catched. However I could not artificially reproduce it by changing the werkzeug.pocoo.org IP in my /etc/hosts. So I'm unsure how to reproduce it

    Getting distribution for 'werkzeug'.
    
    An internal error occurred due to a bug in either zc.buildout or in a
    recipe being used:
    Traceback (most recent call last):
      File "/home/ccomb/buildout-eggs/zc.buildout-1.5.2-py2.7.egg/zc/buildout/buildout.py", line 1805, in main
        getattr(buildout, command)(args)
      File "/home/ccomb/buildout-eggs/zc.buildout-1.5.2-py2.7.egg/zc/buildout/buildout.py", line 584, in install
        installed_files = self[part]._call(recipe.install)
      File "/home/ccomb/buildout-eggs/zc.buildout-1.5.2-py2.7.egg/zc/buildout/buildout.py", line 1297, in _call
        return f()
      File "/home/dadafkas/projets/anybox/anybox.recipe.openerp/anybox/recipe/openerp/__init__.py", line 206, in install
        ws = eggs.install()
      File "/home/ccomb/buildout-eggs/zc.recipe.egg-1.3.2-py2.7.egg/zc/recipe/egg/egg.py", line 145, in install
        reqs, ws = self.working_set()
      File "/home/ccomb/buildout-eggs/zc.recipe.egg-1.3.2-py2.7.egg/zc/recipe/egg/egg.py", line 101, in working_set
        **kw)
      File "/home/ccomb/buildout-eggs/zc.buildout-1.5.2-py2.7.egg/zc/buildout/easy_install.py", line 1090, in install
        return installer.install(specs, working_set)
      File "/home/ccomb/buildout-eggs/zc.buildout-1.5.2-py2.7.egg/zc/buildout/easy_install.py", line 942, in install
        ws, self._always_unzip):
      File "/home/ccomb/buildout-eggs/zc.buildout-1.5.2-py2.7.egg/zc/buildout/easy_install.py", line 723, in _get_dist
        dist, avail = self._satisfied(requirement)
      File "/home/ccomb/buildout-eggs/zc.buildout-1.5.2-py2.7.egg/zc/buildout/easy_install.py", line 484, in _satisfied
        best_available = self._obtain(req, source)
      File "/home/ccomb/buildout-eggs/zc.buildout-1.5.2-py2.7.egg/zc/buildout/easy_install.py", line 639, in _obtain
        if index.obtain(requirement) is None:
      File "/home/dadafkas/projets/distribute/distribute/setuptools/package_index.py", line 342, in obtain
        self.prescan(); self.find_packages(requirement)
      File "/home/dadafkas/projets/distribute/distribute/setuptools/package_index.py", line 327, in find_packages
        self.scan_url(self.index_url + requirement.unsafe_name+'/')
      File "/home/dadafkas/projets/distribute/distribute/setuptools/package_index.py", line 670, in scan_url
        self.process_url(url, True)
      File "/home/dadafkas/projets/distribute/distribute/setuptools/package_index.py", line 225, in process_url
        page = self.process_index(url, page)
      File "/home/dadafkas/projets/distribute/distribute/setuptools/package_index.py", line 302, in process_index
        self.scan_url(new_url)
      File "/home/dadafkas/projets/distribute/distribute/setuptools/package_index.py", line 670, in scan_url
        self.process_url(url, True)
      File "/home/dadafkas/projets/distribute/distribute/setuptools/package_index.py", line 203, in process_url
        f = self.open_url(url, "Download error on %s: %%s -- Some packages may not be found!" % url)
      File "/home/dadafkas/projets/distribute/distribute/setuptools/package_index.py", line 612, in open_url
        return open_with_auth(url)
      File "/home/dadafkas/projets/distribute/distribute/setuptools/package_index.py", line 755, in _socket_timeout
        return func(*args, **kwargs)
      File "/home/dadafkas/projets/distribute/distribute/setuptools/package_index.py", line 797, in open_with_auth
        fp = urllib2.urlopen(request)
      File "/usr/lib/python2.7/urllib2.py", line 126, in urlopen
        return _opener.open(url, data, timeout)
      File "/usr/lib/python2.7/urllib2.py", line 400, in open
        response = self._open(req, data)
      File "/usr/lib/python2.7/urllib2.py", line 418, in _open
        '_open', req)
      File "/usr/lib/python2.7/urllib2.py", line 378, in _call_chain
        result = func(*args)
      File "/usr/lib/python2.7/urllib2.py", line 1207, in http_open
        return self.do_open(httplib.HTTPConnection, req)
      File "/usr/lib/python2.7/urllib2.py", line 1180, in do_open
        r = h.getresponse(buffering=True)
      File "/usr/lib/python2.7/httplib.py", line 1030, in getresponse
        response.begin()
      File "/usr/lib/python2.7/httplib.py", line 407, in begin
        version, status, reason = self._read_status()
      File "/usr/lib/python2.7/httplib.py", line 365, in _read_status
        line = self.fp.readline()
      File "/usr/lib/python2.7/socket.py", line 447, in readline
        data = self._sock.recv(self._rbufsize)
    timeout: timed out
    
    
  5. Christophe Combelles

    I could reproduce randomly on the Python2.7.3rc1 of the Debian, with a timeout 1, but never on Python2.6 nor 2.4, and not even on a hand-compiled 2.7.2 on a Debian Squeeze. On my side, looks like a Python 2.7.3 bug or a Debian bug.

    I got different results from one run the next.

    $ python
    Python 2.7.3rc1 (default, Mar 10 2012, 00:01:06) 
    [GCC 4.6.3] on linux2
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import urllib2
    >>> import socket
    >>> socket.setdefaulttimeout(1)
    >>> urllib2.urlopen("http://trac.pocoo.org/repos/werkzeug/trunk")
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "/usr/lib/python2.7/urllib2.py", line 126, in urlopen
        return _opener.open(url, data, timeout)
      File "/usr/lib/python2.7/urllib2.py", line 400, in open
        response = self._open(req, data)
      File "/usr/lib/python2.7/urllib2.py", line 418, in _open
        '_open', req)
      File "/usr/lib/python2.7/urllib2.py", line 378, in _call_chain
        result = func(*args)
      File "/usr/lib/python2.7/urllib2.py", line 1207, in http_open
        return self.do_open(httplib.HTTPConnection, req)
      File "/usr/lib/python2.7/urllib2.py", line 1177, in do_open
        raise URLError(err)
    urllib2.URLError: <urlopen error timed out>
    >>> urllib2.urlopen("http://trac.pocoo.org/repos/werkzeug/trunk")
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "/usr/lib/python2.7/urllib2.py", line 126, in urlopen
        return _opener.open(url, data, timeout)
      File "/usr/lib/python2.7/urllib2.py", line 400, in open
        response = self._open(req, data)
      File "/usr/lib/python2.7/urllib2.py", line 418, in _open
        '_open', req)
      File "/usr/lib/python2.7/urllib2.py", line 378, in _call_chain
        result = func(*args)
      File "/usr/lib/python2.7/urllib2.py", line 1207, in http_open
        return self.do_open(httplib.HTTPConnection, req)
      File "/usr/lib/python2.7/urllib2.py", line 1180, in do_open
        r = h.getresponse(buffering=True)
      File "/usr/lib/python2.7/httplib.py", line 1030, in getresponse
        response.begin()
      File "/usr/lib/python2.7/httplib.py", line 407, in begin
        version, status, reason = self._read_status()
      File "/usr/lib/python2.7/httplib.py", line 365, in _read_status
        line = self.fp.readline()
      File "/usr/lib/python2.7/socket.py", line 447, in readline
        data = self._sock.recv(self._rbufsize)
    socket.timeout: timed out
    

    So I agree with the wontfix on distribute, but another bug should be reported somewhere else...

  6. Log in to comment