pip gets timeout on large packages

Issue #208 on hold
Manuel Kaufmann created an issue

Hello, I found that pip gets a timeout error when trying to download large packages on a slow Internet connection.

I got this error with ipython package for example:

$ pip install ipython
Collecting ipython
  Retrying (Retry(total=4, connect=None, read=None, redirect=None)) after connection broken by 'ReadTimeoutError("HTTPConnectionPool(host='localhost', port=3141): Read timed out. (read timeout=15)",)': /root/pypi/+f/14b/9c7c5280ded28/ipython-2.3.1-py27-none-any.whl
  Retrying (Retry(total=3, connect=None, read=None, redirect=None)) after connection broken by 'ReadTimeoutError("HTTPConnectionPool(host='localhost', port=3141): Read timed out. (read timeout=15)",)': /root/pypi/+f/14b/9c7c5280ded28/ipython-2.3.1-py27-none-any.whl
  Retrying (Retry(total=2, connect=None, read=None, redirect=None)) after connection broken by 'ReadTimeoutError("HTTPConnectionPool(host='localhost', port=3141): Read timed out. (read timeout=15)",)': /root/pypi/+f/14b/9c7c5280ded28/ipython-2.3.1-py27-none-any.whl
  Retrying (Retry(total=1, connect=None, read=None, redirect=None)) after connection broken by 'ReadTimeoutError("HTTPConnectionPool(host='localhost', port=3141): Read timed out. (read timeout=15)",)': /root/pypi/+f/14b/9c7c5280ded28/ipython-2.3.1-py27-none-any.whl
  Retrying (Retry(total=0, connect=None, read=None, redirect=None)) after connection broken by 'ReadTimeoutError("HTTPConnectionPool(host='localhost', port=3141): Read timed out. (read timeout=15)",)': /root/pypi/+f/14b/9c7c5280ded28/ipython-2.3.1-py27-none-any.whl
  Exception:
  Traceback (most recent call last):
    File "/home/humitos/.virtualenvs/tmp-f47ebfe8a75016d3/local/lib/python2.7/site-packages/pip/basecommand.py", line 232, in main
      status = self.run(options, args)
    File "/home/humitos/.virtualenvs/tmp-f47ebfe8a75016d3/local/lib/python2.7/site-packages/pip/commands/install.py", line 339, in run
      requirement_set.prepare_files(finder)
    File "/home/humitos/.virtualenvs/tmp-f47ebfe8a75016d3/local/lib/python2.7/site-packages/pip/req/req_set.py", line 355, in prepare_files
      do_download, session=self.session,
    File "/home/humitos/.virtualenvs/tmp-f47ebfe8a75016d3/local/lib/python2.7/site-packages/pip/download.py", line 782, in unpack_url
      session,
    File "/home/humitos/.virtualenvs/tmp-f47ebfe8a75016d3/local/lib/python2.7/site-packages/pip/download.py", line 667, in unpack_http_url
      from_path, content_type = _download_http_url(link, session, temp_dir)
    File "/home/humitos/.virtualenvs/tmp-f47ebfe8a75016d3/local/lib/python2.7/site-packages/pip/download.py", line 814, in _download_http_url
      stream=True,
    File "/home/humitos/.virtualenvs/tmp-f47ebfe8a75016d3/local/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 473, in get
      return self.request('GET', url, **kwargs)
    File "/home/humitos/.virtualenvs/tmp-f47ebfe8a75016d3/local/lib/python2.7/site-packages/pip/download.py", line 365, in request
      return super(PipSession, self).request(method, url, *args, **kwargs)
    File "/home/humitos/.virtualenvs/tmp-f47ebfe8a75016d3/local/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 461, in request
      resp = self.send(prep, **send_kwargs)
    File "/home/humitos/.virtualenvs/tmp-f47ebfe8a75016d3/local/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 573, in send
      r = adapter.send(request, **kwargs)
    File "/home/humitos/.virtualenvs/tmp-f47ebfe8a75016d3/local/lib/python2.7/site-packages/pip/_vendor/requests/adapters.py", line 424, in send
      raise ConnectionError(e, request=request)
  ConnectionError: HTTPConnectionPool(host='localhost', port=3141): Max retries exceeded with url: /root/pypi/+f/14b/9c7c5280ded28/ipython-2.3.1-py27-none-any.whl (Caused by ReadTimeoutError("HTTPConnectionPool(host='localhost', port=3141): Read timed out. (read timeout=15)",))
$ pip --version
pip 6.0.7 from /home/humitos/.virtualenvs/tmp-f47ebfe8a75016d3/local/lib/python2.7/site-packages (python 2.7)
$ devpi-server --version
2.1.3
$ 

After that, when devpi finishes the download of ipython I am able to re-run the pip command and install it without any problem.

Is it possible to avoid this timeout on large packages?

Thanks,

Comments (15)

  1. Holger Krekel repo owner

    Thing is that devpi-server first fetches the whole file before sending anything to the client. We could gain ause iterative streaming but it complicates the code which is why we removed it in the first place (once we had streaming).

  2. Holger Krekel repo owner

    at some point devpi-server did support streaming and it worked -- it was just a bit complex and the feature was removed quite some time ago. I guess we should look into re-introducing it in some way.

  3. Diego Russo

    Since the upgrade to 2.4.0 I've been having this kind of errors and they seem quite random. In the devpi-server log file and nginx log file I cannot find anything wrong.

  4. Holger Krekel repo owner

    if you run the server in debug mode, can you post the log when this happens again? Just want to verify it's really the issue which i think it is (i mentioned i above).

  5. Diego Russo

    It seems now much more stable and after the upgrade from 2.4.0 to 2.5.3 we've had just one error and in the log I can see:

    2015-11-24 20:02:36,924 ERROR [req44433] [Rtx1127] serving stale links for u'rbtools', upstream not reachable
    

    which, I think shouldn't raise the error though.

  6. Holger Krekel repo owner

    fix issue208: Mirrored files (PyPI) are streamed to the client while downloading.

    This prevents timeouts in pip etc. The files are only cached if there were no errors and in case there is a checksum, the content matches. Replicas can't stream yet.

    → <<cset fe15c011bce5>>

  7. Log in to comment