1. PyPA
  2. Python Packaging Authority Projects
  3. pypi


Issue #34 invalid

pip install package==version inefficiently crawling all urls

Damien Nozay
created an issue

Attached two files: - download.bad.txt where I'm trying to retrieve a non-existing package from pypi. - download.good-but-bad.txt where I'm trying to retrieve an existing package but things are very sluggish from my network.

First things to pay attention to:

  Will skip URL http://pypi.python.org/simple/django/1.6.777 when looking for download links for django==1.6.777

This means the package maintainer did not provide a subdirectory for that specific version with an associated index.html. Maybe it is a problem in general where no good practices are documented (?). This is the part I would like to see addressed. If all packages had a more limited index.html for specific versions, then crawling would be much faster.

http://pypi.python.org/simple/django/1.6.777 -> http://pypi.python.org/simple/django/1.6.777/index.html -> http://pypi.python.org/simple/django/1.6.777/Django-1.6.777.tar.gz

Because network is so slow for me, I am using collective.eggproxy and that alleviates some of the pain, but it has its own problems too.

  Ignoring link http://b.pypi.python.org/packages/source/D/Django/Django-1.2.6.tar.gz#md5=bff9fc7d871c0b5e6ce1a7babd16847b (from http://b.pypi.python.org/simple/Django/), version 1.2.6 doesn't match ==1.6.777
  Ignoring link http://b.pypi.python.org/packages/source/D/Django/Django-1.3.6.tar.gz#md5=357dbedf41ba6db990fd4be7c86cd80d (from 

Secondly (and I'll file an enhancement request against pip), when asking for a specific version, it tries each link in the package index rather than doing any guesswork. Very often the name conventions is consistent.

http://b.pypi.python.org/packages/source/D/Django/Django-1.3.6.tar.gz#md5=357dbedf41ba6db990fd4be7c86cd80d -> maybe files are all named http://b.pypi.python.org/packages/source/D/Django/Django-X.Y.Z.tar.gz#md5=XXX -> try http://b.pypi.python.org/packages/source/D/Django/Django-1.6.777.tar.gz#md5=XXX

Thank you!

Comments (1)

  1. Log in to comment