Contribute Your Package to the World
single: package index; pypi
single: package index; cheeseshop
The Python Package Index (PyPI)
The Python Package Index (PyPI), formerly known as the Cheeseshop, is to Python
what :term:`CPAN` is to :term:`Perl`: a central repository of projects and
.. note:: XXX put here the Monty Python Cheeseshop extract
`PyPI <http://pypi.python.org>`_ is located at http://pypi.python.org and contains more than 9000 projects registered by developers from the community.
.. image:: images/pypi_screenshot.jpg
Tools like :ref:`Pip <pip_info>` or `zc.buildout
<http://pypi.python.org/pypi/zc.buildout>`_ are using PyPI as the default
location to find :term:`distributions <distribution>` to install. When ``pip
install Foo`` is called, it will browse PyPI to find the latest available
version of the ``Foo`` project using `The Simple Index Protocol`_. If it finds
it, it will download it and install it automatically.
This automatic installation ala :ref:`pip_info` will work only if the distribution is using the Distutils-based structure and contains a ``setup.py`` file.
This means that any serious Python project should use Distutils
(see the :ref:`basics <creation_basics>` for doing this)
and should at the minimum be registered at PyPI. Uploading releases there is
also a good practice.
.. note:: ??? Why should a project at a minimum register at PyPI?
good practice, yes; at a minimum, no
.. index:: package index; registration
Registering a project at PyPI is done by using :term:`Distutils`' ``register``
command. This command performs a simple HTTP post using basic authentication
with the login name and password stored in the ``.pypirc`` file
located in your home directory. This login has to be registered at PyPI, so
you should go there and create an account before running ``register`` for the
Another option is to call ``register`` once on any Distutils-based project.
It will register the new account for you and all you'll have to do is to reply
to the confirmation e-mail PyPI will send you::
$ python setup.py register
warning: register: missing required meta-data: version
We need to know who you are, so please choose either:
1. use your existing login,
2. register as a new user,
3. have the server generate a new password for you (and email it to you), or
Your selection [default 1]:
Once this is done, ``register`` will ask you if you want to save your login
information in the ``.pypirc`` file. By default, this will store the login name
**and** the password::
For security reasons, starting at Python 2.6, you can remove the password from
the file if you want ``register`` to prompt you to type it everytime.
A recent GSoC project called Keyring was created in 2009 in order
to use any available system keyring like KWallet or KeyChain to
store the PyPI password. The project exists and may be used in Distutils
by the ``register`` and ``upload`` commands.
Once your account is ready, registering it at PyPI will create
a new web page there, using the metadata fields of your project.
- **name** will be used as the unique identifier in PyPI. For example, if
your project's name is ``Foo``, its page will be located at
`http://pypi.python.org/pypi/foo`. The rule at PyPI is `first in, first
served`, meaning that once you have registered the `Foo` project, your
login is the owner of the PyPI `foo` identifier and no one else (unless
you authorize them) will be able to register a project under that name.
- **long_description** will be used to
fill that page. An Html to :term:`reStructuredText` rendered will be used
on the field before it is displayed.
A good practice is to use reST, and make sure your ``long_description``
field doesn't contain any reStructuredText syntax error. See
:ref:`rest_example` for a quick introduction on how to write a reST compatible
To perform a check, install ``docutils`` by using Pip
(``pip install docutils``) and run::
$ python setup.py --long-description | rst2html.py > /dev/null
Under Windows, make sure the sys.prefix + 'Scripts/' path is in the ``PATH``
environment variable and run::
$ python.exe setup.py --long-description | rst2html.py > dummy.html
If your text contains any reST error or warning, they will be displayed.
Starting at Python 2.7, you can use the ``check`` command instead of calling
the ``rst2html.py`` script, as long as ``docutils`` is installed::
$ python setup.py check
The check command will check that all fields are compliant before you
register the project at PyPI.
.. index:: package index; uploading
PyPI also allows developers to upload their project's distributions. This can
be done manually via a web form, but also through Distutils by using the
This command will upload freshly created archives via HTTP to PyPI. The
usual way to perform this is to call ``upload`` right after ``register``
and the commands used to create archives. For instance, to upload a source
distribution and update the project's page, one may do::
$ python setup.py register sdist upload
Note that you can upload several archives in one step if wanted::
$ python setup.py register sdist bdist upload
A good practice when uploading distributions at PyPI is to always upload
the source distribution, unless your project is not open source of course.
Binary distribution are optional especially if your project doesn't contain
any extension to be compiled. This will help automatic installers like
:doc:`Pip <pip>` to get and install your project on any platform. Uploading
only a binary distribution will restrict automatic installation to the platform
and Python version it was compiled with.
.. index:: protocol; simple index protocol
The Simple Index protocol
Besides its web pages, PyPI provides a tree structure at
http://pypi.python.org/simple called the ``Simple Index``. This structure
allows installers like :doc:`Pip <pip>` to look for distribution archives.
For example, calling::
$ pip install distribute
will look at the ``http://pypi.python.org/simple/distribute`` page, which
is a list of URLs for the Distribute project. These URLs include:
- the Distribute archives uploaded at PyPI
- a link to the project's home page
- extra links contained in the project's description fields
The Package Index follows these rules (taken from Setuptools' documentation):
1. Its pages are in HTML
2. Individual project version pages' URLs must be of the form
``base/projectname/version``, where ``base`` is the package index's base
URL. The base URL for PyPI is : ``http://pypi.python.org/simple``.
3. Omitting the ``/version`` part of a project page's URL (but keeping the
trailing ``/``) should result in a page that is either:
a) The single active version of that project, as though the version had been
explicitly included, OR
b) A page with links to all of the active version pages for that project.
Depending on the project's configuration, PyPI will display a) or b).
4. Individual project version pages should contain direct links to downloadable
distributions where possible. The project's "long_description" field
may contain URLs that will be displayed.
5. Where available, MD5 information should be added to download URLs by
appending a fragment identifier of the form ``#md5=...``, where ``...`` is
the 32-character hex MD5 digest.
6. Individual project version pages should identify any "homepage" or
"download" URLs using ``rel="homepage"`` and ``rel="download"`` attributes
on the HTML elements linking to those URLs. Use of these attributes will
cause EasyInstall to always follow the provided links, unless it can be
determined by inspection that they are downloadable distributions. If the
links are not to downloadable distributions, they are retrieved, and if they
are HTML, they are scanned for download links. They are *not* scanned for
additional "homepage" or "download" links, as these are only processed for
pages that are part of a package index site.
7. The root URL of the index, if retrieved with a trailing ``/``, must result
in a page containing links to *all* projects' active version pages.
(Note: This requirement is a workaround for the absence of case-insensitive
``safe_name()`` matching of project names in URL paths. If project names are
matched in this fashion (e.g. via the PyPI server, mod_rewrite, or a similar
mechanism), then it is not necessary to include this all-packages listing
8. If a package index is accessed via a ``file://`` URL, then EasyInstall will
automatically use ``index.html`` files, if present, when trying to read a
directory with a trailing ``/`` on the URL.
.. index:: api; PyPI XML-RPC
The XML-RPC interface
.. note:: XXX pointer to PyPI developer doc + demo with yolk
Limitations of the system
Within the last few years, PyPI has become a first class citizen in the Python
development ecosystem. For instance Plone developers that are using
``zc.buildout`` to install Plone-based applications are making hundreds
of calls to PyPI each time they build their projects. That's because many web
applications are using hundreds of small Python distributions nowadays.
This has a bad side effect: if PyPI is down, it becomes impossible to build
some applications. PyPI acts as a ``Single Point of Failure`` (SPOF).
The PyPI server is quite robust and its uptime is probably around 99.99%,
although people hit the SPOF problem once or twice per year. The community
has started to develop strategies to avoid it:
- ``zc.buildout`` comes with a cache that will keep a copy
of downloaded archives on the local system. This avoids getting packages at
PyPI when they were already downloaded once.
- the `collective.eggproxy <http://pypi.python.org/pypi/collective.eggproxy>`_
project provides a cache server that acts like
a proxy between developers and PyPI, and caches PyPI files like
- the `z3c.pypimirror <http://pypi.python.org/pypi/z3c.pypimirror>`_ project
is a mirroring script that syncs a local copy of the simple index so tools
can use it instead of calling PyPI.
In the last year, PEP 381 was added to describe a mirroring protocol and
set up an official "ring" of PyPI mirrors. This system should be ready
by the end of the year 2010, and will provide a DNS entry under
``mirrors.pypi.org`` listing all mirrors IPs. This will allow tools like
:ref:`pip_info` to list all mirrors and pick the closest one, or fall back in
case a server is down.
single: package index; pypi
single: package index; cheeseshop
Private and secondary PyPI servers
.. note:: XXX explain here that other servers can have a pypi feature (like
plone.org, or private)
.. note:: XXX explain that pip can point to any pypi-like server
.. note:: XXX conclude on the multiple-index merge idea