1. Éric Araujo
  2. hhgp


hhgp / source / contributing.txt

Contribute Your Package to the World

.. index::
   single: package index; pypi
   single: package index; cheeseshop

.. _pypi_info:

The Python Package Index (PyPI)

The Python Package Index (PyPI), formerly known as the Cheeseshop, is to Python
what :term:`CPAN` is to :term:`Perl`: a central repository of projects and
:term:`distributions <distribution>`.

.. note:: XXX put here the Monty Python Cheeseshop extract

`PyPI <http://pypi.python.org>`_ is located at http://pypi.python.org and contains more than 9000 projects registered by developers from the community.

.. image:: images/pypi_screenshot.jpg

Tools like :ref:`Pip <pip_info>` or `zc.buildout
<http://pypi.python.org/pypi/zc.buildout>`_ are using PyPI as the default
location to find :term:`distributions <distribution>` to install. When ``pip
install Foo`` is called, it will browse PyPI to find the latest available
version of the ``Foo`` project using `The Simple Index Protocol`_. If it finds
it, it will download it and install it automatically.

This automatic installation ala :ref:`pip_info` will work only if the distribution is using the Distutils-based structure and contains a ``setup.py`` file.

This means that any serious Python project should use Distutils
(see the :ref:`basics <creation_basics>` for doing this)
and should at the minimum be registered at PyPI. Uploading releases there is
also a good practice.

.. note:: ??? Why should a project at a minimum register at PyPI?
   good practice, yes; at a minimum, no

.. index:: package index; registration

Registering projects

Registering a project at PyPI is done by using :term:`Distutils`' ``register``
command. This command performs a simple HTTP post using basic authentication
with the login name and password stored in the ``.pypirc`` file
located in your home directory. This login has to be registered at PyPI, so
you should go there and create an account before running ``register`` for the
first time.

Another option is to call ``register`` once on any Distutils-based project.
It will register the new account for you and all you'll have to do is to reply
to the confirmation e-mail PyPI will send you::

    $ python setup.py register
    running register
    warning: register: missing required meta-data: version
    We need to know who you are, so please choose either:
    1. use your existing login,
    2. register as a new user,
    3. have the server generate a new password for you (and email it to you), or
    4. quit
    Your selection [default 1]:

Once this is done, ``register`` will ask you if you want to save your login
information in the ``.pypirc`` file. By default, this will store the login name
**and** the password::

    index-servers =


For security reasons, starting at Python 2.6, you can remove the password from
the file if you want ``register`` to prompt you to type it everytime.

.. note::

    A recent GSoC project called Keyring was created in 2009 in order
    to use any available system keyring like KWallet or KeyChain to
    store the PyPI password. The project exists and may be used in Distutils
    by the ``register`` and ``upload`` commands.

    See: http://pypi.python.org/pypi/keyring

Once your account is ready, registering it at PyPI will create
a new web page there, using the metadata fields of your project.

- **name** will be used as the unique identifier in PyPI. For example, if
  your project's name is ``Foo``, its page will be located at
  `http://pypi.python.org/pypi/foo`. The rule at PyPI is `first in, first
  served`, meaning that once you have registered the `Foo` project, your
  login is the owner of the PyPI `foo` identifier and no one else (unless
  you authorize them) will be able to register a project under that name.

- **long_description** will be used to
  fill that page. An Html to :term:`reStructuredText` rendered will be used
  on the field before it is displayed.

A good practice is to use reST, and make sure your ``long_description``
field doesn't contain any reStructuredText syntax error. See
:ref:`rest_example` for a quick introduction on how to write a reST compatible

To perform a check, install ``docutils`` by using Pip
(``pip install docutils``) and run::

    $ python setup.py --long-description | rst2html.py > /dev/null

Under Windows, make sure the sys.prefix + 'Scripts/' path is in the ``PATH``
environment variable and run::

    $ python.exe setup.py --long-description | rst2html.py > dummy.html

If your text contains any reST error or warning, they will be displayed.

Starting at Python 2.7, you can use the ``check`` command instead of calling
the ``rst2html.py`` script, as long as ``docutils`` is installed::

    $ python setup.py check

The check command will check that all fields are compliant before you
register the project at PyPI.

.. index:: package index; uploading

Uploading distributions

PyPI also allows developers to upload their project's distributions. This can
be done manually via a web form, but also through Distutils by using the
``upload`` command.

This command will upload freshly created archives via HTTP to PyPI. The
usual way to perform this is to call ``upload`` right after ``register``
and the commands used to create archives. For instance, to upload a source
distribution and update the project's page, one may do::

    $ python setup.py register sdist upload

Note that you can upload several archives in one step if wanted::

    $ python setup.py register sdist bdist upload

A good practice when uploading distributions at PyPI is to always upload
the source distribution, unless your project is not open source of course.
Binary distribution are optional especially if your project doesn't contain
any extension to be compiled. This will help automatic installers like
:doc:`Pip <pip>` to get and install your project on any platform. Uploading
only a binary distribution will restrict automatic installation to the platform
and Python version it was compiled with.

.. index:: protocol; simple index protocol

The Simple Index protocol

Besides its web pages, PyPI provides a tree structure at
http://pypi.python.org/simple called the ``Simple Index``. This structure
allows installers like :doc:`Pip <pip>` to look for distribution archives.

For example, calling::

    $ pip install distribute

will look at the ``http://pypi.python.org/simple/distribute`` page, which
is a list of URLs for the Distribute project. These URLs include:

 - the Distribute archives uploaded at PyPI
 - a link to the project's home page
 - extra links contained in the project's description fields

The Package Index follows these rules (taken from Setuptools' documentation):

1. Its pages are in HTML

2. Individual project version pages' URLs must be of the form
   ``base/projectname/version``, where ``base`` is the package index's base
   URL. The base URL for PyPI is : ``http://pypi.python.org/simple``.

3. Omitting the ``/version`` part of a project page's URL (but keeping the
   trailing ``/``) should result in a page that is either:

   a) The single active version of that project, as though the version had been
      explicitly included, OR

   b) A page with links to all of the active version pages for that project.

   Depending on the project's configuration, PyPI will display a) or b).

4. Individual project version pages should contain direct links to downloadable
   distributions where possible.  The project's "long_description" field
   may contain URLs that will be displayed.

5. Where available, MD5 information should be added to download URLs by
   appending a fragment identifier of the form ``#md5=...``, where ``...`` is
   the 32-character hex MD5 digest.

6. Individual project version pages should identify any "homepage" or
   "download" URLs using ``rel="homepage"`` and ``rel="download"`` attributes
   on the HTML elements linking to those URLs. Use of these attributes will
   cause EasyInstall to always follow the provided links, unless it can be
   determined by inspection that they are downloadable distributions. If the
   links are not to downloadable distributions, they are retrieved, and if they
   are HTML, they are scanned for download links. They are *not* scanned for
   additional "homepage" or "download" links, as these are only processed for
   pages that are part of a package index site.

7. The root URL of the index, if retrieved with a trailing ``/``, must result
   in a page containing links to *all* projects' active version pages.

   (Note: This requirement is a workaround for the absence of case-insensitive
   ``safe_name()`` matching of project names in URL paths. If project names are
   matched in this fashion (e.g. via the PyPI server, mod_rewrite, or a similar
   mechanism), then it is not necessary to include this all-packages listing

8. If a package index is accessed via a ``file://`` URL, then EasyInstall will
   automatically use ``index.html`` files, if present, when trying to read a
   directory with a trailing ``/`` on the URL.

.. index:: api; PyPI XML-RPC

The XML-RPC interface

.. note:: XXX pointer to PyPI developer doc + demo with yolk

Limitations of the system

Within the last few years, PyPI has become a first class citizen in the Python
development ecosystem. For instance Plone developers that are using
``zc.buildout`` to install Plone-based applications are making hundreds
of calls to PyPI each time they build their projects. That's because many web
applications are using hundreds of small Python distributions nowadays.

This has a bad side effect: if PyPI is down, it becomes impossible to build
some applications. PyPI acts as a ``Single Point of Failure`` (SPOF).

The PyPI server is quite robust and its uptime is probably around 99.99%,
although people hit the SPOF problem once or twice per year. The community
has started to develop strategies to avoid it:

- ``zc.buildout`` comes with a cache that will keep a copy
  of downloaded archives on the local system. This avoids getting packages at
  PyPI when they were already downloaded once.

- the `collective.eggproxy <http://pypi.python.org/pypi/collective.eggproxy>`_
  project provides a cache server that acts like
  a proxy between developers and PyPI, and caches PyPI files like
  ``zc.buildout`` does.

- the `z3c.pypimirror <http://pypi.python.org/pypi/z3c.pypimirror>`_ project
  is a mirroring script that syncs a local copy of the simple index so tools
  can use it instead of calling PyPI.

In the last year, PEP 381 was added to describe a mirroring protocol and
set up an official "ring" of PyPI mirrors. This system should be ready
by the end of the year 2010, and will provide a DNS entry under
``mirrors.pypi.org`` listing all mirrors IPs. This will allow tools like
:ref:`pip_info` to list all mirrors and pick the closest one, or fall back in
case a server is down.

.. index::
   single: package index; pypi
   single: package index; cheeseshop

Private and secondary PyPI servers

.. note:: XXX explain here that other servers can have a pypi feature (like
   plone.org, or private)
.. note:: XXX explain that pip can point to any pypi-like server
.. note:: XXX conclude on the multiple-index merge idea