Commits

Vinay Sajip  committed 79ad4f3 Draft

Routine update.

  • Participants
  • Parent commits 1f141f1

Comments (0)

Files changed (4)

File docs/internals.rst

 This is the section containing some discussion of how ``distlib``'s design was
 arrived at, as and when time permits.
 
+The ``locators`` API
+--------------------
+
+This section describes the design of the ``distlib`` API relating to accessing
+distribution metadata, whether stored locally or in indexes like PyPI.
+
+The problem we're trying to solve
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+People who use distributions need to locate, download and install them.
+Distributions can be found in a number of places, such as:
+
+* An Internet index such as `The Python Packages Index (PyPI)
+  <http://pypi.python.org>`_, or a mirror thereof.
+* Other Internet resources, such as the developer's website, or a source
+  code repository such as GitHub, BitBucket, Google Code or similar.
+* File systems, whether local to one computer or shared between several.
+* Distributions which have already been installed, and are available in the
+  ``sys.path`` of a running Python interpreter.
+
+When we're looking for distributions, we don't always know exactly what we
+want: often, we just want the latest version, but it's not uncommon to want
+a specific older version, or perhaps the most recent version that meets some
+constraints on the version. Since we need to be concerned with matching
+versions, we need to consider the version schemes in use (see
+:ref:`version-api`).
+
+It's useful to separate the notion of a *project* from a distribution: The
+project is the version-independent part of the distribution, i.e. it's
+described by the *name* of the distribution and encompasses all released
+distributions which use that name.
+
+We often don't just want a single distribution, either: a common requirement,
+when installing a distribution, is to locate all distributions that it relies
+on, which aren't already installed. So we need a *dependency finder*, which
+itself needs to locate depended-upon distributions, and recursively search for
+dependencies until all that are available have been found.
+
+We may need to distinguish between different types of dependencies:
+
+* Post-installation dependencies. These are needed by the distribution after it
+  has been installed, and is in use.
+* Build dependencies. These are needed for building and/or installing the
+  distribution, but are not needed by the distribution itself after
+  installation.
+* Test dependencies. These are only needed for testing the distribution, but
+  are not needed by the distribution itself after installation.
+
+When testing a distribution, we need all three types of dependencies. When
+installing a distribution, we need the first two, but not the third.
+
+A minimal solution
+^^^^^^^^^^^^^^^^^^
+
+Locating distributions
+~~~~~~~~~~~~~~~~~~~~~~
+
+It seems that the simplest API to locate a distribution would look like
+``locate(requirement)``, where ``requirement`` is a string giving the
+distribution name and optional version constraints. Given that we know that
+distributions can be found in different places, it's best to consider a
+:class:`Locator` class which has a :meth:`locate` method with a corresponding
+signature, with subclasses for each of the different types of location that
+distributions inhabit. It's also reasonable to provide a default locator in
+a module attribute :attr:`default_locator`, and a module-level :func:`locate`
+function which calls the :meth:`locate` method on the default locator.
+
+Since we'll often need to locate all the versions of a project before picking
+one, we can imagine that a locator would need a :meth:`get_project` method for
+fetching all versions of a project; and since we will be likely to want to use
+caching, we can assume there will be a :meth:`_get_project` method to do the
+actual work of fetching the version data, which the higher-level
+:meth:`get_project` will call (and probably cache). So our locator base class
+will look something like this::
+
+    class Locator(object):
+        """
+        Locate distributions.
+        """
+
+        def __init__(self, version_scheme='default'):
+            """
+            Initialise a locator with the specified version scheme.
+            """
+
+        def locate(self, requirement):
+            """
+            Locate the highest-version distribution which satisfies
+            the constraints in ``requirement``, and return a
+            ``Distribution`` instance if found, or else ``None``.
+            """
+
+        def get_project(self, name):
+            """
+            Return all known distributions for a project named ``name``,
+            returning a dictionary mapping version to ``Distribution``
+            instance, or an empty dictionary if nothing was found.
+            Use _get_project to do the actual work, and cache the results for
+            future use.
+            """
+
+        def _get_project(self, name):
+            """
+            Return all known distributions for a project named ``name``,
+            returning a dictionary mapping version to ``Distribution``
+            instance, or an empty dictionary if nothing was found.
+            """
+
+
+Finding dependencies
+~~~~~~~~~~~~~~~~~~~~
+
+A dependency finder will depend on a locator to locate dependencies. A simple
+approach will be to consider a :class:`DependencyFinder` class which takes a
+locator as a constructor argument. It might look something like this::
+
+    class DependenyFinder(object):
+        """
+        Locate dependencies for distributions.
+        """
+
+        def __init__(self, locator):
+            """
+            Initialise an instance, using the specified locator
+            to locate distributions.
+            """
+
+        def find(self, requirement, tests=False):
+            """
+            Find a distribution matching requirement and all distributions
+            it depends on. Use the ``tests`` argument to determine whether
+            distributions used only for testing should be included in the
+            results. Allow ``requirement`` to be either a :class:`Distribution`
+            instance or a string expressing a requirement.
+
+            Return a set of :class:`Distribution` instances and a set of
+            problems.
+
+            The distributions returned should be such that they have the
+            :attr:`required` attribute set to ``True`` if they were
+            from the ``requirement`` passed to ``find()``, and they have the
+            :attr:`build_time_dependency` attribute set to ``True`` unless they
+            are post-installation dependencies of the ``requirement``.
+
+            The problems should be a tuple consisting of the string
+            ``'unsatisfied'`` and the requirement which couldn't be satisfied
+            by any distribution known to the locator.
+            """
+
+
 The ``resources`` API
 ---------------------
 
   [a,,b]
   [a=,b,c]
 
+
+.. _version-api:
+
 The ``version`` API
 -------------------
 

File docs/overview.rst

 How Distlib can help
 --------------------
 
-The idea behind Distlib is expressed in `this python-dev post
+The idea behind Distlib is expressed in `this python-dev mailing-list post
 <http://mail.python.org/pipermail/python-dev/2012-September/121716.html>`_,
 though a different name was suggested for the library. Basically, Distlib
 contains the implementations of the packaging PEPs and other low-level
   the ability to scan for dependencies and building dependency graphs.
 * The package ``distlib.glob``, which implements globbing functionality
   such as the ability to use ``**`` in patterns to specify recursing into
-  subdirectories.
+  subdirectories. (This may be merged into the ``distlib.util`` package.)
 * The package ``distlib.resources``, which allows access to data files stored
   in Python packages, both in the file system and in .zip files.
 * The package ``distlib.scripts``, which allows installing of scripts with

File docs/reference.rst

                     (i.e. with platform-specific directory separators as
                     indicated by ``os.sep``).
 
-The :class:`DependencyGraph` class
-----------------------------------
-
 .. class:: DependencyGraph
 
    This class represents a dependency graph between releases. The nodes are

File docs/tutorial.rst

 to install the package (perhaps invoking with ``sudo`` if you need
 to install to a protected location).
 
+Coverage results are available at:
+
+http://www.red-dove.com/distlib/coverage/
+
+These are updated as and when time permits.
+
+
 First steps
 -----------
 
 ~~~~~~~~~~~~~~~~~~~~~~~
 
 Once you have a :class:`Distribution` instance, you can use it to get more
-information about the distribution. For example, the ``metadata`` attribute
-gives access to the distribution's metadata (see :ref:`use-metadata` for more
-information).
+information about the distribution. For example:
+
+* The ``metadata`` attribute gives access to the distribution's metadata
+  (see :ref:`use-metadata` for more information).
+
+* The ``name_and_version`` attribute shows the name and version in the format
+  ``name (X.Y)``.
+
+* The ``key`` attribute holds the distribution's name in lower-case, as you
+  generally want to search for distributions without regard to case
+  sensitivity.
+
 
 .. _dist-exports:
 
     #!/usr/bin/python
 
     if __name__ == '__main__':
+        import sys, re
+
         def _resolve(module, func):
-            mod = __import__(module)
+            __import__(module)
+            mod = sys.modules[module]
             parts = func.split('.')
             result = getattr(mod, parts.pop(0))
             for p in parts:
             return result
 
         try:
-            import sys, re
             sys.argv[0] = re.sub('-script.pyw?$', '', sys.argv[0])
 
             func = _resolve('foo', 'main')
   :class:`AggregatingLocator` to satisfy requirements from installed
   distributions before looking elsewhere for them.
 
+* :class:`JSONLocator` -- this uses an improved JSON metadata schema and
+  returns data on all versions of a distribution, including dependencies,
+  using a single network request.
+
 * :class:`AggregatingLocator` -- this takes a list of other aggregators and
   delegates finding projects to them. It can either return the first result
   found (i.e. from the first aggregator in the list provided which returns a
 
 This is implemented using the XML-RPC API.
 
-   The Locator API is very bare-bones at the moment, but additional features will
-   be added in due course. A very bare-bones command-line script which exercises
-   these locators is to be found `here <https://gist.github.com/3886402>`_, and
-   feedback will be gratefully received from anyone who tries it out.
+Apart from :class:`JSONLocator`, none of the locators currently returns enough
+metadata to allow dependency resolution to be carried out, but that is a result
+of the fact that metadata relating to dependencies are not indexed, and would
+require not just downloading the distribution archives and inspection of
+contained metadata files, but potentially also introspecting setup.py! This is
+the downside of having vital information only available via keyword arguments
+to the :func:`setup` call: hopefully, a move to fully declarative metadata will
+facilitate indexing it and allowing the provision of improved features.
 
-   None of the locators currently returns enough metadata to allow dependency
-   resolution to be carried out, but that is a result of the fact that metadata
-   relating to dependencies are not indexed, and would require not just downloading
-   the distribution archives and inspection of contained metadata files, but
-   potentially also introspecting setup.py! This is the downside of having vital
-   information only available via keyword arguments to the :func:`setup` call:
-   hopefully, a move to fully declarative metadata will facilitate indexing it and
-   allowing the provision of features currently provided by ``setuptools`` (e.g.
-   hints for downloads -- ``'dependency _links'``).
+The locators will skip binary distributions (``.egg`` files are currently
+treated as binary distributions).
 
-   The locators will skip binary distributions (``.egg`` files are currently
-   treated as binary distributions).
-
-   The PyPI locator classes don't yet support the use of mirrors, but that can be
-   added in due course -- once the basic functionality is working satisfactorily.
+The PyPI locator classes don't yet support the use of mirrors, but that can be
+added in due course -- once the basic functionality is working satisfactorily.
 
 Next steps
 ----------