Commits

sste...@steve-steiners-mac-pro-2.local  committed dab7b5f

Added test_pypi_version.py, added suggest_rational_version docs to README.txt from PEP 386 text, added .pkl extension to .hgignore

  • Participants
  • Parent commits c831ff8

Comments (0)

Files changed (4)

 .*pyc$
 
+.*pkl
 release it.
 
 Furthermore, since Distutils will soon extend the metadata standard, by
-including the `install_requires` field from Setuptools [#requires]_, it
-needs to provide a standard for managing versions.
+including the `install_requires` field from Setuptools, as `Requires-Dist`
+[#requires]_ it needs to provide a standard for managing versions.
 
 Managing version numbers means being able to compare them, and being able to
 make sure each version number is a `valid` number for a given standard.
 
-In Python there are no real restriction yet on how a project should manage
-its versions, and how they should be incremented. They are no standard
-either, even if they are a few conventions widely used, like having a major and
-a minor revision (1.1, 1.2, etc.).
+In Python there are no real restriction yet on how a project should manage its
+versions, and how they should be incremented. They are no standard either,
+even if they are a few conventions widely used, like having a major and a
+minor revision (1.1, 1.2, etc.).
 
 Developers are free to put in the `version` meta-data of their package any
-string they want, and push a new release at PyPI. This version will appear
-as the `latest` for end users.
+string they want, and push a new release at PyPI. This version will appear as
+the `latest` for end users.
 
 Some project are also using dates as their major version numbers, or a custom
 versioning standard that is sometimes quite exotic.
 
 The `LooseVersion` class is quite laxest. From Distutils doc::
 
-    Version numbering for anarchists and software realists.
-    Implements the standard interface for version number classes as
-    described above.  A version number consists of a series of numbers,
-    separated by either periods or strings of letters.  When comparing
-    version numbers, the numeric components will be compared
-    numerically, and the alphabetic components lexically.  The following
-    are all valid version numbers, in no particular order:
+    Version numbering for anarchists and software realists. Implements the
+    standard interface for version number classes as described above. A
+    version number consists of a series of numbers, separated by either
+    periods or strings of letters. When comparing version numbers, the numeric
+    components will be compared numerically, and the alphabetic components
+    lexically. The following are all valid version numbers, in no particular
+    order:
 
         1.5.1
         1.5.2b2
         5.5.kw
         2.0b1pl0
 
-    In fact, there is no such thing as an invalid version number under
-    this scheme; the rules for comparison are simple and predictable,
-    but may not always give the results you want (for some definition
-    of "want").
+    In fact, there is no such thing as an invalid version number under this
+    scheme; the rules for comparison are simple and predictable, but may not
+    always give the results you want (for some definition of "want").
 
 This class makes any version string valid, and provides an algorithm to sort
 them numerically then lexically. It means that anything can be used to version
 
 The `StrictVersion` class is more strict. From the doc::
 
-            Version numbering for anal retentive and software idealists.
-    Implements the standard interface for version number classes as
-    described above.  A version number consists of two or three
-    dot-separated numeric components, with an optional "pre-release" tag
-    on the end.  The pre-release tag consists of the letter 'a' or 'b'
-    followed by a number.  If the numeric components of two version
-    numbers are equal, then one with a pre-release tag will always
-    be deemed earlier (lesser) than one without.
+    Version numbering for anal retentive and software idealists. Implements
+    the standard interface for version number classes as described above. A
+    version number consists of two or three dot-separated numeric components,
+    with an optional "pre-release" tag on the end. The pre-release tag
+    consists of the letter 'a' or 'b' followed by a number. If the numeric
+    components of two version numbers are equal, then one with a pre-release
+    tag will always be deemed earlier (lesser) than one without.
 
-    The following are valid version numbers (shown in the order that
-    would be obtained by sorting according to the supplied cmp function):
+    The following are valid version numbers (shown in the order that would be
+    obtained by sorting according to the supplied cmp function):
 
         0.4       0.4.0  (these two are equivalent)
         0.4.1
     Convert a version string to a chronologically-sortable key
 
     This is a rough cross between Distutils' StrictVersion and LooseVersion;
-    if you give it versions that would work with StrictVersion, then it behaves
-    the same; otherwise it acts like a slightly-smarter LooseVersion. It is
-    *possible* to create pathological version coding schemes that will fool
-    this parser, but they should be very rare in practice.
+    if you give it versions that would work with StrictVersion, then it
+    behaves the same; otherwise it acts like a slightly-smarter LooseVersion.
+    It is *possible* to create pathological version coding schemes that will
+    fool this parser, but they should be very rare in practice.
 
-    The returned value will be a tuple of strings.  Numeric portions of the
+    The returned value will be a tuple of strings. Numeric portions of the
     version are padded to 8 digits so they will compare numerically, but
-    without relying on how numbers compare relative to strings.  Dots are
-    dropped, but dashes are retained.  Trailing zeros between alpha segments
-    or dashes are suppressed, so that e.g. "2.4.0" is considered the same as
+    without relying on how numbers compare relative to strings. Dots are
+    dropped, but dashes are retained. Trailing zeros between alpha segments or
+    dashes are suppressed, so that e.g. "2.4.0" is considered the same as
     "2.4". Alphanumeric parts are lower-cased.
 
     The algorithm assumes that strings like "-" and any alpha string that
-    alphabetically follows "final"  represents a "patch level".  So, "2.4-1"
-    is assumed to be a branch or patch of "2.4", and therefore "2.4.1" is
+    alphabetically follows "final" represents a "patch level". So, "2.4-1" is
+    assumed to be a branch or patch of "2.4", and therefore "2.4.1" is
     considered newer than "2.4-1", which in turn is newer than "2.4".
 
     Strings like "a", "b", "c", "alpha", "beta", "candidate" and so on (that
-    come before "final" alphabetically) are assumed to be pre-release versions,
-    so that the version "2.4" is considered newer than "2.4a1".
+    come before "final" alphabetically) are assumed to be pre-release
+    versions, so that the version "2.4" is considered newer than "2.4a1".
 
     Finally, to handle miscellaneous cases, the strings "pre", "preview", and
     "rc" are treated as if they were "c", i.e. as though they were release
     candidates, and therefore are not as new as a version string that does not
-    contain them, and "dev" is replaced with an '@' so that it sorts lower than
-    than any other pre-release tag.
+    contain them, and "dev" is replaced with an '@' so that it sorts lower
+    than than any other pre-release tag.
 
 In other words, ``parse_version`` will return a tuple for each version string,
-that is compatible with ``StrictVersion`` but also accept arbitrary version and
-deal with them so they can be compared::
+that is compatible with ``StrictVersion`` but also accept arbitrary version
+and deal with them so they can be compared::
 
     >>> from pkg_resources import parse_version as V
     >>> V('1.2')
 versions, which makes it difficult for users to grok the versioning that a
 particular package was using and to provide tools on top of PyPI.
 
-Distutils classes are not really used in Python projects, but the
-Setuptools function is quite spread because it's used by tools like
-`easy_install` [#ezinstall]_, `pip` [#pip]_ or `zc.buildout` [#zc.buildout]_
-to install dependencies of a given project.
+Distutils classes are not really used in Python projects, but the Setuptools
+function is quite spread because it's used by tools like `easy_install`
+[#ezinstall]_, `pip` [#pip]_ or `zc.buildout` [#zc.buildout]_ to install
+dependencies of a given project.
 
-While Setuptools *does* provide a mechanism for comparing/sorting versions,
-it is much preferable if the versioning spec is such that a human can make a
+While Setuptools *does* provide a mechanism for comparing/sorting versions, it
+is much preferable if the versioning spec is such that a human can make a
 reasonable attempt at that sorting without having to run it against some code.
 
 Also there's a problem with the use of dates at the "major" version number
 (e.g. a version string "20090421") with RPMs: it means that any attempt to
-switch to a more typical "major.minor..." version scheme is problematic because
-it will always sort less than "20090421".
+switch to a more typical "major.minor..." version scheme is problematic
+because it will always sort less than "20090421".
 
 Last, the meaning of `-` is specific to Setuptools, while it is avoided in
 some packaging systems like the one used by Debian or Ubuntu.
 verlib
 ======
 
-During Pycon, members of the Python, Ubuntu and Fedora community worked on
-a version standard that would be acceptable for everyone.
+During Pycon, members of the Python, Ubuntu and Fedora community worked on a
+version standard that would be acceptable for everyone.
 
 The pseudo-format supported is::
 
     ...  < V('1.0.post456'))
     True
 
-The trailing ``.post123`` or ``.dev123`` is to allow post- or
-pre-releases.  In the pre-release case one usually "counts up" to the
-release, an example of this is the Python project itself. E.g. after a
-``2.6`` release the development snapshot of the next release would be
-called ``2.7.dev123``.
+The trailing ``.post123`` or ``.dev123`` is to allow post- or pre-releases. In
+the pre-release case one usually "counts up" to the release, an example of
+this is the Python project itself. E.g. after a ``2.6`` release the
+development snapshot of the next release would be called ``2.7.dev123``.
 
-In the post-release schema, used for example by Twisted [#twisted]_,
-you count away from a release.  So after a ``2.6`` release the
-development snapshot of the next release would be ``2.6.post123``.
+In the post-release schema, used for example by Twisted [#twisted]_, you count
+away from a release. So after a ``2.6`` release the development snapshot of
+the next release would be ``2.6.post123``.
 
-Lastly it is possible to have a development snapshot for a
-post-release.  This is a special case mostly provided to allow
-meaningful transitions from some edge cases in Setuptools versioning.
+Lastly it is possible to have a development snapshot for a post-release. This
+is a special case mostly provided to allow meaningful transitions from some
+edge cases in Setuptools versioning.
 
 XXX Check the actual requirement for this.
 
 suggest_rational_version
 ------------------------
 
-XXX explain here suggest_rational_version
+suggest_rational_version is a function that suggests a rational version close
+to the given version string. If you have a version string that isn't rational
+(i.e. RationalVersion doesn't like it) then you might be able to get an
+equivalent (or close) rational version from this function.
+
+This does a number of simple normalizations to the given string, based on
+observation of versions currently in use on PyPI.
+
+There is now a test program, `test_pypi_versions.py` that collects all of the
+entries on PyPi (all 8000+ of them) and produces a quick snapshot of whether
+their current version:
+
+    *   Is already compatible with RationalVersion
+    *   Can have a suggested RationalVersion via suggest_rational_version
+    *   Cannot be processed by suggest_rational_version
+
+As of Thursday, November 26, 2009, that program produced the following output:
+
+    Results:
+    --------
+    Total Packages  :  8515
+    Already Match   :  7478.0 (87.82%)
+    Have Suggestion :  1037.0 (12.18%)
+    No Suggestion   :  0.0 (0.00%)
+
+When a tool needs to work with versions, the best strategy is to use
+suggest_rational_version on the versions string. If this function returns
+None, it means that the provided version is not close enough to the standard
+scheme:
+
+>>> from verlib import suggest_rational_version, RationalVersion
+>>> def validate_version(version):
+...     rversion = suggest_rational_version(version)
+...     if rversion is None:
+...         raise ValueError('Cannot work with "%s"' % version)
+...     return RationalVersion(rversion)
+...
+
+>>> validate_version('2.4rc1')
+RationalVersion('2.4c1')
+
+>>> validate_version('foo')
+Traceback (most recent call last):
+...
+ValueError: Cannot work with "foo"
+
+>>> validate_version('1.24.33')
+RationalVersion('1.24.33')
+
+>>> validate_version('1.24.330pre1')
+RationalVersion('1.24.330c1')
+
+>>> validate_version('2008.12.11')
+Traceback (most recent call last):
+...
+ValueError: Cannot work with "2008.12.11"
+
 
 
 References
    http://twistedmatrix.com/trac/
 
 .. [#requires]
-   xxx
-   
+   http://www.python.org/dev/peps/pep-0386/
 
+

File test_pypi_versions.py

+#
+## test_pypi_versions.py
+##
+##  A very simple test to see what percentage of the current pypi packages
+##  have versions that can be converted automatically by distutils' new
+##  suggest_rational_version() into PEP-386 compatible versions.
+##
+##  Requires : Python 2.5+
+#
+
+try:
+   import cPickle as pickle
+except:
+   import pickle
+
+import xmlrpclib
+import os.path
+
+from verlib import suggest_rational_version
+
+INDEX_PICKLE_FILE = 'pypi-index.pkl'
+VERSION_PICKLE_FILE = 'pypi-version.pkl'
+
+package_info = version_info = []
+
+#
+## Get pypi full package list if the file's not found, else restore it
+#
+if not os.path.exists(INDEX_PICKLE_FILE):
+    print "Retrieving pypi packages..."
+    server = xmlrpclib.Server('http://pypi.python.org/pypi')
+    package_info  = server.search({'name': ''})
+
+    print "Saving package info.."
+    with open(INDEX_PICKLE_FILE, 'wb') as o:
+        pickle.dump(package_info, o)
+else:
+    print "Loading saved pypi data..."
+    with open(INDEX_PICKLE_FILE, 'rb') as f:
+        package_info = pickle.load(f)
+
+#
+## Save pypi package's version info  Get pypi full package list if the file's not found, else restore it
+#
+versions = []
+if not os.path.exists(VERSION_PICKLE_FILE):
+    print "Number of Packages = ", len(package_info)
+    versions = [p['version'] for p in package_info]
+    with open(VERSION_PICKLE_FILE, 'wb') as o:
+        pickle.dump(versions, o)
+else:
+    with open(VERSION_PICKLE_FILE, 'rb') as f:
+        versions = pickle.load(f)
+
+total_versions = len(versions)
+matches = 0.00
+no_sugg = 0.00
+have_sugg = 0.00
+
+for ver in versions:
+    sugg = suggest_rational_version(ver)
+    if ver == sugg:
+        matches += 1
+    elif ver == None:
+        no_sugg += 1
+    else:
+        have_sugg += 1
+
+pct = "(%2.2f%%)"
+print "Results:"
+print "--------"
+print "Total Packages  : ", total_versions
+print "Already Match   : ", matches, pct % (matches/total_versions*100,)
+print "Have Suggestion : ", have_sugg, pct % (have_sugg/total_versions*100,)
+print "No Suggestion   : ", no_sugg, pct % (no_sugg/total_versions*100,)
+
 """
-"Rational" version definition and parsing for DistutilsVersionFight discussion at PyCon 2009.
+"Rational" version definition and parsing for DistutilsVersionFight
+discussion at PyCon 2009.
 """
+
 import sys
 import re
 
     (?P<version>\d+\.\d+)          # minimum 'N.N'
     (?P<extraversion>(?:\.\d+)*)   # any number of extra '.N' segments
     (?:
-        (?P<prerel>[abc])             # 'a'=alpha, 'b'=beta, 'c'=release candidate
+        (?P<prerel>[abc])          # 'a'=alpha, 'b'=beta, 'c'=release candidate
         (?P<prerelversion>\d+(?:\.\d+)*)
     )?
     (?P<postdev>(\.post(?P<post>\d+))?(\.dev(?P<dev>\d+))?)?