Commits

Stan Seibert  committed d2bb2fb

Major documentation update

  • Participants
  • Parent commits bb50cc2

Comments (0)

Files changed (5)

 API Documentation
 =================
 
+.. _shrinkwrap_install_api:
+
 ``shrinkwrap.install``
 ----------------------
 .. automodule:: shrinkwrap.install
+How Shrinkwrap Works
+====================
+
+Shrinkwrap was designed to scratch a particular itch: automating the
+installation of software without using the system package manager is very
+cumbersome.
+
+Nearly all package managers assume they are managing software installation for
+the entire system, and don't allow isolated environments.  `ZeroInstall <http
+://zero-install.sourceforge.net/>`_ is a notable exception which installs
+software into a user's home directory and can manage multiple versions of a
+single package.  However, ZeroInstall is more aimed at providing specific
+applications to an end user, rather than providing a shell environment with a
+set of packages installed and ready for use.
+
+Python's virtualenv and easy_install/pip tools are a great example of a
+userspace packaging solution that creates self-contained, easy-to-use
+environments.  Since we work in mixed environments where C, C++ and Python
+software needs to coexist, shrinkwrap is our attempt to bring all that
+software under one roof.
+
+Shrinkwrap achieves this goal by modifying the virtualenv in three ways:
+
+1. It provides an alternate implementation of the ``python setup.py install``
+command that makes it easy to insert an arbitrary installer function.  This
+function typically downloads source tarballs, unpacks them, compiles and
+installs to the virtualenv base directory.
+
+2. It adds new optional keyword arguments to ``setup()`` to specify the
+installer function, and also to list packages that are build dependencies of
+this package.  See :ref:`package_dependencies` for details.
+
+3. It creates a new ``$VIRTUAL_ENV/env.d`` directory and patches the
+``$VIRTUAL_ENV/bin/activate`` script to source any files found in that
+directory.  This allows shrinkwrap packages to install scripts that make
+changes to the shell environment.  Many of the software packages we work with
+require environment variables to function properly.
+
+
+Installing shrinkwrap packages
+``````````````````````````````
+
+Installing a shrinkwrapped package is identical to installing any package with
+pip.  You can install the package file directly from the filesystem::
+
+	pip install curl-7.27.0.tar.gz
+
+or from a URL::
+
+	pip install http://mtrr.org/packages/curl/curl-7.27.0.tar.gz
+
+or set the URL of extra package repository which contains shrinkwrapped
+packages::
+
+	export PIP_EXTRA_INDEX_URL=http://mtrr.org/packages/
+	pip install curl
+
+If the shrinkwrapped package contains the standard test for the shrinkwrap
+module at the top (see :ref:`built_in_installers`), then shrinkwrap will be
+automatically installed to your virtualenv.
+
+Note that environment files in the ``$VIRTUAL_ENV/env.d/`` directory are only
+sourced when the ``$VIRTUAL_ENV/bin/activate`` script is sourced.  You will
+need to source that script again to refresh your environment if the
+shrinkwrapped package added new environment files.
+
+As true Python packages, shrinkwrapped packages can be used in a
+requirements.txt files passed to pip.  Just place the extra index URL
+argument at the top::
+
+   --extra-index-url http://mtrr.org/packages/
+   curl
+   fftw
+   nose
+
+
+.. _limitations:
+
+Limitations
+```````````
+
+We are the first to admit that shrinkwrap is straining the intended purpose of
+the Python packaging tools.  As a result, shrinkwrap+pip has some
+shortcomings compared to more sophisticated package managers:
+
+* Shrinkwrap can only be used with a virtualenv environment.  Shrinkwrap tests
+  for the presence of ``$VIRTUAL_ENV`` and will throw an exception if it is not
+  found.
+
+* Shrinkwrapped packages have only tested to work with ``pip`` as the installer.
+  We do not support using ``easy_install``, although it might work.
+
+* ``pip uninstall`` does not remove files installed by the package because we
+  do not yet have a way to track changes made by arbitrary installer functions
+  to the filesystem.
+
+* We do not provide any mechanism to have build options selected at
+  installation time (such as 
+  `variants in MacPorts <http://guide.macports.org/chunked/reference.variants.html>`_ or 
+  `USE variables in Gentoo emerge <http://www.gentoo.org/proj/en/devrel/handbook/handbook.xml?part=2&chap=1#doc_chap2_sect5>`_ ).  Repositories of shrinkwrapped packages are easy to make with the
+  ``shrinkwrap`` command line tool, so we encourage you to have different
+  repositories for different kinds of deployment environments.  Then you can
+  tailor the build options of your source packages for each one.
+
+* Shrinkwrap is currently focused on deploying source packages rather than
+  precompiled packages.  There is no reason you can't unpack a tarball of
+  compiled code in a shrinkwrap ``installer()`` function, but we do not 
+  provide any mechanism for having packages compiled for different 
+  architectures in the same repository.  Again, repositories are easy to
+  make, so we would suggest one per architecture if this is your use case.
+  (Selecting platform-specific build options in your ``installer()`` function, 
+  however, is no problem.)

File doc/index.rst

-Welcome to shrinkwrap's documentation!
-======================================
-Shrinkwrap provides tools to create Python package wrappers around non-Python software, and to install such software in a Pythonic way.
+shrinkwrap: Python packaging for everything
+===========================================
+Shrinkwrap provides tools to create lightweight Python package wrappers around non-Python software, and to install such software from project-specific repositories using virtualenv and pip.
 
-Shrinkwrapped packages can be uploaded to a Python package index, facilitating project-specific repositories that leverage familiar tools such as pip and virtualenv.
+A shrinkwrap package is a minimal python package that downloads, compiles and installs software to the virtualenv base directory.  The shrinkwrap package behaves just like a regular python package, so both shrinkwrap and non-shrinkwrap packages can be dependencies of each other.  We find that this greatly simplifies deployment of Python packages that depend on compiled libraries *without* requiring the use of system-wide packaging tools, like apt or yum.  Shrinkwrap is **not** an API wrapper generator like `SWIG <http://www.swig.org/>`_, but does make it easier to install C libraries into a virtualenv before installing a separate Python wrapper around its API.
+
+.. note:: To avoid cluttering PyPI with non-Python software, please do not ever upload shrinkwrap-generated packages there!  Hosting shrinkwrap packages yourself is easy, and described further in :ref:`running_package_index`.
 
 Shrinkwrap is `available on PyPI <http://pypi.python.org/pypi/shrinkwrap/0.7>`_ and can be installed via ``pip install shrinkwrap``, ``easy_install shrinkwrap``, or by downloading the package and installing with ``python setup.py install``.
 
 For those interested in the newest features should use the development version of shrinkwrap, `available on bitbucket <https://bitbucket.org/seibert/shrinkwrap>`_::
 
-    hg clone https://bitbucket.org/seibert/shrinkwrap
+    hg clone http://bitbucket.org/seibert/shrinkwrap
+
+which you can install to a virtualenv::
+
+	pip install -e hg+http://bitbucket.org/seibert/shrinkwrap#egg=shrinkwrap
 
 Contents:
 
 .. toctree::
    :maxdepth: 2
 
+   how
    packaging
    api
+   roadmap
+
 
 Indices and tables
 ==================

File doc/packaging.rst

 
 A shrinkwrap package may either use a built-in installer function (for common installation methods) or define a custom one.
 
+.. _built_in_installers:
+
 Built-in installers
 ```````````````````
-An installer for software using autoconf is available to simplify things. For example, to package `curl`, one may write the following (as, e.g. `curl-7.27.0.py`)::
+
+An installer for software using autoconf is available to simplify things.  The
+autoconf shrinkwrap installer assumes the tarfile unpacks to a directory with the same
+name as the tarfile with .tar.{gz,bz2} removed, contains a configure script,
+and understands the --prefix option to control where the package is installed.
+
+For example, to package `curl`, one may write the following (as, e.g. `curl-7.27.0.py`)::
 
     try:
         from shrinkwrap.install import ShrinkwrapInstall
     except ImportError:
-        import subprocess; subprocess.check_call('pip install shrinkwrap', shell=True)
+        import subprocess; subprocess.check_call('pip install -b $VIRTUAL_ENV/build/build-shrinkwrap shrinkwrap', shell=True)
         from shrinkwrap.install import ShrinkwrapInstall
     from setuptools import setup
 
 
     python bzip2-1.0.6.py install
 
-Here, the ``shrinkwrap_installer`` argument to ``setup()`` is set to our own installer function. The installer uses two shrinkwrap-provided convenience functions, ``download_and_unpack_tarball`` and ``make`` to download, untar, and compile the bzip2 library. By passing extra options to make, the software is installed into the root of the active virtualenv.
+Here, the ``shrinkwrap_installer`` argument to ``setup()`` is set to our own installer function. The installer uses two shrinkwrap-provided convenience functions, ``download_and_unpack_tarball`` and ``make`` to download, untar, and compile the bzip2 library. See :ref:`shrinkwrap_install_api` for a complete list of convenice functions. By passing extra options to make, the software is installed into the root of the active virtualenv.
 
 .. note:: For several examples of custom installers, see `https://bitbucket.org/seibert/shrinkwrap_pkgs <https://bitbucket.org/seibert/shrinkwrap_pkgs>`_.
 
     Examples include getting code from version control, installing with cmake, and customizing install paths.
 
+
+.. _package_dependencies:
+
+Package Dependencies
+````````````````````
+
+The handling of package dependencies in pip (as of version 1.1, anyway) does
+not, unfortunately, meet the requirements for installing compiled software.
+Packages listed in the ``install_requires`` keyword argument to ``setup()``
+will be discovered by pip and installed, but in an arbitrary order.  Packages
+listed in the ``setup_requires`` keyword argument are not actually installed
+to the virtualenv, but rather made available in .egg files in the build
+directory, which also is not a solution for compiled code.
+
+As a result, shrinkwrap adds a new keyword argument, ``shrinkwrap_requires``,
+to ``setup()``. All dependencies listed here are guaranteed to be fully
+installed to the virtualenv before the ``installer()`` function is run.  A
+separate pip process is spawned to install each one, so the dependency string
+can include package versions, just as ``setup_requires`` and
+``install_requires`` allow.  The shrinkwrap dependencies can be any kind of
+package, so if your library uses `SCons <http://www.scons.org/>`_ as the build system, you can list it in the ``shrinkwrap_requires`` field::
+
+    setup(
+        name='foo',
+        version=version,
+        author='Example Author'
+        author_email='author@example.com'
+        shrinkwrap_installer=installer,
+        shrinkwrap_requires=['scons'],
+        cmdclass={'install': ShrinkwrapInstall},
+    )
+
+
+Environment Variables
+`````````````````````
+
+If your package also needs to set some shell environment variables for
+operation, they can be placed in the ``$VIRTUAL_ENV/env.d`` directory, and
+they will be sourced automatically when the ``$VIRTUAL_ENV/bin/activate``
+script is sourced.  See 
+`the package file for CERN's ROOT library <https://bitbucket.org/seibert/shrinkwrap_pkgs/src/127a74fa3a17/root.py#cl-31>`_ 
+for an example of using the ``install_env()`` convenience function.
+
+
 Packaging for a package index
 -----------------------------
-To share shrinkwrap packages via a package index like PyPI, you must create distribution tarballs. Shrinkwrap includes a tool to create these automatically from wrapper packages::
+
+To share shrinkwrap packages via a package index like PyPI (but don't upload
+shrinkwrap packages to PyPI!!), you must create distribution tarballs.
+Shrinkwrap includes a tool to create these automatically from wrapper
+packages::
 
     shrinkwrap createpkg bzip2-1.0.6.py
 
-This will create a tarball suitable for uploading. Wildcards are valid, for tarring up many packages at once.
+This will create a tarball suitable for uploading. Wildcards are valid, for
+tarring up many packages at once.
+
+.. _running_package_index:
 
 Running your own package index
 ------------------------------
-If you wish to serve your own package index, use the ``-p`` option to place output tarballs into one properly-formatted directory::
+
+As mentioned in :ref:`limitations` section, shrinkwrap packages are small and
+easy to deploy.  It is better to have different package repositories when you
+want to build a source package with different options in different situations,
+rather than have One Repository To Rule Them All [#onerepo]_ .
+
+If you wish to serve your own package index, use the ``-p`` option to place
+output tarballs into one properly-formatted directory::
 
     shrinkwrap createpkg -p packages my_shrinkwrap_pkgs/*
 
-Simply serve the `packages` directory on the web, and pip clients can interact with it just like PyPI::
+Simply serve the `packages` directory on the web, and pip clients can interact
+with it just like PyPI::
 
     client$ export PIP_EXTRA_INDEX_URL=http://your-server.com/packages/
     client$ pip install bzip2
 
+Extra package indices can be specified in a requirements.txt file with an option 
+at the top::
+
+    --extra-index-url http://mtrr.org/packages/
+
+
+.. [#onerepo] In order to avoid creating One Repository To Rule Them All, we do not plan to ever offer a repository of "official" shrinkwrap packages. Feel free to copy other people's package files to your repository, however.

File doc/roadmap.rst

+Roadmap
+=======
+
+Near future goals:
+
+* Automated testing suite.  Due to the nature of shrinkwrap, this will require the testing framework to repeatedly create and destroy entire virtualenvs for each test, which requires some non-trivial infrastructure coding.
+
+* SHA1 hash checking of downloaded tar files.
+
+* An interface for shrinkwrap packages to test and report whether the wrapped software is already installed system-wide.  This would allow a shrinkwrap package to optionally skip a lengthy compile and install step when the appropriate system packages are already present.
+
+
+Far future goals:
+
+* Remove all installed files when pip uninstall is called.  This will require detecting and recording all the files added to the virtualenv by the installer() function.