Michael Forbes avatar Michael Forbes committed 2ef790e

Initial commit.

Comments (0)

Files changed (17)

+syntax: glob
+
+.README.html
+# Documentation.  Automatic compilation of the README.rst file to aid editing
+
+.README.html: README.rst
+	rst2html.py $< > $@
+	open $@; sleep 1; open $<
+
+watch:
+	while sleep 1; do make -q rst-html || make rst-html ; done
+
+rst-html: .README.html
+
+# Python Setup
+
+# Updates the requirements to match the version on this machin
+extended_requirements.txt: minimal_requirements.txt
+	pip freeze -r $^ > $@
+
+.PHONY: watch rst-html
+.. -*- rst -*- -*- restructuredtext -*-
+
+.. This file should be written using the restructure text
+.. conventions.  It will be displayed on the bitbucket source page and
+.. serves as the documentation of the directory.
+
+Python Setup
+============
+This meta-project collects all of the python tools I typically use.  It also
+serves as a fairly minimal example of setting up a package the |pip|_ can
+install, and specifying dependencies.
+
+Quick Start
+===========
+If you are impatient and courageous, here is the exceutive summary:
+
+* Install |EPD|_, git_, and GSL_.
+
+.. code:: bash
+
+   pip install virtualenv
+   virtualenv --system-site-packages --distribute ~/.python_environments/epd
+   virtualenv --no-site-packages --distribute ~/.python_environments/clean
+   virtualenv -p /usr/bin/python --system-site-packages --distribute \
+              ~/.python_environments/sys
+
+   cat >> ~/.bashrc <<EOF
+   alias v.epd=". ~/.python_environments/epd/bin/activate"
+   alias v.sys=". ~/.python_environments/sys/bin/activate"
+   alias v.clean=". ~/.python_environments/clean/bin/activate"
+   v.epd
+   EOF
+
+   pip install hg
+
+   mkdir -p ~/src/python/git
+   cd ~/src/python/git
+   git clone http://github.com/gldnspud/virtualenv-pythonw-osx.git
+   cd virtualenv-pythonw-osx
+   python install_pythonw.py /Users/mforbes/.python_environments/epd
+
+Then choose the set of requirements and:
+
+.. code:: bash
+
+   pip install -r all.txt
+
+Requirements
+============
+Here is a list of the various requirements.  These are all disjoint, so you can
+pick and choose.
+
+``doc.txt`` :
+   Various documentation tools like Sphinx_ and associated packages.  I use this
+   for both my code documentation and for things like my website.
+``emacs.txt`` :
+   Various tools for setting up my development environment (I use emacs)
+   including checking tools.
+``debug.txt`` : 
+   Debugging tools, including remote debuggers.
+``profile.txt`` :
+   Profiling tools for optimizing code.
+``testing.txt`` :
+   Testing tools including code coverage.
+``vc.txt`` :
+   Version control tools like mercurial and extensions
+``misc.txt`` :
+   Odds and ends.
+``mmf.txt`` :
+   My source packages for projects.  These will be installed as source
+   distributions.
+``all.txt`` :
+   All of the above.
+
+Here are some additional requirement files:
+
+``EPD.txt`` :
+   The list of requirements frozen from a fresh EPD_ install.
+``freeze.txt`` :
+   Snapshot of my system by running ``pip freeze > freeze.txt``
+
+
+
+Details
+=======
+To use it do the following:
+
+1) Install a version of python.  Many systems have a version preinstalled, so
+   this step is optional.  However, if you plan to do serious development, then
+   I strongly recommend installing the |EPD|_.  There is a free version, and an
+   almost full featured free version for academic use: You can also pay for a
+   comercial version and recieve support.  The EPD_ is very complete, and just
+   works on most common platforms and I highly recommend it.  Make sure you can
+   run the version of python you desire.
+
+   If you install the EPD_, then it will typically add something like the
+   following to your ``~/.bash_login`` or ``~/.profile`` files::
+
+      # Setting PATH for EPD-7.3-2
+      # The orginal version is saved in .bash_login.pysave
+      PATH="/Library/Frameworks/Python.framework/Versions/Current/bin:${PATH}"
+      export PATH
+      
+      MKL_NUM_THREADS=1
+      export MKL_NUM_THREADS
+
+   (If you want to use a multithreaded version of ``numpy``, you will need to
+   change the value of ``MKL_NUM_THREADS``.  See `this discussion`__.)
+
+__ http://stackoverflow.com/q/5260068/1088938
+
+2) Create a virtualenv_.  This will allow you to install new packages in a
+   controlled manner that will not mess with the system version (or the EPD_
+   version).  You can create multiple virtual environments for different
+   projects or associated with different versions of python.  Again, this is
+   highly recommended.  There are several ways of doing this. 
+
+   .. note:: Methods 1) and 2) will install virtualenv_ to the location 
+      specified by the current version of python.  This means that you might
+      need root access, and it will slightly "muck up" you pristine system
+      install. This is generally not a problem, but if it bothers you see step
+      3).
+
+   1) If you have |pip|_ (the new python packageing system), then you can use it
+      to install virtualenv_ as follows::
+
+         pip install virtualenv
+   
+   2) If you do not have |pip|_, you might have ``easy_install``::
+   
+         easy_install virtualenv
+
+   3) If you do not want to muck up your system version of python at all, then
+      you can simply download the file |virtualenv.py|_.  In the commands that
+      follow, replace ``virtualenv`` with ``python virtualenv.py``.
+
+.. |virtualenv.py| replace:: ``virtualenv.py``
+.. _virtualenv.py: https://raw.github.com/pypa/virtualenv/master/virtualenv.py
+
+3) Setup a virtual environment for your work.  You can have many differen
+   environments, so you will need to choose a meaningful name.  I use "epd" for
+   the EPD_ version of python, "sys" for the system version of python, and
+   "clean" for a version using EPD_ but without the site-packages::
+
+       virtualenv --system-site-packages --distribute ~/.python_environments/epd
+       virtualenv --no-site-packages --distribute ~/.python_environments/clean
+       virtualenv -p /usr/bin/python --system-site-packages --distribute \
+                  ~/.python_environments/sys
+
+   Once this virtualenv_ is activated, install packages with pip_ will place all
+   of the installed files in the ``~/.python_environments/epd`` directory.  (You
+   can change this to any convenient location).  The ``--system-site-packages``
+   option allows the virtualenv_ access to the system libraries (in my case, all
+   of the EPD_ goodies).  If you want to test a system for deployment, making
+   sure that it does not have any external dependencies, then you would use the
+   ``--no-site-packages`` option instead.  Run ``virtualenv --help`` for more
+   information.
+
+4) Add some aliases to help you activate virtualenv_ sessions.  I include the
+   following in my ``.bashrc`` file::
+
+      # Some virtualenv related macros
+      alias v.epd=". ~/.python_environments/epd/bin/activate"
+      alias v.sys=". ~/.python_environments/sys/bin/activate"
+      alias v.clean=". ~/.python_environments/clean/bin/activate"
+      v.epd
+
+   You can activate your chosen environment with one of the commands ``v.epd``,
+   ``v.clean``, or ``v.sys``.  The default activation script will insert "(epd)"
+   etc. to your prompt::
+
+      ~ mforbes$ v.epd
+      (epd)~ mforbes$ v.sys
+      (sys)~ mforbes$ deactivate
+      ~ mforbes$
+
+   To get out of the environments, just type ``deactivate`` as shown above.
+   
+   .. note:: If you have an older version of IPython_ (pre 0.13), then you may
+      need to call ``ipython`` from a `function like this`__::
+
+         # Remap ipython if VIRTUAL_ENV is defined
+         function ipython {
+           if [ -n "${VIRTUAL_ENV}" -a -x "${VIRTUAL_ENV}/bin/python" ]; then
+             START_IPYTHON='\
+               import sys; \
+               from IPython.frontend.terminal.ipapp import launch_new_instance;\
+               sys.exit(launch_new_instance())'
+              "${VIRTUAL_ENV}/bin/python" -c "${START_IPYTHON}" "$@"
+            else
+              command ipython "$*"
+            fi
+         }
+
+
+      This deals with issues that IPython_ was not virtualenv_ aware.  The
+      recommended solution is still to install IPython_ in the virtualenv_ using
+      ``pip install ipython``, but then you will need one in each environment.
+      As of IPython_ 0.13, this support is included. (See `this PR`__.)
+
+      If you have not used IPython_ before, then you should have a look.  It has
+      some fantastic features like ``%paste`` and the `IPython notebook`_
+      interface.
+
+__ http://igotgenes.blogspot.fr/2010/01/interactive-sandboxes-using-ipython.html
+__ https://github.com/ipython/ipython/pull/1388/
+
+5) Install mercurial_.  You may already have this (try ``hg --version``).  If
+   not, either install a native distribution (which might have some GUI tools)
+   or install with::
+
+      pip install hg
+
+6) Install git_.  This may not be as easy, but some packages are only available
+   from github_.
+
+7) On Mac OS X you may need to install ``pythonw`` for some GUI applications
+   (like RunSnakeRun_).  You an do this using `this solution`__::
+   
+      mkdir -p ~/src/python/git
+      cd ~/src/python/git
+      git clone http://github.com/gldnspud/virtualenv-pythonw-osx.git
+      cd virtualenv-pythonw-osx
+      python install_pythonw.py /Users/mforbes/.python_environments/epd
+
+   You will have to do this in each virtualenv_ you want to use the GUI apps
+   from.
+
+__ https://github.com/gldnspud/virtualenv-pythonw-osx
+
+8) Non-python prerequisites.  These need to be installed outside of the python
+   environment for some of the required libraries to work.
+
+   * GSL_: This is needed for pygsl_.
+
+
+9) Install various requirements as follows::
+
+   pip install -r requirements/all.txt
+
+.. |EPD| replace:: Enthough Python Distribution
+.. _EPD: http://www.enthought.com/products/epd.php
+.. _mercurial: http://mercurial.selenic.com/
+.. _virtualenv: http://www.virtualenv.org/en/latest/
+.. _IPython: http://ipython.org/
+.. _Ipython notebook: \
+   http://ipython.org/ipython-doc/dev/interactive/htmlnotebook.html
+.. |pip| replace:: ``pip``
+.. _pip: http://www.pip-installer.org/
+.. _git: http://git-scm.com/
+.. _github: https://github.com
+.. _RunSnakeRun: http://www.vrplumber.com/programming/runsnakerun/
+.. _GSL: http://www.gnu.org/software/gsl/
+.. _pygsl: https://bitbucket.org/mforbes/pygsl
+.. _Sphinx: http://sphinx-doc.org/
+
+
+Using |pip|_
+============
+Here are some notes about using |pip|_ that I did not find obvious.
+
+Version Control
+---------------
+It is clear from the `documentation about requirements`__ that you can specify
+version controlled repositories with |pip|_, however, the exact syntax for
+specifying revisions etc. is not so clear.  Examining `the source`__ shows that
+you can specify revisions, tags, etc. as follows::
+
+   # Get the "tip"
+   hg+http://bitbucket.org/mforbes/pymmf#egg=pymmf
+
+   # Get the revision with tag "v1.0" or at the tip of branch "v1.0"
+   hg+https://bitbucket.org/mforbes/pymmf@v1.0#egg=pymmf
+
+   # Get the specified revision exactly
+   hg+https://bitbucket.org/mforbes/pymmf@633be89a#egg=pymmf
+
+What appears after the "@" sign is any valid revision (for mercurial see ``hg
+help revision`` for various options).  Unfortunately, I see no way of specifying
+something like ">=1.1", or ">=633be89a" (i.e. a descendent of a particular
+revision).  (See `issue 782`__)
+
+__ http://www.pip-installer.org/en/latest/requirements.html
+__ https://github.com/pypa/pip/blob/develop/pip/vcs/mercurial.py
+__ https://github.com/pypa/pip/issues/728

requirements/EPD.txt

+Cython==0.16
+EPD==7.3
+Examples==7.3
+GDAL==1.8.1
+Jinja2==2.6
+MDP==3.2
+MKL==10.3
+PIL==1.1.7
+PyOpenGL==3.0.1
+PySide==1.1.0
+PyYAML==3.10
+Pycluster==1.50
+Pygments==1.4
+PythonDoc==2.7.3
+Qt==4.7.3
+Reportlab==2.5
+SQLAlchemy==0.7.6
+Shapely==1.2.14
+SimPy==2.2
+Sphinx==1.1.2
+Twisted==12.0.0
+VTK==5.6.0
+appinst==2.1.1
+apptools==4.1.0
+basemap==1.0.2
+biopython==1.59
+bitarray==0.8.0
+blist==1.3.4
+blockcanvas==4.0.1
+bsdiff4==1.1.1
+casuarius==1.0
+chaco==4.2.0
+cloud==2.4.6
+codetools==4.0.0
+configobj==4.7.2
+coverage==3.5.1
+curl==7.25.0
+distribute==0.6.28
+doclinks==7.3
+docutils==0.8.1
+enable==4.2.0
+enaml==0.2.0
+encore==0.2
+enstaller==4.5.0
+envisage==4.2.0
+epydoc==3.0.1
+ets==4.2.0
+etsdevtools==4.0.0
+etsproxy==0.1.1
+feedparser==5.1.1
+foolscap==0.6.3
+freetype==2.4.4
+fwrap==0.1.1
+graphcanvas==4.0.0
+grin==1.2.1
+h5py==2.0.0
+hdf5==1.8.9
+html5lib==0.90
+idle==2.7.3
+jsonpickle==0.4.0
+keyring==0.9
+lib-netcdf4==4.2
+libgdal==1.8.1
+libjpeg==7.0
+libpng==1.2.40
+libxml2==2.7.8
+libxslt==1.1.26
+lxml==2.3.4
+matplotlib==1.1.0
+mayavi==4.2.0
+netCDF4==1.0
+networkx==1.6
+nose==1.1.2
+numexpr==2.0.1
+numpy==1.6.1
+openpyxl==1.5.8
+pandas==0.7.3
+paramiko==1.7.7.1
+pep8==1.0.1
+ply==3.4
+pyOpenSSL==0.12
+pyaudio==0.2.4
+pycrypto==2.4.1
+pydot==1.0.28
+pyface==4.2.0
+pyfits==3.0.6
+pyflakes==0.5.0
+pygarrayimage==0.0.7
+pyglet==1.1.4
+pyhdf==0.8.3
+pyparsing==1.5.6
+pyproj==1.9.0
+pyserial==2.6
+pytables==2.3.1
+python-dateutil==1.5
+pytz==2011n
+pyzmq==2.1.11
+scikit-learn==0.11
+scikits.image==0.5.0
+scikits.timeseries==0.91.3
+scimath==4.1.0
+scipy==0.10.1
+scons==2.0.1
+statsmodels==0.4.0
+swig==1.3.40
+tornado==2.2
+traits==4.2.0
+traitsui==4.2.0
+virtualenv==1.8.2
+wsgiref==0.1.2
+wxPython==2.8.10.1
+xlrd==0.7.6
+xlwt==0.7.3
+zope.interface==3.8.0

requirements/all.txt

+## These requirements are for the packages I like to have installed.
+
+EPD>=7.3
+
+-r doc.txt
+-r debug.txt
+-r testing.txt
+-r profile.txt
+-r emacs.txt
+-r numeric.txt
+-r vc.txt
+-r package.txt
+-r misc.txt
+-r mmf.txt

requirements/debug.txt

+## Debugging tools
+ipdb>=0.7
+winpdb>=1.4.8

requirements/doc.txt

+## Various tools for generating documentation.
+## I use these packages for my documentation, website, etc.
+docutils>=0.9.1
+Sphinx>=1.1.3
+Jinja2>=2.6
+Pygments>=1.5
+epydoc>=3.0.1
+numpydoc>=0.4
+zope.interface>=3.8.0
+sphinxcontrib-zopeext>=0.2
+rst2rst>=0.3.dev0
+
+hg+ssh://hg@bitbucket.org/mforbes/pweave#egg=mmfweave-dev

Empty file added.

requirements/emacs.txt

+## Development environment: 
+## I use these packages for customizing my development environment (emacs) and
+## checking (lint etc.)
+logilab-astng>=0.24.0
+logilab-common>=0.58.1
+pyflakes>=0.5.0
+rope>=0.9.4
+ropemacs>=0.7
+ropemode>=0.2
+pep8>=1.3.3
+git+http://github.com/pinard/Pymacs.git#egg=Pymacs

requirements/freeze.txt

+## The following requirements were added by pip --freeze:
+CherryPy==3.2.2
+Cython==0.16
+EPD==7.3
+Examples==7.3
+GDAL==1.8.1
+Jinja2==2.6
+MDP==3.2
+MKL==10.3
+-e hg+ssh://hg@bitbucket.org/mforbes/gpe_vortex@0aa39a8a4a4e0aea9ac23364ef693898bc92a666#egg=MMFGPE-dev
+-e hg+ssh://maxwell.phys.washington.edu/current/slda@f5aa01c9547fa72862a4ff15a8a85119f351ee6b#egg=MMFslda-dev
+PIL==1.1.7
+PyOpenGL==3.0.1
+PySide==1.1.0
+PyYAML==3.10
+Pycluster==1.50
+Pygments==1.5
+-e git+http://github.com/pinard/Pymacs.git@4be2c15f423e0a9007fcd3c0468315313f475b27#egg=Pymacs-dev
+PythonDoc==2.7.3
+Qt==4.7.3
+Reportlab==2.5
+RunSnakeRun==2.0.2b1
+SQLAlchemy==0.7.6
+Shapely==1.2.14
+SimPy==2.2
+Sphinx==1.1.3
+SquareMap==1.0.1
+Theano==0.5.0
+Twisted==12.0.0
+VTK==5.6.0
+anfft==0.2
+appinst==2.1.1
+apptools==4.1.0
+basemap==1.0.2
+biopython==1.59
+bitarray==0.8.0
+blist==1.3.4
+blockcanvas==4.0.1
+bsdiff4==1.1.1
+casuarius==1.0
+chaco==4.2.0
+cloud==2.4.6
+codetools==4.0.0
+configobj==4.7.2
+coverage==3.5.1
+curl==7.25.0
+cvxopt==1.1.3
+distribute==0.6.27
+doclinks==7.3
+docutils==0.9.1
+dulwich==0.8.5
+enable==4.2.0
+enaml==0.2.0
+encore==0.2
+enstaller==4.5.0
+envisage==4.2.0
+epydoc==3.0.1
+ets==4.2.0
+etsdevtools==4.0.0
+etsproxy==0.1.1
+feedparser==5.1.1
+foolscap==0.6.3
+freetype==2.4.4
+fwrap==0.1.1
+graphcanvas==4.0.0
+grin==1.2.1
+h5py==2.0.0
+hdf5==1.8.9
+hg-git==0.3.3
+hghooks==1.0
+html5lib==0.90
+idle==2.7.3
+ipdb==0.7
+ipython==0.13
+jsonpickle==0.4.0
+keyring==0.9
+lib-netcdf4==4.2
+libgdal==1.8.1
+libjpeg==7.0
+libpng==1.2.40
+libxml2==2.7.8
+libxslt==1.1.26
+-e hg+https://bitbucket.org/robertkern/line_profiler@94ae5ec83b536b752acac016ba0522cf2f89cef7#egg=line_profiler-dev
+logilab-astng==0.24.0
+logilab-common==0.58.1
+lxml==2.3.4
+matplotlib==1.1.1
+mayavi==4.2.0
+mercurial==2.4-rc
+-e hg+ssh://hg@bitbucket.org/mforbes/pweave@77d452c91badf2f1f5686d3afd547d851da4fecc#egg=mmfweave-dev
+netCDF4==1.0
+networkx==1.6
+-e git+http://github.com/nose-devs/nose.git@a6894e0d23afc3afeb921620a2b61ebf6bbea5c8#egg=nose-dev
+numexpr==2.0.1
+numpy==1.6.1
+numpydoc==0.4
+openpyxl==1.5.8
+pandas==0.7.3
+paramiko==1.7.7.1
+pep8==1.3.3
+pexpect==2.4
+ply==3.4
+pp==1.6.2
+pyOpenSSL==0.12
+pyaudio==0.2.4
+pycrypto==2.4.1
+pydot==1.0.28
+pyface==4.2.0
+pyfits==3.0.6
+pyflakes==0.5.0
+pygarrayimage==0.0.7
+pyglet==1.1.4
+-e hg+ssh://hg@bitbucket.org/mforbes/pygsl@71b97a98a5ce4f618123203a6144a69ed11b7fee#egg=pygsl-dev
+pyhdf==0.8.3
+-e hg+ssh://hg@bitbucket.org/mforbes/pymmf@633be89aafa95c7e2cf6f797ec90cd93a170b258#egg=pymmf-dev
+pyparsing==1.5.6
+pyproj==1.9.0
+pyserial==2.6
+pytables==2.3.1
+python-dateutil==1.5
+pytz==2011n
+pyzmq==2.1.11
+rope==0.9.4
+ropemacs==0.7
+ropemode==0.2
+rst2rst==0.3.dev0
+scikit-learn==0.11
+scikits.bvp1lg==0.2.5
+scikits.image==0.5.0
+scikits.talkbox==0.2.5
+scikits.timeseries==0.91.3
+scimath==4.1.0
+scipy==0.10.1
+scons==2.0.1
+spectrum==0.9
+sphinxcontrib-zopeext==0.2
+statsmodels==0.4.0
+swig==1.3.40
+sympy==0.7.2
+tornado==2.2
+traits==4.2.0
+traitsui==4.2.0
+virtualenv==1.8.2
+winpdb==1.4.8
+wsgiref==0.1.2
+wxPython==2.8.10.1
+xlrd==0.7.6
+xlwt==0.7.3
+yolk==0.4.3
+zope.interface==3.8.0

requirements/misc.txt

+## Miscellaneous
+
+ipython>=0.13
+
+# Webservers/UI/Interaction
+CherryPy>=3.2.2
+pexpect>=2.4

requirements/mmf.txt

+## My source code and packages.  These are installed as source packages
+-e hg+ssh://hg@bitbucket.org/mforbes/gpe_vortex#egg=MMFGPE-dev
+-e hg+ssh://maxwell.phys.washington.edu/current/slda#egg=MMFslda-dev
+-e hg+ssh://hg@bitbucket.org/mforbes/pymmf#egg=pymmf-dev
+

requirements/numeric.txt

+## Numerical Computing:
+## These are useful for numerical computing
+numpy>=1.6.1
+scipy>=0.10.1
+matplotlib>=1.1.1
+sympy>=0.7.2
+
+hdf5>=1.8.9
+
+Theano>=0.5.0
+anfft>=0.2
+
+scikits.bvp1lg>=0.2.5
+
+scikits.talkbox>=0.2.5
+scikits.timeseries>=0.91.3
+spectrum>=0.9
+
+cvxopt>=1.1.3
+
+pp>=1.6.2
+
+swig>=1.3.40
+
+hg+ssh://hg@bitbucket.org/mforbes/pygsl#egg=pygsl-dev

requirements/package.txt

+# Package Management/Environment
+virtualenv>=1.8.2
+yolk>=0.4.3

requirements/profile.txt

+## Profiling tools
+SquareMap>=1.0.1
+RunSnakeRun>=2.0.2b1
+hg+https://bitbucket.org/robertkern/line_profiler#egg=line_profiler-dev

requirements/testing.txt

+## Testing
+coverage>=3.5.1
+nose>=1.1.2
+git+http://github.com/nose-devs/nose.git#egg=nose-dev

requirements/vc.txt

+## Version control.  Mecurial extensions etc.
+mercurial>=2.4-rc
+hg-git>=0.3.3
+hghooks>=1.0
+dulwich>=0.8.5
Tip: Filter by directory path e.g. /media app.js to search for public/media/app.js.
Tip: Use camelCasing e.g. ProjME to search for ProjectModifiedEvent.java.
Tip: Filter by extension type e.g. /repo .js to search for all .js files in the /repo directory.
Tip: Separate your search with spaces e.g. /ssh pom.xml to search for src/ssh/pom.xml.
Tip: Use ↑ and ↓ arrow keys to navigate and return to view the file.
Tip: You can also navigate files with Ctrl+j (next) and Ctrl+k (previous) and view the file with Ctrl+o.
Tip: You can also navigate files with Alt+j (next) and Alt+k (previous) and view the file with Alt+o.