cram / README.txt

======================
 Cram: It's test time
======================

Cram is a functional testing framework for command line applications
based on Mercurial_'s `unified test format`_.

Here's a snippet from ``cram.t`` in `Cram's own test suite`_::

    The $PYTHON environment variable should be set when running this
    test from Python.

      $ [ -n "$PYTHON" ] || PYTHON=python
      $ if [ -n "$COVERAGE" ]; then
      >   coverage erase
      >   alias cram='coverage run -a cram.py'
      > else
      >   alias cram="$PYTHON cram.py"
      > fi
      $ command -v md5 || alias md5=md5sum

    Usage:

      $ cram -h
      [Uu]sage: cram \[OPTIONS\] TESTS\.\.\.

      [Oo]ptions:
        -h, --help            show this help message and exit
        -v, --verbose         show filenames and test status
        -i, --interactive     interactively merge changed test output
        -y, --yes             answer yes to all questions
        -n, --no              answer no to all questions
        -D DIR, --tmpdir=DIR  run tests in DIR
        --keep-tmpdir         keep temporary directories
        -E                    don't reset common environment variables
      $ cram
      [Uu]sage: cram \[OPTIONS\] TESTS\.\.\.
      [1]

The format in a nutshell:

* Cram tests use the ``.t`` file extension.

* Lines beginning with two spaces, a dollar sign, and a space are run
  in the shell.

* Lines beginning with two spaces, a greater than sign, and a space
  allow multi-line commands.

* All other lines beginning with two spaces are considered command
  output.

* Command output in the test is first matched literally with the
  actual output. If it doesn't match, it's then compiled and matched
  as a `Perl-compatible regular expression`_.

* Command output in the test that ends with a percent sign will match
  actual output that doesn't end in a newline.

* Anything else is a comment.

.. _Mercurial: http://mercurial.selenic.com/
.. _unified test format: http://www.selenic.com/blog/?p=663
.. _Cram's own test suite: http://bitbucket.org/brodie/cram/src/tip/tests/cram.t
.. _Perl-compatible regular expression: http://en.wikipedia.org/wiki/Perl_Compatible_Regular_Expressions


Download
--------

* cram-0.3.tar.gz_ (13 KB, requires Python 2.4-2.7 or Python 3.1)

.. _cram-0.3.tar.gz: http://bitheap.org/cram/cram-0.3.tar.gz

Installation
------------

You can use pip_ to install Cram::

    $ sudo pip install cram

Or you can install Cram the old fashioned way::

    $ wget http://bitheap.org/cram/cram-0.3.tar.gz
    $ tar zxvf cram-0.3.tar.gz
    $ cd cram-0.3.tar.gz
    $ sudo python setup.py install

.. _pip: http://pypi.python.org/pypi/pip


Usage
-----

Cram will print a dot for each passing test. If a test fails, a
`unified context diff`_ is printed showing the test's expected output
and the actual output.

For example, if we run cram on `its own example tests`_::

    $ cram examples
    ..
    --- examples/fail.t
    +++ examples/fail.t.out
    @@ -3,11 +3,11 @@
       $ echo 1
       1
       $ echo 1
    -  2
    +  1
       $ echo 1
       1

     Invalid regex:

       $ echo 1
    -  +++
    +  1
    ..

Cram will also write the test with its actual output to
``examples/fail.t.err``.

When you're first writing a test, you might just write the commands
and run the test to see what happens. If you run Cram with ``-i`` or
``--interactive``, you'll be prompted to merge the actual output back
into the test. This makes it easy to quickly prototype new tests.

Note that the following environment variables are reset before tests
are run:

* ``TMPDIR``, ``TEMP``, and ``TMP`` are set to the test runner's
  ``tmp`` directory.

* ``LANG``, ``LC_ALL``, and ``LANGUAGE`` are set to ``C``.

* ``TZ`` is set to ``GMT``.

* ``COLUMNS`` is set to ``80``.

* ``CDPATH`` and ``GREP_OPTIONS`` are set to an empty string.

Cram also provides the following environment variables to tests:

* ``RUNDIR``, set to the directory Cram was run from.

* ``TESTDIR``, set to the test runner's temporary directory.

.. _unified context diff: http://en.wikipedia.org/wiki/Diff#Unified_format
.. _its own example tests: http://bitbucket.org/brodie/cram/src/tip/examples/


News
----

Version 0.3
```````````
* Implemented resetting of common environment variables. This behavior
  can be disabled using the ``-E`` flag.

* Changed the test runner to first make its own overall random
  temporary directory, make ``tmp`` inside of it and set ``TMPDIR``,
  etc. to its path, and run each test with a random temporary working
  directory inside of that.

* Added ``--keep-tmpdir``. Temporary directories are named by test
  filename (along with a random string).

* Added ``-i``/``--interactive`` to merge actual output back to into
  tests interactively.

* Added ability to match command output not ending in a newline by
  suffixing output in the test with ``%``.

Version 0.2
```````````
* Changed the test runner to run tests with a random temporary working
  directory.


Version 0.1
```````````
* Initial release.


Development
-----------

Download the official development repository using Mercurial_::

    hg clone http://bitbucket.org/brodie/cram

Test Cram using Cram::

    make tests

Get a test coverage report using coverage.py_::

    make coverage

Visit Bitbucket_ if you'd like to fork the project, watch for new
changes, or report issues.

.. _Mercurial: http://mercurial.selenic.com/
.. _coverage.py: http://nedbatchelder.com/code/coverage/
.. _Bitbucket: http://bitbucket.org/brodie/cram
Tip: Filter by directory path e.g. /media app.js to search for public/media/app.js.
Tip: Use camelCasing e.g. ProjME to search for ProjectModifiedEvent.java.
Tip: Filter by extension type e.g. /repo .js to search for all .js files in the /repo directory.
Tip: Separate your search with spaces e.g. /ssh pom.xml to search for src/ssh/pom.xml.
Tip: Use ↑ and ↓ arrow keys to navigate and return to view the file.
Tip: You can also navigate files with Ctrl+j (next) and Ctrl+k (previous) and view the file with Ctrl+o.
Tip: You can also navigate files with Alt+j (next) and Alt+k (previous) and view the file with Alt+o.