Source

cram / README.txt

Full commit
Brodie Rao d457c53 



Brodie Rao 2fb0798 

Brodie Rao c7debac 
Brodie Rao cea60f3 




Brodie Rao d457c53 
Brodie Rao 3a2650f 

Brodie Rao d457c53 
Brodie Rao 1e01b78 
Brodie Rao d457c53 

Brodie Rao 3a2650f 
Brodie Rao d457c53 
Brodie Rao 7bf5fc6 
Brodie Rao d457c53 
Brodie Rao c70d79c 
Brodie Rao d457c53 



Brodie Rao c70d79c 
Brodie Rao d457c53 
Brodie Rao 1e01b78 
Brodie Rao 3a2650f 










Brodie Rao d457c53 










myfreeweb 0983fe9 


Brodie Rao d457c53 


Brodie Rao 673deec 
Brodie Rao c70d79c 

Brodie Rao 673deec 

Brodie Rao ceeaaf8 

Brodie Rao c70d79c 


Brodie Rao d457c53 
Brodie Rao e24a395 
Brodie Rao 582832f 

Brodie Rao 4f81195 

Brodie Rao e24a395 

Brodie Rao d457c53 

Brodie Rao c7debac 

Brodie Rao d457c53 
Brodie Rao c70d79c 
Brodie Rao d457c53 

Brodie Rao f56c8d3 


Brodie Rao 112e96e 
Brodie Rao f56c8d3 
Brodie Rao 112e96e 
Brodie Rao f56c8d3 





Brodie Rao ecd937f 
Brodie Rao f56c8d3 


Brodie Rao 112e96e 


Brodie Rao ecd937f 
Brodie Rao f56c8d3 



Brodie Rao d457c53 




Brodie Rao 4d2381f 

Brodie Rao d457c53 
Brodie Rao cea60f3 
Brodie Rao d457c53 
Brodie Rao 4d2381f 



Brodie Rao d457c53 










Brodie Rao c70d79c 
Brodie Rao d457c53 
Brodie Rao 4d2381f 













Brodie Rao d457c53 

Brodie Rao 582832f 





Brodie Rao d457c53 
Brodie Rao d39185c 












Brodie Rao 312ae79 















Brodie Rao 7bf5fc6 
Brodie Rao 312ae79 
Brodie Rao 7bf5fc6 
Brodie Rao 312ae79 
Brodie Rao d457c53 

Brodie Rao 936da7f 
Brodie Rao 582832f 
Brodie Rao 312ae79 


Brodie Rao a905a88 




Brodie Rao 32e2e52 

Brodie Rao a905a88 




Brodie Rao e24a395 


Brodie Rao 4f81195 

Brodie Rao e24a395 























Brodie Rao 4d2381f 

Brodie Rao c70d79c 
Brodie Rao ceeaaf8 
Brodie Rao c70d79c 


Brodie Rao 18e383b 
Brodie Rao ceeaaf8 
Brodie Rao 18e383b 
Brodie Rao c70d79c 
Brodie Rao cea60f3 
Brodie Rao c70d79c 

Brodie Rao 7bf5fc6 




Brodie Rao 7c64dd3 
Brodie Rao 812eb90 


Brodie Rao ac291a4 
Brodie Rao 4d2381f 
Brodie Rao 56360e5 
Brodie Rao 37c124f 

Brodie Rao 4d2381f 
Brodie Rao e349ae7 


Brodie Rao a313a2e 


Brodie Rao ac291a4 
Brodie Rao 620f57a 
Brodie Rao 4d2381f 

Brodie Rao a313a2e 
Brodie Rao 4d2381f 
Brodie Rao a313a2e 
Brodie Rao 620f57a 
Brodie Rao afc75a8 
Brodie Rao eeb73a8 
Brodie Rao 312ae79 
















Brodie Rao afc75a8 
Brodie Rao eeb73a8 
Brodie Rao 312ae79 


Brodie Rao afc75a8 
Brodie Rao eeb73a8 
Brodie Rao 312ae79 


Brodie Rao 936da7f 



















======================
 Cram: It's test time
======================

Cram is a functional testing framework for command line applications
based on Mercurial_'s `unified test format`_.

Cram tests look like snippets of interactive shell sessions. Cram runs
each command and compares the command output in the test with the
command's actual output.

Here's a snippet from `Cram's own test suite`_::

    The $PYTHON environment variable should be set when running this test
    from Python.

      $ [ -n "$PYTHON" ] || PYTHON="`which python`"
      $ if [ -n "$COVERAGE" ]; then
      >   coverage erase
      >   alias cram="`which coverage` run --branch -a $TESTDIR/../cram.py"
      > else
      >   alias cram="$PYTHON $TESTDIR/../cram.py"
      > fi
      $ command -v md5 > /dev/null || alias md5=md5sum

    Usage:

      $ cram -h
      [Uu]sage: cram \[OPTIONS\] TESTS\.\.\. (re)

      [Oo]ptions: (re)
        -h, --help          show this help message and exit
        -V, --version       show version information and exit
        -q, --quiet         don't print diffs
        -v, --verbose       show filenames and test status
        -i, --interactive   interactively merge changed test output
        -y, --yes           answer yes to all questions
        -n, --no            answer no to all questions
        -E, --preserve-env  don't reset common environment variables
        --keep-tmpdir       keep temporary directories
        --shell=PATH        shell to use for running tests
        --indent=NUM        number of spaces to use for indentation

The format in a nutshell:

* Cram tests use the ``.t`` file extension.

* Lines beginning with two spaces, a dollar sign, and a space are run
  in the shell.

* Lines beginning with two spaces, a greater than sign, and a space
  allow multi-line commands.

* Lines beginning with two spaces, an ampersand, and a space start
  background processes that will be killed at the end of the test.

* All other lines beginning with two spaces are considered command
  output.

* Output lines ending with a space and the keyword ``(re)`` are
  matched as `Perl-compatible regular expressions`_.

* Lines ending with a space and the keyword ``(glob)`` are matched
  with a glob-like syntax. The only special characters supported are
  ``*`` and ``?``. Both characters can be escaped using ``\``, and the
  backslash can be escaped itself.

* Output lines ending with either of the above keywords are always
  first matched literally with actual command output.

* Lines ending with a space and the keyword ``(no-eol)`` will match
  actual output that doesn't end in a newline.

* Actual output lines containing unprintable characters are escaped
  and suffixed with a space and the keyword ``(esc)``. Lines matching
  unprintable output must also contain the keyword.

* Anything else is a comment.

.. _Mercurial: http://mercurial.selenic.com/
.. _unified test format: http://www.selenic.com/blog/?p=663
.. _Cram's own test suite: http://bitbucket.org/brodie/cram/src/tip/tests/cram.t
.. _Perl-compatible regular expressions: http://en.wikipedia.org/wiki/Perl_Compatible_Regular_Expressions


Download
--------

* cram-0.5.tar.gz_ (22 KB, requires Python 2.4-2.7 or Python 3.1-3.2)

.. _cram-0.5.tar.gz: http://bitheap.org/cram/cram-0.5.tar.gz

Installation
------------

You can use pip_ to install Cram::

    $ pip install cram

Or you can install Cram the old fashioned way::

    $ wget http://bitheap.org/cram/cram-0.5.tar.gz
    $ tar zxvf cram-0.5.tar.gz
    $ cd cram-0.5.tar.gz
    $ python setup.py install

.. _pip: http://pypi.python.org/pypi/pip


Usage
-----

Cram will print a dot for each passing test. If a test fails, a
`unified context diff`_ is printed showing the test's expected output
and the actual output. Skipped tests (empty tests and tests that exit
with return code ``80``) are marked with ``s`` instead of a dot.

For example, if we run Cram on `its own example tests`_::

    .s.!
    --- /home/brodie/src/cram/examples/fail.t
    +++ /home/brodie/src/cram/examples/fail.t.err
    @@ -3,21 +3,22 @@
       $ echo 1
       1
       $ echo 1
    -  2
    +  1
       $ echo 1
       1

     Invalid regex:

       $ echo 1
    -  +++ (re)
    +  1

     Offset regular expression:

       $ printf 'foo\nbar\nbaz\n\n1\nA\n@\n'
       foo
    +  bar
       baz

       \d (re)
       [A-Z] (re)
    -  #
    +  @
    s.
    # Ran 6 tests, 2 skipped, 1 failed.

Cram will also write the test with its actual output to
``examples/fail.t.err``.

When you're first writing a test, you might just write the commands
and run the test to see what happens. If you run Cram with ``-i`` or
``--interactive``, you'll be prompted to merge the actual output back
into the test. This makes it easy to quickly prototype new tests.

You can specify a default set of options by creating a ``.cramrc``
file. For example::

    [cram]
    verbose = True
    indent = 4

Is the same as invoking Cram with ``--verbose`` and ``--indent=4``.

To change what configuration file Cram loads, you can set the
``CRAMRC`` environment variable. You can also specify command line
options in the ``CRAM`` environment variable.

Note that the following environment variables are reset before tests
are run:

* ``TMPDIR``, ``TEMP``, and ``TMP`` are set to the test runner's
  ``tmp`` directory.

* ``LANG``, ``LC_ALL``, and ``LANGUAGE`` are set to ``C``.

* ``TZ`` is set to ``GMT``.

* ``COLUMNS`` is set to ``80``.

* ``CDPATH`` and ``GREP_OPTIONS`` are set to an empty string.

Cram also provides the following environment variables to tests:

* ``CRAMTMP``, set to the test runner's temporary directory.

* ``TESTDIR``, set to the directory containing the test file.

.. _unified context diff: http://en.wikipedia.org/wiki/Diff#Unified_format
.. _its own example tests: http://bitbucket.org/brodie/cram/src/tip/examples/


News
----

Version 0.6
```````````
* Added support for specifying options in ``.cramrc`` (configurable
  with the ``CRAMRC`` environment variable).

* Added a ``--shell`` option to change the shell tests are run
  with. Contributed by `Kamil Kisiel`_.

* Added the long option ``--preserve-env`` for ``-E``.

.. _Kamil Kisiel: http://kamilkisiel.net/

Version 0.5 (Jan. 8, 2011)
``````````````````````````
* **The test format has changed:** Matching output not ending in a
  newline now requires the ``(no-eol)`` keyword instead of ending the
  line in ``%``.

* Matching output containing unprintable characters now requires the
  ``(esc)`` keyword. Real output containing unprintable characters
  will automatically receive ``(esc)``.

* If an expected line matches its real output line exactly, special
  matching like ``(re)`` or ``(glob)`` will be ignored.

* Regular expressions ending in a trailing backslash are now
  considered invalid.

* Added an ``--indent`` option for changing the default amount of
  indentation required to specify commands and output.

* Added support for specifying command line options in the ``CRAM``
  environment variable.

* The ``--quiet`` and ``--verbose`` options can now be used together.

* When running Cram under Python 3, Unicode-specific line break
  characters will no longer be parsed as newlines.

* Tests are no longer required to end in a trailing newline.

Version 0.4 (Sep. 28, 2010)
```````````````````````````
* **The test format has changed:** Output lines containing regular
  expressions must now end in ``(re)`` or they'll be matched
  literally. Lines ending with keywords are matched literally first,
  however.

* Regular expressions are now matched from beginning to end. In other
  words ``\d (re)`` is matched as ``^\d$``.

* In addition to ``(re)``, ``(glob)`` has been added. It supports
  ``*``, ``?``, and escaping both characters (and backslashes) using
  ``\``.

* **Environment settings have changed:** The ``-D`` flag has been
  removed, ``$TESTDIR`` is now set to the directory containing the
  ``.t`` file, and ``$CRAMTMP`` is set to the test runner's temporary
  directory.

* ``-i``/``--interactive`` now requires ``patch(1)``. Instead of
  ``.err`` files replacing ``.t`` files during merges, diffs are
  applied using ``patch(1)``. This prevents matching regular
  expressions and globs from getting clobbered.

* Previous ``.err`` files are now removed when tests pass.

* Cram now exits with return code ``1`` if any tests failed.

* If a test exits with return code ``80``, it's considered a skipped a
  test. This is useful for intentionally disabling tests when they
  only work on certain platforms or in certain settings.

* The number of tests, the number of skipped tests, and the number of
  failed tests are now printed after all tests are finished.

* Added ``-q``/``--quiet`` to suppress diff output.

* Added `contrib/cram.vim`_ syntax file for Vim. Contributed by `Steve
  Losh`_.

.. _contrib/cram.vim: http://bitbucket.org/brodie/cram/src/tip/contrib/cram.vim
.. _Steve Losh: http://stevelosh.com/

Version 0.3 (Sep. 20, 2010)
```````````````````````````
* Implemented resetting of common environment variables. This behavior
  can be disabled using the ``-E`` flag.

* Changed the test runner to first make its own overall random
  temporary directory, make ``tmp`` inside of it and set ``TMPDIR``,
  etc. to its path, and run each test with a random temporary working
  directory inside of that.

* Added ``--keep-tmpdir``. Temporary directories are named by test
  filename (along with a random string).

* Added ``-i``/``--interactive`` to merge actual output back to into
  tests interactively.

* Added ability to match command output not ending in a newline by
  suffixing output in the test with ``%``.

Version 0.2 (Sep. 19, 2010)
```````````````````````````
* Changed the test runner to run tests with a random temporary working
  directory.

Version 0.1 (Sep. 19, 2010)
```````````````````````````
* Initial release.


Development
-----------

Download the official development repository using Mercurial_::

    hg clone http://bitbucket.org/brodie/cram

Test Cram using Cram::

    make tests

Get a test coverage report using coverage.py_::

    make coverage

Visit Bitbucket_ if you'd like to fork the project, watch for new
changes, or report issues.

.. _Mercurial: http://mercurial.selenic.com/
.. _coverage.py: http://nedbatchelder.com/code/coverage/
.. _Bitbucket: http://bitbucket.org/brodie/cram