====================== Cram: It's test time ====================== Cram is a functional testing framework for command line applications based on Mercurial_'s `unified test format`_. Cram tests look like snippets of interactive shell sessions. Cram runs each command and compares the command output in the test with the command's actual output. Here's a snippet from `Cram's own test suite`_:: The $PYTHON environment variable should be set when running this test from Python. $ [ -n "$PYTHON" ] || PYTHON="`which python`" $ if [ -n "$COVERAGE" ]; then > coverage erase > alias cram="`which coverage` run --branch -a $TESTDIR/../cram.py" > else > alias cram="$PYTHON $TESTDIR/../cram.py" > fi $ command -v md5 > /dev/null || alias md5=md5sum Usage: $ cram -h [Uu]sage: cram \[OPTIONS\] TESTS\.\.\. (re) [Oo]ptions: (re) -h, --help show this help message and exit -V, --version show version information and exit -q, --quiet don't print diffs -v, --verbose show filenames and test status -i, --interactive interactively merge changed test output -y, --yes answer yes to all questions -n, --no answer no to all questions -E, --preserve-env don't reset common environment variables --keep-tmpdir keep temporary directories --shell=PATH shell to use for running tests --indent=NUM number of spaces to use for indentation The format in a nutshell: * Cram tests use the ``.t`` file extension. * Lines beginning with two spaces, a dollar sign, and a space are run in the shell. * Lines beginning with two spaces, a greater than sign, and a space allow multi-line commands. * All other lines beginning with two spaces are considered command output. * Output lines ending with a space and the keyword ``(re)`` are matched as `Perl-compatible regular expressions`_. * Lines ending with a space and the keyword ``(glob)`` are matched with a glob-like syntax. The only special characters supported are ``*`` and ``?``. Both characters can be escaped using ``\``, and the backslash can be escaped itself. * Output lines ending with either of the above keywords are always first matched literally with actual command output. * Lines ending with a space and the keyword ``(no-eol)`` will match actual output that doesn't end in a newline. * Actual output lines containing unprintable characters are escaped and suffixed with a space and the keyword ``(esc)``. Lines matching unprintable output must also contain the keyword. * Anything else is a comment. .. _Mercurial: http://mercurial.selenic.com/ .. _unified test format: http://www.selenic.com/blog/?p=663 .. _Cram's own test suite: http://bitbucket.org/brodie/cram/src/tip/tests/cram.t .. _Perl-compatible regular expressions: http://en.wikipedia.org/wiki/Perl_Compatible_Regular_Expressions Download -------- * cram-0.5.tar.gz_ (22 KB, requires Python 2.4-2.7 or Python 3.1-3.2) .. _cram-0.5.tar.gz: http://bitheap.org/cram/cram-0.5.tar.gz Installation ------------ You can use pip_ to install Cram:: $ pip install cram Or you can install Cram the old fashioned way:: $ wget http://bitheap.org/cram/cram-0.5.tar.gz $ tar zxvf cram-0.5.tar.gz $ cd cram-0.5.tar.gz $ python setup.py install .. _pip: http://pypi.python.org/pypi/pip Usage ----- Cram will print a dot for each passing test. If a test fails, a `unified context diff`_ is printed showing the test's expected output and the actual output. Skipped tests (empty tests and tests that exit with return code ``80``) are marked with ``s`` instead of a dot. For example, if we run Cram on `its own example tests`_:: .s.! --- /home/brodie/src/cram/examples/fail.t +++ /home/brodie/src/cram/examples/fail.t.err @@ -3,21 +3,22 @@ $ echo 1 1 $ echo 1 - 2 + 1 $ echo 1 1 Invalid regex: $ echo 1 - +++ (re) + 1 Offset regular expression: $ printf 'foo\nbar\nbaz\n\n1\nA\n@\n' foo + bar baz \d (re) [A-Z] (re) - # + @ s. # Ran 6 tests, 2 skipped, 1 failed. Cram will also write the test with its actual output to ``examples/fail.t.err``. When you're first writing a test, you might just write the commands and run the test to see what happens. If you run Cram with ``-i`` or ``--interactive``, you'll be prompted to merge the actual output back into the test. This makes it easy to quickly prototype new tests. You can specify a default set of options by creating a ``.cramrc`` file. For example:: [cram] verbose = True indent = 4 Is the same as invoking Cram with ``--verbose`` and ``--indent=4``. To change what configuration file Cram loads, you can set the ``CRAMRC`` environment variable. You can also specify command line options in the ``CRAM`` environment variable. Note that the following environment variables are reset before tests are run: * ``TMPDIR``, ``TEMP``, and ``TMP`` are set to the test runner's ``tmp`` directory. * ``LANG``, ``LC_ALL``, and ``LANGUAGE`` are set to ``C``. * ``TZ`` is set to ``GMT``. * ``COLUMNS`` is set to ``80``. * ``CDPATH`` and ``GREP_OPTIONS`` are set to an empty string. Cram also provides the following environment variables to tests: * ``CRAMTMP``, set to the test runner's temporary directory. * ``TESTDIR``, set to the directory containing the test file. .. _unified context diff: http://en.wikipedia.org/wiki/Diff#Unified_format .. _its own example tests: http://bitbucket.org/brodie/cram/src/tip/examples/ News ---- Version 0.6 ``````````` * Added support for specifying options in ``.cramrc`` (configurable with the ``CRAMRC`` environment variable). * Added a ``--shell`` option to change the shell tests are run with. Contributed by `Kamil Kisiel`_. * Added the long option ``--preserve-env`` for ``-E``. .. _Kamil Kisiel: http://kamilkisiel.net/ Version 0.5 (Jan. 8, 2011) `````````````````````````` * **The test format has changed:** Matching output not ending in a newline now requires the ``(no-eol)`` keyword instead of ending the line in ``%``. * Matching output containing unprintable characters now requires the ``(esc)`` keyword. Real output containing unprintable characters will automatically receive ``(esc)``. * If an expected line matches its real output line exactly, special matching like ``(re)`` or ``(glob)`` will be ignored. * Regular expressions ending in a trailing backslash are now considered invalid. * Added an ``--indent`` option for changing the default amount of indentation required to specify commands and output. * Added support for specifying command line options in the ``CRAM`` environment variable. * The ``--quiet`` and ``--verbose`` options can now be used together. * When running Cram under Python 3, Unicode-specific line break characters will no longer be parsed as newlines. * Tests are no longer required to end in a trailing newline. Version 0.4 (Sep. 28, 2010) ``````````````````````````` * **The test format has changed:** Output lines containing regular expressions must now end in ``(re)`` or they'll be matched literally. Lines ending with keywords are matched literally first, however. * Regular expressions are now matched from beginning to end. In other words ``\d (re)`` is matched as ``^\d$``. * In addition to ``(re)``, ``(glob)`` has been added. It supports ``*``, ``?``, and escaping both characters (and backslashes) using ``\``. * **Environment settings have changed:** The ``-D`` flag has been removed, ``$TESTDIR`` is now set to the directory containing the ``.t`` file, and ``$CRAMTMP`` is set to the test runner's temporary directory. * ``-i``/``--interactive`` now requires ``patch(1)``. Instead of ``.err`` files replacing ``.t`` files during merges, diffs are applied using ``patch(1)``. This prevents matching regular expressions and globs from getting clobbered. * Previous ``.err`` files are now removed when tests pass. * Cram now exits with return code ``1`` if any tests failed. * If a test exits with return code ``80``, it's considered a skipped a test. This is useful for intentionally disabling tests when they only work on certain platforms or in certain settings. * The number of tests, the number of skipped tests, and the number of failed tests are now printed after all tests are finished. * Added ``-q``/``--quiet`` to suppress diff output. * Added `contrib/cram.vim`_ syntax file for Vim. Contributed by `Steve Losh`_. .. _contrib/cram.vim: http://bitbucket.org/brodie/cram/src/tip/contrib/cram.vim .. _Steve Losh: http://stevelosh.com/ Version 0.3 (Sep. 20, 2010) ``````````````````````````` * Implemented resetting of common environment variables. This behavior can be disabled using the ``-E`` flag. * Changed the test runner to first make its own overall random temporary directory, make ``tmp`` inside of it and set ``TMPDIR``, etc. to its path, and run each test with a random temporary working directory inside of that. * Added ``--keep-tmpdir``. Temporary directories are named by test filename (along with a random string). * Added ``-i``/``--interactive`` to merge actual output back to into tests interactively. * Added ability to match command output not ending in a newline by suffixing output in the test with ``%``. Version 0.2 (Sep. 19, 2010) ``````````````````````````` * Changed the test runner to run tests with a random temporary working directory. Version 0.1 (Sep. 19, 2010) ``````````````````````````` * Initial release. Development ----------- Download the official development repository using Mercurial_:: hg clone http://bitbucket.org/brodie/cram Test Cram using Cram:: make tests Get a test coverage report using coverage.py_:: make coverage Visit Bitbucket_ if you'd like to fork the project, watch for new changes, or report issues. .. _Mercurial: http://mercurial.selenic.com/ .. _coverage.py: http://nedbatchelder.com/code/coverage/ .. _Bitbucket: http://bitbucket.org/brodie/cram