1. Jim Dennis
  2. claft

Overview

HTTPS SSH
== claft: Command Line and Filter Test Tool ==

=== Introduction ===

claft is a simple tool for testing command line utilities and UNIX-style
filters.

Given a simple configuration file, claft will run programs, with various
different arguments, confirm that specific patterns (regular expressions)
appear in the output or error stream (stderr), and that the program returned
the appropriate exit value (return code).

=== Example: hello ===

For example if we had a program named //hello// that was intended to be
called with no arguments, print the string: "Hello, World!" and exit with
a zero exit value, then the claft configuration to test its basic functionalty
would be:

	[Basic Functionality]
	Cmd: ./hello
	Output: ^Hello, World!$
	Return: 0

If we wanted to add a test to ensure that our target properly reported an
error message and return an exit value (such as 1) in the case were //hello//
was called with any arguments we might use a configuration such as:


	[DEFAULT]
	Cmd: ./hello
	Return: 0

	[Basic Functionality]
	Output: ^Hello, World!$

	[Error Handling]
	Args: foo
	Errormsg: .*hello takes no command line arguments nor switches.
	Return: 1

We introduce a DEFAULT section (that section name is reserved) so that
we don't have to repeatedly specific the command, return code or other
details.  Then we define our [Basic Functionality] and [Error Handling]
tests.

We can add as many sections as we like.  Each can specify a different command,
different arguments, inputs strings, output and errormsg patterns, and return
codes.

Note: an **"Args:"** directive will be parsed by Python's shlex.split() using
"Posix" mode, and stripping off trailing comments (starting with the first
unquoted/unescapted # character on the line). **"Inputs:"** directives can
be evaluated as Python literal strings.  The interpreter evaluation is only
tried if the lines contain any \ (backslash) characters, which allows us to
embed Python string escape literals such as \n, \t, etc (for newline, tab,
and so on) as well as \0x and \u sequences to represent arbitrary ASCII and
Unicode characters.  Any failed evaluation simply leaves the string as is
(so non-parseable sequences of \ characters are still fed to the target
program exactly as they appear in our configuration file).  **"Output:"** and
**"Errormsg:"** directives are evaluated as Python regular expressions, and
are evaluated as "raw" string literals if they contain any \ (backslash)
characters.  Thus you should be able to type these regular expressions in
your //claft// configuration files as naturallly as you would in a r"..."
literal in your Python code.  In all configuration directives the values
can be continued on multiple lines; as it normal for ConfigParser .ini-style
files. Leading white space will be removed so any inputs or patterns that
require them should have them explicitly encoded using \ sequences such as
\0x20 for space, \t for tabs and so on.  Trailing space should also be
explicitly encoded.

The //claft// command is called with a list of configuration files as
arguments and will iterate over each performing each test in each as it goes.
Thus a list of a dozen configuration files each specifying ten tests would
run 120 separate subprocesses.  Often you would have only one target command
per configuration file, defined in the [DEFAULT] section. However, //claft//
imposes no such restriction.


=== Example: avg ===

If we had a program named //avg// to compute simple arithmetic means
(averages) and using a "0" (zero) input as the terminating sentinel, then
here could be one test suite for it:

        [DEFAULT]
	Cmd: ./avg
	Return: 0

	[Basic Functionality]
        Inputs: 1.0
                2.0
                3
                0
	Output: .* 2\.0000.*

        [No sentinel]
        Inputs: 1
                2
        Errormsg: Warning: Unexpected end of input!
        Output: .* 1\.50000.*
        Return: 1

        [Invalid input]
        Inputs: 1
                foo
        Errormsg: Error: Unable to parse 'foo' as a number
        Return: 127

We specify the output as a regular expression to ignore insignificant
floating point deviations, and we have a second test to confirm that a
warning message is output on //stderr// and a non-zero exit value result
from the lack of terinating sentinel, as specified by our program's
specification.  In this case test specifies that the computation is to be
performed, and the results printed on //stdout// even if the input wasn't
properly terminated. In the last test we confirm the error handling completly
invalid input.  This test suite would run the //avg// program in the current
directory three times and tally up any test failures to report to us.

=== Support ===


The canonical //claft// website is at:

        http://bitbucket.org/jimd/claft/

Any new version can be found there.

This is free software.  There are no end user technical support services.
No helpdesk, phone number, mailing list, nor online "support" forums.

A wiki is provided at:

        http://bitbucket.org/jimd/claft/wiki/Home

This is the primarily place to search for additional information and
documentation about //claft//.  Naturally (as withy any wiki) you're
encouraged to add your own relevant tips, tricks and suggestions there.

This software is intended for programmers and, especially, for those learning
how to program.  As such you're encouraged to read through the source code
and figure out how things are working (or not) on your own.

When you find bugs, you're encouraged isolate them and, if possible fix them.
Bugs can be reported at:

        http://bitbucket.org/jimd/claft/issues/

Please search through existing issues before reporting new ones.  If possible
include a test case which demonstrates the bug you're reporting, and a
patch (Mecurial preferred, but any unidiff patch will be considered).

The author's e-mail address <jimd@starshine.org> should only be used for
legal or administrative issues.  All other relevant communications should be
via the web site, wiki, and tracking system.