== claft: Command Line and Filter Test Tool == === Introduction === claft is a simple tool for testing command line utilities and UNIX-style filters. Given a simple configuration file, claft will run programs, with various different arguments, confirm that specific patterns (regular expressions) appear in the output or error stream (stderr), and that the program returned the appropriate exit value (return code). === Example: hello === For example if we had a program named //hello// that was intended to be called with no arguments, print the string: "Hello, World!" and exit with a zero exit value, then the claft configuration to test its basic functionalty would be: [Basic Functionality] Cmd: ./hello Output: ^Hello, World!$ Return: 0 If we wanted to add a test to ensure that our target properly reported an error message and return an exit value (such as 1) in the case were //hello// was called with any arguments we might use a configuration such as: [DEFAULT] Cmd: ./hello Return: 0 [Basic Functionality] Output: ^Hello, World!$ [Error Handling] Args: foo Errormsg: .*hello takes no command line arguments nor switches. Return: 1 We introduce a DEFAULT section (that section name is reserved) so that we don't have to repeatedly specific the command, return code or other details. Then we define our [Basic Functionality] and [Error Handling] tests. We can add as many sections as we like. Each can specify a different command, different arguments, inputs strings, output and errormsg patterns, and return codes. Note: an **"Args:"** directive will be parsed by Python's shlex.split() using "Posix" mode, and stripping off trailing comments (starting with the first unquoted/unescapted # character on the line). **"Inputs:"** directives can be evaluated as Python literal strings. The interpreter evaluation is only tried if the lines contain any \ (backslash) characters, which allows us to embed Python string escape literals such as \n, \t, etc (for newline, tab, and so on) as well as \0x and \u sequences to represent arbitrary ASCII and Unicode characters. Any failed evaluation simply leaves the string as is (so non-parseable sequences of \ characters are still fed to the target program exactly as they appear in our configuration file). **"Output:"** and **"Errormsg:"** directives are evaluated as Python regular expressions, and are evaluated as "raw" string literals if they contain any \ (backslash) characters. Thus you should be able to type these regular expressions in your //claft// configuration files as naturallly as you would in a r"..." literal in your Python code. In all configuration directives the values can be continued on multiple lines; as it normal for ConfigParser .ini-style files. Leading white space will be removed so any inputs or patterns that require them should have them explicitly encoded using \ sequences such as \0x20 for space, \t for tabs and so on. Trailing space should also be explicitly encoded. The //claft// command is called with a list of configuration files as arguments and will iterate over each performing each test in each as it goes. Thus a list of a dozen configuration files each specifying ten tests would run 120 separate subprocesses. Often you would have only one target command per configuration file, defined in the [DEFAULT] section. However, //claft// imposes no such restriction. === Example: avg === If we had a program named //avg// to compute simple arithmetic means (averages) and using a "0" (zero) input as the terminating sentinel, then here could be one test suite for it: [DEFAULT] Cmd: ./avg Return: 0 [Basic Functionality] Inputs: 1.0 2.0 3 0 Output: .* 2\.0000.* [No sentinel] Inputs: 1 2 Errormsg: Warning: Unexpected end of input! Output: .* 1\.50000.* Return: 1 [Invalid input] Inputs: 1 foo Errormsg: Error: Unable to parse 'foo' as a number Return: 127 We specify the output as a regular expression to ignore insignificant floating point deviations, and we have a second test to confirm that a warning message is output on //stderr// and a non-zero exit value result from the lack of terinating sentinel, as specified by our program's specification. In this case test specifies that the computation is to be performed, and the results printed on //stdout// even if the input wasn't properly terminated. In the last test we confirm the error handling completly invalid input. This test suite would run the //avg// program in the current directory three times and tally up any test failures to report to us. === Support === The canonical //claft// website is at: http://bitbucket.org/jimd/claft/ Any new version can be found there. This is free software. There are no end user technical support services. No helpdesk, phone number, mailing list, nor online "support" forums. A wiki is provided at: http://bitbucket.org/jimd/claft/wiki/Home This is the primarily place to search for additional information and documentation about //claft//. Naturally (as withy any wiki) you're encouraged to add your own relevant tips, tricks and suggestions there. This software is intended for programmers and, especially, for those learning how to program. As such you're encouraged to read through the source code and figure out how things are working (or not) on your own. When you find bugs, you're encouraged isolate them and, if possible fix them. Bugs can be reported at: http://bitbucket.org/jimd/claft/issues/ Please search through existing issues before reporting new ones. If possible include a test case which demonstrates the bug you're reporting, and a patch (Mecurial preferred, but any unidiff patch will be considered). The author's e-mail address <email@example.com> should only be used for legal or administrative issues. All other relevant communications should be via the web site, wiki, and tracking system.