Formal specification for unit testing

This unit test framework supports test automation, sharing of setup and teardown code between tests and aggregation of tests in collections.

With this framework, it is easy to create a unit-testing program which can run all or a subset of available test suites and reports the outcome of the tests with its exit status.

Key concepts

Test case. A test case is the smallest unit of testing. It usually compares an expected behaviour with the actual behaviour for a particular set of inputs.

Test outcome. A test outcome represents the result of a test case, i.e. if the actual behaviour matches the expected behaviour. A test case is failed if its actual behaviour disagrees with the expected behaviour and if it is not marked as an expected failure.

Test fixture. A test fixture represents the preparation needed to perform one or more test cases, and any associated cleanup actions. This may involve, for example, creating temporary directories, logging into a database, or starting a server process.

Test suite. A test suite is a collection of test cases, test suites, or both. It is used to aggregate tests that should be executed together, for instance, the tests for a given component.

Test supervisor. A test supervisor is a component which orchestrates the execution of tests and provides the outcome to the user. The supervisor may use a graphical interface, a textual interface, or return a special value to indicate the results of executing the tests.

User stories

#21 Identify source of failed test case

As a developer, I want to quickly identify the source code of a test case that failed.

Therefore, each test case should be labeled with a unique identifier.

When a series of test cases is automatically generated, the generated identifier should be enough to find the parameters of the test.

#22 Compare obtained value with expected value

As a developer, I want to see the unexpected result that led a test into failure.

Therefore, a test case comparing two values should take a formatting routine as argument to prepare a line in the log file.

#23 Mark expected failures in test cases

As a developer, I want to mark some test cases as expected failure.

A test case which is expected to fail will not invalidate the test if it fails. A separate report for tests expected to fail is maintained.

Therefore each test case should have an attribute “expected to fail”.

#24 List test cases marked as expected failures

As a developer, I want to have an overview of the set of tests which are expected to fail.

Therefore the application should have a special flag to print the names of all tests expected to fail.

#25 Exit code for undefined dependencies of test suites

As a developer, I want to distinguish between a failure of the test suite caused by a failed test-case or by an ill-formed test suite.

If some dependency of a test suite is not defined as it is registered, the program fails with EX_SOFTWARE.

#26 Exit code for failed fixture of test suites

As a developer, I want to distinguish between a failure of the test suite caused by a failed test or by an failed fixture.

If some fixture of a test suite or a test case fails, then the test suite is interrupted and the program fails with EX_UNAVAILABLE.

Formal specification

Bootstrapping code of the application initialises various test cases and test suites, then calls UnitTest.main which implements the application logic.

According to the options selected on the command line, the application will:

1. either perform one or more test suite, print an a posteriori report about results and terminate with exit code 0 (success) if all test cases pass or terminate with exit code 1 (failure) otherwise;

2. or print an a priori report and terminate with exit code 0 (success);

3. or complain that the command line is not well formed, print a short usage note and terminate with exit code 64.

Usage

Usage: unit-testing [-h | -l | suite1 [suite2 [...]]]
 Run unitary tests
Options:
 -h Display a cheerful help message.
 -l List available test suites.
 -x List all test cases marked as expected failures.
Exit Status:
 The unit-testing program exits 0 on success and 1 if a test case failed.

A priori reports

The help screen was described above. This is validated by test-help-screen.

The list of all test suites displays the list of all test suites which are not embedded into other test suites. This is validated by test--suites-list.

The list of all test cases marked as expected failures displays the list of all test cases which are marked as expected failures. This is validated by test-expected-failures-list.

Performing test suites

A test suite is a mixed sequence of test cases and other test suites. Performing a test suite involves its underlying test cases and test suites. The outcome of the test suite depends of the outcome of its underlying elements.

Test case outcomes The outcome of a test case can be represented as follows:

type outcome =
  | Success
  | Failure
  | Not_implemented
  | Skipped
  | Exception of exn

A succesful outcome is represented by Success or Skipped. Other outcomes represent a failure. A test case is passed if its outcome is succesful or if the outcome is a failure but the test case is marked as an expected failure.

Defining a test case A test case is defined by an identifier and some constant predicate.

Performing a test case A test case is perfomed by evaluating its constant predicate in a supervised environment. The predicate might issue diagnostic messages on the standard output of the process, these messages are gathered into the logbook of the unit-testing program. The outcome of the test case is then transformed into a value of type outcome and remembered, so that a given test case will only run once.

Defining a test suite A test suite is defined by an identifier and the items it contains, which are test cases or other test suites. Each item can be decorated with a fixture. A suite can have dependancies. In this case, it will not run if its dependencies did not succeed.

If some dependency of a test suite is not defined as it is registered, the program fails with EX_SOFTWARE.

Some test suites are primary test suites, these are the test suites which are not included in some other test suite.

Performing a test suite If some fixture of a test suite or a test case fails, then the test suite is interrupted and the program fails with EX_UNAVAILABLE.

Each test suite and test case contained in the performed test suite is run and its outcome is compared to the expected outcome. The test case is failed if the outcomes disagree and the test case is not marked as an expected failure. If the outcomes disagreee and the test case is marked as an expected failure, this is reported adequately in the log file.