# SCons Testing Framework

SCons uses extensive automated tests to try to ensure quality. The primary goal is that users should be able to upgrade from version to version without any surprise changes in behavior.

In general, no change goes into SCons unless it has one or more new or modified tests that demonstrably exercise the bug being fixed or the feature being added. There are exceptions to this guideline, but they should be just that, ''exceptions''. When in doubt, make sure it's tested.

## Test Organization

There are three types of SCons tests:

End-to-End Tests
End-to-end tests of SCons are all Python scripts (*.py) underneath the test/ subdirectory. They use the test infrastructure modules in the QMTest subdirectory.
Unit Tests
Unit tests for individual SCons modules live underneath the src/engine/ subdirectory and are the same base name as the module with Tests.py appended--for example, the unit tests for the Builder.py module are in the BuilderTests.py script.
External Tests
For the support of external Tools (in the form of packages, preferably), the testing framework got extended, such that it can run in standalone mode. You can start it from the top-level folder of your Tool's source tree, where it then finds all Python scripts (*.py) underneath the local test/ directory. This implies that Tool tests have to be kept in a folder named test, like for the SCons core.

### Contrasting End-to-End and Unit Tests

In general, anything that we've put into an end-to-end test script should be considered a hardened part of the interface (that is, it's something that a user might do) and should not be broken. Unit tests are now considered more malleable, more for testing internal interfaces that can change so long as we don't break users' SConscript files. (This wasn't always the case, and there's a lot of meaty code in many of the unit test scripts that does, in fact, capture external interface behavior. In general, we should try to move those things to end-to-end scripts as we find them.)

It's more difficult to debug end-to-end tests. You can actually go straight into the Python debugger on the unit test scripts by using the runtest.py --pdb option, but the end-to-end tests treat an SCons invocation as a "black box" and just look for external effects. Simple print statements within the SCons code itself often don't help debug end-to-end because they end up in SCons output that gets compared against expected output and cause a test failure. Probably the most effective technique is to use the internal SCons.Debug.Trace() function, which prints output to /dev/tty on Linux/UNIX systems and con on Windows systems, so you can see what's going on.

### Naming conventions

The end-to-end tests, more or less, stick to the following naming conventions:

1. All tests end with a .py suffix.

2. In the General form we use

Feature.py

for the test of a specified feature; try to keep this description reasonably short

Feature-x.py

for the test of a specified feature using option x

3. The command line option tests take the form

option-x.py

for a lower-case single-letter option

option--X.py

upper-case single-letter option (with an extra hyphen, so the file names will be unique on case-insensitive systems)

option--lo.py

long option; abbreviate the long option name to a few characters

## Running Tests

The standard set of SCons tests are run from the top-level source directory by the runtest.py script. There is a --qmtest option that checks whether the QMTest package is installed on your system. If it can be found, then the runtest.py script will use it to carry out the tests.

Help is available through the -h option:

$python runtest.py -h  To simply run all the tests, use the -a option: $ python runtest.py -a


By default, runtest.py prints a count and percentage message for each test case, along with the name of the test file. If you need the output to be more silent, have a look at the -q, -s and -k options.

You may specifically list one or more tests to be run:

$python runtest.py src/engine/SCons/BuilderTests.py$ python runtest.py test/option-j.py test/Program.py


Folder names are allowed arguments as well, so you can do a

$python runtest.py test/SWIG  to run all SWIG tests only. You can also use the -f option to execute just the tests listed in a specified text file: $ cat testlist.txt
test/option-j.py
test/Program.py
$python runtest.py -f testlist.txt  One test must be listed per line, and any lines that begin with '#' will be ignored (the intent being to allow you, for example, to comment out tests that are currently passing and then uncomment all of the tests in the file for a final validation run). If more than one test is run, the runtest.py script prints a summary of how many tests passed, failed, or yielded no result, and lists any unsuccessful tests. The above invocations all test directly the files underneath the src/ subdirectory, and do not require that a packaging build be performed first. The runtest.py script supports additional options to run tests against unpacked packages in the build/test-*/ subdirectories. If you are testing a separate Tool outside of the SCons source tree, you have to call the runtest.py script in external (stand-alone) mode: $ python ~/scons/runtest.py -e -a


. This ensures that the testing framework doesn't try to access SCons classes needed for some of the internal test cases.

Note, that the actual tests are carried out in a temporary folder each, which gets deleted afterwards. This ensures that your source directories don't get clobbered with temporary files from the test runs. It also means that you can't simply change into a folder to "debug things" after a test has gone wrong. For a way around this, check out the PRESERVE environment variable. It can be seen in action in How to convert old tests below.

## Not Running Tests

If you simply want to check which tests would get executed, you can call the runtest.py script with the -l option:

$python runtest.py -l  Then there is also the -n option, which prints the command line for each single test, but doesn't actually execute them: $ python runtest.py -n


## Finding Tests

When started in standard mode

$python runtest.py test/packaging/sandbox-test.py  , assuming Linux and a bash-like shell. The output should then look something like this: 1/1 (100.00%) /usr/bin/python -tt test/packaging/sandbox-test.py PASSED Preserved directory /tmp/testcmd.4060.twlYNI  and you see that the test files have been kept in the folder /tmp/testcmd.4060.twlYNI, where you can now copy them from to your new fixture folder. Then, in the test script you simply remove all the tedious TestSCons.write() statements and replace them by a single TestSCons.dir_fixture(). Finally, you shouldn't forget to clean up and remove the temporary test directory. ;) ## Test Infrastructure The test API is in QMTest/TestSCons.py. TestSCons is a subclass of TestCommon, which is a subclass of TestCmd; all those python files are in QMTest. Start in QMTest/TestCmd.py for the base API definitions, like how to create files (test.write()) and run commands (test.run()). You want to use TestSCons for the end-to-end tests in test, but TestCmd for the unit tests in the src folder. The match functions work like this: TestSCons.match_re:: match each line with a RE • Splits the lines into a list (unless they already are) • splits the REs at newlines (unless already a list) and puts ^..$ around each
• then each RE must match each line. This means there must be as many REs as lines.
TestSCons.match_re_dotall:: match all the lines against a single RE
• Joins the lines with newline (unless already a string)
• joins the REs with newline (unless it's a string) and puts ^..\$ around the whole thing
• then whole thing must match with python re.DOTALL.

Use them in a test like this:

test.run(..., match=TestSCons.match_re, ...)


or:

test.must_match(..., match=TestSCons.match_re, ...)


## Avoiding Tests based on Tool existence

Here's an easy sample:

#!python
intelc = test.detect_tool('intelc', prog='icpc')
if not intelc:
test.skip_test("Could not load 'intelc' Tool; skipping test(s).\n")


See QMTest/TestSCons.py for the detect_tool method. It calls the tool's generate() method, and then looks for the given prog (tool name by default) in env['ENV']['PATH'].