I complemented the SCons testing framework by some features from the external scons-test-framework for separate Tools and packages:
unified naming of test files, matches Test.py as well as sconstest-.py
sconstest.skip in a directory skips all files at the same level
external mode for separate Tools, option "-e"
short display mode, option "-s"
suppressing output of count and percent progress messages, option "-j"
file and folder fixtures with new methods dir_fixture and file_fixture, respectively
using subprocess module for true "silent" output
I took great care to not break anything for the standard tests, the full "runtest.py -a" looks good on my side.
Note, that when using the "-e" option for external Tools, the following TestSCons methods might show a different behaviour than in normal mode:
Environment(), java_ENV(), detect(), where_is()
...this is due to the lacking support of the SCons core routines.
Pushed an update, where I tried to address all the issues mentioned in the review comments so far.
Some more info about TestSCons.py (l. 283): the "external" mode (-e) exists to support standalone operation of the testing framework. This means you have runtest.py and the QMtest folder somewhere on your machine, without any other SCons packages/modules installed and in sight. Hence the "if not external" sentinels...
Running tests in parallel is not that hard to implement, IMO (I now reserved the -j option for it). Would be worth a try, perhaps some fiddling is needed to avoid mangled output on the screen.
Stopping the tests with CTRL-C works fine under my Ubuntu Linux, so it might be a Windows/subprocess issue. Can you try again with the new changeset?
I will start documentation of the testing framework next. The text will contain a description of how to call basic tests (internal/external), usage of the fixture methods and describe the conversion from old tests. Usage of the TestSCons methods too, of course.
Leaves the question: Where should this document go? Replace the old description in the Wiki, or should I create a new Docbook document, alongside the user and reference manuals?
One simple place to add doc is in test/README. I'm a big fan of keeping the doc near the code, where someone using it would be able to find it. Wiki is the worst place for it. Separate doc is probably overkill. If you want external mode (only runtest.py and QMtest) to contain it, then put it in QMtest/README-test-framework.txt (?) and point to it from QMtest/README.txt.
We really shouldn't call that dir QMtest anymore; we're not actually using the QMtest tool. Oh well.