Extension of testing framework (#2862)

#33 Merged at 653d622
  1. dirkbaechle

I complemented the SCons testing framework by some features from the external scons-test-framework for separate Tools and packages:

  • unified naming of test files, matches Test.py as well as sconstest-.py
  • sconstest.skip in a directory skips all files at the same level
  • external mode for separate Tools, option "-e"
  • short display mode, option "-s"
  • suppressing output of count and percent progress messages, option "-j"
  • file and folder fixtures with new methods dir_fixture and file_fixture, respectively
  • using subprocess module for true "silent" output

I took great care to not break anything for the standard tests, the full "runtest.py -a" looks good on my side. Note, that when using the "-e" option for external Tools, the following TestSCons methods might show a different behaviour than in normal mode:

Environment(), java_ENV(), detect(), where_is()

...this is due to the lacking support of the SCons core routines.

Comments (5)

  1. SCons repo owner

    Minor nit: Line 429 in runtest.py - Can you change to be a list comprehension rather than list(map())?

    Question: Line 283 TestSCons.py: Why only set SCons.NODE.FS.default_fs if not external? (same question for Environment()

    TestCmd.py: (line 902) Is it necessary to wrap os.environ.get() in a try, if you supply a default value?

    Otherwise looks reasonable. Next I'll try running it.. :)

    1. dirkbaechle author

      Pushed an update, where I tried to address all the issues mentioned in the review comments so far.

      Some more info about TestSCons.py (l. 283): the "external" mode (-e) exists to support standalone operation of the testing framework. This means you have runtest.py and the QMtest folder somewhere on your machine, without any other SCons packages/modules installed and in sight. Hence the "if not external" sentinels...

      Running tests in parallel is not that hard to implement, IMO (I now reserved the -j option for it). Would be worth a try, perhaps some fiddling is needed to avoid mangled output on the screen.

      Stopping the tests with CTRL-C works fine under my Ubuntu Linux, so it might be a Windows/subprocess issue. Can you try again with the new changeset?

      I will start documentation of the testing framework next. The text will contain a description of how to call basic tests (internal/external), usage of the fixture methods and describe the conversion from old tests. Usage of the TestSCons methods too, of course. Leaves the question: Where should this document go? Replace the old description in the Wiki, or should I create a new Docbook document, alongside the user and reference manuals?

      1. Gary Oberbrunner

        One simple place to add doc is in test/README. I'm a big fan of keeping the doc near the code, where someone using it would be able to find it. Wiki is the worst place for it. Separate doc is probably overkill. If you want external mode (only runtest.py and QMtest) to contain it, then put it in QMtest/README-test-framework.txt (?) and point to it from QMtest/README.txt.

        We really shouldn't call that dir QMtest anymore; we're not actually using the QMtest tool. Oh well.

        1. dirkbaechle author

          Did some more work and added a short doc text in reST at QMTest/test-framework.rst. Is it okay like this and are the two converted examples enough?