test system does not pause if test fails to run

Create issue
Issue #1242 new
Roland Haas created an issue

when running all tests in a thorn, the test system does not stop for user input when a test completely fails to run (or produces no output files) eg due to an error in the parameter file.

  Test GRHydro: tov_carpetevolutionmask2
    "Carpet mesh refined, one TOV, Gamma-law, use evolution mask"

  Issuing mpirun -np 2 exe/cactus_bns_all arrangements/Zelmani/GRHydro/test/tov_carpetevolutionmask2.par
Cactus exited with error code 1
Please check the logfile TEST/bns_all/GRHydro/tov_carpetevolutionmask2.log

  No files created in test directory

  Failure: 6 files missing, 0 files compared, 0 differ

Had the test run but produced different files, the system would have stopped allowing he user to inspect the results.

For a thorn with many tests this can make it hard to spot these errors.


Comments (3)

  1. Frank Löffler
    • removed comment

    Instead of stopping I would probably prefer here to produce a summary of the test results in the end, like what is done when running all tests. What do you think?

  2. Roland Haas reporter
    • removed comment

    Having it produce a summary would be even more helpful. Having to stop at each failing test when running all in a thorn can be occasionally tedious, yes.

  3. Ian Hinder
    • removed comment

    I always found it very strange that the system behaved differently if you asked it to run all tests or not. I think the summary should be output in all cases. That being said, when SimFactory runs the tests, it always runs "all" tests. The reason is that it is always noninteractive, and the summary log is useful. Additionally, because SimFactory needs to have all the input data available at the time the job runs, i.e. it must not rely on having the home filesystem available and in the same state as when the job was submitted, it actually copies the reference data for the requested tests into the job output directory. This is how it filters which tests are supposed to be run, and then it can run "all tests" and generate the summary output. I always use simfactory when running tests, as I don't find the interactive prompting very useful, preferring to examine the output and diffs etc myself in a text editor.

  4. Log in to comment