ctest results in a lot of failed or timed-out tests

[A Schuetz]

After installing BDSIM and all required programs, I was running

ctest -E LONG

A lot of the tests failed or timed-out. Then I run ctest again, and even more tests were failing. See the attached text file.

Does that mean, something went wrong with the installation?

Thanks for your help! Anne

[Laurie Nevay]

Hello Anne, are you using geant4.10.03? If so, can you please use Geant4.10.02 (p01 - p03), which should be fine. We see long running overlap checks with 4.10.3 just now - it still works, just the tests time out. I haven't tested BDSIM with Geant4.10.3 yet - we'll officially support it soon.

[A Schuetz]

Hey Laurie, Yes, I was using Geant4.10.03. I switched to Geant4.10.02.03 and that seems to not give any failed or timed-out test when doing ctest. Thanks for your help!

[Laurie Nevay]

As discussed - due to lack of support for Geant4.10.03 just now. Will be supported shortly.

[Jochem Snuverink]

Thanks Anne for the report. There is already an issue for this open: #151. As long as you have "option, checkOverlaps=0" Geant4.10.3 should probably be fine to use.

[A Schuetz]

After switching to Geant4.10.02.03, ctest still gives failed and timed-out tests, way less than before though:

94% tests passed, 13 tests failed out of 236

Label Time Summary: LONG = 378.96 sec (12 tests)

Total Test time (real) = 4440.77 sec

The following tests FAILED: 63 - root-options (Failed) 66 - root-event (Failed) 123 - 2_magnet_poles_circular (Timeout) 147 - magnet_one_of_each_base (Timeout) 197 - electron_250GeV_copper_ANALYSIS (Failed) 198 - electron_250GeV_copper_COMPARE (Failed) 199 - electron_250GeV_stainless_ANALYSIS (Failed) 200 - electron_250GeV_stainless_COMPARE (Failed) 201 - electron_250GeV_titanium_ANALYSIS (Failed) 202 - electron_250GeV_titanium_COMPARE (Failed) 203 - electron_250GeV_tungsten_ANALYSIS (Failed) 204 - electron_250GeV_tungsten_COMPARE (Failed) 235 - Manual (Failed) Errors while running CTest

Is there still something wrong?

[Jochem Snuverink]

That looks mostly alright.

Very occasionally we have still some stuck particles which cause test to timeout, Laurie is working on that.

The root tests are not directly related to BDSIM but check if ROOT is working. These are failing for me at my desktop too as a matter of fact (even though I have a working ROOT version). This probably needs some better configuration checks to see if it loads right libraries.

I don't know why the manual building fails, probably a LaTeX problem. I presume this is not an issue for you.

The COMPARE tests check if the physics output is consistent in a statistical sense to what it was before, however the configuration files need updating and the limits are likely too strict. This is planned for version 1.0.

The ANALYSIS tests use rebdsim (the ROOT output analysis tool). There might be something wrong with that. I am not an expert there though. In case you would like to rebdsim, could you post the output or error message of:

ctest -R electron_250GeV_copper_ANALYSIS -VV

p.s. in case you are running the develop version, you can check the daily status here: http://abp-cdash.web.cern.ch/abp-cdash/index.php?project=BDSIM

[A Schuetz]

ctest -R electron_250GeV_copper_ANALYSIS -VV

has the following output message:

test 212 Start 212: electron_250GeV_copper_ANALYSIS

212: Test command: /space/vol2/bdsim-build/analysis/rebdsim "electron_250GeV_copper.txt" 212: Test timeout computed to be: 1500 212: rebdsim> configuration file name : electron_250GeV_copper.txt 212: Config::Instance> No instance present, construct 212: Error in <TFile::TFile>: file ./electron_250GeV_copper_event.root does not exist 212: DataLoader::BuildTreeNameList> no such file "./electron_250GeV_copper_event.root" 1/1 Test #212: electron_250GeV_copper_ANALYSIS ...***Failed 0.77 sec

0% tests passed, 1 tests failed out of 1

Label Time Summary: LONG = 0.77 sec (1 test)

Total Test time (real) = 0.98 sec

The following tests FAILED: 212 - electron_250GeV_copper_ANALYSIS (Failed) Errors while running CTest

ROOT and LaTex are defenitely working for me, but it still seems like the root file could not be produced for some reason?

[Jochem Snuverink]

Thanks. Okay that meant the output root from the previous test was not present (I had assumed it was).

Could you run that one also:

ctest -R electron_250GeV_copper -VV

This will run the test (will take a while), the ANALYSIS test and the COMPARE test.

[A Schuetz]

All of the tests passed there. :)

The following tests passed: electron_250GeV_copper_LONG electron_250GeV_copper_ANALYSIS electron_250GeV_copper_COMPARE electron_250GeV_copper

100% tests passed, 0 tests failed out of 4

Label Time Summary: LONG = 1043.17 sec (3 tests)

Total Test time (real) = 1056.42 sec

[Jochem Snuverink]

hmm okay, but these failed before.. :-)

Just to check root, could you also post the log root-options:

ctest -R root-options -VV

[Jochem Snuverink]

Ahhh sorry I missed that you ran ctest -E LONG. Then it makes sense that the ANALYSIS and COMPARE tests were failing since the input data for these tests were missing. The newly recommended way to run the short tests is (for next time):

ctest -LE LONG

(LE for Label Exclude) Instead of putting LONG in the name, we have now labeled the tests LONG. That should give a more consistent set of tests.

[A Schuetz]

The BDSIM documentation (http://www.pp.rhul.ac.uk/bdsim/manual/installation.html#building) suggests to run

ctest -E LONG

[A Schuetz]

ctest -R root-options -VV fails with Error regular expression found in output

Do you want the full output?

[Jochem Snuverink]

Ah oops. I was just looking up what was in the manual. I will fix it.

[Jochem Snuverink]

Okay. The test checks for the expression "Error" and fails if it finds that. What is the full output?

[A Schuetz]

test 64 Start 64: root-options

64: Test command: /space/vol1/root/bin/root "-b" "-q" "-l" "optionsTest.C"

64: Test timeout computed to be: 1500

64: 64: Processing optionsTest.C...

64: Loading BDSIM Libraries 64: Loading libbdsimRootEvent 64: Loading librebdsimLib 64: BDISM libraries loading complete

64: In file included from OptionsDict dictionary payload:10:

64: /space/vol2/bdsim/analysis/Options.hh:6:10: fatal error: 'BDSOutputROOTEventOptions.hh' file not found

64: #include "BDSOutputROOTEventOptions.hh" 64: ^ 64: Error in <TInterpreter::AutoParse>: Error parsing payload code for class Options with content:

64: 64: #line 1 "OptionsDict dictionary payload"

64: 64: #ifndef G__VECTOR_HAS_CLASS_ITERATOR 64: #define G__VECTOR_HAS_CLASS_ITERATOR 1 64: #endif 64: #ifndef ROOTBUILD 64: #define ROOTBUILD 1 64: #endif 64:

64: #define _BACKWARD_BACKWARD_WARNING_H 64: #include "/space/vol2/bdsim/analysis/Options.hh" 64:

64: #undef _BACKWARD_BACKWARD_WARNING_H 64:

64: /space/vol2/bdsim-build/examples/features/analysis/0_output/./createOptions.C:3:20: error: allocation of incomplete type 'Options' 64: Options o = new Options(); 64: ^~~ 64: OptionsDict dictionary forward declarations' payload:5:90: note: forward declaration of 'Options' 64: class attribute((annotate("$clingAutoload$/space/vol2/bdsim/analysis/Options.hh"))) Options; 64:
^ 64: /space/vol2/bdsim-build/examples/features/analysis/0_output/./createOptions.C:4:3: warning: deleting pointer to incomplete type 'Options' may cause undefined behavior [-Wdelete-incomplete] 64: delete o; 64: ^ ~ 64: OptionsDict dictionary forward declarations' payload:5:90: note: forward declaration of 'Options' 64: class attribute((annotate("$clingAutoload$/space/vol2/bdsim/analysis/Options.hh"))) Options; 64:
^ 1/1 Test #64: root-options .....................**Failed Error regular expression found in output. Regex=[Error] 0.68 sec

0% tests passed, 1 tests failed out of 1

Total Test time (real) = 0.87 sec

The following tests FAILED: 64 - root-options (Failed) Errors while running CTest

[A Schuetz]

The header file it is looking for "fatal error: 'BDSOutputROOTEventOptions.hh' file not found" is here:

../bdsim/include/BDSOutputROOTEventOptions.hh

[Jochem Snuverink]

That will take some time to understand, it is possibly a bug. For now I wouldn't worry about it, since the other tests have run well. For the record which version of ROOT do you have?

[A Schuetz]

I compiled and run it with ROOT 6.06/04

[Jochem Snuverink]

manual is updated as discussed above in commit cd277f8

[Jochem Snuverink]

The root-options test failing is a bug. I have opened a separate issue for that: #173.

[Laurie Nevay]

I've patched this issue with ROOT for now, but it's ongoing. I wasn't sure when I replied which branch you were on. In master we have a mix of cmake labels for tests and LONG in the file name, but in develop we've now moved entirely to the label system (well some names remain, but aren't used).