#82 Merged at 71d8332
  1. Sam Skillman

This involves an upgrade to the answer testing framework used in Enzo, which utilizes functionality from yt (yt-project.org). This includes both functional upgrades as well as a vastly improved documentation of the testing framework. There are many ways of identifying the set of tests to run, which is documented. The three primary settings are denoted through --suite={quick, push, full}. Tests up to and including the push suite have been uploaded to an Amazon cloud instance, allowing users to compare to a gold standard. Alternatively, answer testing can be done while comparing to local results. Testing between local results generally yields tests that all pass. Some tests may fail at the bitwise level if compilers/optimizations are changed. Testing between local and gold (cloud) results may produce more test failures due to the differences in compilers/optimizations.

Current issues/enhancements will be added to the issue tracker, and addressed as needed.

Issues: Zeus hydro fails with SAB (SAB should ideally be used). Tests for which this fails: ExtremeAdvectionTest, GravityTestSphere GravityTest also fails with SAB.

When comparing, the following are particularly sensitive to compiler/optimization settings: Cosmology/Hydro/AdiabaticExpansion CollideTest ProtostellarCollapse_Std PhotonTestAMR

The full suite takes a very long time (~2 days on -g optimization). This should be investigated, but for now not used.

Comments (5)

  1. Nathan Goldbaum

    Hi Sam, thanks for putting this all together.

    When I run the test suite on my laptop, 35 tests fail with an error inside the testing framework: http://paste.yt-project.org/show/2902/

    I'm also seeing about a hundred fewer failures than when I last ran the quick suite this afternoon.

    All of the rest of the failures are due to small mismatches above the relative tolerances or mismatching hashes.

    1. MattT

      @ngoldbaum the error you pasted is hiding a test failure. There are no outputs, or there is only one.

  2. Sam Skillman author

    Hi Nathan,

    That problem is (I think) due to the SAB issue mentioned above. If you look in estd.out of the failed run, it should show up there.

    1. Nathan Goldbaum

      That was the problem! Given that these are issues with enzo rather than the testing suite, I think this PR should be accepted as is. I'm glad that so many of the tests pass cleanly out of the box :)