Add Cosmology Simulation to test suite

#419 Merged at d4cee19
Repository
enzo-dev-cosmotest
Branch
week-of-code
Repository
enzo-dev
Branch
week-of-code
Author
  1. Britton Smith
Reviewers
Description

This PR adds three new cosmology simulations and tests that can be run using the test runner.

The amr_cosmology simulation is derived from run/CosmologySimulation/AMRCosmology, but is updated to use initial conditions from MUSIC. A number of tests will run on this simulation, including total mass measures, halo mass function, phase plots, radial profiles, and total number of outputs.

The dm_only simulation is the same base as amr_cosmology, but with baryons turned off. This includes the halo mass function test.

The amr_nested_cosmology simulation has 1 initial nested level of refinement around the most massive halo with must-refine-particles within 3 virial radii of the halo at z = 1.8. Tests are the same as amr_cosmology.

These tests generate their own results files, which are saved in the same location as the answer testing database used by the test runner. We decided to do it this way because it gave us much more flexibility in saving and comparing results and because the current answer testing infrastructure in yt is being phased out. This can serve as an example of how to re-engineer the test suite to operate outside of that, if and when the time comes to do that.

Finally, the initial conditions are hosted on the yt Hub here. In the bitbucket pipelines testing script, we now download those before running the tests so that we don't have to install MUSIC.

  • Commit status

Comments (20)

  1. John Regan

    My only comment is on the Cloudy file that needs to be copied across from Grackle. Would it be better to just include it as part of the run directory?

    1. Brian O'Shea

      So am I. The PR czars need to round up a third reviewer! (Attention @jwise77 and @gbryan )

      1. John Wise

        I looked at my email when pinging reviewers, and I said that I’d be the 3rd reviewer! Hah. I’ll take a look tomorrow.

  2. John Wise

    This is all great! I read through the new tests, and I think it’s a good addition to the suite. Nice work!

    I will merge it after it passes the test suite.

  3. John Wise

    @brittonsmith I tried to run the test suite on this PR, but the new tests aren’t running because the Grackle datafile and ICs aren’t generated automatically. I just wanted to double check that I will have to generate the ICs and copy the Grackle cooling table before running the test suite. I see that these files are set up to be downloaded in the automated testing (`run/test.sh`), though, and won’t be a problem when we move to Github.

    1. Britton Smith author

      @jwise77 , yeah, you need to generate or download the IC files and the Grackle file to run locally. The easiest thing would probably be to run the girder client command in the notes.txt files in each directory. That will download all necessary files from the yt hub. It is also possible to download the entire IC collection at once. It may be worth adding a note to this effect.

      At the moment, the yt hub is still coming back online after some outages from the machines supporting it, so the download isn’t working right now. I’ll send an update when this changes.

    2. Britton Smith author

      @jwise77 , the yt hub is back up now, so this should now be runnable from the automated testing.

  4. John Wise

    Just an update on this PR: test_runner.py wasn't running these simulations automatically because the ICs had to be downloaded/generated by hand. Also, they're prone to be deleted if the test suite is run with --clobber. I've updated test_runner.py and the new cosmology .enzotest files to download the ICs before running the simulation. I’ll create a pull request to @brittonsmith 's fork which can then be incorporated into this PR. Afterwards I can run the test suite, generate a new gold standard, and accept.

    1. Brian O'Shea

      John, the outputs of the test suite have some errors:

      Complete!
      Total time: 2616.259208 seconds.
      See /root/enzo_test/0d7a2b9d82db/test_results.txt for a summary of all tests.
      Passes:             2230.
      Failures:           0.
      Errors:             83.
      Sims not finishing: 0.
      

      Is this to be expected?

      1. John Wise

        If they’re related to the new cosmology tests, then yes. Can you upload the results somewhere?

        1. Brian O'Shea

          I just clicked on the testing button at the top of the PR - the info is at https://bitbucket.org/brittonsmith/enzo-dev-cosmotest/addon/pipelines/home#!/results/10 , click on the “bash test.sh” in the right panel. Upon further inspection (now that I’ve actually had sleep and coffee) it’s clear that all of the errors are from the new cosmology tests, and are “no old answer available” errors (which is what one would expect to happen in this circumstances). I think this is PR good to go!

  5. Greg Bryan

    OK, we have three approvals, the tests pass (except those that are expected to fail). This seems ready to go – I will merge by the end of the day unless I hear otherwise. Thanks all!

    1. Britton Smith author

      @gbryan , once you merge this, I recommend tagging a new gold standard and changing the GOLD_STANDARD_TAG value in test.sh and either pushing that straight this repo or issuing a new PR.

      1. Greg Bryan

        Good suggestion – this is now merged and I just pushed through a PR with the update to a new gold standard.