criteria, including (but not limited to) the physics included, the
estimated runtime of the test, and the dimensionality. For
convenience, three pre-created, overlapping sets of tests are
+provided. For each set of tests, the test suite can automatically
+pull the "gold standard" results from a remote server; or one
+can generate their own standard locally against which she can compare.
-1. The "quick suite" (``--
quicksuite= True``). This is composed of
+1. The "quick suite" (``--suite=``). This is composed of
small calculations that test critical physics packages both
alone and in combination. The intent of this package is to be run
-relatively frequently (multiple times a day) to ensure that bugs have
-not been introduced during the code development process. All runs
-in the quick suite use no more than a single processor. The total
-run time should be about 25 minutes. The gold standard results for
-the quick suite alone can be downloaded
+automatically and relatively frequently (multiple times a day) on
+a remote server to ensure that bugs have not been introduced during the code
+development process. All runs in the quick suite use no more than
+a single processor. The total run time should be about 25 minutes.
2. The "push suite" (``--pushsuite=True``). This is a slightly
large set of tests, encompassing all of the quick suite and
modules. The intent of this package is to provide a thorough validation
of the code prior to changes being pushed to the main repository. The
total run time is roughly 90 minutes and all simulations use only a single
-processor. The gold standard results for the push suite can be downloaded
3. The "full suite" (``--fullsuite=True``). This encompasses essentially
all of test simulations contained within the run directory. This suite
situations, and is intended to be run prior to major changes being pushed
to the stable branch of the code. A small number of simulations in the full
suite are designed to be run on 2 processors and will take multiple hours to
-complete. The total run time is roughly 36 hours. The gold standard results
-for the full suite can be downloaded
+complete. The total run time is roughly 36 hours.
How to run the test suite
-The Enzo test suite is run within the ``run/`` subdirectory of the
-Enzo source distribution, using the ``test_runner.py`` file. To
-run the test suite, follow these instructions:
-1. Before running the test suite, you should download the "gold
-standard" results for the
-`push <http://enzo-project.org/tests/gold_standard_push.tar.gz>`_, or
-suites and untar that file into a convenient directory.
-2. Compile Enzo. The gold standard calculations use the default
+1. Compile Enzo. The gold standard calculations use the default
compiler settings that can be restored with ``make default``.
If you use significantly different compilation options
-(higher-level optimization in particular) you may see somewhat
-different outputs that will result in failed tests.
-3. Go into the ``run/`` subdirectory in the Enzo repository and
-type the following command:
+(high-level optimization in particular) you may see somewhat
+different outputs that will result in failed tests. To compile
+enzo with the standard settings, complete these commands:
- ./test_runner.py --quicksuite=True --compare-dir=/path/to/gold_standard \
+ cd <enzo_root>/src/enzo
-In this comand, ``--quicksuite=True`` instructs the test runner to
+Make sure you move the resulting enzo.exe file to somewhere in
+2. Get yt. The enzo tests are generated and compared using the
+yt analysis suite. If you do not yet have yt, visit
+http://yt-project.org/#getyt for installation instructions.
+If you already have yt and yt is in your path, make sure you're using
+the most up-to-date version by running the following command:
+3. Generate the test files. The testing suite operates by
+creating a number of standard tests for each test problem, but
+you need to generate the test files first by executing the following
+ python make_new_tests.py
+3. Run the testing suite. While remaining in the ``run/``
+subdirectory, you can initiate the generation of the quicksuite test
+simulations and comparison of them against the gold standard by
+running the following commands:
+ python test_runner.py --suite=quick -o <external_directory_where_tests_will_reside>
+In this comand, ``--suite=quick`` instructs the test runner to
use the quick suite (other possible keyboards here are
pushsuite= True`` and ``-- fullsuite= True``).
+``--suite=`` and ``--suite=``).
``--output-dir=/enzo/test/directory`` instructs the test runner to
-write output to the user-specified directory, and
-``--compare-dir=/path/to/gold_standard`` instructs the test runner
-to use the set of data files in the listed directory as a gold
-standard for comparison. It is also possible to choose sets of tests
-that are sorted by dimensionality, physics modules, runtime, number of
-processors required, and other criteria. A single named test can be run
-by giving ``--name=<name of test>``. Type ``./test_runner.py
---help`` for a more complete listing.
+write output to a user-specified directory (preferably outside of the
+enzo root hierarchy). For a full description of the many flags
+associated with test_runner.py, see the section on running more tests below.
+4. Review the results. While the test_runner is executing, you should
+see the results coming up at the terminal in real time, but you can review
+these results in a file output at the end of the run. The test_runner
+generates a couple subdirectories in the output directory you provided it.
+These will look something like this:
+ ls <external_directory_where_tests_will_reside>
+ gold_quick fe7d4e298cb2
+The first subdirectory is where the gold standard downloaded itself
+on to your system. The files in that directory are python ''shelve''
+objects. The second subdirectory is named by the unique hash of the
+version of enzo that you used to run your tests. Within this directory,
+you should see the individual test problems that you ran, and you should
+see a file called test_results.txt. This file contains statistics on
+all of the tests the passed and failed, as well as the reasons why
+failures occurred. If you get a test failure with a brand new version
+of the code (i.e. no modifications), you should report your results
+to the enzo-users email list. However, if you have modified the source
+and you receive some failures, you can use test_results.txt to track down
+the source of the problems.
How to add a new test to the library
(with extension ``.enzotest``). The latter contains a set of
parameters that specify the properties of the test. Consider the test
suite parameter file for InteractingBlastWaves, which can be found in the
./test_runner.py --repo=/path/to/desired/enzo/repo \