Wiki

Clone wiki

pflotran / Depreciated / Documentation / RegressionTests

The test manager for PFLOTRAN is a python program that is responsible for reading a configuration file, identifying the tests declared in the file, running PFLOTRAN on the appropriate input files, and then comparing the results to a known "gold standard" output file.

Running the test manager

The test manager can be run in two ways, either as part of the build system using "make" or manually.

Calling the test manager through make with make rtest runs a full set of regression tests intended to identify when changes to the code cause significant changes to PFLOTRAN's results.

$ cd ${PFLOTRAN_DIR}/src/pflotran
$ make rtest

The full regression and unit test suite can be run with make test.

NOTE: The regression test manager requires python 2.7.x or python >= 3.2 . The version of python used by "make test" is specified by your petsc configuration. A different python executable is called by setting PYTHON on the command line:

$ make PYTHON=/opt/local/bin/python2.7 rtest
# or
$ make PYTHON=python3.3 rtest

Call the test manager manually to isolate a particular test or suite:

$ cd ${PFLOTRAN_DIR}/regression_tests
$ python regression-tests.py --executable ../src/pflotran/pflotran \
    --config-file shortcourse/copper_leaching/cu_leaching.cfg \
    --tests cu_leaching

Some important command line arguments when running manually are:

  • executable : the path to the PFLOTRAN executable
  • mpiexec : the name of the executable for launching parallel jobs, (mpiexec, mpirun, aprun, etc).
  • config-file : the path to the configuration file containing the tests you want to run
  • tests : a list of test names that should be run
  • suites : a list of test suites that should be run

The full list of command line options and a brief description can be found by running with the "--help" flag:

$ python regression-tests.py --help

Running the test manager on a batch system

The regression test suite can be run on batch systems by submitting the "make rtest" command as part of a batch script:

cat > pflotran-test-batch.sh <<EOF
#!/bin/sh
#PBS -N pflotran-test-batch
#PBS -l nodes=1:ppn=16:lr3,qos=lr3_esd2
#PBS -q lr_condo
#PBS -A lr_esd2
#PBS -l walltime=00:30:00
#PBS -o pflotran-test-batch.stdout
#PBS -e pflotran-test-batch.stderr
#PBS -j oe
#PBS -M someone@somewhere.com
#PBS -m bea
#PBS -r n
#PBS -V

module unload intel
module load intel/2013.2.146
module load mkl/2013.2.146
module load openmpi/1.6.3-intel
module load cmake/2.8.7
export PETSC_ARCH=lrc-intel-2013-fort-debug
export PETSC_DIR=/global/home/users/someone/cfs-someone/petsc/petsc-dev

module load python/2.7.3

cd /global/home/users/someone/cfs-someone/pflotran-dev/regression_tests

make PYTHON=python2.7 rtest
EOF

Test Output

The screen output of the test manager is limited to a "progress bar" and summary information. Additional information including the shell commands to manually run pflotran for a test and reasons for failure are written to a log file in the regression test directory: "pflotran-tests-DATE_TIME.testlog". Test logs are verbose and are intended to be searched using grep or a text editor. Use a case insensitive search for "fail", "error", or "warn".

The output of the regression test manager will look something like:

/usr/bin/python regression-tests.py -e ../src/pflotran/pflotran  \    (1)
             --suite standard \
             --config-files ascem/batch/batch.cfg ascem/1d/1d-calcite/1d-calcite.cfg \
                            default/543/543.cfg default/anisothermal/anisothermal.cfg \
                            default/column/column.cfg default/infiltrometer/infiltrometer.cfg \
                            default/condition/condition.cfg default/multicontinuum/multicontinuum.cfg \
                            ngee/ngee.cfg shortcourse/copper_leaching/cu_leaching.cfg
Running pflotran regression tests :
  Test log file : pflotran-tests-2013-02-25_17-39-49.testlog          (2)
............FFF......F......F..
----------------------------------------------------------------------
Regression test summary:
    Total run time: 455.791 [s]
    Tests run : 31                                                    (3)
    Tests failed : 5
    Errors : 4

Where

  1. regression test command, including the test suite and list of configuration files
  2. name of test log with detailed test information
  3. summary for the current run

Note: when "warnings" or "errors" are reported, this is a sign that the regression test manager could not do EXACTLY what was requested. It is either an error in the test manager input, an error in the test manager itself, or pflotran exited with an error condition and did not generate a regression file for comparison.

Creating New Tests

The steps for creating a new test are:

  1. Create the PFLOTRAN input files. Good tests are small and fast. Try to isolate a single process of the code.
  2. Add a regression block to the pflotran input file to generate the regression file.
  3. Add your test to an existing configuration file by adding a test block. The test name should be the input file name without the ".in" extension.
  4. Add your test name to the appropriate test suites.
  5. Verify the output of your test problem, then copy the *.regression file to *.regression.gold
  6. run your test through the test manager to verify that it passes.
  7. Try running the test on a different machine to determine whether any tolerances need to be adjusted to prevent inappropriate failures.

make test runs the "standard" and "standard-parallel" suites. There are also suites and make rules for flow, transport, geochemistry, mesh.

Updated