[JOSS review] Automated tests

This is the response to one of the items in the JOSS review checklist:

Automated tests: Are there automated tests or manual steps described so that the function of the software can be verified?

1. No instruction on how to run tests locally

I can find the documentation of CI tests here, but I can't find the instruction about how to run the unit tests (serial & MPI), regression tests, and demo tests on local machines.

Though I (or users) can find the information of the tests from config.yaml. But I think it will be nice to have this information explicitly mentioned in the documentation, so users and developers can know how to run tests locally before pushing changes online or even making a pull request.

2. One unit test of the serial tests failed

ocellaris/tests/test_io.py::test_restart_file_io failed. I used Singularity to do the tests. The version of Singularity is 3.1.0, and I'm using Arch Linux with kernel 5.0.2-arhc1-1-ARCH. The following is how I did the serial unit tests.

First, build a Singularity image based on the commit 55cf646 and get into a container shell

$ wget https://bitbucket.org/ocellarisproject/ocellaris/raw/55cf646f9811513646d8cca549b0716f4af49546/containers/Singularity
$ sudo singularity build ocellaris.img ./Singularity
$ singularity shell ocellaris.img

Next, I checked the version of the Ocellaris installed in the Singularity container. I confirmed the version is the commit 55cf646f9811513646d8cca549b0716f4af49546, which is the current master branch.

$ cd /source/ocellaris
$ git rev-parse HEAD

And the return is 55cf646f9811513646d8cca549b0716f4af49546.

Finally, I ran the serial tests of the Ocellaris source code in the container:

$ cd ~
$ python3 -m pytest -v /source/ocellaris/tests/ --instafail --timeout=300 --durations=10 --junitxml=~/serial.xml

The error message from the failed test:

E       AssertionError: assert 'ocellaris:\n...    dt: 1.0\n' == 'ocellaris:\n ...start: 42.0\n'
E           ocellaris:
E               type: input
E               version: 1.0
E           mesh:
E               Nx: 10
E               Ny: 10
E               type: Rectangle...
E         
E         ...Full output truncated (26 lines hidden), use '-vv' to show

/source/ocellaris/tests/test_io.py:89: AssertionError
------------------------------------------------ Captured stdout call ------------------------------------------------
Validating input using YSchema
    The input file is not valid! Ocellaris will still continue since this may be a false negative (the validation schema is not perfect), but beware!                                                                                       
    ERROR: time: Missing required keys {'tmax'} in the Time sub-schema

3. One unit test of the MPI tests failed

ocellaris/tests/test_io.py::test_restart_file_io also failed for MPI cases. The command for launching the MPI tests is

$ mpirun -n 3 python3 -m pytest -v /source/ocellaris/tests/ --maxfail=1 --timeout=300 --durations=10 --junitxml=~/MPI.xml

4. All regression tests passed

But in the Singularity container, the regression tests require to write some kind of log files to the directory /soruce/ocellaris, which requires root privilege. So I have to copy the whole /source/ocellaris/cases folder to some where else for running regression tests.

5. All demo tests passed

But again, demo tests require file I/O to /source/ocellaris, so I have to copy the whole directory to somewhere else.

1. No instruction on how to run tests locally

2. One unit test of the serial tests failed

3. One unit test of the MPI tests failed

4. All regression tests passed

5. All demo tests passed

Comments (4)