- edited description
Improve testing framework
About
Currently we have a number of tests written for freud, but we have a number of issues that need to be addressed:
- It is unclear how much code is actually being tested and how much untested code we have.
- We have test that are nondeterministic.
- Even where tests exist, it is unclear how much of the actual physics involved is being tested.
Specific issues
This list should be updated as more are found
- Nondeterministic test in test_order_cubatic.test_disordered has been switched to use a random seed, but we should be testing with an actually disordered system generated deterministically.
Proposals
- Add coverage testing for python and cython code.
- This is ongoing in the coverage branch.
- Add coverage for CPP testing.
- Need to do more research here, but gcov seems like a potential option for this.
- Split tests for APIs, code consistency, etc from validation of methods.
- We need to write more validation tests I would guess, at present our coverage is low.
- Separate tests that require specific non-random inputs from those that can use deterministic but arbitrary values.
Comments (11)
-
reporter -
reporter @harperic @bdice any input you have would be appreciated. I wonder if splitting validation from other unit tests is actually necessary in our case, but I think it's worth considering at least to ensure proper testing coverage.
-
reporter - marked as enhancement
-
As of the latest release (v0.8.2), the Python tests have been substantially cleaned up. To the best of my knowledge, all behavior is deterministic with the possible exception of
#171. -
reporter @harperic I think this could be one area that you could tackle much more effectively than Bradley or I. Since you have more experience with freud, do you think you could write validation tests (when you think you have time)? I plan to go through and document which modules have sufficient testing and which ones do not, but for modules that you wrote you could write simple sanity checks more quickly than we would. Currently, freud is sorely lacking in that kind of check.
-
Another non-deterministic test:
test_order_Nematic
test_imperfect
just failed: https://circleci.com/bb/glotzer/freud/1014 -
I would be happy to head up the tests. I can do the majority of them and will request help writing specific tests from the specific authors of some of the more complex branches e.g. Steinhardt or environment
-
-
assigned issue to
-
assigned issue to
-
- changed component to testing-validation
-
- changed version to 1.0
-
- changed status to resolved
- Log in to comment