- removed comment
Hydro_InitExcision sphere_pugh_ppm test fails
The Hydro_InitExcision sphere_pugh_ppm test fails for me when run on 1 process on an Ubuntu 16.04 machine. The diffs are attached.
Keyword:
Comments (20)
-
-
reporter - removed comment
Yes, this is with the ubuntu.cfg currently in Simfactory. I will try the Debian configuration or with -ffast-math removed.
-
reporter - changed status to resolved
- removed comment
I can confirm that the test passes (as do other failures in NaNChecker.nancount) when I remove -ffast-math from ubuntu.cfg.
-
- changed status to open
- removed comment
It should not be necessary to avoid
-ffast-math
to make the tests pass. In Cactus, we have always taken the approach that the order in which floating-point expressions are evaluated should not matter, and that we do not want to rely on IEEE semantics when it comes to NaN (or similar exceptional values).While it would be nice to use this part of the standard, it (a) slows things down considerably, and (b) the standard doesn't guarantee bitwise identical results anyway.
-
- removed comment
Test suites containing nans have to be avoided then (a good idea anyway), because -ffast-math is not guaranteed to produce them. The only test that I could find using a quick search that contains nans is CT_MultiLevel/test/boostedpuncture.
-
- removed comment
Non concerning nans: the absolute differences in the velocity here are larger than 10e-7. This is usually considered too large for double-precision hydro simulations. Only looking at the Cactus-diff-output I cannot judge why that is, but I suspect these values are supposed to be zero, or close to it, but -ffast-math seems to produce a considerably larger error. This is a problem.
I don't think we are likely to find a workaround for this in time for the release (code that works better even with fast math). I would suggest to remove --fast-math from the Ubuntu option list for the time being.
-
- removed comment
Replying to [comment:5 knarf]:
The only test that I could find using a quick search that contains nans is CT_MultiLevel/test/boostedpuncture.
-
- removed comment
Compiler behaviour is not OS-dependent -- this is probably triggered by a combination of compiler version, compiler flags, and CPU capabilities. I argue against remove -ffast-math from the options; instead, we should either have suitably stable tests, or sufficiently large tolerances.
-
reporter - removed comment
Replying to [comment:5 knarf]:
Test suites containing nans have to be avoided then (a good idea anyway), because -ffast-math is not guaranteed to produce them. The only test that I could find using a quick search that contains nans is CT_MultiLevel/test/boostedpuncture.
The NaNChecker.nancount test is also expected to produce NaNs, and fails with ubuntu.cfg when -ffast-math is enabled. This is a somewhat different case, however, as the whole point of the test is to check for NaNs.
-
- changed milestone to ET_2018_02
- removed comment
-
- changed status to open
- assigned issue to
- removed comment
-
- assigned issue to
- removed comment
-
- removed comment
I am not sure the failure is still happening. See https://build.barrywardell.net/job/EinsteinToolkit/lastCompletedBuild/testReport/ which lists only * SphericalHarmonicRecon.regression_test/2procs * SphericalHarmonicReconGen.SpEC-dat-test/2procs * SphericalHarmonicReconGen.SpEC-h5-test/2procs as failures. Of course this maybe compiler, processor and option list specific.
-
- removed comment
This test might fail with -march=native and depend on the CPU being used. We might see failures when we test on multiple machines for the release. Leaving this ticket open for now.
-
- removed comment
There are no failures on Jenkins but there are some on ET supported clusters: http://einsteintoolkit.org/testsuite_results/index.php lists some where it does fails: * bethe * comet * hydra * stampede2-skx
-
- assigned issue to
- removed comment
-
- changed milestone to ET_2019_10
- edited description
-
- changed milestone to ET_2020_04
-
- changed milestone to ET_2020_05
-
- changed milestone to ET_2020_11
- Log in to comment
Do you use the standard Ubuntu configuration out of Simfactory? Can you try the debian configuration? The main difference I see could be that the Ubuntu configuration uses the -ffast-math option, while the Debian doesn't.