- changed milestone to ET_2013_05
- removed comment
disable hyperthreading in loopcontrol by default
when running with both openmp and sufficiently many threads that hyperthreading threads are used, many tests using LoopControl (Cartoon, RotatingSymmety180, RotatingSymmetry90) fail.
This can be tracked down to disabling hyperthreading support in LoopControl (ie. turning of hyperthreading makes things work).
In particular on bethe with smt and 8 physical cores:
The Cartoon/test_cartoon_2.par test shows differences from the recorded results when run with 16 threads (but not with 8 threads). If I then go ahead and disable OMP in all ML source files but ML_BSSN_enforce and comment out the #include "loopcontrol.h", then the difference goes away. Adding back #include "loopcontrol.h" brings back the error. Some further experimenting with LoopControl's options shows that indeed the use_smt_threads option is what causes problems. If I turn it off things work fine even with a vanilla source tree. Otherwise relative differences are on the order 1e-7 and absolute 1e-11 (in momx_z_[2][2].xg). Without smt the results are identical to the stored values.
The issue only occurs in combination of OpenMP, vectorization and hyperthreading. The issue is independent of the compiler (both intel 13 and gcc 4.4 show the same behaviour), and vectorization (sse2) and many threads (up to 4 times the number of physical cores) works fine on non-smt machines.
I propose to disable LoopControl::use_smt_threads by default. Note that we cannot completely remove it since apparently for Vesta (a Blue Gene/Q) smt is required to get and multi-threading at all.
Keyword: LoopControl
Comments (4)
-
-
reporter - removed comment
Applied to Carpet master in hash a4dc204e712f58cabfc270a5cb158ff9664485cf applied to release branch as hash c8bbd52a3ddfe031ba4789bee3a8ab72ecab4e9f to 2013_05 branch.
-
- removed milestone
- removed comment
-
reporter - changed status to resolved
- removed comment
This has been fixed as part of Carpet hash cce526ef043dda6115818e54120e8d1410d2d7c5 "LoopControl: Implement automatic performance improvements". The option now defaults to "yes" again.
- Log in to comment
I suggest to do this for the release, and to continue debugging afterwards.