Make it possible to disable OpenMP in the `cctk_Loop.h` loop macros

[Erik Schnetter]

The looping macros defined in cctk_Loop.h use OpenMP unconditionally. If a thorn is not using OpenMP, or is using OpenMP in a manner incompatible with cctk_Loop.h, then this can lead to segfaults. For example, cctk_Loop.h uses omp single (that prevents certain code from executing) or omp for (which parallelizes part of the loops). There should be a mechanism to disable this OpenMP parallelization.

[Erik Schnetter]

This pull request https://bitbucket.org/cactuscode/cactus/pull-requests/102/cctk_looph-allow-caller-to-disable-openmp addresses this.

[Erik Schnetter]

I forgot to say: For example, CarpetX uses OpenMP in a manner incompatible with cctk_Loop.h, leading to such segfaults.

[Roland Haas]

Somewhat ugly, but seems to be the only solution.

Naming scheme (OMP or OPENMP) is consistent with the existing CCTK_DISABLE_OMP_COLLAPSE configure option.

I would suggest to extend the ifndef to include also the case where _OPENMP is not defined (ie

• fopenmp

is not in effect):

#  if(defined(_OPENMP) && !defined(CCTK_LOOP_DISABLE_OMP))
#    define CCTK_PRAGMA_OMP(x) _Pragma(x)
#  else
#    define CCTK_PRAGMA_OMP(x)
#  endif

since this will avoid the “warning: ignoring #pragma omp for” warnings when compiling without

• fopenmp

. It does require, like all such checks for OPENMP, that

• fopenmp

is part of FPPFLAGS and CPPFLAGS (required already for MAKEDEPEND to work which uses CPP and FPP only) so runs slight risk of always disabling OpenMP in CCTK_LOOP if CPPFLAGS and FPPFLAGS are not set correctly (know-architectures does not set them, unfortunately).

Other than that please apply.

[Roland Haas]

@eschnett said:

Is that (accidentally switching off OpenMP) not quite dangerous?

Also, not all OpenMP warnings will go away, since I left the omp critical in. These are used for error messages. (One should probably wrap the bodies of CCTK_ERROR and friends instead of asking the callers to do so, but that’s another point.) I’m inclined to leave out the defined _OPENMP.

‌

[Roland Haas]

Uff, I had not realized it would leave OpenMP statements behind.

Yes, please leave out the defined _OPENMP as a more comprehensive solution is needed.

As for the danger of silently disabling OpenMP: yes, that was definitely a concern.

Right now we kind of cheat since even if CPPFLAGS does not contain -fopenmp it really only affects the MAKEDEPEND call in the Makefile since C/C++ preprocessing is handled on-the-fly by the compiler wrappers to which we pass both CPPFLAGS and CFLAGS/ CXXFLAGS. If these two stages were actually separate (or if there was a wrapper that would take say --cppflags = ““ and --cflags = ““ then we’ woul be in trouble already now since LoopControl (likely, I have not checked right now) contains the same #ifdef _OPENMP constructs to wrap around omp_num_threads() calls.

For Fortran code this is much more obvious since there FPP is indeed a separate invocation from the compiler.

So: please apply (the original version).

[Roland Haas]

Actually…. since the define only affects the loops, would it be possible to name it CCTK_LOOP_PPRAGMA_OMP?

[Erik Schnetter]

I will rename the macro.

[Erik Schnetter]

I renamed the macro.