CCTK_LOOP3_INT and friends broken for Fortran OpenMP code

[Roland Haas]

In http://lists.einsteintoolkit.org/pipermail/users/2020-February/007298.html @Miguel Zilhão reported that the CCTK_LOOP3_INT family macros cannot be used in Fortran using OpenMP:

i'm trying to use the CCTK_LOOP3_INT macro in a Fortran 90 code. following the example in
Carpet/TestLoopControl, i've done:

#include "cctk.h"
#include "cctk_Arguments.h"
#include "cctk_Parameters.h"
#include "cctk_Functions.h"

#include "loopcontrol.h"

subroutine myroutine( CCTK_ARGUMENTS )
   use loopcontrol
   implicit none
   DECLARE_CCTK_ARGUMENTS
   DECLARE_CCTK_PARAMETERS
   DECLARE_CCTK_FUNCTIONS

   CCTK_LOOP3_INT_DECLARE(int3)

   (...)

   CCTK_LOOP3_INT(int3, i, j, k)

      (...)

   CCTK_ENDLOOP3_INT(int3)
end subroutine myroutine

this seems to work fine. however, if i add the OpenMP statements to the loop in the "usual" way:

   !$OMP PARALLEL DO COLLAPSE(3) &
   !$OMP PRIVATE(i,j,k, (...))
   CCTK_LOOP3_INT(int3, i, j, k)

    (...)

   CCTK_ENDLOOP3_INT(int3)
   !$OMP END PARALLEL DO

the code fails to compile with a bunch of errors:

Error: Unexpected !$OMP SINGLE statement at (1)

Error: Unexpected assignment statement at (1)

Error: Unexpected !$OMP END SINGLE statement at (1)

(...)

‌

[Roland Haas]

A (partial?) fix is provided in rhaas/loops

[Roland Haas]

I have not yet gotten to LoopControl proper, but I have at least made the plain Cactus CCTK_LOOP macros work (the one you get if you do not have #include "loopcontrol.h").

They are in the branch "rhaas/loops" of the flesh and I have also updated / fixed / generated the tests in TestLoop where I have checked that Fortran and C code (which I have not touched) produce the same result.

You can use them via:

cd repos/flesh
git fetch
git checkout rhaas/loops
cd ../repos/cactustest
git fetch
git checkout rhaas/loops

then build (no need to clean).

You will have to take care of making your loop variables private on your own (unless you trust OpenMP + Fortran to be smart enough to do so for you, I believe the standard claims is should be... but that may only apply to the loop directly following the "omp do" directive) and use CCTK_LOOP3_XXX_OMP_PRIVATE(name). See TestLoopFortran.F90 for examples (the default(none) is purely optional and just helped me debug things).

So basically:

!$OMP PARALLEL private(i,j,k) CCTK_LOOP3_INT_OMP_PRIVATE(int3)
CCTK_LOOP3_INT(int3, i,j,k)
   ! do something
CCTK_ENDLOOP3_INT(int3)
!$OMP END PARALLEL

The calling syntax of the PRIVATE and DECLARE macros may still change. Right now the C LOOP macros internally declare the loop and normal variables (i, ni) while my current set of Fortran DECLARE and PRIVATE variables do not declare them or make them private (they cannot, they do not now know their name).

So the options are to have either (1):

integer :: i,ni
CCTK_LOOP1_DECLARE(foo)
!$OMP PARALLEL private(i,ni) CCTK_LOOP1_BND_OMP_PRIVATE(int1)
CCTK_LOOP1_INT(int1, i,ni)
   ! do something
CCTK_ENDLOOP1_INT(int1)
!$OMP END PARALLEL

or (2):

CCTK_LOOP1_DECLARE(foo, i,ni)
!$OMP PARALLEL CCTK_LOOP1_BND_OMP_PRIVATE(int1,i,ni)
CCTK_LOOP1_INT(int1, i,ni)
   ! do something
CCTK_ENDLOOP1_INT(int1)
!$OMP END PARALLEL

ie carry a couple of extra arguments (2*dim) in the DECLARE and PRIVATE macros but rest assured that all variables are properly declared private no matter what.

I believe LoopControl went with (2) but Cactus' cctk_Loop.h (which was broken before as far as I can tell) seems to have aimed for (1).

Having just implemented (1) I am actually leaning towards (2) right now, but then I do not actually use the LOOP macros in Fortran.

[Roland Haas]

I have implemented a version (1.5) which is halfway in between (1) and (2) ie requires that users declare i,j,k and ni,nj,nk themselves but list them in the OMP_PRIVATE etc macros. This means that one can use “nice” names for the user visible loop variables even when there are multiple loops, which is not possible when all variables are declared by the DECLARE_XXX macros and still has some safety checks in place ensuring that the variables are marked as private to OpenMP. Example code would be in CactusTest/TestLoop in the rhaas/loops branch:

subroutine TestLoopFortran_intbnd(CCTK_ARGUMENTS)
  implicit none
  DECLARE_CCTK_ARGUMENTS
  DECLARE_CCTK_FUNCTIONS
  DECLARE_CCTK_PARAMETERS

  CCTK_LOOP3_INTBND_DECLARE(intbnd3)
  integer   :: i,j,k
  integer   :: n_i,n_j,n_k
  CCTK_REAL :: fsum

  call CCTK_INFO("TestLoopFortran_intbnd")
  fsum = 0
  !$OMP PARALLEL &
  !$OMP default(none) shared(cctkgh, cctk_ash, cctk_lsh, cctk_tile_min, cctk_tile_max, r) &
  !$OMP reduction(+: fsum) &
  !$omp CCTK_LOOP3_INTBND_OMP_PRIVATE(intbnd3, i,j,k, n_i,n_j,n_k)
  CCTK_LOOP3_INTBND(intbnd3, i,j,k, n_i,n_j,n_k)
     fsum = fsum + r(i,j,k)
  CCTK_ENDLOOP3_INTBND(intbnd3)
  !$OMP END PARALLEL
  fsum_intbnd = fsum
end subroutine TestLoopFortran_intbnd

‌

[Roland Haas]

These should now all work in:

• https://bitbucket.org/cactuscode/cactus/pull-requests/116/handle-openmp-private-stuff-for-cct_loop
• https://bitbucket.org/cactuscode/cactustest/pull-requests/4/testloop-use-openmp-and-openmp-macros
• https://bitbucket.org/eschnett/carpet/pull-requests/39

Note that none of these addresses #2385.

[Roland Haas]

Please review.

[Miguel Zilhão]

This now seems to work for me.

[Roland Haas]

Thank you.

Applied as git hash 6926742e "TestLoopControl: add OpenMP code to test" of carpet

Applied as git hash 1457d7c "TestLoop: adapt to OMP_PRIVATE macros accepting normal direction" of cactustest

Applied as git hash a57b2e79 "Cactus: regenerate loop macros" of cactus