CCTK_LOOP3_INT and friends broken for Fortran OpenMP code

Create issue
Issue #2386 open
Roland Haas created an issue

In http://lists.einsteintoolkit.org/pipermail/users/2020-February/007298.html @Miguel Zilhão reported that the CCTK_LOOP3_INT family macros cannot be used in Fortran using OpenMP:

i'm trying to use the CCTK_LOOP3_INT macro in a Fortran 90 code. following the example in
Carpet/TestLoopControl, i've done:

#include "cctk.h"
#include "cctk_Arguments.h"
#include "cctk_Parameters.h"
#include "cctk_Functions.h"

#include "loopcontrol.h"

subroutine myroutine( CCTK_ARGUMENTS )
   use loopcontrol
   implicit none
   DECLARE_CCTK_ARGUMENTS
   DECLARE_CCTK_PARAMETERS
   DECLARE_CCTK_FUNCTIONS

   CCTK_LOOP3_INT_DECLARE(int3)

   (...)

   CCTK_LOOP3_INT(int3, i, j, k)

      (...)

   CCTK_ENDLOOP3_INT(int3)
end subroutine myroutine

this seems to work fine. however, if i add the OpenMP statements to the loop in the "usual" way:

   !$OMP PARALLEL DO COLLAPSE(3) &
   !$OMP PRIVATE(i,j,k, (...))
   CCTK_LOOP3_INT(int3, i, j, k)

    (...)

   CCTK_ENDLOOP3_INT(int3)
   !$OMP END PARALLEL DO

the code fails to compile with a bunch of errors:

Error: Unexpected !$OMP SINGLE statement at (1)

Error: Unexpected assignment statement at (1)

Error: Unexpected !$OMP END SINGLE statement at (1)

(...)

Comments (5)

  1. Roland Haas reporter

    I have not yet gotten to LoopControl proper, but I have at least made the plain Cactus CCTK_LOOP macros work (the one you get if you do not have #include "loopcontrol.h").

    They are in the branch "rhaas/loops" of the flesh and I have also updated / fixed / generated the tests in TestLoop where I have checked that Fortran and C code (which I have not touched) produce the same result.

    You can use them via:

    cd repos/flesh
    git fetch
    git checkout rhaas/loops
    cd ../repos/cactustest
    git fetch
    git checkout rhaas/loops
    

    then build (no need to clean).

    You will have to take care of making your loop variables private on your own (unless you trust OpenMP + Fortran to be smart enough to do so for you, I believe the standard claims is should be... but that may only apply to the loop directly following the "omp do" directive) and use CCTK_LOOP3_XXX_OMP_PRIVATE(name). See TestLoopFortran.F90 for examples (the default(none) is purely optional and just helped me debug things).

    So basically:

    !$OMP PARALLEL private(i,j,k) CCTK_LOOP3_INT_OMP_PRIVATE(int3)
    CCTK_LOOP3_INT(int3, i,j,k)
       ! do something
    CCTK_ENDLOOP3_INT(int3)
    !$OMP END PARALLEL
    

    The calling syntax of the PRIVATE and DECLARE macros may still change. Right now the C LOOP macros internally declare the loop and normal variables (i, ni) while my current set of Fortran DECLARE and PRIVATE variables do not declare them or make them private (they cannot, they do not now know their name).

    So the options are to have either (1):

    integer :: i,ni
    CCTK_LOOP1_DECLARE(foo)
    !$OMP PARALLEL private(i,ni) CCTK_LOOP1_BND_OMP_PRIVATE(int1)
    CCTK_LOOP1_INT(int1, i,ni)
       ! do something
    CCTK_ENDLOOP1_INT(int1)
    !$OMP END PARALLEL
    

    or (2):

    CCTK_LOOP1_DECLARE(foo, i,ni)
    !$OMP PARALLEL CCTK_LOOP1_BND_OMP_PRIVATE(int1,i,ni)
    CCTK_LOOP1_INT(int1, i,ni)
       ! do something
    CCTK_ENDLOOP1_INT(int1)
    !$OMP END PARALLEL
    

    ie carry a couple of extra arguments (2*dim) in the DECLARE and PRIVATE macros but rest assured that all variables are properly declared private no matter what.

    I believe LoopControl went with (2) but Cactus' cctk_Loop.h (which was broken before as far as I can tell) seems to have aimed for (1).

    Having just implemented (1) I am actually leaning towards (2) right now, but then I do not actually use the LOOP macros in Fortran.

  2. Roland Haas reporter

    I have implemented a version (1.5) which is halfway in between (1) and (2) ie requires that users declare i,j,k and ni,nj,nk themselves but list them in the OMP_PRIVATE etc macros. This means that one can use “nice” names for the user visible loop variables even when there are multiple loops, which is not possible when all variables are declared by the DECLARE_XXX macros and still has some safety checks in place ensuring that the variables are marked as private to OpenMP. Example code would be in CactusTest/TestLoop in the rhaas/loops branch:

    subroutine TestLoopFortran_intbnd(CCTK_ARGUMENTS)
      implicit none
      DECLARE_CCTK_ARGUMENTS
      DECLARE_CCTK_FUNCTIONS
      DECLARE_CCTK_PARAMETERS
    
      CCTK_LOOP3_INTBND_DECLARE(intbnd3)
      integer   :: i,j,k
      integer   :: n_i,n_j,n_k
      CCTK_REAL :: fsum
    
      call CCTK_INFO("TestLoopFortran_intbnd")
      fsum = 0
      !$OMP PARALLEL &
      !$OMP default(none) shared(cctkgh, cctk_ash, cctk_lsh, cctk_tile_min, cctk_tile_max, r) &
      !$OMP reduction(+: fsum) &
      !$omp CCTK_LOOP3_INTBND_OMP_PRIVATE(intbnd3, i,j,k, n_i,n_j,n_k)
      CCTK_LOOP3_INTBND(intbnd3, i,j,k, n_i,n_j,n_k)
         fsum = fsum + r(i,j,k)
      CCTK_ENDLOOP3_INTBND(intbnd3)
      !$OMP END PARALLEL
      fsum_intbnd = fsum
    end subroutine TestLoopFortran_intbnd
    

  3. Log in to comment