Issues link against Kadath thorns and Intel MKL due to link ordering

[tootle]

A recurring issue that keeps coming up with the KadathThorn/Importer is centered around linking when using intel MKL.  It is worth noting that this is not restricted to the ETK, but I had not been able to reproduce the error myself until now.

The problem is resolved if I specify all of the dependencies of Kadath in a specific order in the final build command that outputs the cactus executable. e.g, adding the following directly after -lkadath:

-lgsl -lgslcblas -lfftw3 -L${MKLROOT}/lib/ -Wl,--start-group -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lmkl_blacs_intelmpi_lp64 -Wl,--end-group -lgomp -lpthread -lmkl_scalapack_lp64 -lm -ldl

results in a working executable on the local HPC in Idaho. Excluding this link order or adding it somewhere else in the command other than directly after -lkadath will result in an executable that yields undefined behavior. Note: the undefined behavior leads to a crash during ID import from all known cases (in and out of the ETK) where this problem has come up.

Solution:

1. Ability to specify a controlled link order in the options.cfg file (LDFLAGS does not work)
2. Ability to specify a controlled link order during the build script of KadathThorn

‌

[tootle]

@Erik Schnetter @Roland Haas Do you have ideas for either solution?

[Roland Haas]

Is dislike both :-) But, as @Steven R. Brandt , will be happy to confirm, that is my default answer and may well change.

Let me think about this a bit. I would have hoped using LDFLAGS would have worked, but this does not seem to be the case for you.

[Steven R. Brandt]

I’ve had trouble controlling the linking process on multiple occasions. It would be nice to have more control over it.

[tootle]

@Roland Haas any updates on whether or not this will be possible to do? Until this is resolved there's no point in pushing further updates to the importer as MKL is prevalent on HPC systems.

[Roland Haas]

How about adding ${FFTW3_LIBS} to KadathThorn's LIBS output in detect.sh ? Like HDF5 does:

          HDF5_LIB_DIRS="$HDF5_LIB_DIRS $MPI_LIB_DIRS"
          HDF5_INC_DIRS="$HDF5_INC_DIRS $MPI_INC_DIRS"
          HDF5_LIBS="$HDF5_LIBS $MPI_LIBS"

That lets you ensure that FFTW3_LIBS is right after Kadath’s. This should work for static linking. For dynamic linking adding it to LD should work. So it’s a cluster specific solution (at least for LD) but so is this issue you are encountering.

[Roland Haas]

Testing this I can nudge things from (the not correct):

g++ -o "/data/rhaas/postdoc/gr/cactus/ET_trunk/exe/cactus_fuka"  -fopenmp -Wall -g -std=gnu++17 -O2 -fopenmp -Wall -rdynamic  "/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/fuka/datestamp.o" -Wl,--whole-archive -L/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/fuka/lib -lthorn_ADMBase -lthorn_ADMCoupling -lthorn_ADMMacros -lthorn_AEILocalInterp -lthorn_AHFinderDirect -lthorn_BLAS -lthorn_Boost -lthorn_Boundary -lthorn_Carpet -lthorn_CarpetIOHDF5 -lthorn_CarpetIOScalar -lthorn_CarpetInterp -lthorn_CarpetLib -lthorn_CarpetMask -lthorn_CarpetReduce -lthorn_CarpetSlab -lthorn_CartGrid3D -lthorn_CoordBase -lthorn_CoordGauge -lthorn_CycleClock -lthorn_FFTW3 -lthorn_GSL -lthorn_HDF5 -lthorn_HydroBase -lthorn_IOASCII -lthorn_IOBasic -lthorn_IOUtil -lthorn_InitBase -lthorn_KadathImporter -lthorn_KadathThorn -lthorn_LAPACK -lthorn_LocalInterp2 -lthorn_LocalReduce -lthorn_LoopControl -lthorn_MPI -lthorn_MoL -lthorn_NaNChecker -lthorn_PUGH -lthorn_SpaceMask -lthorn_SphericalSurface -lthorn_StaticConformal -lthorn_SymBase -lthorn_SystemTopology -lthorn_Time -lthorn_Timers -lthorn_TmunuBase -lthorn_Vectors -lthorn_hwloc -lthorn_zlib -lthorn_Cactus -lthorn_CactusBindings -Wl,--no-whole-archive -L/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/fuka/scratch/external/KadathThorn/lib -L/opt/intel/oneapi/mkl/2021.3.0/lib/intel64 -L/opt/intel/oneapi/mkl/2021.3.0/lib/intel64 -L/usr/lib/x86_64-linux-gnu/hdf5/openmpi -L/usr/lib/x86_64-linux-gnu/openmpi/lib -L/usr/lib/x86_64-linux-gnu/openmpi/lib -L/usr/lib/x86_64-linux-gnu/openmpi/lib -L/usr/lib/gcc/x86_64-linux-gnu/13 -Wl,-rpath,/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/fuka/scratch/external/KadathThorn/lib -Wl,-rpath,/opt/intel/oneapi/mkl/2021.3.0/lib/intel64 -Wl,-rpath,/opt/intel/oneapi/mkl/2021.3.0/lib/intel64 -Wl,-rpath,/usr/lib/x86_64-linux-gnu/hdf5/openmpi -Wl,-rpath,/usr/lib/x86_64-linux-gnu/openmpi/lib -Wl,-rpath,/usr/lib/x86_64-linux-gnu/openmpi/lib -Wl,-rpath,/usr/lib/x86_64-linux-gnu/openmpi/lib -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/13 -lkadath -lmkl_scalapack_lp64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lmkl_scalapack_lp64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lgsl -lgslcblas -lm -lfftw3 -lboost_filesystem -lboost_system -lhdf5hl_fortran -lhdf5_fortran -lhdf5_hl -lhdf5 -lmpi -lhwloc -levent_core -levent_pthreads -lm -lz -lsz -lz -lmpi_cxx -lmpi -lm -lz -lmpi_cxx -lmpi -lhwloc -lm -ludev -lpthread -lz -lm -lm -lm -lm -lm -lm -lm -lm -lcrypt -lgfortran

to the (more correct):

g++ -o "/data/rhaas/postdoc/gr/cactus/ET_trunk/exe/cactus_fuka"  -fopenmp -Wall -g -std=gnu++17 -O2 -fopenmp -Wall -rdynamic  "/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/fuka/datestamp.o" -Wl,--whole-archive -L/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/fuka/lib -lthorn_ADMBase -lthorn_ADMCoupling -lthorn_ADMMacros -lthorn_AEILocalInterp -lthorn_AHFinderDirect -lthorn_BLAS -lthorn_Boost -lthorn_Boundary -lthorn_Carpet -lthorn_CarpetIOHDF5 -lthorn_CarpetIOScalar -lthorn_CarpetInterp -lthorn_CarpetLib -lthorn_CarpetMask -lthorn_CarpetReduce -lthorn_CarpetSlab -lthorn_CartGrid3D -lthorn_CoordBase -lthorn_CoordGauge -lthorn_CycleClock -lthorn_FFTW3 -lthorn_GSL -lthorn_HDF5 -lthorn_HydroBase -lthorn_IOASCII -lthorn_IOBasic -lthorn_IOUtil -lthorn_InitBase -lthorn_KadathImporter -lthorn_KadathThorn -lthorn_LAPACK -lthorn_LocalInterp2 -lthorn_LocalReduce -lthorn_LoopControl -lthorn_MPI -lthorn_MoL -lthorn_NaNChecker -lthorn_PUGH -lthorn_SpaceMask -lthorn_SphericalSurface -lthorn_StaticConformal -lthorn_SymBase -lthorn_SystemTopology -lthorn_Time -lthorn_Timers -lthorn_TmunuBase -lthorn_Vectors -lthorn_hwloc -lthorn_zlib -lthorn_Cactus -lthorn_CactusBindings -Wl,--no-whole-archive -L/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/fuka/scratch/external/KadathThorn/lib -L/opt/intel/oneapi/mkl/2021.3.0/lib/intel64 -L/opt/intel/oneapi/mkl/2021.3.0/lib/intel64 -L/opt/intel/oneapi/mkl/2021.3.0/lib/intel64 -L/usr/lib/x86_64-linux-gnu/hdf5/openmpi -L/usr/lib/x86_64-linux-gnu/openmpi/lib -L/usr/lib/x86_64-linux-gnu/openmpi/lib -L/usr/lib/x86_64-linux-gnu/openmpi/lib -L/usr/lib/gcc/x86_64-linux-gnu/13 -Wl,-rpath,/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/fuka/scratch/external/KadathThorn/lib -Wl,-rpath,/opt/intel/oneapi/mkl/2021.3.0/lib/intel64 -Wl,-rpath,/opt/intel/oneapi/mkl/2021.3.0/lib/intel64 -Wl,-rpath,/opt/intel/oneapi/mkl/2021.3.0/lib/intel64 -Wl,-rpath,/usr/lib/x86_64-linux-gnu/hdf5/openmpi -Wl,-rpath,/usr/lib/x86_64-linux-gnu/openmpi/lib -Wl,-rpath,/usr/lib/x86_64-linux-gnu/openmpi/lib -Wl,-rpath,/usr/lib/x86_64-linux-gnu/openmpi/lib -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/13 -lkadath -lfftw3 -lgsl -lgslcblas -lm -lmkl_scalapack_lp64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lboost_filesystem -lboost_system -lmkl_scalapack_lp64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lmkl_scalapack_lp64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lgsl -lgslcblas -lm -lfftw3 -lboost_filesystem -lboost_system -lhdf5hl_fortran -lhdf5_fortran -lhdf5_hl -lhdf5 -lmpi -lhwloc -levent_core -levent_pthreads -lm -lz -lsz -lz -lmpi_cxx -lmpi -lm -lz -lmpi_cxx -lmpi -lhwloc -lm -ludev -lpthread -lz -lm -lm -lm -lm -lm -lm -lm -lm -lcrypt -lgfortran

Change to KadathThorn:

diff --git a/src/detect.sh b/src/detect.sh
index 9f80ed6..d5be1a0 100755
--- a/src/detect.sh
+++ b/src/detect.sh
@@ -89,9 +89,10 @@ echo "END MESSAGE"
 # Pass options to Cactus
 echo "BEGIN MAKE_DEFINITION"
 echo "KADATH_DIR      = ${KADATH_DIR}"
-echo "KADATH_INC_DIRS = ${KADATH_INC_DIRS}"
-echo "KADATH_LIB_DIRS = ${KADATH_LIB_DIRS}"
-echo "KADATH_LIBS     = ${KADATH_LIBS}"
+echo "KADATH_INC_DIRS = ${KADATH_INC_DIRS} ${FFTW3_INC_DIRS} ${GSL_INC_DIRS} ${LAPACK_INC_DIRS} ${BOOST_INC_DIRS}"
+echo "KADATH_LIB_DIRS = ${KADATH_LIB_DIRS}" ${FFTW3_LIB_DIRS} ${GSL_LIB_DIRS} ${LAPACK_LIB_DIRS} ${BOOST_LIB_DIRS}
+# keep the order like this, since MKL (LAPACK) can contain a (partial) FFTW3 which conflicts
+echo "KADATH_LIBS     = ${KADATH_LIBS} ${FFTW3_LIBS} ${GSL_LIBS} ${LAPACK_LIBS} ${BOOST_LIBS}"
 echo "END MAKE_DEFINITION"

 echo 'INCLUDE_DIRECTORY $(KADATH_INC_DIRS)'

‌

[Roland Haas]

Please review.

[tootle]

@Roland Haas Indeed that worked! Here is the final version with minor modification:

diff --git a/src/detect.sh b/src/detect.sh
index 9f80ed6..3fe691b 100755
--- a/src/detect.sh
+++ b/src/detect.sh
@@ -82,16 +82,18 @@ KADATH_LIBS="kadath"

 echo "BEGIN MESSAGE"
 echo "KadathThorn: Detect settings - KADATH_DIR : ${KADATH_DIR}"
-echo "KadathThorn: Detect settings - KADATH_LIB_DIRS : ${KADATH_LIB_DIRS}"
-echo "KadathThorn: Detect settings - KADATH_LIBS : ${KADATH_LIBS}"
+echo "KadathThorn: Detect settings - KADATH_INC_DIRS : ${KADATH_INC_DIRS}"
+echo "KadathThorn: Detect settings - KADATH_LIB_DIRS : ${KADATH_LIB_DIRS} ${GSL_LIB_DIRS} ${FFTW3_LIB_DIRS} ${BLAS_LIB_DIRSS} ${LAPACK_LIB_DIRS} ${BOOST_LIB_DIRS}"
+echo "KadathThorn: Detect settings - KADATH_LIBS : ${KADATH_LIBS} ${GSL_LIBS} ${FFTW3_LIBS} ${BLAS_LIBS} ${LAPACK_LIBS} ${BOOST_LIBS}"
 echo "END MESSAGE"

 # Pass options to Cactus
 echo "BEGIN MAKE_DEFINITION"
 echo "KADATH_DIR      = ${KADATH_DIR}"
 echo "KADATH_INC_DIRS = ${KADATH_INC_DIRS}"
-echo "KADATH_LIB_DIRS = ${KADATH_LIB_DIRS}"
-echo "KADATH_LIBS     = ${KADATH_LIBS}"
+echo "KADATH_LIB_DIRS = ${KADATH_LIB_DIRS} ${GSL_LIB_DIRS} ${FFTW3_LIB_DIRS} ${BLAS_LIB_DIRSS} ${LAPACK_LIB_DIRS} ${BOOST_LIB_DIRS}"
+# keep the order like this, since MKL (LAPACK) can contain a (partial) FFTW3 which conflicts
+echo "KADATH_LIBS     = ${KADATH_LIBS} ${GSL_LIBS} ${FFTW3_LIBS} ${BLAS_LIBS} ${LAPACK_LIBS} ${BOOST_LIBS}"
 echo "END MAKE_DEFINITION"

 echo 'INCLUDE_DIRECTORY $(KADATH_INC_DIRS)'

‌

[Roland Haas]

Looks good to me. Please apply.

[Roland Haas]

This has not yet been applied, has it?

[tootle]

This is still in the dev branch as I need to test it more thoroughly with the build script. I also ran into an issue on frontier where the external libs linked against didn’t match the libs the fuka library was built with when the external libraries found the local libraries on their own. In this case FUKA was built independently of the ETK build. I think if it is built from scratch during the ET build process and I can ensure the importer functions correctly, then this will be sufficient for now. If people want to build ETK with a prebuilt FUKA, then they will need to be more careful - which I’ll add to the documentation once the details are flushed out