LORENE hard codes OpenMP options into compile options

Create issue
Issue #1125 resolved
Roland Haas created an issue

Currently LOERENE's configuration.sh contains

CXXFLAGS = ${CXXFLAGS} ${CPPFLAGS} ${CPP_OPENMP_FLAGS}

which always uses OpenMP independent of the option list's OPENMP=yes/no setting.

The attached patch duplicates (unfortunately there seems to be no way to access this variables from within a per-thorn configuration script) the logic used in Cactus' configure to set OPENMP_MODE based on OPENMP. Without it compilation of a Cactus fails at link if OpenMP is not selected:

g++ -o "/home/lovelace/trunk/Cactus/exe/cactus_sim" -DMPICH_IGNORE_CXX_SEEK -DKRANC_VECTORS  -g3 -march=native -std=gnu++0x -rdynamic -O2    "/home/lovelace/trunk/Cactus/configs/sim/datestamp.o" -L/home/lovelace/trunk/Cactus/configs/sim/lib -lthorn_Cactus -lthorn_CactusBindings -lthorn_CarpetEvolutionMask -lthorn_IOHDF5 -lthorn_IOHDF5Util -lthorn_CarpetIOScalar -lthorn_ML_WaveToy -lthorn_Meudon_Mag_NS -lthorn_HTTPDExtra -lthorn_HTTPD -lthorn_RotatingSymmetry90 -lthorn_NoExcision -lthorn_Hydro_InitExcision -lthorn_ML_BSSN_Helper -lthorn_GaugeWave -lthorn_NullPsiInt -lthorn_hwloc -lthorn_PUGHInterp -lthorn_Cartoon2D -lthorn_SetMask_SphericalSurface -lthorn_EHFinder -lthorn_EOS_IdealFluid -lthorn_ModifiedSchwarzschildBL -lthorn_Multipole -lthorn_CarpetTracker -lthorn_SphericalHarmonicRecon -lthorn_QuasiLocalMeasures -lthorn_TGRtensor -lthorn_Exact -lthorn_Periodic -lthorn_IOJpeg -lthorn_CarpetRegrid -lthorn_TestGlobalReduce -lthorn_CarpetInterp2 -lthorn_Vaidya2 -lthorn_GRHydro_InitData -lthorn_TestLoop -lthorn_LegoExcision -lthorn_ADMMass -lthorn_ADM -lthorn_CarpetSlab -lthorn_NoMPI -lthorn_PunctureTracker -lthorn_CarpetRegrid2 -lthorn_IDScalarWaveElliptic -lthorn_IDScalarWaveCXX -lthorn_IDScalarWaveC -lthorn_IDScalarWave -lthorn_ML_ADMQuantities -lthorn_ML_BSSN -lthorn_WaveToyExtra -lthorn_CarpetMask -lthorn_ML_CCZ4_Helper -lthorn_ML_CCZ4 -lthorn_AHFinderDirect -lthorn_Meudon_Bin_BH -lthorn_ADMConstraints -lthorn_ML_ADMConstraints -lthorn_PUGHSlab -lthorn_TestFortranDependencies1 -lthorn_TwoPunctures -lthorn_CalcK -lthorn_IOBasic -lthorn_NullExact -lthorn_NullConstr -lthorn_NullNews -lthorn_CarpetInterp -lthorn_IDBrillData -lthorn_Extract -lthorn_WaveBinarySource -lthorn_WaveToyFreeF90 -lthorn_WaveToyF90 -lthorn_WaveToyF77 -lthorn_WaveToyCXX -lthorn_WaveToyC -lthorn_IDAxiOddBrillBH -lthorn_CarpetIOBasic -lthorn_CarpetIOASCII -lthorn_Minkowski -lthorn_SlabTest -lthorn_Dissipation -lthorn_SphericalSurface -lthorn_EOS_Hybrid -lthorn_EOS_Polytrope -lthorn_DistortedBHIVP -lthorn_Meudon_Bin_NS -lthorn_LORENE -lthorn_LAPACK -lthorn_IDAxiBrillBH -lthorn_EllSOR -lthorn_NewRad -lthorn_IOASCII -lthorn_NullSHRExtract -lthorn_NullEvolve -lthorn_NullInterp -lthorn_NullDecomp -lthorn_NullGrid -lthorn_AHFinder -lthorn_RotatingDBHIVP -lthorn_SphericalHarmonicDecomp -lthorn_WeylScal4 -lthorn_IDConstraintViolate -lthorn_TOVSolver -lthorn_GRHydro -lthorn_TmunuBase -lthorn_SpaceMask -lthorn_EOS_Omni -lthorn_IDAnalyticBH -lthorn_ShiftedGaugeWave -lthorn_CarpetIOHDF5 -lthorn_HDF5 -lthorn_NaNChecker -lthorn_PUGHReduce -lthorn_PUGH -lthorn_CarpetReduce -lthorn_Carpet -lthorn_CarpetLib -lthorn_ADMAnalysis -lthorn_zlib -lthorn_libjpeg -lthorn_Time -lthorn_TestTypes -lthorn_TestTimers -lthorn_TestTable -lthorn_TestStrings -lthorn_TestSchedule -lthorn_TestReduce -lthorn_TestPar -lthorn_TestMath -lthorn_TestLocalReduce -lthorn_TestInclude2 -lthorn_TestInclude1 -lthorn_TestFreeF90 -lthorn_TestFpointerNULL -lthorn_TestFortranDependencies2 -lthorn_TestCoordinates -lthorn_TestComplex -lthorn_TestArrays -lthorn_TerminationTrigger -lthorn_TATelliptic -lthorn_SystemStatistics -lthorn_SymBase -lthorn_SummationByParts -lthorn_Socket -lthorn_ReflectionSymmetry -lthorn_OpenSSL -lthorn_NullVars -lthorn_Norms -lthorn_Nice -lthorn_MoL -lthorn_ML_WaveToy_Test -lthorn_ML_CCZ4_Test -lthorn_ML_BSSN_Test -lthorn_LocalReduce -lthorn_LocalInterp -lthorn_InterpToArray -lthorn_Fortran -lthorn_FFTW3 -lthorn_EllBase -lthorn_EinsteinExact_Test -lthorn_EOS_Base -lthorn_CoordGauge -lthorn_Constants -lthorn_BLAS -lthorn_AEILocalInterp -lthorn_ADMMacros -lthorn_ADMCoupling -lthorn_Formaline -lthorn_RotatingSymmetry180 -lthorn_Slab -lthorn_Hydro_Analysis -lthorn_HydroBase -lthorn_InitBase -lthorn_KerrSchild -lthorn_LoopControl -lthorn_Vectors -lthorn_GSL -lthorn_GenericFD -lthorn_IDLinearWaves -lthorn_IDFileADM -lthorn_IOUtil -lthorn_StaticConformal -lthorn_ADMBase -lthorn_Boundary -lthorn_TimerReport -lthorn_MPI -lthorn_Noise -lthorn_CartGrid3D -lthorn_CoordBase -lthorn_Cactus -lthorn_CactusBindings -lthorn_Formaline-cactus-source -L/home/lovelace/trunk/Cactus/configs/sim/scratch/external/hwloc/lib -L/home/lovelace/trunk/Cactus/configs/sim/scratch/external/LORENE/Lib -L/home/lovelace/trunk/Cactus/configs/sim/scratch/external/LAPACK -L/home/lovelace/trunk/Cactus/configs/sim/scratch/external/HDF5/lib -L/home/lovelace/trunk/Cactus/configs/sim/scratch/external/zlib/lib -L/home/lovelace/trunk/Cactus/configs/sim/scratch/external/libjpeg/lib -L/home/lovelace/trunk/Cactus/configs/sim/scratch/external/FFTW3/lib -L/home/lovelace/trunk/Cactus/configs/sim/scratch/external/BLAS -L/home/lovelace/trunk/Cactus/configs/sim/scratch/external/GSL/lib -L/usr/lib -Wl,-rpath,/home/lovelace/trunk/Cactus/configs/sim/scratch/external/hwloc/lib -Wl,-rpath,/home/lovelace/trunk/Cactus/configs/sim/scratch/external/LORENE/Lib -Wl,-rpath,/home/lovelace/trunk/Cactus/configs/sim/scratch/external/LAPACK -Wl,-rpath,/home/lovelace/trunk/Cactus/configs/sim/scratch/external/HDF5/lib -Wl,-rpath,/home/lovelace/trunk/Cactus/configs/sim/scratch/external/zlib/lib -Wl,-rpath,/home/lovelace/trunk/Cactus/configs/sim/scratch/external/libjpeg/lib -Wl,-rpath,/home/lovelace/trunk/Cactus/configs/sim/scratch/external/FFTW3/lib -Wl,-rpath,/home/lovelace/trunk/Cactus/configs/sim/scratch/external/BLAS -Wl,-rpath,/home/lovelace/trunk/Cactus/configs/sim/scratch/external/GSL/lib -Wl,-rpath,/usr/lib -lpthread -lhwloc -llorene_export -llorene -llorenef77 -llapack -lhdf5_hl_cpp -lhdf5_cpp -lhdf5hl_fortran -lhdf5_fortran -lhdf5_hl -lhdf5 -lz -lm -lz -ljpeg -lfftw3 -lblas -lgsl -lgslcblas -lm -lmpich -lfmpich -lcrypt -ldl -lgfortran
/home/lovelace/trunk/Cactus/configs/sim/scratch/external/LORENE/Lib/liblorene_export.a(bin_ns_aux.o): In function `Bin_NS':
/home/lovelace/trunk/Cactus/configs/sim/scratch/build/LORENE/Lorene/Export/C++/Source/bin_ns_aux.C:333: undefined reference to `GOMP_parallel_start'
/home/lovelace/trunk/Cactus/configs/sim/scratch/build/LORENE/Lorene/Export/C++/Source/bin_ns_aux.C:333: undefined reference to `GOMP_parallel_end'
/home/lovelace/trunk/Cactus/configs/sim/scratch/external/LORENE/Lib/liblorene_export.a(bin_ns_aux.o): In function `Bin_NS::Bin_NS(int, double const*, double const*, double const*, char const*) [clone ._omp_fn.0]':
/home/lovelace/trunk/Cactus/configs/sim/scratch/build/LORENE/Lorene/Export/C++/Source/bin_ns_aux.C:334: undefined reference to `omp_get_num_threads'
/home/lovelace/trunk/Cactus/configs/sim/scratch/build/LORENE/Lorene/Export/C++/Source/bin_ns_aux.C:334: undefined reference to `omp_get_thread_num'
/usr/lib/libmpich.so: undefined reference to `MPL_trid'
/usr/lib/libmpich.so: undefined reference to `MPL_trvalid'
/usr/lib/libmpich.so: undefined reference to `MPL_env2int'
/usr/lib/libmpich.so: undefined reference to `MPL_trrealloc'
/usr/lib/libmpich.so: undefined reference to `MPL_trspace'
/usr/lib/libmpich.so: undefined reference to `MPL_trDebugLevel'
/usr/lib/libmpich.so: undefined reference to `MPL_TrSetMaxMem'
/usr/lib/libmpich.so: undefined reference to `MPL_trlevel'
/usr/lib/libmpich.so: undefined reference to `MPL_trmalloc'
/usr/lib/libmpich.so: undefined reference to `MPL_putenv'
/usr/lib/libmpich.so: undefined reference to `MPL_env2bool'
/usr/lib/libmpich.so: undefined reference to `MPL_env2range'
/usr/lib/libmpich.so: undefined reference to `MPL_trcalloc'
/usr/lib/libmpich.so: undefined reference to `MPL_trfree'
/usr/lib/libmpich.so: undefined reference to `MPL_env2str'
/usr/lib/libmpich.so: undefined reference to `MPL_trstrdup'
/usr/lib/libmpich.so: undefined reference to `MPL_trdump'
/usr/lib/libmpich.so: undefined reference to `MPL_trinit'
collect2: ld returned 1 exit status
make[1]: *** [/home/lovelace/trunk/Cactus/exe/cactus_sim] Error 1
make[1]: Leaving directory `/home/lovelace/trunk/Cactus'
make: *** [sim] Error 2

The MPL errors are due to a missing link library for libmpl in the Ubuntu.cfg file (separate ticket).

Also note that both the old and the patched version actually use "CPP_OPENMP_FLAGS" not "CXX_OPENMP_FLAGS" in the CXX options. This looks like an error to me.

Keyword: LORENE

Comments (8)

  1. Erik Schnetter
    • removed comment

    CPP_OPENMP_FLAGS and CXX_OPENMP_FLAGS are different. In this case, probably both are wanted.

    I don't see why someone would need to explicitly add OPENMP flags to CXXFLAGS. I don't see how the code could access a version of CXXFLAGS that doesn't already have CPP_OPENMP_FLAGS in it. Can this logic just be removed?

  2. Roland Haas reporter
    • removed comment

    I suspect that the CXXFLAGS that the script sees is the raw one from the options list. Namely when I tested various way to get OPENMP flags to work, I could have an options list that said OPENMP=yes but CXXFLAGS would not see the openmp options unless I explicitly added them. I just tried this and that seems indeed to be the case.

    I agree though that a much nicer way of handling this would be to pass all the defines that Cactus' configure defines to the individual thorn's configure.sh scripts. Ie present the fully expanded CXX_FLAGS (incl. optimization options etc) to the per-thorn configuration scripts.

    LORENE seems to actually use OpenMP at least rev 23 says "add some openmp support to lorene evaluation onto list of points". Given how slow LORENE is otherwise this is highly desirable for OpenMP eneabled builds.

  3. Erik Schnetter
    • changed status to open
    • removed comment

    In this case I suggest to apply the patch, but to include both CPP_OPENMP_FLAGS and CXX_OPENMP_FLAGS.

  4. Roland Haas reporter
    • changed status to open

    #1345 makes this no longer necessary and the combination of

    • git hash 691f01bf "Cactus: remove support for F77 flags from linux architecture" of cactus
    • git hash 9f46397b "Cactus: Compile all Fortran files as Fortran 90 files" of cactus
    • git hash e8f2db41 "Drop explicit support for Fortran 77 in Cactus" of cactus
    • git hash 01c74e2 "generic: add F90 optimization options, remove all F77 options" of simfactory2

    broke the removal due to F77_OPENMP_FLAGS not being set from F90_OPENMP_FLAGS. A fix for the broken stuff is fairly simple (and I have a patch).

  5. Log in to comment