Problem with building on Ubuntu 14.04

[anonymous]

Hello,

Following the instructions on the page: https://docs.einsteintoolkit.org/et-docs/Simplified_Tutorial_for_New_Users I failed to compile ET on the Ubuntu 14.04 In the file ubuntu.cfg paths to mpi libs or includes do not exist, The MPICH2 package is installed (sudo apt-get install mpich2)

It could be better to change mpich to openmpi?

Keyword: MPI
Keyword: build

[Roland Haas]

Maxim,

you seem to have some leftover options in /home/bmv/.cactus/config that point to openmpi. If you would like to continue using ubuntu.cfg you either need to remove those options from the file or install OpenMPI (since the options in /home/bmv/.cactus/config take precendence over those in ubuntu.cfg).

[anonymous]

Dear Rhaas,

Thank you for your swift response. I have installed OpenMPI. As you recommended I rename config file and run following commands:

mv ~/.cactus/config ~/.cactus/config_old
./simfactory/bin/sim cleanup
./simfactory/bin/sim setup --optionlist=ubuntu.cfg --runscript debian.sh
./simfactory/bin/sim build --thornlist=manifest/einsteintoolkit.th 2&> sim_2a_log.txt

After that operations I got error message:

Creating cactus_2 in /home/bmv/utils/soft/ET/Cactus/exe from EinsteinAnalysis/ADMAnalysis EinsteinBase/ADMBase EinsteinBase/ADMCoupling EinsteinBase/ADMMacros AEIThorns/ADMMass AEIThorns/AEILocalInterp EinsteinAnalysis/AHFinder EinsteinAnalysis/AHFinderDirect ExternalLibraries/BLAS CactusBase/Boundary EinsteinAnalysis/CalcK Carpet/Carpet Carpet/CarpetEvolutionMask Carpet/CarpetIOASCII Carpet/CarpetIOBasic Carpet/CarpetIOHDF5 Carpet/CarpetIOScalar Carpet/CarpetIntegrateTest Carpet/CarpetInterp Carpet/CarpetInterp2 Carpet/CarpetLib Carpet/CarpetMask Carpet/CarpetProlongateTest Carpet/CarpetReduce Carpet/CarpetRegrid Carpet/CarpetRegrid2 Carpet/CarpetRegridTest Carpet/CarpetSlab Carpet/CarpetTracker CactusBase/CartGrid3D CactusNumerical/Cartoon2D EinsteinBase/Constants CactusBase/CoordBase EinsteinBase/CoordGauge Carpet/CycleClock CactusExamples/DemoInterp CactusNumerical/Dissipation EinsteinInitialData/DistortedBHIVP EinsteinAnalysis/EHFinder EinsteinBase/EOS_Base EinsteinEOS/EOS_Hybrid EinsteinEOS/EOS_IdealFluid EinsteinEOS/EOS_Omni EinsteinEOS/EOS_Polytrope EinsteinExact/EinsteinExact_Test CactusElliptic/EllBase CactusElliptic/EllSOR EinsteinInitialData/Exact EinsteinAnalysis/Extract ExternalLibraries/FFTW3 CactusExamples/FleshInfo CactusUtils/Formaline CactusBase/Fortran EinsteinEvolve/GRHydro EinsteinEvolve/GRHydro_InitData ExternalLibraries/GSL EinsteinExact/GaugeWave KrancNumericalTools/GenericFD ExternalLibraries/HDF5 CactusConnect/HTTPD CactusConnect/HTTPDExtra CactusExamples/HelloWorld Carpet/HighOrderWaveTest EinsteinBase/HydroBase EinsteinAnalysis/Hydro_Analysis EinsteinInitialData/Hydro_InitExcision EinsteinInitialData/IDAnalyticBH EinsteinInitialData/IDAxiBrillBH EinsteinInitialData/IDAxiOddBrillBH EinsteinInitialData/IDBrillData EinsteinInitialData/IDConstraintViolate EinsteinInitialData/IDFileADM EinsteinInitialData/IDLinearWaves CactusWave/IDScalarWave CactusWave/IDScalarWaveC CactusWave/IDScalarWaveCXX CactusWave/IDScalarWaveElliptic CactusExamples/IDWaveMoL CactusBase/IOASCII CactusBase/IOBasic CactusPUGHIO/IOHDF5 CactusPUGHIO/IOHDF5Util CactusIO/IOJpeg CactusBase/IOUtil CactusBase/InitBase CactusNumerical/InterpToArray EinsteinExact/KerrSchild ExternalLibraries/LAPACK ExternalLibraries/LORENE CactusNumerical/LocalInterp CactusNumerical/LocalReduce Carpet/LoopControl McLachlan/ML_ADMConstraints McLachlan/ML_ADMQuantities McLachlan/ML_BSSN McLachlan/ML_BSSN_Helper McLachlan/ML_BSSN_Test McLachlan/ML_CCZ4 McLachlan/ML_CCZ4_Helper McLachlan/ML_CCZ4_Test McLachlan/ML_WaveToy McLachlan/ML_WaveToy_Test ExternalLibraries/MPI CactusUtils/MemSpeed EinsteinInitialData/Meudon_Bin_BH EinsteinInitialData/Meudon_Bin_NS EinsteinInitialData/Meudon_Mag_NS EinsteinExact/Minkowski CactusNumerical/MoL EinsteinExact/ModifiedSchwarzschildBL EinsteinAnalysis/Multipole CactusUtils/NaNCatcher CactusUtils/NaNChecker EinsteinEvolve/NewRad CactusUtils/Nice EinsteinInitialData/NoExcision CactusUtils/NoMPI CactusNumerical/Noise CactusNumerical/Norms PITTNullCode/NullConstr PITTNullCode/NullDecomp PITTNullCode/NullEvolve PITTNullCode/NullExact PITTNullCode/NullGrid PITTNullCode/NullInterp PITTNullCode/NullNews PITTNullCode/NullPsiInt PITTNullCode/NullSHRExtract PITTNullCode/NullVars ExternalLibraries/OpenSSL EinsteinAnalysis/Outflow ExternalLibraries/PAPI CactusPUGH/PUGH CactusPUGH/PUGHInterp CactusPUGH/PUGHReduce CactusPUGH/PUGHSlab CactusNumerical/Periodic LSUThorns/PeriodicCarpet CactusExamples/Poisson AEIThorns/PunctureTracker LSUThorns/QuasiLocalMeasures Carpet/ReductionTest Carpet/ReductionTest2 Carpet/ReductionTest3 CactusNumerical/ReflectionSymmetry Carpet/RegridSyncTest EinsteinInitialData/RotatingDBHIVP CactusNumerical/RotatingSymmetry180 CactusNumerical/RotatingSymmetry90 CactusExamples/SampleBoundary CactusExamples/SampleIO EinsteinUtils/SetMask_SphericalSurface EinsteinExact/ShiftedGaugeWave CactusNumerical/Slab CactusNumerical/SlabTest CactusConnect/Socket CactusNumerical/SpaceMask PITTNullCode/SphericalHarmonicDecomp PITTNullCode/SphericalHarmonicRecon CactusNumerical/SphericalSurface EinsteinBase/StaticConformal LSUThorns/SummationByParts CactusBase/SymBase AEIThorns/SystemStatistics CactusElliptic/TATelliptic EinsteinUtils/TGRtensor EinsteinInitialData/TOVSolver CactusUtils/TerminationTrigger CactusTest/TestArrays Carpet/TestCarpetGridInfo CactusTest/TestComplex CactusTest/TestCoordinates CactusTest/TestFortranCrayPointers CactusTest/TestFortranDependencies1 CactusTest/TestFortranDependencies2 CactusTest/TestFpointerNULL CactusTest/TestFreeF90 CactusTest/TestGlobalReduce CactusTest/TestInclude1 CactusTest/TestInclude2 CactusNumerical/TestLocalReduce CactusTest/TestLoop Carpet/TestLoopControl CactusTest/TestMath CactusTest/TestMoL CactusTest/TestPar CactusTest/TestReduce CactusTest/TestSchedule CactusTest/TestStrings CactusTest/TestTable CactusTest/TestTimers CactusTest/TestTypes CactusBase/Time CactusExamples/TimerInfo CactusUtils/TimerReport Carpet/Timers EinsteinBase/TmunuBase AEIThorns/Trigger EinsteinInitialData/TwoPunctures EinsteinExact/Vaidya2 LSUThorns/Vectors CactusWave/WaveBinarySource CactusExamples/WaveMoL CactusExamples/WaveToy1DF77 CactusExamples/WaveToy2DF77 CactusWave/WaveToyC CactusWave/WaveToyCXX CactusWave/WaveToyExtra CactusWave/WaveToyF77 CactusWave/WaveToyF90 CactusWave/WaveToyFreeF90 EinsteinAnalysis/WeylScal4 ExternalLibraries/hwloc ExternalLibraries/libjpeg ExternalLibraries/zlib
Formaline: Creating git master repository...
Formaline: Creating git local repository...
Formaline: Pushing source tree to master git repository...
Formaline: Pushing to local repository /home/bmv/utils/soft/ET/Cactus/../CactusSourceJar.git...
Formaline: Optimising git repository (slow only the first time)...
Formaline: Done.
Formaline: (Formaline has finished updating the git repositories.
Formaline:  If the build process seems to hang, some other command
Formaline:  is still running, e.g. the final link stage.)
/home/bmv/utils/soft/ET/Cactus/configs/2/scratch/external/hwloc/lib/libhwloc.a(topology-linux.o): In function `hwloc_linux_set_area_membind':
/home/bmv/utils/soft/ET/Cactus/configs/2/scratch/build/hwloc/hwloc-1.7.2/src/topology-linux.c:1230: undefined reference to `mbind'
/home/bmv/utils/soft/ET/Cactus/configs/2/scratch/build/hwloc/hwloc-1.7.2/src/topology-linux.c:1211: undefined reference to `mbind'
/home/bmv/utils/soft/ET/Cactus/configs/2/scratch/external/hwloc/lib/libhwloc.a(topology-linux.o): In function `hwloc_linux_set_thisthread_membind':
/home/bmv/utils/soft/ET/Cactus/configs/2/scratch/build/hwloc/hwloc-1.7.2/src/topology-linux.c:1289: undefined reference to `migrate_pages'
/home/bmv/utils/soft/ET/Cactus/configs/2/scratch/build/hwloc/hwloc-1.7.2/src/topology-linux.c:1301: undefined reference to `set_mempolicy'
/home/bmv/utils/soft/ET/Cactus/configs/2/scratch/build/hwloc/hwloc-1.7.2/src/topology-linux.c:1278: undefined reference to `set_mempolicy'
/home/bmv/utils/soft/ET/Cactus/configs/2/scratch/external/hwloc/lib/libhwloc.a(topology-linux.o): In function `hwloc_linux_find_kernel_max_numnodes':
/home/bmv/utils/soft/ET/Cactus/configs/2/scratch/build/hwloc/hwloc-1.7.2/src/topology-linux.c:1334: undefined reference to `get_mempolicy'
/home/bmv/utils/soft/ET/Cactus/configs/2/scratch/external/hwloc/lib/libhwloc.a(topology-linux.o): In function `hwloc_linux_get_area_membind':
/home/bmv/utils/soft/ET/Cactus/configs/2/scratch/build/hwloc/hwloc-1.7.2/src/topology-linux.c:1432: undefined reference to `get_mempolicy'
/home/bmv/utils/soft/ET/Cactus/configs/2/scratch/external/hwloc/lib/libhwloc.a(topology-linux.o): In function `hwloc_linux_get_thisthread_membind':
/home/bmv/utils/soft/ET/Cactus/configs/2/scratch/build/hwloc/hwloc-1.7.2/src/topology-linux.c:1379: undefined reference to `get_mempolicy'
collect2: error: ld returned 1 exit status
make[1]: *** [/home/bmv/utils/soft/ET/Cactus/exe/cactus_2] Error 1
make: *** [2] Error 2

Nevertheless the example code start to work

# modify parameter file for smaller memory footprint using sed
 sed '/CoordBase::d[xyz]/s/8/12/' <par/static_tov.par >par/static_tov_small.par
 # start simulation, watch log output
 ./simfactory/bin/sim submit static_tov --parfile=par/static_tov_small.par --procs=1 --walltime=8:0:0
 ./simfactory/bin/sim show-output --follow static_tov

Could you please clarify that mean this Error massage? Should I ignore it?

[Roland Haas]

The error you see is caused by the code no being linked against the NUMA library. Since hwloc's compilation scripts changed a bit between the last release (Herschel) and the current development version, the first we'd need to know if which version of the ET you are trying to use: Herschel or the development version? If you followed the tutorial and downloaded https://bitbucket.org/einsteintoolkit/manifest/raw/ET_2014_11/einsteintoolkit.th then you have Herschel.

Can you post the content of the file /home/bmv/utils/soft/ET/Cactus/configs/2/bindings/Configuration/Capabilities/make.HWLOC.defn please? It will show if numa is listed in HWLOC_LIBS . Also the output of dpkg --list would be helpful (as an attachement, please).

I am very surprised that it actually ran since errors would seem to have prevented Cactus from linking properly. The only way this could happen would be if some utility did not build properly (in which case you may choose to ignore it).

Actually looking at you command used (the "2&>" part) you seem to have created a new configuration called "2" and the one that you successfully submitted was the one called "sim". Can you check if you have a bin/cactus_sim but no bin/cactus_2 , please?

[Roland Haas]

Update priority to blocker since it prevents compilation (on Ubuntu machines under certain circumstances).

[Frank Löffler]

in ET call: decided to add a numa dependency on general linux option lists.

[anonymous]

Some lib names and paths should be corrected. The following setup works fine on ubuntu 14.04

# Ubuntu's libhwloc-dev is broken; see
# <https://trac.einsteintoolkit.org/ticket/1717>
HWLOC_DIR = BUILD

MPI_DIR      = /usr
# Ubuntu < 14.04 uses mpich2, >= 14.04 uses mpich
MPI_INC_DIRS = /usr/include/mpi /usr/include/mpich
MPI_LIB_DIRS = /usr/lib
MPI_LIBS     = mpi mpif90 mpi++ mpif77

PTHREADS = yes

[Roland Haas]

Replying to [comment:6 maxim.barkov@…]:

Some lib names and paths should be corrected. The following setup works fine on ubuntu 14.04

`

Ubuntu's libhwloc-dev is broken; see

https://trac.einsteintoolkit.org/ticket/1717

HWLOC_DIR = BUILD

MPI_DIR = /usr

Ubuntu < 14.04 uses mpich2, >= 14.04 uses mpich

MPI_INC_DIRS = /usr/include/mpi /usr/include/mpich MPI_LIB_DIRS = /usr/lib MPI_LIBS = mpi mpif90 mpi++ mpif77

PTHREADS = yes

` From the names of the files it seems as if you are using the OpenMPI package and not mpich package. Since the option list is for mpich the changes would break it for any users that follow the instructions in the file.

Frank: since you created the configuration initially, and since Ubuntu actually uses openmpi as the package to supply mpi-default-dev do you object switch the option list over to openmpi.

[Roland Haas]

In order to fix both #1764 we will probably need some workaround for ubuntu/debian to normally use static linking (which we want) and only switch to dynamic linking when there is no libnuma.a due to the broken hwloc-dev package (see comment:171717).

This is a mess (on multiple levels).

[Frank Löffler]

Reportedly builds now work.

[Frank Löffler]

Replying to [comment:7 rhaas]:

Frank: since you created the configuration initially, and since Ubuntu actually uses openmpi as the package to supply mpi-default-dev do you object switch the option list over to openmpi.

I don't care either way. Both MPI implementations usually work and I don't really care which is our "default" for any system, as long as it works.