Cannot compile the ET on Datura

Create issue
Issue #1668 closed
Ian Hinder created an issue

Compilation of the ET fails on Datura while configuring MPI (see also #1667). When running the mpic++ command directly, I get

[ianhin@login-damiana EinsteinToolkitGit]$ /cluster/openmpi/SL6/1.7.2/intel14/bin/mpic++
/cluster/openmpi/SL6/1.7.2/intel14/bin/mpic++: error while loading shared libraries: libimf.so: cannot open shared object file: No such file or directory

This happens because the compiler library is not on LD_LIBRARY_PATH:

[ianhin@login-damiana EinsteinToolkitGit]$ ldd /cluster/openmpi/SL6/1.7.2/intel14/bin/mpic++
    linux-vdso.so.1 =>  (0x00007fff2a1ff000)
    libopen-pal.so.5 => /cluster/openmpi/SL6/1.7.2/intel14/lib/libopen-pal.so.5 (0x00007fc755f21000)
    libm.so.6 => /lib64/libm.so.6 (0x00007fc755c90000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007fc755a8c000)
    librt.so.1 => /lib64/librt.so.1 (0x00007fc755884000)
    libnsl.so.1 => /lib64/libnsl.so.1 (0x00007fc75566a000)
    libutil.so.1 => /lib64/libutil.so.1 (0x00007fc755467000)
    libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fc755251000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fc755033000)
    libc.so.6 => /lib64/libc.so.6 (0x00007fc754ca0000)
    /lib64/ld-linux-x86-64.so.2 (0x00007fc75620e000)
    libimf.so => not found
    libsvml.so => not found
    libirng.so => not found
    libintlc.so.5 => not found

because when OpenMPI was compiled, its rpath was not set to point to the compiler installation directory.

This is because the compiler module is not loaded. The reason it is not loaded in the envsetup for the machine is that this this is specific to the version of the compiler, and one might want to use a different compiler via a different optionlist.

Since the optionlist is the only place for compiler-specific options, how can I set LD_LIBRARY_PATH so that mpicxx can run?

As a last resort, I suppose I can set all the required MPI variables in the optionlist, but MPI_DIR used to work with the old configure script. I would like it to work again now.

Keyword:

Comments (5)

  1. Roland Haas
    • removed comment

    Envsetup is also compiler specific. For example the Cray machines which use a common CC wrapper for all compilers decide which compiler is actually used based on the loaded modules. See eg. the bluewaters files in simfactory.

  2. Ian Hinder reporter
    • removed comment

    I think you are referring to the fact that since you need to load modules to use bluewaters, you are forced to have one machine definition per compiler, in order to use multiple envsetup variables? I think having a "machine" for each compiler (or other optional things, such as mpi version) is the right way to solve this. At the moment, this requires a lot of duplicated text in the machine definition files. This could be solved by allowing machine definitions to inherit from other machines. So bluewaters-cray.ini would just be

    [bluewaters-cray]
    inherit-from = bluewaters
    envsetup        = source /etc/profile && module unload PrgEnv-cray PrgEnv-gnu PrgEnv-intel PrgEnv-pathscale PrgEnv-pgi && module load PrgEnv-cray && module load atp && module swap cce cce/8.1.5 && module load fftw/3.3.0.1 && module load hdf5/1.8.8 && module load papi/5.0.1
    

    Wanting to "mix and match" different options for a given machine and give them a name seems like quite a useful thing to be able to do in general. The name "machine" might be a little confusing if the concept is extended to mean a combination of machines and choices of the options to use on those machines.

  3. Frank Löffler
    • removed comment

    So, the right way to solve this would be to load the right module on Datura, possibly providing different "machines" for different configurations? If so, this could be done, this ticket closed, and a request could be opened for the "include" feature in simfactory (maybe even with a patch? :) )

  4. Ian Hinder reporter
    • changed status to resolved
    • removed comment

    I have instead set the MPI options explicitly in the optionlist, so that you still don't have to load the module. It's all a bit of a mess; simfactory 3 will have something better.

  5. Log in to comment