MPI thorn should auto configure

Create issue
Issue #1648 closed
Steven R. Brandt created an issue

The attached patch upgrades the MPI configuration thorn in a number of ways. First, and most importantly, it automatically configures mpi based on information obtained from mpicc when MPI_DIR isn't specified.

Configuration is only disabled by setting MPI_DIR = NONE.

In addition to MPI_INSTALL_DIR, it will look at CACTUS_EXT_INSTALL_DIR. Frank says that at one point a general external directory for applications was discussed.

My feeling is that having the install land in configs is usually not what people want. This version emits an error message if you set MPI_DIR to BUILD, but don't set one of the above install dirs. If you really want MPI under configs, you can set MPI_INSTALL_DIR=CONFIGS.

I anticipate pushback on my MPI_DIR=NONE, and MPI_INSTALL_DIR=CONFIGS suggestions, but I thought I'd suggest them regardless.


Comments (22)

  1. Ian Hinder
    • removed comment

    The auto-configuration is a nice idea.

    I think the MPI thorn should behave like all the other external library thorns, and continue to install in configs by default if you set MPI_DIR to BUILD, without having to set any new variables. The fact that this is possibly not the best long-term solution is outweighed by the fact that this is how it works now (and everyone should not have to update their optionlists for this), and is consistent with the other thorns.

    For reference, I believe that logic behind having the libraries in configs is that they are built using the Cactus build environment (compilers, flags, libs etc) and this is different for each configuration. They may not work with other configurations. I think Erik has a clearer picture of the issues relating to this than I do, though. Erik's proposal is that libraries are built outside of Cactus, e.g. using simfactory. I'm not sure whether that mechanism uses the same compilers as Cactus or not.

  2. Steven R. Brandt reporter
    • removed comment

    This patch addresses Ian's comment. I believe this patch will make it easier to use MPI on new systems and/or people's laptops/desktops.

  3. Erik Schnetter
    • removed comment

    I agree with both of Steve's suggestions. However, I recommend that they should be implemented in the same way as is already done in other thorns:

    The variable CACTUS_EXT_INSTALL_DIR is superfluous. If you want to install MPI somewhere outside Cactus, then you set MPI_INSTALL_DIR accordingly. If this variable is not set, MPI should be installed into the configs directory.

    If you do not want MPI to be build, then one should set MPI_DIR to NO_BUILD. This should effectively disable the configuration logic, assuming that all necessary compiler flags are defined automatically.

  4. Steven R. Brandt reporter
    • removed comment

    The idea of CACTUS_EXT_INSTALL_DIR is that you could set the directory once, and all external thorns would build there. Maybe that should be a separate ticket/discussion.

  5. Ian Hinder
    • removed comment

    I think my only remaining issue with the patch is that it introduces a configure script written in Perl, which is different to the other library configure scripts. While this makes me uncomfortable, and might make it harder to debug (I'm not very familiar with Perl), the additional feature of automatically probing the MPI directories makes it worth committing on balance. If Erik is now OK with the current patch, then so am I.

  6. Erik Schnetter
    • changed status to open
    • removed comment

    MPI_DIR=NO_BUILD should not search.

    Apart from this change the patch seems fine. The Perl script has a much better structure than our Bash scripts.

  7. Roland Haas
    • changed status to open
    • marked as
    • removed comment

    The script fails if only MPI_DIR is given. It should then itself set MPI_INC_DIRS and MPI_LIB_DIRS based on that. Currently I get:

    $ MPI_DIR=/usr/lib/openmpi perl arrangements/ExternalLibraries/MPI/
    MPI was manually configured.
    CCTK_MPI 1
    HAVE_MPI 1
    CCTK_MPI     = 1
    HAVE_MPI     = 1
    MPI_DIR      = /usr/lib/openmpi
    MPI_LIBS     =
    LIBRARY           $(MPI_LIBS)
  8. Ian Hinder
    • changed status to open
    • removed comment

    The result of this change has been to cause mpicxx to be run when you specify MPI_DIR. Why is this needed? If I specify MPI_DIR, then I don't think the compiler wrapper script needs to be run. Can't you deduce the default include and lib directories from this? If they don't exist, you can then try to run the compiler wrapper, but (see #1668) this is less likely to work reliably.

  9. Erik Schnetter
    • removed comment

    I implemented deducing defaults from MPI_DIR. However, MPI_LIBS cannot be auto-deduced.

    One option would be to fail loudly.

    Another option would be to set LD_LIBRARY_PATH and equivalents (for other operating systems) automatically based on the library directory for MPI_DIR.

    We could also use pkg-info to look for the respective libraries.

    What did the code do previously? How did it determine the libraries? Did it just assume that it was an OpenMPI installation?

  10. Ian Hinder
    • removed comment

    The previous version is here:;a=blob;;h=6f32d25a601057e6ecb4076e7a89543357c4712c;hb=f6bf8a9340aaf18c11656e406ac016a5d097581f

    It probes some places, and hard-codes the libs to mpi mpi_cxx.

    I would say that guessing MPI_DIR/lib and MPI_DIR/include would be a fairly good guess. For the libraries, "mpi" sounds good. Does it make sense to put logic supporting each of the common MPI implementations into the script, or try to get the compiler wrapper approach to work?

  11. Erik Schnetter
    • removed comment

    The include and library directories are already guessed.

    Guessing just "mpi" works only for OpenMPI. We either need to hard-code various MPI implementations, or need to make the wrapper approach work, or need to declare that trying to use an MPI implementation where one can't run "mpicxx" is sign of a broken implementation or a configuration error.

  12. Steven R. Brandt reporter
    • removed comment

    Both OpenMPI and Mpich provide a mechanism to reveal their command line options when they are run with either --showme or -config_info, and most machines use one of these two. That was the rationale for the script. It means that most laptops and desktops can configure out of the box.

    If those mechanisms fail, I invoke mpicxx with "sh -x" in the hopes that mpicxx is a shell script and that I can find the compile line in stderr. Obviously, all this will fail if there is an mpicxx and it doesn't work.

    However, it doesn't seem surprising to me that there are some machines that fail all these attempts to auto configure. One can set everything manually for those machines, and mpicxx won't be run.

  13. Frank Löffler
    • removed comment

    Replying to [comment:12 hinder]:

    The result of this change has been to cause mpicxx to be run when you specify MPI_DIR. Why is this needed?

    Apart from the question whether this is needed (which has been answered): is there a problem with this approach? If it does not work for some cases we obviously have to look into it.

  14. Erik Schnetter
    • removed comment

    I think the problem is that the new system fails in some corner cases, which are different from the corner cases in which the old system failed. In principle, using mpicxx is a sound approach, as it is "the" approach explicitly supported by the MPI implementation.

  15. Ian Hinder
    • removed comment

    The approach of using mpicxx to automatically probe for the required settings is good. The problem arises because of simfactory, and workarounds to its problems, but it is there nonetheless.

  16. Ian Hinder
    • changed status to resolved
    • removed comment

    I think the problems mentioned above are just quirks of the new system, so I am closing this ticket (I was the one who reopened it in the first place). If the above discussion includes a report of a still-existing problem, it should be reported in a new defect ticket.

  17. Log in to comment