xl compilers: mpif.h could not be located at: []

Issue #144 new
James Ostrander
created an issue

Hi, I'm trying to compile petsc on a ppc64le system on RHEL 7.1 with xlf/xlc/xlc++.

Configuring, making and testing with the default (no) options works fine.

Output of ./configure --CC=xlc --CXX=xlc++ --FC=xlf --CFLAGS="-qarch=pwr8 -qtune=pwr8 -O3 -qhot" --CXXFLAGS="-qarch=pwr8 -qtune=pwr8 -O3 -qhot" --FFLAGS="-qarch=pwr8 -qtune=pwr8 -O3 -qhot"

===============================================================================
            Configuring PETSc to compile on your system
===============================================================================
TESTING: FortranMPICheck from config.packages.MPI(config/BuildSystem/config/packages/MPI.py:371)       *******************************************************************************
        UNABLE to CONFIGURE with GIVEN OPTIONS    (see configure.log for details):
-------------------------------------------------------------------------------
Fortran error! mpif.h could not be located at: []
*******************************************************************************

Output of [u0017592@sys-83179 petsc]$ sudo find / -name "mpif.h"

/usr/local/include/mpif.h
/home/u0017592/projects/ompi/ompi/include/mpif.h
/home/u0017592/projects/petsc/include/petsc/mpiuni/mpif.h

As you can see, it's in my include folder. Do I need to tell it specifically where to find it? Any idea why this worked with the default options but not with the xl compilers?

My MPI was compiled with xl, by the way.

Comments (16)

  1. James Ostrander reporter

    Hi Matthew,

    I worked around the issue by using:

    ./configure --CC=xlc --CXX=xlc++ --FC=xlf --CFLAGS="-qarch=pwr8 -qtune=pwr8 -O3 -qhot -qinclude=/usr/local/include/mpif.h" --CXXFLAGS="-qarch=pwr8 -qtune=pwr8 -O3 -qhot -qinclude=/usr/local/include/mpif.h" --FFLAGS="-qarch=pwr8 -qtune=pwr8 -O3 -qhot -qinclude=/usr/local/include/mpif.h"
    

    It seems strange that I would have to manually include mpif.h though, since it's in a common folder. Maybe this is a quirk of the xl compilers, but compiling other projects with xl*, I haven't had to do this.

    Here is the configure.log output.

    I ran into another problem after the workaround, but since it appears to be unrelated, I'll put it in a separate issue.

  2. James Ostrander reporter

    On second thought, having investigated a little, it seems like both might be related to the way mpi and xl are used.

    Command:

    ./configure --CC=xlc --CXX=xlc++ --FC=xlf --CFLAGS="-qarch=pwr8 -qtune=pwr8 -O3 -qhot -qinclude=/usr/local/include/mpif.h" --CXXFLAGS="-qarch=pwr8 -qtune=pwr8 -O3 -qhot -qinclude=/usr/local/include/mpif.h" --FFLAGS="-qarch=pwr8 -qtune=pwr8 -O3 -qhot -qinclude=/usr/local/include/mpif.h"
    

    Stdout:

        ===============================================================================
                    Configuring PETSc to compile on your system
        ===============================================================================
        TESTING: checkCCompiler from config.setCompilers(config/BuildSystem/config/setCompilers.py:553)                                                    *******************************************************************************
                UNABLE to CONFIGURE with GIVEN OPTIONS    (see configure.log for details):
        -------------------------------------------------------------------------------
        C compiler you provided with -CC=xlc does not work.
        Cannot compile C with xlc.
        *******************************************************************************
    

    configure.log:

    See configure-xl-include.log
    
  3. James Ostrander reporter

    mpif.h:

    ! -*- fortran -*-
    !
    ! Copyright (c) 2004-2006 The Trustees of Indiana University and Indiana
    !                         University Research and Technology
    !                         Corporation.  All rights reserved.
    ! Copyright (c) 2004-2005 The University of Tennessee and The University
    !                         of Tennessee Research Foundation.  All rights
    !                         reserved.
    ! Copyright (c) 2004-2005 High Performance Computing Center Stuttgart,
    !                         University of Stuttgart.  All rights reserved.
    ! Copyright (c) 2004-2005 The Regents of the University of California.
    !                         All rights reserved.
    ! Copyright (c) 2006-2014 Cisco Systems, Inc.  All rights reserved.
    ! $COPYRIGHT$
    !
    ! Additional copyrights may follow
    !
    ! $HEADER$
    !
    !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
    ! WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING
    !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
    !
    ! Do ***not*** copy this file to the directory where your Fortran
    ! fortran application is compiled unless it is absolutely necessary!  Most
    ! modern Fortran compilers now support the -I command line flag, which
    ! tells the compiler where to find .h files (specifically, this one).  For
    ! example:
    !
    !      shell$ mpifort foo.f -o foo -I$OMPI_HOME/include
    !
    ! will probably do the trick (assuming that you have set OMPI_HOME
    ! properly).
    !
    ! That being said, OMPI's "mpifort" wrapper compiler should
    ! automatically include the -I option for you.  The following command
    ! should be equivalent to the command listed above:
    !
    !      shell$ mpifort foo.f -o foo
    !
    ! You should not copy this file to your local directory because it is
    ! possible that this file will be changed between versions of Open MPI.
    ! Indeed, this mpif.h is incompatible with the mpif.f of other
    ! implementations of MPI.  Using this mpif.h with other implementations
    ! of MPI, or with other versions of Open MPI will result in undefined
    ! behavior (to include incorrect results, segmentation faults,
    ! unexplainable "hanging" in your application, etc.).  Always use the
    ! -I command line option instead (or let mpifort do it for you).
    !
    !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
    ! WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING
    !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
    
          include 'mpif-config.h'
          include 'mpif-constants.h'
          include 'mpif-handles.h'
          include "mpif-io-constants.h"
          include "mpif-io-handles.h"
          include 'mpif-externals.h'
          include 'mpif-sentinels.h'
          include 'mpif-sizeof.h'
    
  4. BarryFSmith

    The -qinclude=/usr/local/include/mpif.h is definitely problematic in CFLAGS and CXXFLAGS

    What if you never use that option and just use the option -I/usr/local/include in the FFLAGS variable?

    Please post the configure.log here if that fails.

  5. James Ostrander reporter

    Thanks for the response, Barry -- I also had that idea since the content of mpif.h file was very obviously not C code. When I run ./configure --CC=xlc --CXX=xlc++ --FC=xlf --CFLAGS="-qarch=pwr8 -qtune=pwr8 -O3 -qhot" --CXXFLAGS="-qarch=pwr8 -qtune=pwr8 -O3 -qhot" --FFLAGS="-qarch=pwr8 -qtune=pwr8 -O3 -qhot -qinclude=/usr/local/include/mpif.h", the result is...

    ===============================================================================
                 Configuring PETSc to compile on your system
    ===============================================================================
    TESTING: FortranMPICheck from config.packages.MPI(config/BuildSystem/config/packages/MPI.py:371)                                                   *******************************************************************************
             UNABLE to CONFIGURE with GIVEN OPTIONS    (see configure.log for details):
    -------------------------------------------------------------------------------
    Fortran error! mpif.h could not be located at: []
    *******************************************************************************
    

    configure.log:

    See configure-xl-include-fflags-only.log
    
  6. BarryFSmith

    Don't use -qinclude use -I The problem with -qinclude is that it just includes the additional file. The compiler still looks for the mpif.h included by the file and cannot find it so stops.

  7. James Ostrander reporter

    I tried that as well and had no luck - going beyond that, I honed in on a more minimized example:

    test.F90:

    program main
    #include "mpif.h"
    end
    
    [u0017592@sys-83519 petsc]$ gfortran test.F90 -I/home/u0017592/projects/ompi/ompi/include
    mpif-sizeof.h:75.48:
        Included at /home/u0017592/projects/ompi/ompi/include/mpif.h:61:
        Included at test.F90:3:
    
          COMPLEX(REAL128), DIMENSION(1,1,1,1,1,1,1,*)::x
                                                    1
    Error: Array specification at (1) has more than 7 dimensions
    mpif-sizeof.h:82.48:
        Included at /home/u0017592/projects/ompi/ompi/include/mpif.h:61:
        Included at test.F90:3:
    
          COMPLEX(REAL128), DIMENSION(1,1,1,1,1,1,1,1,*)::x
                                                    1
    Error: Array specification at (1) has more than 7 dimensions
    mpif-sizeof.h:89.48:
        Included at /home/u0017592/projects/ompi/ompi/include/mpif.h:61:
        Included at test.F90:3:
    
          COMPLEX(REAL128), DIMENSION(1,1,1,1,1,1,1,1,1,*)::x
                                                    1
    Error: Array specification at (1) has more than 7 dimensions
    mpif-sizeof.h:96.48:
        Included at /home/u0017592/projects/ompi/ompi/include/mpif.h:61:
        Included at test.F90:3:
    
          COMPLEX(REAL128), DIMENSION(1,1,1,1,1,1,1,1,1,1,*)::x
                                                    1
    Error: Array specification at (1) has more than 7 dimensions
    mpif-sizeof.h:103.48:
        Included at /home/u0017592/projects/ompi/ompi/include/mpif.h:61:
        Included at test.F90:3:
    
          COMPLEX(REAL128), DIMENSION(1,1,1,1,1,1,1,1,1,1,1,*)::x
                                                    1
    Error: Array specification at (1) has more than 7 dimensions
    mpif-sizeof.h:110.48:
        Included at /home/u0017592/projects/ompi/ompi/include/mpif.h:61:
        Included at test.F90:3:
    
          COMPLEX(REAL128), DIMENSION(1,1,1,1,1,1,1,1,1,1,1,1,*)::x
                                                    1
    Error: Array specification at (1) has more than 7 dimensions
    mpif-sizeof.h:117.48:
        Included at /home/u0017592/projects/ompi/ompi/include/mpif.h:61:
        Included at test.F90:3:
    
          COMPLEX(REAL128), DIMENSION(1,1,1,1,1,1,1,1,1,1,1,1,1,*)::x
                                                    1
    Error: Array specification at (1) has more than 7 dimensions
    mpif-sizeof.h:124.48:
        Included at /home/u0017592/projects/ompi/ompi/include/mpif.h:61:
        Included at test.F90:3:
    
          COMPLEX(REAL128), DIMENSION(1,1,1,1,1,1,1,1,1,1,1,1,1,1,*)::x
                                                    1
    Error: Array specification at (1) has more than 7 dimensions
    mpif-sizeof.h:187.47:
        Included at /home/u0017592/projects/ompi/ompi/include/mpif.h:61:
        Included at test.F90:3:
    
          COMPLEX(REAL32), DIMENSION(1,1,1,1,1,1,1,*)::x
                                                   1
    Error: Array specification at (1) has more than 7 dimensions
    mpif-sizeof.h:194.47:
        Included at /home/u0017592/projects/ompi/ompi/include/mpif.h:61:
        Included at test.F90:3:
    
          COMPLEX(REAL32), DIMENSION(1,1,1,1,1,1,1,1,*)::x
                                                   1
    Error: Array specification at (1) has more than 7 dimensions
    mpif-sizeof.h:201.47:
        Included at /home/u0017592/projects/ompi/ompi/include/mpif.h:61:
        Included at test.F90:3:
    
          COMPLEX(REAL32), DIMENSION(1,1,1,1,1,1,1,1,1,*)::x
                                                   1
    Error: Array specification at (1) has more than 7 dimensions
    mpif-sizeof.h:208.47:
        Included at /home/u0017592/projects/ompi/ompi/include/mpif.h:61:
        Included at test.F90:3:
    
          COMPLEX(REAL32), DIMENSION(1,1,1,1,1,1,1,1,1,1,*)::x
                                                   1
    Error: Array specification at (1) has more than 7 dimensions
    mpif-sizeof.h:215.47:
        Included at /home/u0017592/projects/ompi/ompi/include/mpif.h:61:
        Included at test.F90:3:
    
          COMPLEX(REAL32), DIMENSION(1,1,1,1,1,1,1,1,1,1,1,*)::x
                                                   1
    Error: Array specification at (1) has more than 7 dimensions
    mpif-sizeof.h:222.47:
        Included at /home/u0017592/projects/ompi/ompi/include/mpif.h:61:
        Included at test.F90:3:
    
          COMPLEX(REAL32), DIMENSION(1,1,1,1,1,1,1,1,1,1,1,1,*)::x
                                                   1
    Error: Array specification at (1) has more than 7 dimensions
    mpif-sizeof.h:229.47:
        Included at /home/u0017592/projects/ompi/ompi/include/mpif.h:61:
        Included at test.F90:3:
    
          COMPLEX(REAL32), DIMENSION(1,1,1,1,1,1,1,1,1,1,1,1,1,*)::x
                                                   1
    Error: Array specification at (1) has more than 7 dimensions
    mpif-sizeof.h:236.47:
        Included at /home/u0017592/projects/ompi/ompi/include/mpif.h:61:
        Included at test.F90:3:
    
          COMPLEX(REAL32), DIMENSION(1,1,1,1,1,1,1,1,1,1,1,1,1,1,*)::x
                                                   1
    Error: Array specification at (1) has more than 7 dimensions
    mpif-sizeof.h:299.47:
        Included at /home/u0017592/projects/ompi/ompi/include/mpif.h:61:
        Included at test.F90:3:
    
          COMPLEX(REAL64), DIMENSION(1,1,1,1,1,1,1,*)::x
                                                   1
    Error: Array specification at (1) has more than 7 dimensions
    mpif-sizeof.h:306.47:
        Included at /home/u0017592/projects/ompi/ompi/include/mpif.h:61:
        Included at test.F90:3:
    
          COMPLEX(REAL64), DIMENSION(1,1,1,1,1,1,1,1,*)::x
                                                   1
    Error: Array specification at (1) has more than 7 dimensions
    mpif-sizeof.h:313.47:
        Included at /home/u0017592/projects/ompi/ompi/include/mpif.h:61:
        Included at test.F90:3:
    
          COMPLEX(REAL64), DIMENSION(1,1,1,1,1,1,1,1,1,*)::x
                                                   1
    Error: Array specification at (1) has more than 7 dimensions
    mpif-sizeof.h:320.47:
        Included at /home/u0017592/projects/ompi/ompi/include/mpif.h:61:
        Included at test.F90:3:
    
          COMPLEX(REAL64), DIMENSION(1,1,1,1,1,1,1,1,1,1,*)::x
                                                   1
    Error: Array specification at (1) has more than 7 dimensions
    mpif-sizeof.h:327.47:
        Included at /home/u0017592/projects/ompi/ompi/include/mpif.h:61:
        Included at test.F90:3:
    
          COMPLEX(REAL64), DIMENSION(1,1,1,1,1,1,1,1,1,1,1,*)::x
                                                   1
    Error: Array specification at (1) has more than 7 dimensions
    mpif-sizeof.h:334.47:
        Included at /home/u0017592/projects/ompi/ompi/include/mpif.h:61:
        Included at test.F90:3:
    
          COMPLEX(REAL64), DIMENSION(1,1,1,1,1,1,1,1,1,1,1,1,*)::x
                                                   1
    Error: Array specification at (1) has more than 7 dimensions
    mpif-sizeof.h:341.47:
        Included at /home/u0017592/projects/ompi/ompi/include/mpif.h:61:
        Included at test.F90:3:
    
          COMPLEX(REAL64), DIMENSION(1,1,1,1,1,1,1,1,1,1,1,1,1,*)::x
                                                   1
    Error: Array specification at (1) has more than 7 dimensions
    mpif-sizeof.h:348.47:
        Included at /home/u0017592/projects/ompi/ompi/include/mpif.h:61:
        Included at test.F90:3:
    
          COMPLEX(REAL64), DIMENSION(1,1,1,1,1,1,1,1,1,1,1,1,1,1,*)::x
                                                   1
    Error: Array specification at (1) has more than 7 dimensions
    mpif-sizeof.h:411.46:
        Included at /home/u0017592/projects/ompi/ompi/include/mpif.h:61:
        Included at test.F90:3:
    
          INTEGER(INT16), DIMENSION(1,1,1,1,1,1,1,*)::x
                                                  1
    Error: Array specification at (1) has more than 7 dimensions
    Fatal Error: Error count reached limit of 25.
    

    So, not necessarily related to this xlf or even this project at all. I did try configuring mpi with --with-f90-max-array-dim=7 but that doesn't seem to have made any difference (the default is 4 anyway so this shouldn't have been a problem if I'm understanding the explanation of that flag correctly).

  8. James Ostrander reporter

    sudo ./configure --CC=xlc --CXX=xlc++ --FC=xlf --CFLAGS="-qarch=pwr8 -qtune=pwr8 -O3 -qhot" --CXXFLAGS="-qarch=pwr8 -qtune=pwr8 -O3 -qhot" --FFLAGS="-qarch=pwr8 -qtune=pwr8 -O3 -qhot -I/usr/local/include"

    Results in configure.log: See mpi_init-not-located.log

    The smallest version of this I could reproduce is as follows:

    test.F90:

    program main
    #include "mpif.h"
    integer ierr
    call mpi_init(ierr)
    end
    
    [u0017592@sys-83519 petsc]$ xlf test.F90 -I/usr/local/include
    ** main   === End of Compilation 1 ===
    1501-510  Compilation successful for file test.F90.
    test.o: In function `main':
    test.F90:(.text+0x18): undefined reference to `mpi_init'
    

    However, if I use mpif90 instead of xlf:

    [u0017592@sys-83519 petsc]$ mpif90 test.F90
    ** main   === End of Compilation 1 ===
    1501-510  Compilation successful for file test.F90.
    

    And I don't even have to manually include -I/usr/local/include in order for it to find mpif.h.

  9. James Ostrander reporter

    Tried setting mpif90 as my fortran compiler - doesn't work either:

    sudo ./configure --CC=xlc --CXX=xlc++ --FC=mpif90 --CFLAGS="-qarch=pwr8 -qtune=pwr8 -O3 -qhot" --CXXFLAGS="-qarch=pwr8 -qtune=pwr8 -O3 -qhot" --FFLAGS="-qarch=pwr8 -qtune=pwr8 -O3 -qhot"

    See compiler-mpif90-doesnt-work.log

  10. Log in to comment