Help desired to install mpi4py

[Former user]

Hello,

I tried to install mpi4py on my laptop (which is running Linux Fedora 18) using the command "sudo -E pip install mpi4py". I have both openmpi-1.6.3-7.fc18.x86_64 and mpich-3.0.4-3.fc18.x86_64 installed.

Here is the error that showed up:

Downloading/unpacking mpi4py
  Running setup.py (path:/tmp/pip_build_root/mpi4py/setup.py) egg_info for package mpi4py

Installing collected packages: mpi4py
  Running setup.py install for mpi4py
    MPI configuration: [mpi] from 'mpi.cfg'
    checking for library 'pthread' ...
    gcc -fPIC -fno-strict-aliasing -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -c _configtest.c -o _configtest.o
    gcc _configtest.o -lpthread -o _configtest
    success!
    removing: _configtest.c _configtest.o _configtest
    checking for library 'mpe' ...
    gcc -fPIC -fno-strict-aliasing -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -c _configtest.c -o _configtest.o
    gcc _configtest.o -lmpe -lpthread -o _configtest
    /bin/ld: cannot find -lmpe
    collect2: error: ld returned 1 exit status
    failure.
    removing: _configtest.c _configtest.o
    checking for library 'lmpe' ...
    gcc -fPIC -fno-strict-aliasing -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -c _configtest.c -o _configtest.o
    gcc _configtest.o -llmpe -lpthread -o _configtest
    /bin/ld: cannot find -llmpe
    collect2: error: ld returned 1 exit status
    failure.
    removing: _configtest.c _configtest.o
    building 'mpe' dylib library
    gcc -fPIC -fno-strict-aliasing -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -c src/pmpi-mpe.c -o build/temp.linux-x86_64-2.7/src/pmpi-mpe.o
    src/pmpi-mpe.c:2:17: fatal error: mpi.h: No such file or directory
    compilation terminated.
    warning: build_clib: building library "mpe" failed

    warning: build_clib: command 'gcc' failed with exit status 1

    checking for library 'vt-mpi' ...
    gcc -fPIC -fno-strict-aliasing -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -c _configtest.c -o _configtest.o
    gcc _configtest.o -lvt-mpi -o _configtest
    /bin/ld: cannot find -lvt-mpi
    collect2: error: ld returned 1 exit status
    failure.
    removing: _configtest.c _configtest.o
    checking for library 'vt.mpi' ...
    gcc -fPIC -fno-strict-aliasing -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -c _configtest.c -o _configtest.o
    gcc _configtest.o -lvt.mpi -o _configtest
    /bin/ld: cannot find -lvt.mpi
    collect2: error: ld returned 1 exit status
    failure.
    removing: _configtest.c _configtest.o
    building 'vt' dylib library
    gcc -fPIC -fno-strict-aliasing -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -c src/pmpi-vt.c -o build/temp.linux-x86_64-2.7/src/pmpi-vt.o
    src/pmpi-vt.c:2:17: fatal error: mpi.h: No such file or directory
    compilation terminated.
    warning: build_clib: building library "vt" failed

    warning: build_clib: command 'gcc' failed with exit status 1

    checking for library 'vt-mpi' ...
    gcc -fPIC -fno-strict-aliasing -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -c _configtest.c -o _configtest.o
    gcc _configtest.o -lvt-mpi -o _configtest
    /bin/ld: cannot find -lvt-mpi
    collect2: error: ld returned 1 exit status
    failure.
    removing: _configtest.c _configtest.o
    building 'vt-mpi' dylib library
    gcc -fPIC -fno-strict-aliasing -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -c src/pmpi-vt-mpi.c -o build/temp.linux-x86_64-2.7/src/pmpi-vt-mpi.o
    src/pmpi-vt-mpi.c:2:17: fatal error: mpi.h: No such file or directory
    compilation terminated.
    warning: build_clib: building library "vt-mpi" failed

    warning: build_clib: command 'gcc' failed with exit status 1

    checking for library 'vt-hyb' ...
    gcc -fPIC -fno-strict-aliasing -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -c _configtest.c -o _configtest.o
    gcc _configtest.o -lvt-hyb -o _configtest
    /bin/ld: cannot find -lvt-hyb
    collect2: error: ld returned 1 exit status
    failure.
    removing: _configtest.c _configtest.o
    building 'vt-hyb' dylib library
    gcc -fPIC -fno-strict-aliasing -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -c src/pmpi-vt-hyb.c -o build/temp.linux-x86_64-2.7/src/pmpi-vt-hyb.o
    src/pmpi-vt-hyb.c:2:17: fatal error: mpi.h: No such file or directory
    compilation terminated.
    warning: build_clib: building library "vt-hyb" failed

    warning: build_clib: command 'gcc' failed with exit status 1

    MPI configuration: [mpi] from 'mpi.cfg'
    checking for MPI compile and link ...
    checking for header 'mpi.h' ...
    gcc -fPIC -fno-strict-aliasing -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -I/usr/include/python2.7 -c _configtest.c -o _configtest.o
    _configtest.c:2:17: fatal error: mpi.h: No such file or directory
    compilation terminated.
    failure.
    removing: _configtest.c _configtest.o
    failure.
    error: Cannot find 'mpi.h' header. Check your configuration!!!
    Complete output from command /usr/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip_build_root/mpi4py/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-xv19Kw-record/install-record.txt --single-version-externally-managed --compile:
    running install

running build

running build_src

running build_py

creating build

creating build/lib.linux-x86_64-2.7

creating build/lib.linux-x86_64-2.7/mpi4py

copying src/rc.py -> build/lib.linux-x86_64-2.7/mpi4py

copying src/__init__.py -> build/lib.linux-x86_64-2.7/mpi4py

creating build/lib.linux-x86_64-2.7/mpi4py/include

creating build/lib.linux-x86_64-2.7/mpi4py/include/mpi4py

copying src/include/mpi4py/mpi4py.MPI_api.h -> build/lib.linux-x86_64-2.7/mpi4py/include/mpi4py

copying src/include/mpi4py/mpi4py.MPI.h -> build/lib.linux-x86_64-2.7/mpi4py/include/mpi4py

copying src/include/mpi4py/mpi4py.h -> build/lib.linux-x86_64-2.7/mpi4py/include/mpi4py

copying src/include/mpi4py/MPI.pxd -> build/lib.linux-x86_64-2.7/mpi4py/include/mpi4py

copying src/include/mpi4py/mpi_c.pxd -> build/lib.linux-x86_64-2.7/mpi4py/include/mpi4py

copying src/include/mpi4py/__init__.pxd -> build/lib.linux-x86_64-2.7/mpi4py/include/mpi4py

copying src/include/mpi4py/__init__.pyx -> build/lib.linux-x86_64-2.7/mpi4py/include/mpi4py

copying src/include/mpi4py/mpi.pxi -> build/lib.linux-x86_64-2.7/mpi4py/include/mpi4py

copying src/include/mpi4py/mpi4py.i -> build/lib.linux-x86_64-2.7/mpi4py/include/mpi4py

copying src/MPI.pxd -> build/lib.linux-x86_64-2.7/mpi4py

copying src/mpi_c.pxd -> build/lib.linux-x86_64-2.7/mpi4py

running build_clib

MPI configuration: [mpi] from 'mpi.cfg'

checking for library 'pthread' ...

gcc -fPIC -fno-strict-aliasing -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -c _configtest.c -o _configtest.o

gcc _configtest.o -lpthread -o _configtest

success!

removing: _configtest.c _configtest.o _configtest

checking for library 'mpe' ...

gcc -fPIC -fno-strict-aliasing -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -c _configtest.c -o _configtest.o

gcc _configtest.o -lmpe -lpthread -o _configtest

/bin/ld: cannot find -lmpe

collect2: error: ld returned 1 exit status

failure.

removing: _configtest.c _configtest.o

checking for library 'lmpe' ...

gcc -fPIC -fno-strict-aliasing -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -c _configtest.c -o _configtest.o

gcc _configtest.o -llmpe -lpthread -o _configtest

/bin/ld: cannot find -llmpe

collect2: error: ld returned 1 exit status

failure.

removing: _configtest.c _configtest.o

building 'mpe' dylib library

creating build/temp.linux-x86_64-2.7

creating build/temp.linux-x86_64-2.7/src

gcc -fPIC -fno-strict-aliasing -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -c src/pmpi-mpe.c -o build/temp.linux-x86_64-2.7/src/pmpi-mpe.o

src/pmpi-mpe.c:2:17: fatal error: mpi.h: No such file or directory

compilation terminated.

warning: build_clib: building library "mpe" failed

warning: build_clib: command 'gcc' failed with exit status 1

checking for library 'vt-mpi' ...

gcc -fPIC -fno-strict-aliasing -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -c _configtest.c -o _configtest.o

gcc _configtest.o -lvt-mpi -o _configtest

/bin/ld: cannot find -lvt-mpi

collect2: error: ld returned 1 exit status

failure.

removing: _configtest.c _configtest.o

checking for library 'vt.mpi' ...

gcc -fPIC -fno-strict-aliasing -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -c _configtest.c -o _configtest.o

gcc _configtest.o -lvt.mpi -o _configtest

/bin/ld: cannot find -lvt.mpi

collect2: error: ld returned 1 exit status

failure.

removing: _configtest.c _configtest.o

building 'vt' dylib library

gcc -fPIC -fno-strict-aliasing -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -c src/pmpi-vt.c -o build/temp.linux-x86_64-2.7/src/pmpi-vt.o

src/pmpi-vt.c:2:17: fatal error: mpi.h: No such file or directory

compilation terminated.

warning: build_clib: building library "vt" failed

warning: build_clib: command 'gcc' failed with exit status 1

checking for library 'vt-mpi' ...

gcc -fPIC -fno-strict-aliasing -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -c _configtest.c -o _configtest.o

gcc _configtest.o -lvt-mpi -o _configtest

/bin/ld: cannot find -lvt-mpi

collect2: error: ld returned 1 exit status

failure.

removing: _configtest.c _configtest.o

building 'vt-mpi' dylib library

gcc -fPIC -fno-strict-aliasing -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -c src/pmpi-vt-mpi.c -o build/temp.linux-x86_64-2.7/src/pmpi-vt-mpi.o

src/pmpi-vt-mpi.c:2:17: fatal error: mpi.h: No such file or directory

compilation terminated.

warning: build_clib: building library "vt-mpi" failed

warning: build_clib: command 'gcc' failed with exit status 1

checking for library 'vt-hyb' ...

gcc -fPIC -fno-strict-aliasing -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -c _configtest.c -o _configtest.o

gcc _configtest.o -lvt-hyb -o _configtest

/bin/ld: cannot find -lvt-hyb

collect2: error: ld returned 1 exit status

failure.

removing: _configtest.c _configtest.o

building 'vt-hyb' dylib library

gcc -fPIC -fno-strict-aliasing -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -c src/pmpi-vt-hyb.c -o build/temp.linux-x86_64-2.7/src/pmpi-vt-hyb.o

src/pmpi-vt-hyb.c:2:17: fatal error: mpi.h: No such file or directory

compilation terminated.

warning: build_clib: building library "vt-hyb" failed

warning: build_clib: command 'gcc' failed with exit status 1

running build_ext

MPI configuration: [mpi] from 'mpi.cfg'

checking for MPI compile and link ...

checking for header 'mpi.h' ...

gcc -fPIC -fno-strict-aliasing -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -I/usr/include/python2.7 -c _configtest.c -o _configtest.o

_configtest.c:2:17: fatal error: mpi.h: No such file or directory

compilation terminated.

failure.

removing: _configtest.c _configtest.o

failure.

error: Cannot find 'mpi.h' header. Check your configuration!!!

----------------------------------------
Cleaning up...
Command /usr/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip_build_root/mpi4py/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-xv19Kw-record/install-record.txt --single-version-externally-managed --compile failed with error code 1 in /tmp/pip_build_root/mpi4py
Storing debug log for failure in /home/emmanuel/.pip/pip.log

I found online several discussions addressing this issue: - https://groups.google.com/forum/#!topic/mpi4py/18kCmnKJQWs - https://groups.google.com/forum/#!topic/mpi4py/ZSqyqstSmxU - https://groups.google.com/forum/#!msg/mpi4py/_dodHQXWdFU/a9U1f9ulHJUJ

So tried to follow some suggestions:

1/ I added the following to the system file "~/.profile": export MPI_DIR=/usr/lib64/mpich export PATH=/usr/lib64/mpich/bin:$PATH export LD_LIBRARY_PATH=/usr/lib64/mpich/lib:$LD_LIBRARY_PATH export DYLD_LIBRARY_PATH=/usr/lib64/mpich/lib:$DYLD_LIBRARY_PATH

2/ Running the command "which mpicc" returns: /usr/lib64/mpich/bin/mpicc

3/ Running the command "mpicc -show" returns: cc -m64 -O2 -fPIC -Wl,-z,noexecstack -I/usr/include/mpich-x86_64 -L/usr/lib64/mpich/lib -Wl,-rpath -Wl,/usr/lib64/mpich/lib -lmpich -lopa -lmpl -lrt -lpthread

Please let me know what additional information you may need. Thank you in advance for your help.

[Lisandro Dalcin]

Did you try to install this as root or using sudo? mpi4py should pick mpicc automatically, but it seems it has not. This happens only if your mpicc command is not found. If you are using sudo, use the -E flag, ie. "sudo -E pip install mpi4py". But I really recommend to install mpi4py in your user account, ie. "pip install --user mpi4py"

What Linux distribution are you using? Have you checked within your distro's package manager about availability of mpi4py?

[Former user]

Thank you very much for the prompt reply.

I tried either way: "sudo -E pip install mpi4py" and "pip install --user mpi4py". Both commands led to the same error.

I am using Fedora. I have checked the availability of mpi4py within the package manager: mpi4py-common-1.3-6.fc18: installed mpi4py-docs-1.3-6.fc18: installed mpi4py-mpich-1.3-6.fc18: installed mpi4py-mpich2-1.3-5.fc18 not installed because obsoleted by the previous one mpi4py-opnempi-1.3-6.fc18: installed python3-mpi4py-mpich-1.3-6.fc18: not installed because I use Python 2.7 python3-mpi4py-mpich2-1.3-5.fc18: not installed because I use Python 2.7 python3-mpi4py-openpmi-1.3-g.fc18: not installed because I use Python 2.7

[Former user]

To follow up, some packages dealing with MPICH and Open MPI were already pre-installed on my Linux distribution (Fedora 18). However, the package management system did not provide the missing packages so that MPICH / Open MPI can work. Hence the following steps to be able to use mpi4py with MPICH:

1/ # yum install mpich-devel mpich-autoload

2/ # pip install mpi4py

3/ To avoid that the execution of a program gets stuck with "ssh: Could not resolve hostname laptop: Name or service not known" (issue with the network configuration):

a/ $ cat /etc/hosts This should return something like:

127.0.0.1 localhost.localdomain localhost

::1 localhost6.localdomain6 localhost6

b/ # gedit /etc/hosts in order to modify the file to ("laptop" is added at the end of the first line):

127.0.0.1 localhost.localdomain localhost laptop

::1 localhost6.localdomain6 localhost6

That did the trick for me. Thanks Lisandro for helping me.

[Thomas Spura]

mpich-autoload doesn't do more than a module load mpi/mpich-x86_64. If you would have done that before, then you can also directly use the mpi4py packages provided from fedora:

1. yum install mpi4py-mpich
2. module load mpi/mpich-x86_64
3. python -c "import mpi4py"

[Former user]

Okay, thanks for the info.

[Herrera Alexandre]

FOR UBUNTU USERS WITH THE SAME ERROR TRY :

sudo apt install mpich

then run :

sudo pip3 install mpi4py

It will work without errors, you can now :

python3

import mpi4py

[Thomas Spura]

sudo pip interferes with the packages with apt, which is the reason I mostly avoid pip in these cases and would use virtual environments in this case.

[Lisandro Dalcin]

Or use pip install --user <package> to install under ~/.local.