Cannot `pip install mpi4py` with conda

[Kevin Paul]

First, I'm sorry if this is the wrong place to post this issue. I cannot find out where the appropriate place should be. (For example, if this turns out to be a conda package issue, please point me there.)

I have a problem installing mpi4py with pip inside of a conda environment. I have created a conda environment and tried installing mpi4py with pip using the following:

conda create -y --name testenv python=3.6 mpich
conda activate testenv
pip install mpi4py

which results in the following error:

error: Cannot compile MPI programs. Check your configuration!!!

If I then try:

.../envs/testenv/bin/mpicc: line 278: x86_64-apple-darwin13.4.0-clang: command not found

I have then tried installing the conda compilers with conda install -y compilers and attempting pip install mpi4py again, which results in:

error: Cannot link MPI programs. Check your configuration!!!

I am using the conda-forge channel (with defaults).

Any thoughts?

[Lisandro Dalcin]

I guess you are missing the mpich-mpicc package, this one is a metapackage that depends in the right compiler. Why are you installing mpi4py with pip? Maybe all you need is to conda install mpi4py (available in the conda-forge channel) ?

[Kevin Paul]

Oh, and I forgot to mention that this is on MacOS. (I think this procedure works on Linux.)

Again, I'm not sure if this is a problem with the conda mpich package on OSX or the conda compilers package. This just seems like it should work "out of the box."

[Lisandro Dalcin]

I have to insist:

(base) [dalcinl@localhost ~]$ conda search mpich-mpicc --info
Loading channels: done
mpich-mpicc 3.2.1 h79c3c89_1008
-------------------------------
file name   : mpich-mpicc-3.2.1-h79c3c89_1008.tar.bz2
name        : mpich-mpicc
version     : 3.2.1
build       : h79c3c89_1008
build number: 1008
size        : 6 KB
license     : MPICH
subdir      : osx-64
url         : https://conda.anaconda.org/conda-forge/osx-64/mpich-mpicc-3.2.1-h79c3c89_1008.tar.bz2
md5         : b6ca5e8e6f25a1d243371df6a11c52bd
timestamp   : 2019-02-05 15:33:05 UTC
dependencies: 
  - clang_osx-64
  - mpich 3.2.1 h71abe9c_1008

[Lisandro Dalcin]

Anyway, you installed compilers. That should have been enough. Could you attach a verbose build log so I can take a look?

[Kevin Paul]

Full log from the pip install mpi4py command.

[Kevin Paul]

I still get the same errors, even with mpich-mpicc installed.

The errors are the same as when I install mpich and compilers separately, which makes sense because it looks like mpich-mpicc is just a metapackage with the mpich and compilers dependency.

To answer your original question about "why am I doing this"? Well, I realize that I can just do conda install mpich mpi4py and it will work. The conda mpi4py package installs and works fine.

However, I have a package that depends on mpi4py that I want to be installable via pip. And I have mpi4py listed as a dependency. ...And, while I could just install the dependencies with conda, it seems like this should work?

So, this isn't critical, obviously, but...well...it bugs me when something seems like it should work but doesn't.

[Kevin Paul]

Any further thoughts on this?

[Lisandro Dalcin]

OK, after looking at the full build log, now I see this is totally unrelated to mpi4py/MPICH, the problem is with the macOS SDKs. conda-forge packages are built with an old SDK, but you of course have the newest one, and this is probably incompatible with the conda compiler package. At this point, the only thing you may want to try is to export MPICH_CC=clang CC=clang to force mpicc and distutils to use your system's clang compiler. Maybe this way it will work, but maybe not.

[Lisandro Dalcin]

A final observation: mpi4py is pip-installable, all what is required is a working MPI installation and mpicc on your PATH env variable. In fact, the mpi4py's conda-forge package is built using pip: https://github.com/conda-forge/mpi4py-feedstock/blob/master/recipe/meta.yaml#L16

[Kevin Paul]

Lisandro, thank you! I appreciate your looking into this in such detail.

I agree that mpi4py is installable with pip. In fact, that is how I used to install it before I started using conda. (And I have many collaborators that depend on this capability here at NCAR.) So, yes, this is clearly not a problem with mpi4py.

Maybe I should post this issue on the compilers feedstock and ask about issues with the newest macOS SDK.

Thanks, again! I appreciate the feedback.

[Kevin Paul]

Oh, and just as a reference, this lead me to look at the following issue on the compilers feedstock, which lead me to this issue on the mpi feedstock. This issue is exactly the problem I was seeing above! And minrk suggests the following solution:

export CONDA_BUILD_SYSROOT=/

which works! With this trick, you can do the following without any problems:

export CONDA_BUILD_SYSROOT=/
conda create --name mpitest python=3.6 mpich-mpicc
conda activate mpitest
pip install mpi4py

and it will work as expected. Fantastic!

[Lisandro Dalcin]

Great! Thanks for the tip! For sure I'll need to use it at some point.