Cannot `pip install mpi4py` with conda
First, I'm sorry if this is the wrong place to post this issue. I cannot find out where the appropriate place should be. (For example, if this turns out to be a conda package issue, please point me there.)
I have a problem installing mpi4py
with pip
inside of a conda environment. I have created a conda environment and tried installing mpi4py with pip using the following:
conda create -y --name testenv python=3.6 mpich
conda activate testenv
pip install mpi4py
which results in the following error:
error: Cannot compile MPI programs. Check your configuration!!!
If I then try:
.../envs/testenv/bin/mpicc: line 278: x86_64-apple-darwin13.4.0-clang: command not found
I have then tried installing the conda compilers with conda install -y compilers
and attempting pip install mpi4py
again, which results in:
error: Cannot link MPI programs. Check your configuration!!!
I am using the conda-forge channel (with defaults).
Any thoughts?
Comments (14)
-
-
reporter Oh, and I forgot to mention that this is on MacOS. (I think this procedure works on Linux.)
Again, I'm not sure if this is a problem with the conda mpich package on OSX or the conda compilers package. This just seems like it should work "out of the box."
-
I have to insist:
(base) [dalcinl@localhost ~]$ conda search mpich-mpicc --info Loading channels: done mpich-mpicc 3.2.1 h79c3c89_1008 ------------------------------- file name : mpich-mpicc-3.2.1-h79c3c89_1008.tar.bz2 name : mpich-mpicc version : 3.2.1 build : h79c3c89_1008 build number: 1008 size : 6 KB license : MPICH subdir : osx-64 url : https://conda.anaconda.org/conda-forge/osx-64/mpich-mpicc-3.2.1-h79c3c89_1008.tar.bz2 md5 : b6ca5e8e6f25a1d243371df6a11c52bd timestamp : 2019-02-05 15:33:05 UTC dependencies: - clang_osx-64 - mpich 3.2.1 h71abe9c_1008
-
Anyway, you installed
compilers
. That should have been enough. Could you attach a verbose build log so I can take a look? -
reporter - attached pip_install_mpi4py.log
Full log from the
pip install mpi4py
command. -
reporter I still get the same errors, even with
mpich-mpicc
installed.The errors are the same as when I install
mpich
andcompilers
separately, which makes sense because it looks likempich-mpicc
is just a metapackage with thempich
andcompilers
dependency.To answer your original question about "why am I doing this"? Well, I realize that I can just do
conda install mpich mpi4py
and it will work. The condampi4py
package installs and works fine.However, I have a package that depends on
mpi4py
that I want to be installable viapip
. And I havempi4py
listed as a dependency. ...And, while I could just install the dependencies with conda, it seems like this should work?So, this isn't critical, obviously, but...well...it bugs me when something seems like it should work but doesn't.
-
reporter - marked as minor
-
reporter Any further thoughts on this?
-
OK, after looking at the full build log, now I see this is totally unrelated to mpi4py/MPICH, the problem is with the macOS SDKs. conda-forge packages are built with an old SDK, but you of course have the newest one, and this is probably incompatible with the conda compiler package. At this point, the only thing you may want to try is to
export MPICH_CC=clang CC=clang
to forcempicc
and distutils to use your system'sclang
compiler. Maybe this way it will work, but maybe not. -
A final observation: mpi4py is pip-installable, all what is required is a working MPI installation and
mpicc
on yourPATH
env variable. In fact, the mpi4py's conda-forge package is built using pip: https://github.com/conda-forge/mpi4py-feedstock/blob/master/recipe/meta.yaml#L16 -
- changed status to invalid
-
reporter Lisandro, thank you! I appreciate your looking into this in such detail.
I agree that
mpi4py
is installable withpip
. In fact, that is how I used to install it before I started usingconda
. (And I have many collaborators that depend on this capability here at NCAR.) So, yes, this is clearly not a problem withmpi4py
.Maybe I should post this issue on the
compilers
feedstock and ask about issues with the newest macOS SDK.Thanks, again! I appreciate the feedback.
-
reporter Oh, and just as a reference, this lead me to look at the following issue on the
compilers
feedstock, which lead me to this issue on thempi
feedstock. This issue is exactly the problem I was seeing above! And minrk suggests the following solution:export CONDA_BUILD_SYSROOT=/
which works! With this trick, you can do the following without any problems:
export CONDA_BUILD_SYSROOT=/ conda create --name mpitest python=3.6 mpich-mpicc conda activate mpitest pip install mpi4py
and it will work as expected. Fantastic!
-
Great! Thanks for the tip! For sure I'll need to use it at some point.
- Log in to comment
I guess you are missing the
mpich-mpicc
package, this one is a metapackage that depends in the right compiler. Why are you installing mpi4py with pip? Maybe all you need is toconda install mpi4py
(available in the conda-forge channel) ?