Error when installing mpi4py with pip on macOS

[Caleb Ju]

I am trying to install mpi4py with python3.8 using pip and virtualenv on my Macbook. When I run pip install ---no-cache-dir mpi4p, I get an error of error: Cannot compile MPI programs. Check your configuration!!! See attached “LOGFILE” starting at line 254.

After reading other similar issues here and online, I am unsure what is going wrong. Following the instructions from the website, I have a working MPI (mpich using brew) and when I run which mpicc I get /usr/local/bin/mpicc. I am also able to compile C code with it, so my suspect is mpicc is not at fault.

I’ve tried various tweaks, including trying to run without virtualenv, modifying MPI_CC, and I’ve tried building with python setup.py build, but they all result in the same error. Any thoughts on how to proceed? Should I be using a different compiler? Am I missing a dependency? Is pip not appropriate here? Any help would be greatly appreciated.

‌

[Lisandro Dalcin]

Well, you are using Homebrew Python, they decided to build Python in a way that is quite problematic, at least for mpi4py, which tries to replace the compiler commands distutils uses for building extension modules.

First you should try to run the following in a terminal:

pushd /tmp
rm -f tmp.c && touch tmp.c
xcrun -sdk macosx clang -arch x86_64 -c tmp.c

If the above commands succeed without error, then try installing mpi4py the following way:

export OMPI_CC=xcrun
export MPICH_CC=xcrun
python3 -m pip install --no-cache-dir mpi4py

‌

[Caleb Ju]

Thanks for the response. Your fix helped me move past my error but presented a new one, namely mpicc was broken. I did not save logfiles, but mpicc was complaining about xcrun and when I ran pip install --no-cache-dir mpi4py, I had error: Cannot link MPI programs. Check your configuration!!!, so maybe it was related to #118.

Fortunately, your comment of Homebrew Python being problematic helped me find an alternative solution. I installed another version of Python using pyenv and now mpi4py can be installed.

[Lisandro Dalcin]

No, I don’t think the problem is related to #118. You were using Homebrew Python, not Anaconda Python. Anyway, using Anaconda Python (minconda installer) and conda-forge packages is another quick and convenient way of using mpi4py, you conda install -c conda-forge mpi4py and you get everything you need, MPI included.

[jik yan]

hi, I have a same promble with python3.7, and mpicc by run brew install open-mpi

And when I run /usr/local/bin/mpicc -sdk macosx clang -arch x86_64 -c tmp.c, I get an error like this:

clang: error: unknown argument: '-sdk'
clang: error: no such file or directory: 'macosx'
clang: error: no such file or directory: 'clang'

I can‘t find args -sdk in /usr/local/bin/mpicc --help. And I can compile tmp.c, when I delete -sdk macosx clang.

‌

I run /usr/local/bin/mpicc --version, and get this:
Apple clang version 11.0.3 (clang-1103.0.32.62)
Target: x86_64-apple-darwin19.4.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

‌

Can you give some suggestion about my mpicc version or how to delete sdk arguments when I building mpi4py

‌

‌

[Caleb Ju]

I don’t have an answer to your problem, but I do have some questions as I initially faced these same errors (for instance, I also had “clang:error: …” ).

First, what do you get when you run which mpicc? Is it /usr/local/bin/mpicc?

Although this is just an ad-hoc fix, how did you install Python? Was it through brew? As Lisandro pointed in his first response, brew’s build of Python is problematic for mpi4py. Perhaps try using a different python. For me, I installed another Python via pyenv and this removed the clang errors.

‌

[Caleb Ju]

Also it might be helpful to include logfiles of the pip install (if that’s what you using, I’m actually not sure how you are installing mpi4py here), which can be created by adding --log <your log filename>.

‌

[Lisandro Dalcin]

BTW, If you manage to build wheel file with a pyenv Python, then I expect that wheel file to work just fine if installed in Homebrew Python. Could any of you try that?

[Lisandro Dalcin]

Could you tell me the output of the following two lines executed in a terminal?

python3 -c "import sysconfig; print(sysconfig.get_config_var('CC'))"
python3 -c "import sysconfig; print(sysconfig.get_config_var('LDSHARED'))"

‌

[Lisandro Dalcin]

Folks, do you have the Xcode Command Line Tools installed? Can you double check running xcode-select --install and install them if they are not yet installed?

I’m still clueless about where xcrun originates.

[Caleb Ju]

I’ll try my best here. I switched to my system’s python (/usr/bin/python). I reinstalled brew from my earlier attempts and am using python3.7.3 here.

First off, wheel does not work for me. Attached here is the log file.

When I run the two lines above with the system’s python, I get

xcrun -sdk macosx clang    -arch x86_64
xcrun -sdk macosx clang    -arch x86_64 -bundle -undefined dynamic_lookup

Switching over to pyenv which can install mpi4py (I’m using python3.6.9 specifically due to extraneous reasons), the two same lines result in

clang
clang -bundle -undefined dynamic_lookup -L/usr/local/opt/readline/lib -L/usr/local/opt/readline/lib -L/usr/local/opt/openssl@1.1/lib -L/Users/calebju/.pyenv/versions/3.6.9/lib -L/usr/local/opt/readline/lib -L/usr/local/opt/readline/lib -L/usr/local/opt/openssl@1.1/lib -L/Users/calebju/.pyenv/versions/3.6.9/lib

I already have Xcode command line tools installed (I also have Xcode if that is relevant).

[Lisandro Dalcin]

Sorry, but you confused me. The first two lines, those with xcrun are From system (Apple) Python from /usr/bin/python, or Homebrew Python? Note that Homebrew Python should be located in /usr/local/bin, double check your $PATH and run hash -r in your bash shell.

In the LOG file you linked, I’m not sure you are really using Homebrew Python. I think you are still using system Python.

To make sure you are using Homebrew Python, run like this

/usr/local/bin/python3 -m pip install mpi4py --log LOG

Does that work?

[Caleb Ju]

Wow, yes, running

/usr/local/bin/python3 -m pip install mpi4py --log LOG

worked for me.

Edit:

When I run

python3 -c "import sysconfig; print(sysconfig.get_config_var('CC'))"
python3 -c "import sysconfig; print(sysconfig.get_config_var('LDSHARED'))"

with /usr/local/bin/python3, I get

clang
clang -bundle -undefined dynamic_lookup -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk

‌

[Lisandro Dalcin]

So, there you have. Please consider runing cd /usr/local/bin; sudo ln -s python3 python; so that you never ever again have to deal with system Python.

[Jasmyn Z]

Hi Lisandro,

I have ran conda install -c conda-forge mpi4py successfully on MacOS, but why when I try this command : from mpi4py import MPI, it said

---

ImportError Traceback (most recent call last)
<ipython-input-28-ac4b05198e5f> in <module>
----> 1 from mpi4py import MPI

ImportError: cannot import name 'MPI' from 'mpi4py'

Can you help me? Thanks!!

[Lisandro Dalcin]

Dear Jazmin Z, mpi4py development moved to GitHub. Please post your issues to mpi4py’s mailing list in Google Groups ([e-mail]). All that being said, I bet your IPython is not running under the python executable installed within your conda environment, you can confirm it with print(sys.executable).

[Jasmyn Z]

Hi Lisandro,

Thank you for your reply! I will post my issue to the group. And it said name 'sys' is not defined after I try print(sys.executable). By the way, what method you suggested to use when installing the mpi4y on macOS at anaconda with python 3.8?

Thanks!

[Lisandro Dalcin]

You have to import sys first! I assumed you knew that sys is a builtin Python module, and that you have to import it first to use it.

I think you have to install ipython in your conda environment, otherwise your ipython is working with a different python executable installed elsewhere.

[Jasmyn Z]

Here shows that : /Users/jasmyn/opt/anaconda3/bin/python

[Lisandro Dalcin]

What’s the output of the following two lines?

import mpi4py
print(mpi4py)

‌

[Jasmyn Z]

Because I used a lot of methods that didn't work, but downloaded a lot of different packages, SO I cleaned them all by using:
conda uninstall mpi4py
conda uninstall ompenmpi

Now, it shows No module named 'mpi4py' now.

Then I tried to use :

brew install --build-from-source openmpi
pip install mpi4py

But it shown error when I enter : pip install mpi4py

‌

Then, I used:

brew install mpi4py

to do it. Although it didn’t show error, it also shown No module named 'mpi4y' when I enter 'import mpi4y' in jupyter-notebook now.

‌

‌

‌

[Lisandro Dalcin]

You are doing a complete and total mess. You are mixing conda and Homebrew, and you cannot simply do that and expect things to work. You need to pick one, and stick to that one.

If you pick Homebrew, then get rid of your conda install (or at least make it sure the install is not active, check with echo $PATH). Then you can brew install mpi4py, and then Homebrew should install any required dependencies. After you do that, you can run the following to test it worked:

mpiexec -n 4 python3 -m mpi4py.bench helloworld

‌

[Jasmyn Z]

Great!

I tried that and the test results show:

Successfully built mpi4py

Installing collected packages: mpi4py

Successfully installed mpi4py-3.1.3

(base) jasmyn@JasmyndeMacBook-Pro ~ % mpiexec -n 4 python3 -m mpi4py.bench helloworld

Hello, World! I am process 0 of 4 on JasmyndeMacBook-Pro.local.

Hello, World! I am process 1 of 4 on JasmyndeMacBook-Pro.local.

Hello, World! I am process 2 of 4 on JasmyndeMacBook-Pro.local.

Hello, World! I am process 3 of 4 on JasmyndeMacBook-Pro.local.

Thank you very much!!!!

‌