ImportError: PATH_TO_VENV/lib/python2.7/site-packages/mpi4py/MPI.so: undefined symbol: PyUnicodeUCS2_FromStringAndSize

[renyuneyun]

I'm using virtualenv (for python2.7) to manage the python environment.
They used to work fine, as well as the mpi4py library (both building and using).

Today, when I switch to a new cluster, I cloned the source code and built it (inside virtualenv). These steps went without any issues.
However, when I try to do some tests, the error occurs.

(I've manually replaced the very long path to venv with PATH_TO_VENV.)

Python 2.7.5 (default, Nov 19 2015, 14:51:11) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import mpi4py
>>> from mpi4py import MPI
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: PATH_TO_VENV/lib/python2.7/site-packages/mpi4py/MPI.so: undefined symbol: PyUnicodeUCS2_FromStringAndSize
>>>

As I'm always inside virtualenv, it doesn't seem to be the common cause (conflictions of python installations) of this kind of error.
And because it works fine in other platforms (I've tested today), I'm not sure if this is a mpi4py problem. But here should be the most relevant place to ask (at least, a start point).

Is this because I missed something when building it? I simply used (because I don't know what else could be used)

python setup.py build
python setup.py install

No configuration files were modified by me. (The file mpi.cfg states I should manually modify it to point to the correct location of mpi, but I never edited it and everything works fine previously. There is no big difference between previous platforms and the current platform - mpi is always not in /usr/{,local}/{bin,include,lib})

This time I used module load some_mpi_version to load a mpi build. It works without issues on one of them called openmpi/1.8.5, but the error occurs this time when I used another one.

I also read issue #63 and tried python setup.py build --configure. The problem remains the same, though the time used in building mpi4py became longer and it generated some extra outputs (and files).

[Lisandro Dalcin]

Maybe you mpi4py was shomehow built with a different Python runtime than the one you are using in your virtualenv? What's the output of ldd /path/to/MPI.so ? Anyway, this does not look like an mpi4py issue, not even an MPI issue. Maybe you have a bad LD_LIBRARY_PATH environment variable?

I would suggest to debug this issue by not using a virtualenv. Instead, using the "system" Python, do:

$ python setup.py install --user

and mpi4py will get installed under ~/.local

[renyuneyun]

Thanks for your help :)

The output of ldd /path/to/MPI.so is:

        linux-vdso.so.1 =>  (0x00007ffed1da6000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f6b2ce7b000)
        libpython2.7.so.1.0 => /lib64/libpython2.7.so.1.0 (0x00007f6b2cab4000)
        libmpi.so.12 => /exports/igmm/software/pkg/el7/mpi/gcc/mpich/3.1.4/lib/libmpi.so.12 (0x00007f6b2c654000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f6b2c438000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f6b2c075000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f6b2d436000)
        libutil.so.1 => /lib64/libutil.so.1 (0x00007f6b2be72000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f6b2bb70000)
        librt.so.1 => /lib64/librt.so.1 (0x00007f6b2b967000)
        libgpfs.so => /lib64/libgpfs.so (0x00007f6b2b752000)
        libgfortran.so.3 => /lib64/libgfortran.so.3 (0x00007f6b2b42f000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f6b2b218000)
        libquadmath.so.0 => /lib64/libquadmath.so.0 (0x00007f6b2afdc000)

Not sure how to interpret it. It seems it's using system python? (But isn't the python version in the venv the same as the system python version?)

I just did python setup.py install --user, but I still get that ImportError.

[Lisandro Dalcin]

Are you sure you created your venv with /usr/bin/python ? The problem here should be that mpi4py was built with Python headers that do not correspond to /lib64/libpython2.7.so.1.0.

• What's the output of ldd PATH_TO_VENV/bin/python?
• What's the value of sys.maxunicode in your venv Python interpreter?

[renyuneyun]

The command I used to create the virtual environment is simply virtualenv venv, and virtualenv --help tells me the default one is /usr/bin/python.

$ which virtualenv
/usr/bin/virtualenv
$ which python
/usr/bin/python

$ ldd PATH_TO_VENV/bin/python
        linux-vdso.so.1 =>  (0x00007ffd6ed73000)
        libpython2.7.so.1.0 => /lib64/libpython2.7.so.1.0 (0x00007fa1bb54f000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fa1bb332000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007fa1bb12e000)
        libutil.so.1 => /lib64/libutil.so.1 (0x00007fa1baf2b000)
        libm.so.6 => /lib64/libm.so.6 (0x00007fa1bac28000)
        libc.so.6 => /lib64/libc.so.6 (0x00007fa1ba866000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fa1bb935000)

Python 2.7.5 (default, Nov 19 2015, 14:51:11) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.maxunicode
1114111

The strange thing is that by default there is no pip, but virtualenv could create virtual environments with pip installed.

I also found there is anaconda installed, so I tried it just now. But the error is the same.
The differences are:

• python version from 2.7.5 to 2.7.13
• the libpython2.7.so.1.0 line of ldd is the conda one PATH_TO_MY_CONDA_ENV/lib/libpython2.7.so.1.0 (0x00007f7008c33000)

[Lisandro Dalcin]

So your value of sys.maxunicode confirms your python is a "wide unicode" build, which is the usual setup for Linux. However, mpi4py is being built in "narrow unicode" mode, and it refers to PyUnicodeUCS2_XXX symbols instead of PyUnicodeUCS4_XXX symbols.

What's the output of grep Py_UNICODE_SIZE /usr/include/python2.7/pyconfig*.h ?

BTW, every time you switch your python interpreter (either to virtualenv or conda), in the mpi4py source tree, you should rm -r build to remove previous builds. Could you do that ant try to build&install again? Maybe you were installing an outdated build.

Also, clear the pip cache with rm -r ~/.cache/pip (Python 2 has no way to distinguish between wide and narrow unicode builds, so the pip cache may get populated with incompatible binaries).

[renyuneyun]

The output is:

$ grep Py_UNICODE_SIZE /usr/include/python2.7/pyconfig*.h
usr/include/python2.7/pyconfig-64.h:#define Py_UNICODE_SIZE 4

Yes I did make clean every time I swith to a new interpreter (and MPI library) - it contains the step rm -r build.
Removing ~/.cache/pip doesn't seem to help. I tried to remove it after installing mpi4py with no luck, and the tried to delete it before building also with no luck.

I also tried creating a python3 environment with conda, but still get error, though a different one:

Python 3.6.1 |Continuum Analytics, Inc.| (default, May 11 2017, 13:09:58) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from mpi4py import MPI
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: PATH_TO_MY_CONDA_PY3_ENV/lib/python3.6/site-packages/mpi4py/MPI.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _Py_ZeroStruct
>>> 

Could it be something that I didn't do to build mpi4py? (Actually I don't know anything extra can do to build mpi4py, except for python setup.py build)

[Lisandro Dalcin]

Something weird is going on in your system. If mpi4py is built with Python 3, then there should not be any reference to _Py_ZeroStruct in the extension module, that is a Python 2 symbol.

Now I definitely believe something is broken in your build environment. Do you have the LD_LIBRARY_PATH variable set in your shell? Did you try to build and run any other Python package with extension modules? Could you attach a file with the full output of a fresh build with python setup.py build to take a look at the compiler/linker invocations?

[renyuneyun]

There are LD_LIBRARY_PATH, but files under that directory doesn't seem to be affective.

$ echo $LD_LIBRARY_PATH
/exports/igmm/software/pkg/el7/mpi/gcc/mpich/3.1.4/lib/:/exports/applications//gridengine/2011.11p1_155/lib/linux-x64

I'm not sure if I understand it correctly, but I tried numpy which seems to work correctly.

The attachment is the output of python setup.py build. Here is stderr:

/usr/bin/ld: cannot find -llmpe
collect2: error: ld returned 1 exit status
/usr/bin/ld: cannot find -lvt-mpi
collect2: error: ld returned 1 exit status
/usr/bin/ld: cannot find -lvt.mpi
collect2: error: ld returned 1 exit status
/usr/bin/ld: cannot find -lvt-mpi
collect2: error: ld returned 1 exit status
/usr/bin/ld: cannot find -lvt.mpi
collect2: error: ld returned 1 exit status
/usr/bin/ld: cannot find -lvt-hyb
collect2: error: ld returned 1 exit status
/usr/bin/ld: cannot find -lvt.ompi
collect2: error: ld returned 1 exit status

[Lisandro Dalcin]

Sorry for taking so long to be back at your issue. Still having trouble? Could you figure out what's going on?

[Lisandro Dalcin]

@renyuneyun Ping. Any news?

[renyuneyun]

Hi, there.

The problem wasn't solved, but I switched to another MPI environment (SGE module) and that one worked correctly.

Still not sure why the errors happened.

[Lisandro Dalcin]

Chances are high that the previous build environment has a broken Python build. Thanks!.