ImportError: PATH_TO_VENV/lib/python2.7/site-packages/mpi4py/MPI.so: undefined symbol: PyUnicodeUCS2_FromStringAndSize
I'm using virtualenv (for python2.7) to manage the python environment.
They used to work fine, as well as the mpi4py library (both building and using).
Today, when I switch to a new cluster, I cloned the source code and built it (inside virtualenv). These steps went without any issues.
However, when I try to do some tests, the error occurs.
(I've manually replaced the very long path to venv with PATH_TO_VENV
.)
Python 2.7.5 (default, Nov 19 2015, 14:51:11)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import mpi4py
>>> from mpi4py import MPI
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: PATH_TO_VENV/lib/python2.7/site-packages/mpi4py/MPI.so: undefined symbol: PyUnicodeUCS2_FromStringAndSize
>>>
As I'm always inside virtualenv, it doesn't seem to be the common cause (conflictions of python installations) of this kind of error.
And because it works fine in other platforms (I've tested today), I'm not sure if this is a mpi4py problem. But here should be the most relevant place to ask (at least, a start point).
Is this because I missed something when building it? I simply used (because I don't know what else could be used)
python setup.py build
python setup.py install
No configuration files were modified by me. (The file mpi.cfg
states I should manually modify it to point to the correct location of mpi, but I never edited it and everything works fine previously. There is no big difference between previous platforms and the current platform - mpi is always not in /usr/{,local}/{bin,include,lib}
)
This time I used module load some_mpi_version
to load a mpi build. It works without issues on one of them called openmpi/1.8.5, but the error occurs this time when I used another one.
I also read issue #63 and tried python setup.py build --configure
. The problem remains the same, though the time used in building mpi4py became longer and it generated some extra outputs (and files).
Comments (13)
-
-
reporter Thanks for your help :)
The output of
ldd /path/to/MPI.so
is:linux-vdso.so.1 => (0x00007ffed1da6000) libdl.so.2 => /lib64/libdl.so.2 (0x00007f6b2ce7b000) libpython2.7.so.1.0 => /lib64/libpython2.7.so.1.0 (0x00007f6b2cab4000) libmpi.so.12 => /exports/igmm/software/pkg/el7/mpi/gcc/mpich/3.1.4/lib/libmpi.so.12 (0x00007f6b2c654000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f6b2c438000) libc.so.6 => /lib64/libc.so.6 (0x00007f6b2c075000) /lib64/ld-linux-x86-64.so.2 (0x00007f6b2d436000) libutil.so.1 => /lib64/libutil.so.1 (0x00007f6b2be72000) libm.so.6 => /lib64/libm.so.6 (0x00007f6b2bb70000) librt.so.1 => /lib64/librt.so.1 (0x00007f6b2b967000) libgpfs.so => /lib64/libgpfs.so (0x00007f6b2b752000) libgfortran.so.3 => /lib64/libgfortran.so.3 (0x00007f6b2b42f000) libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f6b2b218000) libquadmath.so.0 => /lib64/libquadmath.so.0 (0x00007f6b2afdc000)
Not sure how to interpret it. It seems it's using system python? (But isn't the python version in the venv the same as the system python version?)
I just did
python setup.py install --user
, but I still get thatImportError
. -
Are you sure you created your venv with
/usr/bin/python
? The problem here should be that mpi4py was built with Python headers that do not correspond to/lib64/libpython2.7.so.1.0
.- What's the output of
ldd PATH_TO_VENV/bin/python
? - What's the value of
sys.maxunicode
in your venv Python interpreter?
- What's the output of
-
reporter The command I used to create the virtual environment is simply
virtualenv venv
, andvirtualenv --help
tells me the default one is/usr/bin/python
.$ which virtualenv /usr/bin/virtualenv $ which python /usr/bin/python
$ ldd PATH_TO_VENV/bin/python linux-vdso.so.1 => (0x00007ffd6ed73000) libpython2.7.so.1.0 => /lib64/libpython2.7.so.1.0 (0x00007fa1bb54f000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fa1bb332000) libdl.so.2 => /lib64/libdl.so.2 (0x00007fa1bb12e000) libutil.so.1 => /lib64/libutil.so.1 (0x00007fa1baf2b000) libm.so.6 => /lib64/libm.so.6 (0x00007fa1bac28000) libc.so.6 => /lib64/libc.so.6 (0x00007fa1ba866000) /lib64/ld-linux-x86-64.so.2 (0x00007fa1bb935000)
Python 2.7.5 (default, Nov 19 2015, 14:51:11) [GCC 4.8.5 20150623 (Red Hat 4.8.5-4)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> sys.maxunicode 1114111
The strange thing is that by default there is no pip, but virtualenv could create virtual environments with pip installed.
I also found there is anaconda installed, so I tried it just now. But the error is the same.
The differences are:- python version from 2.7.5 to 2.7.13
- the
libpython2.7.so.1.0
line ofldd
is the conda onePATH_TO_MY_CONDA_ENV/lib/libpython2.7.so.1.0 (0x00007f7008c33000)
-
So your value of
sys.maxunicode
confirms your python is a "wide unicode" build, which is the usual setup for Linux. However, mpi4py is being built in "narrow unicode" mode, and it refers toPyUnicodeUCS2_XXX
symbols instead ofPyUnicodeUCS4_XXX
symbols.What's the output of
grep Py_UNICODE_SIZE /usr/include/python2.7/pyconfig*.h
?BTW, every time you switch your python interpreter (either to virtualenv or conda), in the mpi4py source tree, you should
rm -r build
to remove previous builds. Could you do that ant try to build&install again? Maybe you were installing an outdated build.Also, clear the pip cache with
rm -r ~/.cache/pip
(Python 2 has no way to distinguish between wide and narrow unicode builds, so the pip cache may get populated with incompatible binaries). -
reporter The output is:
$ grep Py_UNICODE_SIZE /usr/include/python2.7/pyconfig*.h usr/include/python2.7/pyconfig-64.h:#define Py_UNICODE_SIZE 4
Yes I did
make clean
every time I swith to a new interpreter (and MPI library) - it contains the steprm -r build
.
Removing~/.cache/pip
doesn't seem to help. I tried to remove it after installing mpi4py with no luck, and the tried to delete it before building also with no luck.I also tried creating a python3 environment with conda, but still get error, though a different one:
Python 3.6.1 |Continuum Analytics, Inc.| (default, May 11 2017, 13:09:58) [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux Type "help", "copyright", "credits" or "license" for more information. >>> from mpi4py import MPI Traceback (most recent call last): File "<stdin>", line 1, in <module> ImportError: PATH_TO_MY_CONDA_PY3_ENV/lib/python3.6/site-packages/mpi4py/MPI.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _Py_ZeroStruct >>>
Could it be something that I didn't do to build mpi4py? (Actually I don't know anything extra can do to build mpi4py, except for
python setup.py build
) -
Something weird is going on in your system. If mpi4py is built with Python 3, then there should not be any reference to
_Py_ZeroStruct
in the extension module, that is a Python 2 symbol.Now I definitely believe something is broken in your build environment. Do you have the
LD_LIBRARY_PATH
variable set in your shell? Did you try to build and run any other Python package with extension modules? Could you attach a file with the full output of a fresh build withpython setup.py build
to take a look at the compiler/linker invocations? -
reporter - attached output
There are
LD_LIBRARY_PATH
, but files under that directory doesn't seem to be affective.$ echo $LD_LIBRARY_PATH /exports/igmm/software/pkg/el7/mpi/gcc/mpich/3.1.4/lib/:/exports/applications//gridengine/2011.11p1_155/lib/linux-x64
I'm not sure if I understand it correctly, but I tried numpy which seems to work correctly.
The attachment is the output of
python setup.py build
. Here is stderr:/usr/bin/ld: cannot find -llmpe collect2: error: ld returned 1 exit status /usr/bin/ld: cannot find -lvt-mpi collect2: error: ld returned 1 exit status /usr/bin/ld: cannot find -lvt.mpi collect2: error: ld returned 1 exit status /usr/bin/ld: cannot find -lvt-mpi collect2: error: ld returned 1 exit status /usr/bin/ld: cannot find -lvt.mpi collect2: error: ld returned 1 exit status /usr/bin/ld: cannot find -lvt-hyb collect2: error: ld returned 1 exit status /usr/bin/ld: cannot find -lvt.ompi collect2: error: ld returned 1 exit status
-
Sorry for taking so long to be back at your issue. Still having trouble? Could you figure out what's going on?
-
@renyuneyun Ping. Any news?
-
reporter Hi, there.
The problem wasn't solved, but I switched to another MPI environment (SGE module) and that one worked correctly.
Still not sure why the errors happened.
-
Chances are high that the previous build environment has a broken Python build. Thanks!.
-
- changed status to closed
- Log in to comment
Maybe you mpi4py was shomehow built with a different Python runtime than the one you are using in your virtualenv? What's the output of
ldd /path/to/MPI.so
? Anyway, this does not look like an mpi4py issue, not even an MPI issue. Maybe you have a badLD_LIBRARY_PATH
environment variable?I would suggest to debug this issue by not using a virtualenv. Instead, using the "system" Python, do:
and mpi4py will get installed under ~/.local