Issue while distutils install mpi4py using pypy

[Alain Cady]

Hi there,

I'm currently trying to get mpi4py 1.3.1 working with pypy, but setuping leads to following stacktrace:

/usr/bin/pypy setup.py build
running build
running build_src
running build_py
creating /var/tmp/portage/dev-python/mpi4py-1.3.1/work/mpi4py-1.3.1-pypy/build/lib/mpi4py
copying src/__init__.py -> /var/tmp/portage/dev-python/mpi4py-1.3.1/work/mpi4py-1.3.1-pypy/build/lib/mpi4py
copying src/rc.py -> /var/tmp/portage/dev-python/mpi4py-1.3.1/work/mpi4py-1.3.1-pypy/build/lib/mpi4py
creating /var/tmp/portage/dev-python/mpi4py-1.3.1/work/mpi4py-1.3.1-pypy/build/lib/mpi4py/include
creating /var/tmp/portage/dev-python/mpi4py-1.3.1/work/mpi4py-1.3.1-pypy/build/lib/mpi4py/include/mpi4py
copying src/include/mpi4py/mpi4py.h -> /var/tmp/portage/dev-python/mpi4py-1.3.1/work/mpi4py-1.3.1-pypy/build/lib/mpi4py/include/mpi4py
copying src/include/mpi4py/mpi4py.MPI.h -> /var/tmp/portage/dev-python/mpi4py-1.3.1/work/mpi4py-1.3.1-pypy/build/lib/mpi4py/include/mpi4py
copying src/include/mpi4py/mpi4py.MPI_api.h -> /var/tmp/portage/dev-python/mpi4py-1.3.1/work/mpi4py-1.3.1-pypy/build/lib/mpi4py/include/mpi4py
copying src/include/mpi4py/__init__.pxd -> /var/tmp/portage/dev-python/mpi4py-1.3.1/work/mpi4py-1.3.1-pypy/build/lib/mpi4py/include/mpi4py
copying src/include/mpi4py/mpi_c.pxd -> /var/tmp/portage/dev-python/mpi4py-1.3.1/work/mpi4py-1.3.1-pypy/build/lib/mpi4py/include/mpi4py
copying src/include/mpi4py/MPI.pxd -> /var/tmp/portage/dev-python/mpi4py-1.3.1/work/mpi4py-1.3.1-pypy/build/lib/mpi4py/include/mpi4py
copying src/include/mpi4py/__init__.pyx -> /var/tmp/portage/dev-python/mpi4py-1.3.1/work/mpi4py-1.3.1-pypy/build/lib/mpi4py/include/mpi4py
copying src/include/mpi4py/mpi.pxi -> /var/tmp/portage/dev-python/mpi4py-1.3.1/work/mpi4py-1.3.1-pypy/build/lib/mpi4py/include/mpi4py
copying src/include/mpi4py/mpi4py.i -> /var/tmp/portage/dev-python/mpi4py-1.3.1/work/mpi4py-1.3.1-pypy/build/lib/mpi4py/include/mpi4py
copying src/MPI.pxd -> /var/tmp/portage/dev-python/mpi4py-1.3.1/work/mpi4py-1.3.1-pypy/build/lib/mpi4py
copying src/mpi_c.pxd -> /var/tmp/portage/dev-python/mpi4py-1.3.1/work/mpi4py-1.3.1-pypy/build/lib/mpi4py
warning: build_py: byte-compiling is disabled, skipping.

running build_clib
MPI configuration: [mpi] from 'mpi.cfg'
MPI C compiler:    /usr/bin/mpicc
MPI C++ compiler:  /usr/bin/mpicxx
MPI F77 compiler:  /usr/bin/mpif77
MPI F90 compiler:  /usr/bin/mpif90
Traceback (most recent call last):
  File "<builtin>/app_main.py", line 75, in run_toplevel
  File "setup.py", line 588, in <module>
    main()
  File "setup.py", line 585, in main
    run_setup()
  File "setup.py", line 459, in run_setup
    **metadata)
  File "/var/tmp/portage/dev-python/mpi4py-1.3.1/work/mpi4py-1.3.1-pypy/conf/mpidistutils.py", line 495, in setup
    return fcn_setup(**attrs)
  File "/usr/lib64/pypy/lib-python/2.7/distutils/core.py", line 151, in setup
    dist.run_commands()
  File "/usr/lib64/pypy/lib-python/2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/usr/lib64/pypy/lib-python/2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/usr/lib64/pypy/lib-python/2.7/distutils/command/build.py", line 127, in run
    self.run_command(cmd_name)
  File "/usr/lib64/pypy/lib-python/2.7/distutils/cmd.py", line 326, in run_command
    self.distribution.run_command(command)
  File "/usr/lib64/pypy/lib-python/2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/var/tmp/portage/dev-python/mpi4py-1.3.1/work/mpi4py-1.3.1-pypy/conf/mpidistutils.py", line 895, in run
    configure_compiler(self.compiler, config)
  File "/var/tmp/portage/dev-python/mpi4py-1.3.1/work/mpi4py-1.3.1-pypy/conf/mpidistutils.py", line 230, in configure_compiler
    mpicc=mpicc, mpicxx=mpicxx, mpild=mpild)
  File "/var/tmp/portage/dev-python/mpi4py-1.3.1/work/mpi4py-1.3.1-pypy/conf/mpidistutils.py", line 121, in customize_compiler
    cxx = cxx .replace('-pthread', '')
AttributeError: 'NoneType' object has no attribute 'replace'

I'm not familiar enough with distutils to tell where the error came from. I'm currently using Pypy 2.6.0.

[Lisandro Dalcin]

Why are you trying to use the old release 1.3.1? Could you try 2.0.0 instead?

[Alain Cady]

While developing on other(s) platform(s), target system is RHEL 7.1 which provides mpi4py 1.3.1 ; so we based our development on this version.

Installing mpi4py 2.0.0 with pypy, using very same methods, works just fine.

[Lisandro Dalcin]

Well, mpi4py 1.3.1 predates pypy 2.6.0, and a new mpi4py release is out, so I'm not going to take any action on this issue.

That being said, you should be able to workaround the failure by editing mpidistutils.py and adding a line with let say cxx = 'g++' right above the offending line. As mpi4py does not build with C++ by default, this should be enough.

[Alain Cady]

Hi,

Thanks to your tips, I manage to get a little further:

Defining cxx, ld and ccshared (to '-fPIC') enable mpi4pypy compile/install, but importing fails when ldopening MPI.so, because of missing PyByteArray_Resize...

Fixing that seems a bit tricky, and pointless so will try to use mpi4py 2.0.0. Anyway, thanks for help!

Just for the sake of precision: mpi4py RPM came from EPEL repo, RHEL one doesn't provide any.