openmpi-based mpi4py runtests.py error

Issue #78 resolved
Timothy Jim
created an issue

I'm not sure if this is the right place to post this but I am having trouble with running the unit tests on myu mpi4py install. I am running Ubuntu 16.04 and have got a working implementation of opempi 3.0.0 installed. Following https://mpi4py.readthedocs.io/en/stable/install.html I installed mpi4py using:

pip install mpi4py

It appears to have successfully picked up on my setup, and I could run the test

mpiexec -n 4 python -m mpi4py helloworld

I couldn't find where pip keeps the source files, so I downloaded the source from mpi4py github to run the unit tests, as suggested in the install guide. However, I had the following error below. Is this a problem? Kind regards.

tjim@DESKTOP-TA3P0PS:~/Downloads/mpi4py-master$ mpiexec -n 4 python test/runtests.py
[2@DESKTOP-TA3P0PS] Python 3.6 (/home/tjim/anaconda3/bin/python)
[2@DESKTOP-TA3P0PS] MPI 3.1 (Open MPI 3.0.0)
[3@DESKTOP-TA3P0PS] Python 3.6 (/home/tjim/anaconda3/bin/python)
[2@DESKTOP-TA3P0PS] mpi4py 2.0.0 (/home/tjim/anaconda3/lib/python3.6/site-packages/mpi4py)
[0@DESKTOP-TA3P0PS] Python 3.6 (/home/tjim/anaconda3/bin/python)
[0@DESKTOP-TA3P0PS] MPI 3.1 (Open MPI 3.0.0)
[0@DESKTOP-TA3P0PS] mpi4py 2.0.0 (/home/tjim/anaconda3/lib/python3.6/site-packages/mpi4py)
[3@DESKTOP-TA3P0PS] MPI 3.1 (Open MPI 3.0.0)
[1@DESKTOP-TA3P0PS] Python 3.6 (/home/tjim/anaconda3/bin/python)
[3@DESKTOP-TA3P0PS] mpi4py 2.0.0 (/home/tjim/anaconda3/lib/python3.6/site-packages/mpi4py)
[1@DESKTOP-TA3P0PS] MPI 3.1 (Open MPI 3.0.0)
[1@DESKTOP-TA3P0PS] mpi4py 2.0.0 (/home/tjim/anaconda3/lib/python3.6/site-packages/mpi4py)
EEEEEEEE.sE...ss........F.FEEE.EE.EFEEEEEE.s....FEEE................FFFFEEEEEEEEEEEE............FFF.EEEEEE.EEE............FFF.EEEEEE.EEE............FFFEFEEEEEEEE.E............E..EEEEEEEE.EE...F.E...E......E................F..EE.E.......EEE.....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................EEEE................E....E.E................E...........EEEE........................EEEE........................................................................................................s.............s.............s.............s................................................................................................................................................................................................................................................................................................................................................................................................................................E...E............................................................................................................................................................................E.........EE.........E....................E...E.................................................................................................................ssssssssssssssssssssssss.....................................................ssssss.........ssssss..........ssssss.........ssssss...............ssssss.......s...s.................ssssss......s...s...........ssssss.........s.....s..........Fatal Python error: exception in user-defined reduction operation
Traceback (most recent call last):
  File "MPI/opimpl.pxi", line 99, in mpi4py.MPI.op_user_mpi (src/mpi4py.MPI.c:19531)
  File "MPI/opimpl.pxi", line 90, in mpi4py.MPI.op_user_py (src/mpi4py.MPI.c:19417)
  File "test/test_op.py", line 41, in mysum
    return mysum_buf(ba, bb, dt)
  File "test/test_op.py", line 35, in mysum_buf
    b[:] = mysum_obj(asarray('i', a), asarray('i', b))
ValueError: memoryview assignment: lvalue and rvalue have different structures
Fatal Python error: exception in user-defined reduction operation
Traceback (most recent call last):
  File "MPI/opimpl.pxi", line 99, in mpi4py.MPI.op_user_mpi (src/mpi4py.MPI.c:19531)
  File "MPI/opimpl.pxi", line 90, in mpi4py.MPI.op_user_py (src/mpi4py.MPI.c:19417)
  File "test/test_op.py", line 41, in mysum
    return mysum_buf(ba, bb, dt)
  File "test/test_op.py", line 35, in mysum_buf
    b[:] = mysum_obj(asarray('i', a), asarray('i', b))
ValueError: memoryview assignment: lvalue and rvalue have different structures
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 3 in communicator MPI_COMM_WORLD
with errorcode 1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
[DESKTOP-TA3P0PS:04099] 1 more process has sent help message help-mpi-api.txt / mpi-abort
[DESKTOP-TA3P0PS:04099] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages

Comments (33)

  1. Lisandro Dalcin

    You should install mpi4py from the git master, pip install https://bitbucket.org/mpi4py/mpi4py/get/master.tar.gz. The master branch have some improvements that workaround issues with Python's memoryview behavior across versions.

  2. Timothy Jim reporter

    @Lisandro Dalcin I tried out your suggestion above, but it seems that it has broken the helloworld test above too. When I try to run

    mpiexec -n 4 python -m mpi4py helloworld
    

    I get the following error. What may have happened?

    tjim@DESKTOP-TA3P0PS:~$ mpiexec -n 4 python -m mpi4py helloworld 
    Traceback (most recent call last):
      File "/home/tjim/anaconda3/lib/python3.6/runpy.py", line 193, in _run_module_as_main
        "__main__", mod_spec)
      File "/home/tjim/anaconda3/lib/python3.6/runpy.py", line 85, in _run_code
        exec(code, run_globals)
      File "/home/tjim/anaconda3/lib/python3.6/site-packages/mpi4py/__main__.py", line 7, in <module>
        main()
      File "/home/tjim/anaconda3/lib/python3.6/site-packages/mpi4py/run.py", line 196, in main
        run_command_line(args)
      File "/home/tjim/anaconda3/lib/python3.6/site-packages/mpi4py/run.py", line 47, in run_command_line
        run_path(sys.argv[0], run_name='__main__')
      File "/home/tjim/anaconda3/lib/python3.6/runpy.py", line 261, in run_path
        code, fname = _get_code_from_file(run_name, path_name)
      File "/home/tjim/anaconda3/lib/python3.6/runpy.py", line 236, in _get_code_from_file
        code = compile(f.read(), fname, 'exec')
    ValueError: source code string cannot contain null bytes
    Traceback (most recent call last):
      File "/home/tjim/anaconda3/lib/python3.6/runpy.py", line 193, in _run_module_as_main
        "__main__", mod_spec)
      File "/home/tjim/anaconda3/lib/python3.6/runpy.py", line 85, in _run_code
        exec(code, run_globals)
      File "/home/tjim/anaconda3/lib/python3.6/site-packages/mpi4py/__main__.py", line 7, in <module>
        main()
      File "/home/tjim/anaconda3/lib/python3.6/site-packages/mpi4py/run.py", line 196, in main
        run_command_line(args)
      File "/home/tjim/anaconda3/lib/python3.6/site-packages/mpi4py/run.py", line 47, in run_command_line
        run_path(sys.argv[0], run_name='__main__')
      File "/home/tjim/anaconda3/lib/python3.6/runpy.py", line 261, in run_path
        code, fname = _get_code_from_file(run_name, path_name)
      File "/home/tjim/anaconda3/lib/python3.6/runpy.py", line 236, in _get_code_from_file
        code = compile(f.read(), fname, 'exec')
    ValueError: source code string cannot contain null bytes
    Traceback (most recent call last):
      File "/home/tjim/anaconda3/lib/python3.6/runpy.py", line 193, in _run_module_as_main
        "__main__", mod_spec)
      File "/home/tjim/anaconda3/lib/python3.6/runpy.py", line 85, in _run_code
        exec(code, run_globals)
      File "/home/tjim/anaconda3/lib/python3.6/site-packages/mpi4py/__main__.py", line 7, in <module>
        main()
      File "/home/tjim/anaconda3/lib/python3.6/site-packages/mpi4py/run.py", line 196, in main
        run_command_line(args)
      File "/home/tjim/anaconda3/lib/python3.6/site-packages/mpi4py/run.py", line 47, in run_command_line
        run_path(sys.argv[0], run_name='__main__')
      File "/home/tjim/anaconda3/lib/python3.6/runpy.py", line 261, in run_path
        code, fname = _get_code_from_file(run_name, path_name)
      File "/home/tjim/anaconda3/lib/python3.6/runpy.py", line 236, in _get_code_from_file
        code = compile(f.read(), fname, 'exec')
    ValueError: source code string cannot contain null bytes
    Traceback (most recent call last):
      File "/home/tjim/anaconda3/lib/python3.6/runpy.py", line 193, in _run_module_as_main
        "__main__", mod_spec)
      File "/home/tjim/anaconda3/lib/python3.6/runpy.py", line 85, in _run_code
        exec(code, run_globals)
      File "/home/tjim/anaconda3/lib/python3.6/site-packages/mpi4py/__main__.py", line 7, in <module>
        main()
      File "/home/tjim/anaconda3/lib/python3.6/site-packages/mpi4py/run.py", line 196, in main
        run_command_line(args)
      File "/home/tjim/anaconda3/lib/python3.6/site-packages/mpi4py/run.py", line 47, in run_command_line
        run_path(sys.argv[0], run_name='__main__')
      File "/home/tjim/anaconda3/lib/python3.6/runpy.py", line 261, in run_path
        code, fname = _get_code_from_file(run_name, path_name)
      File "/home/tjim/anaconda3/lib/python3.6/runpy.py", line 236, in _get_code_from_file
        code = compile(f.read(), fname, 'exec')
    ValueError: source code string cannot contain null bytes
    
  3. Timothy Jim reporter

    @Lisandro Dalcin Thanks for the update and information. The two tests you suggested appear to be running fine. Can I assume that this means I have a working install? Or is there a more comprehensive test I should run? Thanks again for the help.

  4. Timothy Jim reporter

    @Lisandro Dalcin I am actually unsure where pip has installed mpi4py, so I just downloaded the tests from the github master. Actually, I wanted to download mpi4py mainly to use SU2; however, which installing SU2, I get the Error: Unable to find 'mpi4py/mpi4py.i'. - where is mpi4py located when installed by pip? Thanks and regards.

  5. Timothy Jim reporter

    I realised that it is because SU2 uses python2.7, hence it's not picking up the mpi4py install in the 3.6 directories. Can I also use pip install https://bitbucket.org/mpi4py/mpi4py/get/master.tar.gz to install the latest mpi4py for python2.7 too?

  6. Timothy Jim reporter

    I just created a python 2.7 environment and installed mpi4py as before. I noticed that for the helloworld test case on 4 cores, it prints "Hello, world...process 0 of 1 on..." 4 times. Is this the expected behavior? (I thought I remembered differently from before).

    From which directory might I be able to perform the runtests command? I'm not sure where pip puts the test directory.

  7. Lisandro Dalcin

    Something should be wrong. It seems you installed mpi4py with some MPI implementation, but then you are executing with an mpiexec command corresponding to a different MPI. About running tests, pip do not install the test scripts, you have to explicitly clone the repository with git and run them as you did before.

  8. Lisandro Dalcin

    You will have to figure out yourself, try mpiexec --help or mpiexec -help. Rerunning the installs may help, but maybe you have a mess in your environment or system. You should check that the mpicc and mpiexec commands are in the same directory path and they correspond each other.

  9. Timothy Jim reporter

    Yes - you were completely correct; the wrong mpiexec was being picked up. For reference for anyone who has Paraview installed, please see below.

    Running which mpicc showed my openmpi install, at /opt/openmpi/openmpi-3.0.0/bin/mpicc but which mpiexec had hooked onto my Paraview mpiexec at /opt/paraview/ParaView-5.4.1-Qt5-OpenGL2-MPI-Linux-64bit/bin/mpiexec. Swapping my $PATH order solved the issue and running the process again gives the expected behaviour:

    tjim@DESKTOP-TA3P0PS:/opt/SU2/SU2v5.0.0/bin$ mpiexec -n 4 python -m mpi4py.bench helloworld
    Hello, World! I am process 0 of 4 on DESKTOP-TA3P0PS.
    Hello, World! I am process 1 of 4 on DESKTOP-TA3P0PS.
    Hello, World! I am process 2 of 4 on DESKTOP-TA3P0PS.
    Hello, World! I am process 3 of 4 on DESKTOP-TA3P0PS.
    
  10. Timothy Jim reporter

    @Lisandro Dalcin I'm getting some strange behaviour when running in a conda environment - since the project I need to run uses python2.7, I have set up a new conda environemnt which defaults to python2.7. I installed mpi4py in this environment as before and (with the environment active) it successfully runs mpiexec -n 4 python -m mpi4py.bench helloworld and ringtest, but for some reason, it fails when attempting to run runtests.py with the following:

    (su2) tjim@DESKTOP-TA3P0PS:/opt/mpi4py/mpi4py_src$ mpiexec -n 4 python test/runtests.py
    Traceback (most recent call last):
      File "test/runtests.py", line 240, in <module>
        sys.exit(main())
      File "test/runtests.py", line 227, in main
        package = import_package(options, pkgname)
      File "test/runtests.py", line 103, in import_package
        import mpi4py.MPI
    ImportError: No module named MPI
    Traceback (most recent call last):
      File "test/runtests.py", line 240, in <module>
        sys.exit(main())
      File "test/runtests.py", line 227, in main
        package = import_package(options, pkgname)
      File "test/runtests.py", line 103, in import_package
        import mpi4py.MPI
    ImportError: No module named MPI
    Traceback (most recent call last):
      File "test/runtests.py", line 240, in <module>
        sys.exit(main())
      File "test/runtests.py", line 227, in main
        package = import_package(options, pkgname)
      File "test/runtests.py", line 103, in import_package
        import mpi4py.MPI
    ImportError: No module named MPI
    Traceback (most recent call last):
      File "test/runtests.py", line 240, in <module>
        sys.exit(main())
      File "test/runtests.py", line 227, in main
        package = import_package(options, pkgname)
      File "test/runtests.py", line 103, in import_package
        import mpi4py.MPI
    ImportError: No module named MPI
    

    This is strange since I can start up python in the terminal and successfully import mpi4py.MPI

    Is this something you have come across?

  11. Lisandro Dalcin

    Are you sure the python command corresponds with the one in your conda environment? I really don't know, you have to understand it is quite hard for me to guess all the issues you may have without access to your machine. Try running mpiexec -n 4 python -c "import sys;print(sys.executable)" to be sure you are using the right Python binary.

  12. Timothy Jim reporter

    Yes, I'm fairly sure it the right python

    (su2) tjim@DESKTOP-TA3P0PS:/opt/mpi4py/mpi4py_src$ mpiexec -n 4 python -c "import sys;print(sys.executable)"
    /home/tjim/anaconda3/envs/su2/bin/python
    /home/tjim/anaconda3/envs/su2/bin/python
    /home/tjim/anaconda3/envs/su2/bin/python
    /home/tjim/anaconda3/envs/su2/bin/python
    

    This is the same python location as the one I get when calling which python with the environment activated.

    I had a check and it seems that mpi4py is installed in /home/tjim/anaconda3/envs/su2/lib/python2.7/site-packages/mpi4py/include/mpi4py/.

    Thanks for the support so far - I appreciate it's a bit difficult to debug without being here!

  13. Timothy Jim reporter

    Ok, for some reason, that solved the issue. Here is the output:

    (su2) tjim@DESKTOP-TA3P0PS:/opt/mpi4py/mpi4py_src$ git clean -dxf
    Removing .eggs/
    Removing build/
    Removing conf/cythonize.pyc
    Removing conf/mpiconfig.pyc
    Removing conf/mpidistutils.pyc
    Removing conf/mpiregexes.pyc
    Removing conf/mpiscanner.pyc
    Removing src/lib-mpi/config/config.h
    Removing src/mpi4py.MPI.c
    Removing src/mpi4py/include/mpi4py/mpi4py.MPI.h
    Removing src/mpi4py/include/mpi4py/mpi4py.MPI_api.h
    

    After that, mpiexec -n 4 python test/runtests.py runs just fine (aside from the pickle errors seen earlier)

  14. Timothy Jim reporter

    Yes, I'm still getting them at the moment. I'm getting the below message on running git checkout master && git pull - I suppose I don't need to commit any changes

    (su2) tjim@DESKTOP-TA3P0PS:/opt/mpi4py/mpi4py_src$ git checkout master && git pull
    M   mpi.cfg
    Already on 'master'
    Your branch is up-to-date with 'origin/master'.
    remote: Counting objects: 667, done.
    remote: Compressing objects: 100% (428/428), done.
    remote: Total 667 (delta 545), reused 312 (delta 235)
    Receiving objects: 100% (667/667), 85.75 KiB | 0 bytes/s, done.
    Resolving deltas: 100% (545/545), completed with 135 local objects.
    From https://bitbucket.org/mpi4py/mpi4py
       5845476..0a6ee1d  master     -> origin/master
    Updating 5845476..0a6ee1d
    error: Your local changes to the following files would be overwritten by merge:
        mpi.cfg
    Please, commit your changes or stash them before you can merge.
    Aborting
    
  15. Log in to comment