MPI.Win.Create fails in singleton mode

[Adam Lanman]

I would like to use RMA to write data to different spots in a numpy array held on one process. I found a version of the following online:

from  mpi4py  import  MPI
import time
import  numpy  as np
# adapted from https://sea.ucar.edu/sites/default/files/4_mpi4py.pdf

comm = MPI.COMM_WORLD
rank = comm.rank
n = np.zeros (10, dtype=complex).reshape((2,5))
if rank == 0:
    win = MPI.Win.Create (n, comm=MPI.COMM_WORLD)
else:
    win = MPI.Win.Create (None , comm=MPI.COMM_WORLD)

if rank ==0:
    print("[%02d] Original  data %s" % (rank , n))

win.Fence()
if rank !=0:
    data = np.arange(10,  dtype = np.complex)
    for i in range(5):
       # time.sleep(np.random.randint(3))
        win.Accumulate(data[5:], 0, target=2 * 16, op=MPI.SUM)
win.Fence()

if rank ==0:
    print("[%02d] Received  data %s" % (rank , n))

win.Free()

‌

Calling this rma_test.py, if I run this with mpirun -n 2 python rma_test.py, it behaves as expected. However, if I run it without mpirun (in singleton mode), or with mpirun -n 1 python rma_test.py, it returns the errors below:

The OSC pt2pt component does not support MPI_THREAD_MULTIPLE in this release.
Workarounds are to run on a single node, or to use a system with an RDMA
capable network such as Infiniband.

Traceback (most recent call last):
File "work_rma_gather2.py", line 11, in <module>
win = MPI.Win.Create (n, comm=MPI.COMM_WORLD)
File "mpi4py/MPI/Win.pyx", line 72, in mpi4py.MPI.Win.Create
mpi4py.MPI.Exception: MPI_ERR_WIN: invalid window

‌

My installation is the latest mpi4py and OpenMPI from anaconda. This could very easily be some OpenMPI quirk I’m not aware of… but I can’t find it documented anywhere. I can work around this if necessary, but it would be nice if I could use the same Window syntax regardless of the number of processes.

[Lisandro Dalcin]

I think the problem is precicely the one described in the error message. The OSC (one-sided communication, a.k.a RMA remote memory accesss) Open MPI component doesn’t/cannot work if MPI is initialized with MPI_THREAD_MULTIPLE thread support, which is what mpi4py does by default (mpi4py could pick a more conservative default, but come on, we are in 2020, the MPI-2 standard more than 20 years old by now)

Try adding the following two lines to the VERY BEGINNING of your script, this way mpi4py will initialize MPI with no thread support (plain old MPI_Init()) :

import mpi4py
mpi4py.rc.threads = False

Or you can try this one, that gives the highest level of thread support before multiple.

import mpi4py
mpi4py.rc.thread_level = 'serialized'

Please note that you can edit mpi4py/__init__.py in the install location to change the rc defaults, just search for them. Definitely not pretty, but good enough to workaround the underlying issue, which is IMHO way uglier, and would force you to add the stanza above in every script you write.

If these tips fix your problems, please close this issue as resolved.

[Adam Lanman]

Interesting… both of your suggestions do fix the problem. Do you have any idea of why this only comes up when running with one MPI processs?

Thanks for your help!

[Adam Lanman]

Switching to "serialized" thread level fixed this issue.

[Lisandro Dalcin]

No, I don’t know why this error triggers with only one MPI process. Maybe @Jeff Squyres can shed some light…

‌

[Jeff Squyres]

The pt2pt OSC module is probably chosen when you run with a single process because there’s no RDMA supported (or emulated) code path available (as there would be in the rdma OSC module).

[Lisandro Dalcin]

@Jeff Squyres Does this means that the rdma OSC module cannot work under a single-process situation? Or something deeper is going on there? Maybe a broken install? Or some kind of “gotcha” situation in Open MPI’s module selection priorities?