1. mpi4py Avatar mpi4py
  2. mpi4py
  3. mpi4py
  4. Issues

Multiple test errors with OpenMPI 3.0

Issue #87 closed
Former user created an issue

I’m trying to build mpi4py with OpenMPI 3.0, but it fails during tests:

running test
[0@archange] Python 3.6 (/usr/bin/python)
[0@archange] MPI 3.1 (Open MPI 3.0.0)
[0@archange] mpi4py 3.0.0 (build/lib.linux-x86_64-3.6/mpi4py)
....s.........................................................................................................................................................................................................................................................................................................ssssssssssssssssssssssss..........................................................ss...................................................................ssssssssssss............................................................sss................................................................................................................................................................................................................ssssss...........ssssss......s...s........................................................................................................................................s...ss.............................................................--------------------------------------------------------------------------
There are not enough slots available in the system to satisfy the 2 slots
that were requested by the application:
  /usr/bin/python

Either request fewer slots for your application, or make more slots available
for use.
--------------------------------------------------------------------------
E--------------------------------------------------------------------------
All nodes which are allocated for this job are already filled.
--------------------------------------------------------------------------
EEEEE[archange:00982] 4 more processes have sent help message help-orte-rmaps-base.txt / orte-rmaps-base:all-available-resources-used
[archange:00982] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
[archange:00979] PMIX ERROR: UNPACK-PAST-END in file client/pmix_client_spawn.c at line 215
[archange:00979] PMIX ERROR: UNPACK-PAST-END in file client/pmix_client.c at line 115
EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE..........................................................................................................................
======================================================================
ERROR: testArgsOnlyAtRootMultiple (test_spawn.TestSpawnSelf)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_spawn.py", line 198, in testArgsOnlyAtRootMultiple
    info=INFO, root=self.ROOT)
  File "mpi4py/MPI/Comm.pyx", line 1574, in mpi4py.MPI.Intracomm.Spawn_multiple
mpi4py.MPI.Exception: MPI_ERR_SPAWN: could not spawn processes

======================================================================
ERROR: testCommSpawn (test_spawn.TestSpawnSelf)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_spawn.py", line 66, in testCommSpawn
    info=self.INFO, root=self.ROOT)
  File "mpi4py/MPI/Comm.pyx", line 1534, in mpi4py.MPI.Intracomm.Spawn
mpi4py.MPI.Exception: MPI_ERR_SPAWN: could not spawn processes

======================================================================
ERROR: testCommSpawnMultiple (test_spawn.TestSpawnSelf)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_spawn.py", line 128, in testCommSpawnMultiple
    info=INFO, root=self.ROOT)
  File "mpi4py/MPI/Comm.pyx", line 1574, in mpi4py.MPI.Intracomm.Spawn_multiple
mpi4py.MPI.Exception: MPI_ERR_SPAWN: could not spawn processes

======================================================================
ERROR: testCommSpawnMultipleDefaults1 (test_spawn.TestSpawnSelf)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_spawn.py", line 142, in testCommSpawnMultipleDefaults1
    child = self.COMM.Spawn_multiple(COMMAND, ARGS)
  File "mpi4py/MPI/Comm.pyx", line 1574, in mpi4py.MPI.Intracomm.Spawn_multiple
mpi4py.MPI.Exception: MPI_ERR_SPAWN: could not spawn processes

======================================================================
ERROR: testCommSpawnMultipleDefaults2 (test_spawn.TestSpawnSelf)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_spawn.py", line 156, in testCommSpawnMultipleDefaults2
    child = self.COMM.Spawn_multiple(COMMAND, ARGS, 1, MPI.INFO_NULL)
  File "mpi4py/MPI/Comm.pyx", line 1574, in mpi4py.MPI.Intracomm.Spawn_multiple
mpi4py.MPI.Exception: MPI_ERR_SPAWN: could not spawn processes

======================================================================
ERROR: testErrcodes (test_spawn.TestSpawnSelf)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_spawn.py", line 81, in testErrcodes
    errcodes=errcodes)
  File "mpi4py/MPI/Comm.pyx", line 1534, in mpi4py.MPI.Intracomm.Spawn
mpi4py.MPI.Exception: MPI_ERR_SPAWN: could not spawn processes

======================================================================
ERROR: testErrcodesMultiple (test_spawn.TestSpawnSelf)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_spawn.py", line 177, in testErrcodesMultiple
    errcodes=errcodelist)
  File "mpi4py/MPI/Comm.pyx", line 1574, in mpi4py.MPI.Intracomm.Spawn_multiple
mpi4py.MPI.Exception: MPI_ERR_SPAWN: could not spawn processes

======================================================================
ERROR: testNoArgs (test_spawn.TestSpawnSelf)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_spawn.py", line 111, in testNoArgs
    info=self.INFO, root=self.ROOT)
  File "mpi4py/MPI/Comm.pyx", line 1534, in mpi4py.MPI.Intracomm.Spawn
mpi4py.MPI.Exception: MPI_ERR_SPAWN: could not spawn processes

======================================================================
ERROR: testNoArgsMultiple (test_spawn.TestSpawnSelf)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_spawn.py", line 220, in testNoArgsMultiple
    info=INFO, root=self.ROOT)
  File "mpi4py/MPI/Comm.pyx", line 1574, in mpi4py.MPI.Intracomm.Spawn_multiple
mpi4py.MPI.Exception: MPI_ERR_SPAWN: could not spawn processes

======================================================================
ERROR: testArgsOnlyAtRoot (test_spawn.TestSpawnSelfMany)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_spawn.py", line 94, in testArgsOnlyAtRoot
    info=self.INFO, root=self.ROOT)
  File "mpi4py/MPI/Comm.pyx", line 1534, in mpi4py.MPI.Intracomm.Spawn
mpi4py.MPI.Exception: MPI_ERR_SPAWN: could not spawn processes

======================================================================
ERROR: testArgsOnlyAtRootMultiple (test_spawn.TestSpawnSelfMany)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_spawn.py", line 198, in testArgsOnlyAtRootMultiple
    info=INFO, root=self.ROOT)
  File "mpi4py/MPI/Comm.pyx", line 1574, in mpi4py.MPI.Intracomm.Spawn_multiple
mpi4py.MPI.Exception: MPI_ERR_SPAWN: could not spawn processes

======================================================================
ERROR: testCommSpawn (test_spawn.TestSpawnSelfMany)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_spawn.py", line 66, in testCommSpawn
    info=self.INFO, root=self.ROOT)
  File "mpi4py/MPI/Comm.pyx", line 1534, in mpi4py.MPI.Intracomm.Spawn
mpi4py.MPI.Exception: MPI_ERR_SPAWN: could not spawn processes

======================================================================
ERROR: testCommSpawnMultiple (test_spawn.TestSpawnSelfMany)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_spawn.py", line 128, in testCommSpawnMultiple
    info=INFO, root=self.ROOT)
  File "mpi4py/MPI/Comm.pyx", line 1574, in mpi4py.MPI.Intracomm.Spawn_multiple
mpi4py.MPI.Exception: MPI_ERR_SPAWN: could not spawn processes

======================================================================
ERROR: testCommSpawnMultipleDefaults1 (test_spawn.TestSpawnSelfMany)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_spawn.py", line 142, in testCommSpawnMultipleDefaults1
    child = self.COMM.Spawn_multiple(COMMAND, ARGS)
  File "mpi4py/MPI/Comm.pyx", line 1574, in mpi4py.MPI.Intracomm.Spawn_multiple
mpi4py.MPI.Exception: MPI_ERR_SPAWN: could not spawn processes

======================================================================
ERROR: testCommSpawnMultipleDefaults2 (test_spawn.TestSpawnSelfMany)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_spawn.py", line 156, in testCommSpawnMultipleDefaults2
    child = self.COMM.Spawn_multiple(COMMAND, ARGS, 1, MPI.INFO_NULL)
  File "mpi4py/MPI/Comm.pyx", line 1574, in mpi4py.MPI.Intracomm.Spawn_multiple
mpi4py.MPI.Exception: MPI_ERR_SPAWN: could not spawn processes

======================================================================
ERROR: testErrcodes (test_spawn.TestSpawnSelfMany)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_spawn.py", line 81, in testErrcodes
    errcodes=errcodes)
  File "mpi4py/MPI/Comm.pyx", line 1534, in mpi4py.MPI.Intracomm.Spawn
mpi4py.MPI.Exception: MPI_ERR_SPAWN: could not spawn processes

======================================================================
ERROR: testErrcodesMultiple (test_spawn.TestSpawnSelfMany)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_spawn.py", line 177, in testErrcodesMultiple
    errcodes=errcodelist)
  File "mpi4py/MPI/Comm.pyx", line 1574, in mpi4py.MPI.Intracomm.Spawn_multiple
mpi4py.MPI.Exception: MPI_ERR_SPAWN: could not spawn processes

======================================================================
ERROR: testNoArgs (test_spawn.TestSpawnSelfMany)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_spawn.py", line 111, in testNoArgs
    info=self.INFO, root=self.ROOT)
  File "mpi4py/MPI/Comm.pyx", line 1534, in mpi4py.MPI.Intracomm.Spawn
mpi4py.MPI.Exception: MPI_ERR_SPAWN: could not spawn processes

======================================================================
ERROR: testNoArgsMultiple (test_spawn.TestSpawnSelfMany)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_spawn.py", line 220, in testNoArgsMultiple
    info=INFO, root=self.ROOT)
  File "mpi4py/MPI/Comm.pyx", line 1574, in mpi4py.MPI.Intracomm.Spawn_multiple
mpi4py.MPI.Exception: MPI_ERR_SPAWN: could not spawn processes

======================================================================
ERROR: testArgsOnlyAtRoot (test_spawn.TestSpawnWorld)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_spawn.py", line 94, in testArgsOnlyAtRoot
    info=self.INFO, root=self.ROOT)
  File "mpi4py/MPI/Comm.pyx", line 1534, in mpi4py.MPI.Intracomm.Spawn
mpi4py.MPI.Exception: MPI_ERR_SPAWN: could not spawn processes

======================================================================
ERROR: testArgsOnlyAtRootMultiple (test_spawn.TestSpawnWorld)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_spawn.py", line 198, in testArgsOnlyAtRootMultiple
    info=INFO, root=self.ROOT)
  File "mpi4py/MPI/Comm.pyx", line 1574, in mpi4py.MPI.Intracomm.Spawn_multiple
mpi4py.MPI.Exception: MPI_ERR_SPAWN: could not spawn processes

======================================================================
ERROR: testCommSpawn (test_spawn.TestSpawnWorld)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_spawn.py", line 66, in testCommSpawn
    info=self.INFO, root=self.ROOT)
  File "mpi4py/MPI/Comm.pyx", line 1534, in mpi4py.MPI.Intracomm.Spawn
mpi4py.MPI.Exception: MPI_ERR_SPAWN: could not spawn processes

======================================================================
ERROR: testCommSpawnMultiple (test_spawn.TestSpawnWorld)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_spawn.py", line 128, in testCommSpawnMultiple
    info=INFO, root=self.ROOT)
  File "mpi4py/MPI/Comm.pyx", line 1574, in mpi4py.MPI.Intracomm.Spawn_multiple
mpi4py.MPI.Exception: MPI_ERR_SPAWN: could not spawn processes

======================================================================
ERROR: testCommSpawnMultipleDefaults1 (test_spawn.TestSpawnWorld)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_spawn.py", line 142, in testCommSpawnMultipleDefaults1
    child = self.COMM.Spawn_multiple(COMMAND, ARGS)
  File "mpi4py/MPI/Comm.pyx", line 1574, in mpi4py.MPI.Intracomm.Spawn_multiple
mpi4py.MPI.Exception: MPI_ERR_SPAWN: could not spawn processes

======================================================================
ERROR: testCommSpawnMultipleDefaults2 (test_spawn.TestSpawnWorld)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_spawn.py", line 156, in testCommSpawnMultipleDefaults2
    child = self.COMM.Spawn_multiple(COMMAND, ARGS, 1, MPI.INFO_NULL)
  File "mpi4py/MPI/Comm.pyx", line 1574, in mpi4py.MPI.Intracomm.Spawn_multiple
mpi4py.MPI.Exception: MPI_ERR_SPAWN: could not spawn processes

======================================================================
ERROR: testErrcodes (test_spawn.TestSpawnWorld)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_spawn.py", line 81, in testErrcodes
    errcodes=errcodes)
  File "mpi4py/MPI/Comm.pyx", line 1534, in mpi4py.MPI.Intracomm.Spawn
mpi4py.MPI.Exception: MPI_ERR_SPAWN: could not spawn processes

======================================================================
ERROR: testErrcodesMultiple (test_spawn.TestSpawnWorld)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_spawn.py", line 177, in testErrcodesMultiple
    errcodes=errcodelist)
  File "mpi4py/MPI/Comm.pyx", line 1574, in mpi4py.MPI.Intracomm.Spawn_multiple
mpi4py.MPI.Exception: MPI_ERR_SPAWN: could not spawn processes

======================================================================
ERROR: testNoArgs (test_spawn.TestSpawnWorld)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_spawn.py", line 111, in testNoArgs
    info=self.INFO, root=self.ROOT)
  File "mpi4py/MPI/Comm.pyx", line 1534, in mpi4py.MPI.Intracomm.Spawn
mpi4py.MPI.Exception: MPI_ERR_SPAWN: could not spawn processes

======================================================================
ERROR: testNoArgsMultiple (test_spawn.TestSpawnWorld)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_spawn.py", line 220, in testNoArgsMultiple
    info=INFO, root=self.ROOT)
  File "mpi4py/MPI/Comm.pyx", line 1574, in mpi4py.MPI.Intracomm.Spawn_multiple
mpi4py.MPI.Exception: MPI_ERR_SPAWN: could not spawn processes

======================================================================
ERROR: testArgsOnlyAtRoot (test_spawn.TestSpawnWorldMany)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_spawn.py", line 94, in testArgsOnlyAtRoot
    info=self.INFO, root=self.ROOT)
  File "mpi4py/MPI/Comm.pyx", line 1534, in mpi4py.MPI.Intracomm.Spawn
mpi4py.MPI.Exception: MPI_ERR_SPAWN: could not spawn processes

======================================================================
ERROR: testArgsOnlyAtRootMultiple (test_spawn.TestSpawnWorldMany)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_spawn.py", line 198, in testArgsOnlyAtRootMultiple
    info=INFO, root=self.ROOT)
  File "mpi4py/MPI/Comm.pyx", line 1574, in mpi4py.MPI.Intracomm.Spawn_multiple
mpi4py.MPI.Exception: MPI_ERR_SPAWN: could not spawn processes

======================================================================
ERROR: testCommSpawn (test_spawn.TestSpawnWorldMany)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_spawn.py", line 66, in testCommSpawn
    info=self.INFO, root=self.ROOT)
  File "mpi4py/MPI/Comm.pyx", line 1534, in mpi4py.MPI.Intracomm.Spawn
mpi4py.MPI.Exception: MPI_ERR_SPAWN: could not spawn processes

======================================================================
ERROR: testCommSpawnMultiple (test_spawn.TestSpawnWorldMany)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_spawn.py", line 128, in testCommSpawnMultiple
    info=INFO, root=self.ROOT)
  File "mpi4py/MPI/Comm.pyx", line 1574, in mpi4py.MPI.Intracomm.Spawn_multiple
mpi4py.MPI.Exception: MPI_ERR_SPAWN: could not spawn processes

======================================================================
ERROR: testCommSpawnMultipleDefaults1 (test_spawn.TestSpawnWorldMany)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_spawn.py", line 142, in testCommSpawnMultipleDefaults1
    child = self.COMM.Spawn_multiple(COMMAND, ARGS)
  File "mpi4py/MPI/Comm.pyx", line 1574, in mpi4py.MPI.Intracomm.Spawn_multiple
mpi4py.MPI.Exception: MPI_ERR_SPAWN: could not spawn processes

======================================================================
ERROR: testCommSpawnMultipleDefaults2 (test_spawn.TestSpawnWorldMany)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_spawn.py", line 156, in testCommSpawnMultipleDefaults2
    child = self.COMM.Spawn_multiple(COMMAND, ARGS, 1, MPI.INFO_NULL)
  File "mpi4py/MPI/Comm.pyx", line 1574, in mpi4py.MPI.Intracomm.Spawn_multiple
mpi4py.MPI.Exception: MPI_ERR_SPAWN: could not spawn processes

======================================================================
ERROR: testErrcodes (test_spawn.TestSpawnWorldMany)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_spawn.py", line 81, in testErrcodes
    errcodes=errcodes)
  File "mpi4py/MPI/Comm.pyx", line 1534, in mpi4py.MPI.Intracomm.Spawn
mpi4py.MPI.Exception: MPI_ERR_SPAWN: could not spawn processes

======================================================================
ERROR: testErrcodesMultiple (test_spawn.TestSpawnWorldMany)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_spawn.py", line 177, in testErrcodesMultiple
    errcodes=errcodelist)
  File "mpi4py/MPI/Comm.pyx", line 1574, in mpi4py.MPI.Intracomm.Spawn_multiple
mpi4py.MPI.Exception: MPI_ERR_SPAWN: could not spawn processes

======================================================================
ERROR: testNoArgs (test_spawn.TestSpawnWorldMany)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_spawn.py", line 111, in testNoArgs
    info=self.INFO, root=self.ROOT)
  File "mpi4py/MPI/Comm.pyx", line 1534, in mpi4py.MPI.Intracomm.Spawn
mpi4py.MPI.Exception: MPI_ERR_SPAWN: could not spawn processes

======================================================================
ERROR: testNoArgsMultiple (test_spawn.TestSpawnWorldMany)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_spawn.py", line 220, in testNoArgsMultiple
    info=INFO, root=self.ROOT)
  File "mpi4py/MPI/Comm.pyx", line 1574, in mpi4py.MPI.Intracomm.Spawn_multiple
mpi4py.MPI.Exception: MPI_ERR_SPAWN: could not spawn processes

----------------------------------------------------------------------
Ran 1134 tests in 2.956s

FAILED (errors=39, skipped=59)
error: test

I’m not sure what is causing this, but there are no issues with OpenMPI 2.1.

Comments (4)

  1. Lisandro Dalcin

    Not mpi4py's fault, but recent changes in Open MPI related to oversubscription. Did you try googling the error? https://stackoverflow.com/questions/35704637/mpirun-not-enough-slots-available

    Try this way (or try the --oversubscribe option mentioned in the SO thread linked above):

    export OMPI_MCA_rmaps_base_oversubscribe=yes
mpiexec -n 1 python test/runtests.py
mpiexec -n 5 python test/runtests.py

    If it works, please close this issue. Otherwise, ping me back.

  2. ArchangeGabriel

    Yes, I had meanwhile found information on stackexchange (https://stackoverflow.com/questions/42522060/dompi-error-there-are-not-enough-slots-available-in-the-system-to-satisfy-the-2#42523509), but had no idea how to use that here.

    Tests are run through python setup.py test, so I wasn’t sure how to pass those arguments to mpi commands.

    However, your OMPI_MCA env var did the trick: I have not tested mpiexec -n 1 nor mpiexec -n 5 —and I think those would have worked even before since I have 8 cores on this machine—, but my above command works now. ;)

    Sorry for the noise and thanks for the trick!

  5. Log in to comment
Assignee
–
Type
bug
Priority
minor
Status
closed
Votes
0
Watchers
None
Jira: the preferred issue tracker for Bitbucket. Join the team!