Difficulties with spawning new processes on the victim's node

George Bosilca reporter

This issue is rooted in OMPI and is due to the forwarding of job-level constraints from the original job to all spawnees. In this particular case adding "-npernode 1" restricts all future processes from sharing a node, across all jobid handled by the same HNP. In a normal MPI application such behavior might be desired, but in context of ULFM we need to be able to reuse nodes, which means to respawn processes on a node where older processes failed.

Multiple solution might be envisioned, but I think the cleanest solution is to provide an info key to prevent the original job parameters inheritance. I have create an OMPI issue related to this topic open-mpi/ompi#5376.

2018-07-04T18:30:00+00:00

Comments (2)

George Bosilca reporter
This issue is rooted in OMPI and is due to the forwarding of job-level constraints from the original job to all spawnees. In this particular case adding "-npernode 1" restricts all future processes from sharing a node, across all jobid handled by the same HNP. In a normal MPI application such behavior might be desired, but in context of ULFM we need to be able to reuse nodes, which means to respawn processes on a node where older processes failed.

Multiple solution might be envisioned, but I think the cleanest solution is to provide an info key to prevent the original job parameters inheritance. I have create an OMPI issue related to this topic open-mpi/ompi#5376.
- 2018-07-04T18:30:00+00:00
Aurelien Bouteiller
- changed status to resolved
open-mpi/ompi#5376 has been imported, as well as fixing the 'oversubscribe' non-propagation issue; this should resolve the problem.
- 2018-10-25T21:10:11+00:00
Log in to comment