**.py error:unrecognized arguments: **.py when using mpiexec and mpi4py

Issue #86 new
yrl
created an issue

Here's the command:

$ mpiexec -n 4 python -m mpi4py.futures loopall.py --size_lb 2 --size_ub 2 --size_amount 1 --kkk_lb 7000 --kkk_ub 7000 --kkk_amount 1 --l_lb 0.02 --l_ub 0.02 --l_amount 1 --k 5

And here's the error notes:

usage: loopall.py [-h] [--size_lb SIZE_LB] [--size_ub SIZE_UB] [--size_amount 
SIZE_AMOUNT] [--kkk_lb KKK_LB] [--kkk_ub KKK_UB] [--kkk_amount KKK_AMOUNT]
 [--l_lb L_LB] [--l_ub L_UB] [--l_amount L_AMOUNT] [--k K]
loopall.py: error: unrecognized arguments: loopall.py
 usage: loopall.py [-h] [--size_lb SIZE_LB] [--size_ub SIZE_UB] [--size_amount SIZE_AMOUNT] [--kkk_lb KKK_LB] [--kkk_ub KKK_UB] [--kkk_amount KKK_AMOUNT] [--l_lb L_LB] [--l_ub L_UB] [--l_amount L_AMOUNT] [--k K] loopall.py: 
error: unrecognized arguments: loopall.py 
usage: loopall.py [-h] [--size_lb SIZE_LB] [--size_ub SIZE_UB] [--size_amount SIZE_AMOUNT] [--kkk_lb KKK_LB] [--kkk_ub KKK_UB] [--kkk_amount KKK_AMOUNT] [--l_lb L_LB] [--l_ub L_UB] [--l_amount L_AMOUNT] [--k K] loopall.py: 
error: unrecognized arguments: loopall.py -------------------------------------------------------------------------- MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD with errorcode 2.

    NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. You may or may not see output from other processes, depending on
    exactly when Open MPI kills them.

    -------------------------------------------------------------------------- mpiexec has exited due to process rank 1 with PID 19320 on node yang-ubuntu exiting improperly. There are two reasons this could occur:

        this process did not call "init" before exiting, but others in the job did. This can cause a job to hang indefinitely while it waits for all processes to call "init". By rule, if one process calls "init", then ALL processes must call "init" prior to termination.

        this process called "init", but exited without calling "finalize". By rule, all processes that call "init" MUST call "finalize" prior to exiting or it will be considered an "abnormal termination"

    This may have caused other processes in the application to be terminated by signals sent by mpiexec (as reported here).

And here's my CODE:

#!python

import numpy as np
from mpi4py.futures import MPIPoolExecutor
import argparse
parser = argparse.ArgumentParser()

parser.add_argument("--size_lb", help="the lower bound of the size of the system", type=int)
parser.add_argument("--size_ub", help="the upper bound of the size of the system", type=int)
parser.add_argument("--size_amount", help="the amount of the size of the system", type=int)
parser.add_argument("--kkk_lb", help="the lower bound of the kkk of the system", type=int)
parser.add_argument("--kkk_ub", help="the upper bound of the kkk of the system", type=int)
parser.add_argument("--kkk_amount", help="the amount of the kkk of the system", type=int)
parser.add_argument("--l_lb", help="the lower bound of the l of the system", type=float)
parser.add_argument("--l_ub", help="the upper bound of the l of the system", type=float)
parser.add_argument("--l_amount", help="the amount of the l of the system", type=int)
parser.add_argument("--k", help="the total amount of steps of the simulation", type=int)


args = parser.parse_args()

size_pool = np.linspace(args.size_lb,args.size_ub,num=args.size_amount)
kkk_pool = np.linspace(args.kkk_lb,args.kkk_ub,args.kkk_amount)
l_pool = np.linspace(args.l_lb,args.l_ub,args.l_amount)

parameters=[]

for i in range(args.size_amount):
    for j in range(args.kkk_amount):
        for l in range (args.l_amount):
            parameters.append([size_pool[i],kkk_pool[j],l_pool[l],args.k,i*j*l])


def SUBall(prms):   
    omitted

if __name__ == '__main__':

    with MPIPoolExecutor() as executor:
        results=executor.map(SUBall, parameters)
        for result in results:
       np.savez('population{}.npz'.format(result['index']),population=result['population'])

Comments (6)

  1. yrl reporter

    well, I got around this by giving up the argparse part, instead I used built in argument.So, every time I want to change the arguments, I have to change the source code, for now.

  2. Lisandro Dalcin

    Oh, now I see where the problem is. You have to protect argument parsing too, like this:

    import numpy as np
    from mpi4py.futures import MPIPoolExecutor
    
    def SUBall(prms):   
        return None # Put here your code
    
    if __name__ == '__main__':
    
        import argparse
        parser = argparse.ArgumentParser()
    
        parser.add_argument("--size_lb", help="the lower bound of the size of the system", type=int)
        parser.add_argument("--size_ub", help="the upper bound of the size of the system", type=int)
        parser.add_argument("--size_amount", help="the amount of the size of the system", type=int)
        parser.add_argument("--kkk_lb", help="the lower bound of the kkk of the system", type=int)
        parser.add_argument("--kkk_ub", help="the upper bound of the kkk of the system", type=int)
        parser.add_argument("--kkk_amount", help="the amount of the kkk of the system", type=int)
        parser.add_argument("--l_lb", help="the lower bound of the l of the system", type=float)
        parser.add_argument("--l_ub", help="the upper bound of the l of the system", type=float)
        parser.add_argument("--l_amount", help="the amount of the l of the system", type=int)
        parser.add_argument("--k", help="the total amount of steps of the simulation", type=int)
    
    
        args = parser.parse_args()
    
        size_pool = np.linspace(args.size_lb,args.size_ub,num=args.size_amount)
        kkk_pool = np.linspace(args.kkk_lb,args.kkk_ub,args.kkk_amount)
        l_pool = np.linspace(args.l_lb,args.l_ub,args.l_amount)
    
        parameters=[]
    
        for i in range(args.size_amount):
            for j in range(args.kkk_amount):
                for l in range (args.l_amount):
                    parameters.append([size_pool[i],kkk_pool[j],l_pool[l],args.k,i*j*l])
    
        with MPIPoolExecutor() as executor:
            results = executor.map(SUBall, parameters)
            for result in results:
                np.savez('population{}.npz'.format(result['index']),population=result['population'])
    

    Could you please try it and confirm whether it worked on your side?

  3. Log in to comment