OverflowError: integer 2768896564 does not fit in 'int'

[Marco Buongiorno Nardelli]

Hi, I am trying to scatter large numpy arrays but if the dimensions exceed some limit I get this error:

  File "MPI/Comm.pyx", line 1276, in mpi4py.MPI.Comm.bcast (src/mpi4py.MPI.c:108819)
  File "MPI/msgpickle.pxi", line 612, in mpi4py.MPI.PyMPI_bcast (src/mpi4py.MPI.c:47005)
  File "MPI/msgpickle.pxi", line 119, in mpi4py.MPI.Pickle.dump (src/mpi4py.MPI.c:40840)
  File "MPI/msgbuffer.pxi", line 35, in mpi4py.MPI.downcast (src/mpi4py.MPI.c:29070)
OverflowError: integer 2768896564 does not fit in 'int'

My understanding is that python should handle such int just fine. sys.maxint gives me a max integer = 9223372036854775807!

I am attaching the section of the code where I am doing this operations, Any advise would be much appreciated. Thanks, Marco

[Lisandro Dalcin]

This is not Python, but an MPI issue. Many MPI calls are limited to buffer sizes of ~2G entries, there is nothing mpi4py can do about it. The usual way to workaround this issue is to use user-defined datatypes to "chunk" your buffers. Some additional info in https://github.com/jeffhammond/BigMPI

[Marco Buongiorno Nardelli]

Thank you very much! I already adopted your suggestion and divided the arrays in chunks.