MPIPoolExecutor & dill

[Minas Karamanis]

Hi all,

‌

I’m trying to parallelise a class' method using the MPIPoolExecutor map function but I’m getting the following error message:

‌

TypeError: can't pickle _thread.lock objects

‌

As far as I know, this means that the function that I’m trying to parallelise is not pickleable. Is there a way of combining MPIPoolExecutor with dill to make this pickleable?

‌

Cheers,

‌

[Lisandro Dalcin]

It is not the function, but something that you are passing as a function argument that is not pickleable. Try to add this code at the beginning of your main script. I do not promise it will work, but it is a start. Note that this hack will not preserve the locked state of the lock instance (though I’m not sure you want to pass that state when copying the object). Look at the documentation of pickle and copyreg modules for further information.

‌

from threading import Lock                                              
import copyreg                                                          
copyreg.pickle(type(Lock()), lambda x: (Lock, ()))  

PS: This is definitely not an issue related to mpi4py. These kinds of user-level/beginner questions are better suited for our mailing list at Google Groups <[e-mail]>.

‌