Feature/try to improve subcommunicator creation in fields local with mpi undefined

Merged
#307 · Created  · Last updated

Merged pull request

Merged in feature/try_to_improve_subcommunicator_creation_in_fields_local_with_mpi_undefined (pull request #307)

ec9fe24·Author: ·Closed by: ·2020-06-05

Description

  • Update utils to feature branch for now

  • Improve creation of subcommunicators in fields_local

    The main change here is to use MPI_UNDEFINED as the colour passed to split for processors for which we don't need the resulting subcommunicator. This should help reduce the number of subcommunicators we create only to then destroy.

    This may help with performance but is primarily motivated by the fact that some mpi implementations may not be able to recover all the resources assigned to a created communicator (or excessive communicator creation can lead to resource exhaustion).

  • Initial commit of approach to remove some subcommunicator creation in fields_local

    This introduces the new calculation for (hacky) testing but doesn't remove the old approach

  • Switch over to new approach to calculating work sharing in fields_local

    Also tidy up the associated new code slightly

  • Remove some unused use statements

The fields local module creates a number of subcommunicators during initialisation. Some of these are used throughout the advance stage whilst others are almost immediately destroyed. This PR attempts to reduce the number of communicators that are destroyed and then freed during initialisation. There are two parts to this. The first is to use MPI_UNDEFINED as the colour when splitting communicators for any processor that doesn’t need to use the resulting communicator (instead it gets MPI_COMM_NULL), this avoids having to create and free communicators on the processors which don’t need them. The second part is to replace the sub-communicator creation in setup of the work distribution algorithm with an alternative calculation. These sub-communicators were essentially only being used to reorder process ranks in order to aid load balancing (as the work is distributed in rank order). The replacement code is more verbose but avoids creating and destroying a large number of communicators.

 

0 attachments

0 comments

Loading commits...