Add in-place and vector reductions

[Mathias Jacquelin]

Very frequently, when one is trying to reduce / allreduce a large chunk of data, it is advantageous to do it "in place". I do not think our API allows to do that.

One thing that could be done, is to use the move syntax to "move" the input value and place the modified content in the future, that way we avoid making an extra copy.

Any take on this ?

[Amir Kamil]

Can you clarify what use case you're asking for? Something like vector reductions, or just the ability to pass the input by pointer?

I imagine supporting both with an MPI-like interface:

template <typename T, typename BinaryOp >
future<> allreduce(const T *src, T *dst, size_t count, BinaryOp &&op,
                   team &team = world());

However, unless this is high priority for one of our apps, I don't think we can promise this in the near term.

[Mathias Jacquelin]

Yes, reducing a vector is good example. I think it applies for both versions of the API, let's assume I do:

std::vector<double> buffer; .... broadcast( buffer, 0 );

Can I do this in place ? This is a use case that we will HAVE to handle as well I think.

Best,

Mathias Jacquelin Research Scientist Lawrence Berkeley National Laboratory [e-mail] 1-510-495-2605

[Amir Kamil]

Yes, we have a vector broadcast interface:

template <typename T>
future<> broadcast(T *buffer, std::size_t count,
                   intrank_t sender, team &team = world());

[Dan Bonachea]

At the 9-12 meeting we agreed to defer this issue.

[Dan Bonachea]

This issue was discussed in the 1/10/18 meeting and we agreed to defer it to the September release.

[Paul Hargrove]

The title and comments touch on 2 issues. This first is a vector API for reduction, and is the topic of this comment.

As Amir noted in a 20178-08-30 comment, we do have a vector broadcast.
However, there is not currently a vector interface for reduce as Mathias has requested.

I believe possible to use a vector as the type and a lambda as the operator to achieve the same interface. However, if vector reductions with the basic types and standard operators are ever to be off-loaded, then an explicit vector form (like Amir proposes in the 1st comment on this issue) will be required.

I would like the issue of the inclusion/exclusion of this API decided soon, because it impact my priorities for GASNet-EX collectives.

[Paul Hargrove]

The title and comments touch on 2 issues. This second is "in-place" support for reduction, and is the topic of this comment.

I think for memory scaling reasons, it is desirable to support the equivalent of MPI_IN_PLACE for all collective operations where is it well defined. At the moment, that appears only to impact the vector allreduce with the prototype Amir provides in the first comment on this issue (except for the removal of const from src). However, the "in-place" behavior is potentially relevant for other collectives we may add, such as gather-to-all and exchange (aka all-to-all).

[Dan Bonachea]

This issue was triaged at the 2018-06-13 Pagoda meeting and assigned a new milestone/priority.

We noted that GEX should have support for vector (ie multi-field) reductions in time for the upcoming release, and we should expose at least that feature in UPC++ collectives.

[Dan Bonachea]

Resolved in 2d9680f (pull request #16)