Team constructors need progress level: user

[Dan Bonachea]

Currently the only user-callable team constructor is team::split. In the future there will be others. All will share the property (inherited from GASNet and analogous to MPI communicators) that they are blocking collective operations over the parent team. In addition, many will internally require barrier-equivalent synchronization -- team::split in particular requires this property by the nature of its data flow semantics.

team::split is currently defined to progress level: internal, but I think this is very likely to cause deadlock in surprising ways. Consider:

auto fut = upcxx::rpc(0, []() { return x; });
wait(fut);
upcxx::world().split(0, upcxx::rank_me());

This simple program is very likely to deadlock, because rank 0 will complete its own RPC and enter the blocking collective, where it spins on internal progress while the other ranks wait for it to process an RPC it will never see.

This is exactly why upcxx::barrier() has progress level: user.

I think team::split (and any future team constructors or other blocking collective operations) need to be progress level: user, for the same reason.

[Former user]

Agreed, this was an oversight.

[Amir Kamil]

Is there also an implied fence? Should there be a "C++ memory ordering" clause for team::split()?

[Dan Bonachea]

The data dependencies of team::split directly imply barrier semantics over the parent team. I don't see a downside to providing the same memory ordering guarantees.

In the future we might introduce other "weaker" team constructors that still need to be collective over the parent team but need not imply barrier synchronization. Those will probably still need to have progress level user, but might not make sense as a memory synchronization point.

[Amir Kamil]

If the memory ordering is semantically implied, then we should guarantee it so that users aren't encouraged to add their own unnecessary synchronization.

[Amir Kamil]

Fix Issue #72: Team constructors need progress level: user.

→ <<cset b07ba810759c>>