Clarify behavior of future/promise for synchronously completed RMA

[Dan Bonachea]

The UPC++ RMA interface is generalized to abstract away affinity in rput and rget. The asynchrony support is useful for overlapping communication latencies, however for the case where the global_ptr argument actually is_local(), the RMA operation could (and probably should) be converted to synchronous load/store accesses on hardware shared memory - meaning the data movement operation could technically be "done" before the initiation call returns.

The question involves the semantics of the returned future/promise when an RMA operation was synchronously completed during initiation (most likely due to it being .local()). Specifically:

1. Do we allow a future-based RMA injection to return a readied future?
2. Do we allow a promised-based RMA injection to fulfill the promise before returning?

I don't see anything normative prohibiting these as a valid implementation. In fact, the first example in section 5.2 seems to imply this is possible. If we allow this behavior, then we really need to be clear about it because:

1. In the first case it means any callback the user chains on an RMA future with .then() could execute immediately (ie outside upcxx::progress)
2. In the second case it means promise fulfillment might trigger callback execution before an rget initiation returns (ie outside upcxx::progress). For an rput it means the subsequent call to promise::finalize_anonymous() could trigger callbacks (with no intervening call to upcxx::progress).

Both of these could be surprising to users if they expect RMA notifications to occur exclusively during a later call to progress. For example, if we allow this behavior, then this text from 6.3 should probably be adjusted:

An important aspect to clarify is that notification of completion only happens during user-level progress. Even if an operation completes early, the application cannot learn this fact without entering user-progress. For futures and promises, only when the initiating thread (persona actually) enters user level progress will the future or promise change its state (be readied or fulfilled).

in particular, this text clearly states that completion won't trigger callbacks asynchronously, but it doesn't really cover the case where an RMA was synchronously completed before return from the initiation call (in which cases there may be no "state change" required).

[Former user]

Yes this needs to be addressed. I think users would be happiest if immediately completed operations had their completions deferred until the next user progress (no return of readied futures). The same applies for rpc's to self, those also should not execute immediately and should be deferred until progress.

Though it might seem like a performance loss to artificially delay notifications, I think there might be a chance that it could actually be helpful to users expecting the same code to perform on both distriubted and non-distributed (single node) runs. If a single-node run were to complete everything synchronously and notify immediately, the user's callbacks would be fired in depth-first manner. While in distriubted runs they would fill up queues waiting for the nic and process completions breadth-first. By deferring synchronous completion we at least give breadth-first behavior in both scenarios leading to a more predictable runtime.