Clarify progress level contract for most functions

[Dan Bonachea]

Progress chapter contains:

Each UPC++ function’s contract to the user contains its progress guarantee level. ...
For the user, understanding which functions perform these progress spins becomes crucial, since any invocation of user-level progress may execute rpcs or local callbacks.

Elsewhere there is wording like:

Serialization/deserialization may not call any UPC++ routine with a progress level other than none.

However, the progress contract currently only appears to be unambiguously specified for progress(), discharge() and flush(). If the intent is something like "when not explicitly stated all other functions have a progress level X", then this needs to be clearly and directly stated somewhere.

There's currently some imprecise wording about the "most common progress guarantee made by UPC++ functions", but that doesn't answer the question of the exact progress guarantee of any particular function whose API ref says nothing about progress.

In particular, I'd like to know the progress guarantee level of rput and rget initiation. For example, if an rput or rget RMA completes synchronously inside the initiation call (eg because the global_ptr was local()), is that initiation permitted to fulfill promises and fire callbacks before returning?

Same question if RMA injection encounters network backpressure.

Same question for loopback rpc injection or lpc to an active persona.

[Dan Bonachea]

Related question for dist_object constructors, which says:

Continuations attached to the future returned from dist_id<T>::when_local will fire here.

The verb "fire" here does not appear to be formally defined. Does that mean that user code in .then() continuations chained on futures created by when_local might actually run before the dist_object constructor returns? Or only that promises are fulfilled and chained callbacks will run within the next progress(progress_level_user)?

Assuming the former, does that mean the dist_object constructor has a progress-level of user, implying that other, completely-unrelated callbacks in the ready state for active personas may also execute?

[Former user]

I added progress levels to all calls.

I will clean up the "fire" wording.

The progress semantics are the former case, where only the promises are fulfilled during this constructor. Promise fulfillment implies that continuations hanging from the when_local() future would execute immediately, as in before the constructor returns. This does not mean the constructor makes user level progress. Unrelated rpc's will not execute. It might be friendlier to change the semantics to your second scenario where we defer fulfilling the promises until the next time the master enters user progress. I am going to leave this issue open until I hear some opinions about this.

[Dan Bonachea]

Chapter 5 is still missing annotations, some of which are non-obvious.

In particular, promise fulfill and finalize are definitely not progress level none, based on their description

[Former user]

Chapter 5 is missing per-function annotations. I took a shortcut and added text under the "API Reference" title saying that all functions are progress=none.

Even though they run continuations, fulfill and finalize are definitely upcxx progress=none. Their execution engine is considered distinct from upcxx.

[Dan Bonachea]

I think this concept of a distinct execution engine for promise fulfillment needs to somehow be clarified further. The current spec makes it sound like callbacks only ever execute in progress().

Here's at least one potentially-confusing statement:

1806 If lev == progress_level_user then this thread is also used to execute any
1807 available user actions for the personas currently active. Actions include:
1808 1. Future/promise fulfillment. By the execution model of futures this induces
1809 callback cascade.

If I understand correctly this is not saying that "Future/promise fulfillment" is ALWAYS a user action deferred to progress, this bullet is actually referring specifically to "Future/promise fulfillment" performed on behalf of asynchronous RMA completions (and any transitively activated by these and the other actions of progress)

[Former user]

Reworded that last bullet per Dan.