clarify finalize() behavior wrt shared heap and library objects

[Dan Bonachea]

Current semantics for finalize include:

After the number of calls to finalize matches the number of calls to init, no UPC++ function can be invoked until the library is reinitialized by a subsequent call to init. ....
If this call matches the call to init that placed UPC++ in an initialized state, then this call uninitializes the UPC++ library.

This makes it clear that a finalize which uninitializes the library also prohibits subsequent calls to upcxx until re-init. However there are alot of details missing here that need to be filled in.

Specifically:

1. What happens to user objects allocated from the shared heap when the library is uninitialized? (eg: int *p = upcxx::allocate(sizeof(int)); upcxx::finalize()) Might the backing storage for *p disappear? If yes, does that mean those objects are guaranteed to be recycled, or might they result in leaked memory? If no, does a later re-init of the library run the risk of reallocating that same space, or will the allocator in the new instance respect previously allocated objects?

2. What happens to library objects in private memory? Eg if the user has a promise, dist_object, team, etc in global variables, are those objects automatically invalidated by the call to finalize()? It's not enough to prohibit method calls on them, because the question remains whether they become valid for use again after a library re-init without a new constructor call. What about DefaultConstructed objects that may have been created before the first init()? (issue #90)

3. What about destructors on library objects? If the client has some library objects on the stack when he calls finalize, does that mean he's prohibited from leaving that static scope because it might invoke prohibited destructor calls on those objects? The Hello World example in section 8.2 technically violates this because it returns from main with a future on the stack frame after calling finalize(). persona_scopes are also especially tricky here, since they have non-trivial destructor semantics when they go out of scope. Same question for library objects stored in private heap memory that the user wants to delete after finalize(). This point has the potential to make finalization un-usable in practice, unless we can harden all our object destructors to handle this case gracefully and specify that guarantee.

4. What about thread persona objects? The library implicitly creates a persona for each OS thread. If the app has stored references to personas (which is a requirement for using lpc() or cross-thread continuations), are those references invalid after a finalize and reinit? Or will they continue to reference the same threads? This one is especially problematic because it means the app potentially has to wake any idle/sleeping worker threads in the system to make them discard any persona references before tearing down the library (and re-exchange them after re-init).

[Dan Bonachea]

At the 8-9-17 Pagoda meeting we resolved:

1. Upon library tear-down, any objects still residing in shared memory immediately become invalid (without calling destructors), and their storage is guaranteed to eventually be recycled.
2. Upon library tear-down, any library objects (aside from DefaultConstructed ones) immediately become invalid and any associated state is discarded.
3. Destructors for library objects are guaranteed to work correctly at all times, even after library tear-down has rendered an object invalid. All other library calls (except init) remain prohibited while the library is in an uninitialized state.
4. The default persona object created for each OS thread (upcxx::default_persona()) is a special case. Library tear-down renders them temporarily invalid for use (and any continuations still queued to a persona violate the quiescence precondition, implying UB). After a library re-init, these personas automatically become valid again and still correspond to the default persona for the same OS thread. The same probably applies to the default persona_scope

This all needs to be added to the spec and sanity-checked in implementation.

[Former user]

I think we need to take focus off the DefaultConstructible trait, and instead view all ctors and dtors like any other function calls wrt which are valid in and out of the initialized context. TriviallyConstructible/Destructible are important traits, clearly any operation which is trivial is safe outside upcxx. Also, any function with progress level other than none is also unsafe outside upcxx.

global_ptr: null-ctor is ok outside upcxx while anything that accepts real memory isnt. the dtor is trivial so safe.

future/promise: all safe except future.wait(), because that has user-progress level.

persona/persona_scope: all safe even lpc's. Queue resources are acquired at thread creation time, or first use and freed at thread destruction.

dist_object: both ctor and dtor are unsafe since we may want them to communicate with the team.

[Mathias Jacquelin]

So how do we propose to write this up ?

I personally think it is best to have a paragraph stating whether a given datatype is safe/unsafe outside upcxx, but also have a summary in the description of upcxx::init and upcxx::finalize where we would be listing all safe datatypes, unsafe datatypes, in the same way than @jbachan is describing it in his previous message.

[Dan Bonachea]

This issue was discussed in the 1/10/18 meeting, and Brian agreed to add the resolutions recorded in the first comment to the spec by March.

[Dan Bonachea]

In the 2018-03-21 meeting, we agreed to defer this to next release, given the current implementation release does not support reinitialization.

[Dan Bonachea]

See pull request #10

[Dan Bonachea]

This issue was triaged at the 2018-06-13 Pagoda meeting and assigned a new milestone/priority.

We noted that specing the behavior after finalize is mostly academic until we commit to implementing support for tearing down and re-initializing the library. we solicit stakeholder input regarding the importance of this feature.