Document shared heap controls

[Dan Bonachea]

We need to document how to control the shared heap size of a UPC++ program in our implementation.

Some starter text:

GASNET_MAX_SEGSIZE defines the maximum amount of space available for the GASNet shared segment per process, and is often something you can set once as a site-wide setting, based on the memory resources and ulimit restrictions of the system. Some conduits (notably ibv and aries) have additional settings that interact here, but usually the defaults for those are sufficient.

UPCXX_SEGMENT_MB is the size of the shared memory heap each UPC++ rank actually creates at startup. The value needs to be strictly less than GASNET_MAX_SEGSIZE (there's a few MB of overhead in there). This value determines how much shared memory a rank can allocate before shared memory allocation fails (subject to the usual caveats regarding allocator fragmentation, etc), and is something you might want to tune on a per-run basis to control how much of the heap is reserved for shared memory versus available for the private heap.

This probably belongs in a document describing implementation-specific behaviors.

[Paul Hargrove]

It is worth noting that additional logic in upcxx-run could potentially help enforce the GASNET_MAX_SEGSIZE > UPCXX_SEGMENT_MB constraint. That may only make sense, however, if GASNET_MAX_SEGSIZE is not already set.

[Steven Hofmeyr]

I have added this to the end of the programming guide, in a section on advanced runtime configuration. Do we need it elsewhere?

[Dan Bonachea]

I have added this to the end of the programming guide, in a section on advanced runtime configuration. Do we need it elsewhere?

Would say the answer depends on the resolution of issue #99.

I think the repo should contain instructions in markdown or text format for how to control the shared heap - currently that's environment variables. If upcxx-run is upgraded to provide a different syntax for this, then this bug becomes documenting how to use that syntax.

On a related topic, I would advocate the upcxx-run syntax be:

upcxx-run -shared-heap 1024MB -n <ranks> <exe> <args...>

(where the "MB" suffix on the value can also be "KB" or "GB") instead of the current PG:

upcxx-run -shared-heap-mb 1024 -n <ranks> <exe> <args...>

GASNet already has a function for parsing these memory sizes from the environment and exposes it via gasnett_getenv_int_withdefault, which the UPC++ runtime should probably just use. See docs in README-tools. In particular, std::getenv is not guaranteed to retrieve variables set by the spawning console on all distributed-memory spawners - this is why we have gasnet_getenv.

CC: @jdbachan

[Steven Hofmeyr]

This was added to the programmer's guide a while back