Consider renaming internal UPCXX_ macros

I'm starting to grow concerned over the growth and conflation of our UPC++ macro space. I'll categorize the current usages as follows:

Cat 1: We have around a half-dozen "specified" UPCXX_ macros, eg:

UPCXX_SERIALIZED_FIELDS
UPCXX_SPEC_VERSION
UPCXX_VERSION
UPCXX_DEFER_COMPLETION

Cat 2. We have a set of macros that are explicitly documented as implementation-defined:

UPCXX_THREADMODE
UPCXX_CODEMODE
UPCXX_NETWORK_*

Cat 3. Another set of macros that are not explicitly documented, but probably should be:

UPCXX_CUDA_ENABLED
UPCXX_MAX_VALUE_SIZE
UPCXX_MAX_RPC_ARG_SIZE

Cat 4. Another set of macros that are undocumented, but used in tests and are relatively "stable":

UPCXX_ASSERT_ALWAYS
UPCXX_ASSERT_ENABLED - side-note, should probably be replaced with UPCXX_CODEMODE in tests
UPCXX_BACKEND_GASNET_SEQ - side-note, should probably be replaced with UPCXX_THREADMODE in tests
UPCXX_BACKEND - side-note, should probably be excised from tests

Cat 5. Finally there's a rapidly growing list of undocumented macros that are entirely internal to the implementation, that have no guarantees of stability and are not intended for user code:

Nearly everything in gen_include/upcxx/upcxx_config.hpp, eg :
- UPCXX_ISSUE400_WORKAROUND
- UPCXX_HAVE___BUILTIN_ASSUME_ALIGNED
- UPCXX_GASNET_NATIVE_NP_ALLOC_REQ_MEDIUM
- UPCXX_MPSC_QUEUE_ATOMIC
UPCXX_UNIFORM_LOCAL_VTABLES
UPCXX_HAS_OPERATION_CX_AS_BLOCKING
UPCXX_AD_METHODS
UPCXX_GPTR_CHK
UPCXX_ETYPE
UPCXX_ASSERT_MASTER_CURRENT_IFSEQ
Around 100 other macros used internally by the UPC++ headers

My concern is that currently these all live in the same UPCXX_ macro namespace, providing no obvious distinction regarding which of the five categories a given macro falls into. Worse, the lack of documentation for category 3..5 means that a user "fishing" for undocumented macros has no way to distinguish whether a given macro falls into category 3 or 5.

I propose the following (in priority order):

We rename all the macros in category 5 into a unique prefix, for example UPCXXI_ (where the "I" stands for "internal"). This has the purpose of immediately breaking any external code relying on these macros we really don't want used outside the runtime, and "giving notice" to all future macro fishermen that these are unstable.
Update implementation-defined.md to clearly state that all UPCXXI_* macros are off-limits to client code, and that client-code should never define symbols into UPCXX_* or UPCXXI_* (except where a given symbol is explicitly documented as a user-provided setting)
Add documentation for category 3 macros in implementation-defined.md (promoting them to category 2)
"Prune" category 4 by replacing test uses with documented macros, and possibly even replacing outdated ones in the implementation.

Consider renaming internal UPCXX_ macros

Comments (7)

Macros defined by the UPC++ spec

Macros defined in `implementation-defined.md`

Macros that probably should be defined (@Dan Bonachea 's category 3)

Macros that are stable (@Dan Bonachea 's category 4)

Environment variables (currently undocumented)

Additional macros used in tests (in addition to the 4 macro categories above)

Environment variables used in tests

Macros only used in tests (not in the upcxx source)

Macros that we might consider defining (in addition to @Dan Bonachea 's)

Ident strings (global variables)