Provide memberof(global_ptr<T>, member_designator)

[Dan Bonachea]

Problem statement

An end user question on the mailing list had highlighted a gap in our global_ptr API.

Consider this code:

struct QueueNode {
 int64_t field1;
 int32_t field2;
};
...
global_ptr<QueueNode> gp = ...; // fetch pointer to some remote object

How would one construct a global_ptr to perform an rget on a single field of this remote object?

For the first field, one can get away with something like:

global_ptr<int64_t> gp_f1 = reinterpret_pointer_cast<int64_t>(gp);

However this only happens to work for the first field and only because this is a standard-layout class (making the addresses of the object and first field pointer-interconvertible, see [basic.compound])

For the second and subsequent fields, one needs something nastier like:

global_ptr<int32_t> gp_f2 = reinterpret_pointer_cast<int32_t>(
  reinterpret_pointer_cast<char>(gp) + offsetof(QueueNode, field2));

However this seems error-prone and due to [support.types.layout] is still only guaranteed to work for standard-layout classes - which notably excludes classes containing a reference field or a field with different access control.

History

UPC++ 0.1 provided a macro for handling this situation:

  // obj is a global_ref_base or a global_ptr of the object.
  // m is a field/member of the global object.
  #define upcxx_memberof(obj, m)

and I think we should consider providing a similar macro.

There was some discussion about this feature in early design stages, but review of historical emails seems to indicate we did not fully examine the consequences of this omission in the context of the current/final design of UPC++.

Proposal:

#define upcxx_memberof(gptr_obj, member_designator) ...

Preconditions:

There would probably need to be some preconditions on T, but I'm not yet sure what those would be.

Semantics:

The first argument gptr_obj must be a global_ptr<T,Kind> referencing a valid (potentially remote) object. The second argument member_designator is a field designator following rules analogous to offsetof() from <cstddef>. The result is a global_ptr<U,Kind> referencing the selected field in the same object, where U is the type of the field in T indicated by member_designator.

Example:

global_ptr<int32_t> gp_f2 = upcxx_memberof(gptr, field2);

Discussion

This has to be implemented as a macro (for the same reasons as offsetof), so it has an explicit prefix to prevent name collisions (we cannot rely on C++ namespaces for macros).

[john bachan]

Great. Potential implementation?

#define upcxx_memberof(gp, FIELD) \
  ::upcxx::global_ptr<decltype(::std::declval<decltype(gp)::element_type>().FIELD), decltype(gp)::memory_kind>( \
    ::upcxx::detail::internal_only(), \
    gp.rank_, \
    reinterpret_cast<decltype(::std::declval<decltype(gp)::element_type>().FIELD)*>( \
      reinterpret_cast<::std::uintptr_t>(gp.raw_ptr_) + offsetof(decltype(gp)::element_type, FIELD) \
    ), \
    gp.device_)

[Dan Bonachea]

@john bachan said:

Potential implementation

Yes something like that is what I had in mind.

How do we feel about the fact that upcxx_memberof would inherit the same restrictions that C++ puts on offsetof, namely that it only works for standard-layout classes? Based on this interesting thread it appears at least gcc and clang issue loud warnings (or errors?) when users try to violate this rule, which would also include invalid uses of upcxx_memberof using this implementation.

An alternative would be an implementation or separate entry point (upcxx_dynamic_memberof?) that actually communicated and used the & operator on the field in question to construct a gptr. This should have the benefit of working for all types (although the results might be surprising in classes overloading operator&), but the downside of injecting an RPC communication latency that the user might have been trying to avoid by constructing a gptr for use in RMA.

[Amir Kamil]

C++17 has conditional support for offsetof on non-standard-layout types. I think we should just provide the non-communicating version and specify that T must be a type on which the underlying compiler supports offsetof.

[john bachan]

Concur with @Amir Kamil

[Dan Bonachea]

@Amir Kamil said:

provide the non-communicating version and specify that T must be a type on which the underlying compiler supports offsetof.

This sounds reasonable, however as it did not become conditionally supported until C++17 (it was previously simply undefined behavior), users cannot necessarily rely upon getting a diagnostic for its use on a non-conforming type. Similarly, the spec-mandated documentation for conditionally supported behavior may be poor or non-existent (e.g. here is the conditionally supported documentation for gcc 9.1.0)

All this unfortunately means the user's first indication that something's gone wrong may be silent data corruption.

Perhaps we could partially mitigate this by at least statically asserting offsetof(decltype(gp)::element_type, FIELD) < sizeof(decltype(gp)::element_type). That helps ensure that random bogus outputs from offsetof at least cannot result in our memberof returning a pointer to memory outside the boundaries of the containing struct. It might also encourage stricter conformance checking from the compiler if we force the assertion to be evaluated at compile time.

[Paul Hargrove]

FYI PGI-19.10 has grown picky:

mpicxx -std=c++11 -D_GNU_SOURCE=1 -I/home/data2/upcnightly/dirac/EX-dirac-ibv-pgi_1910/work/dbg/upcxx/.nobs/art/13d7940579106f14bdb27be6e0e507ac43dfdbac -DUPCXX_ASSERT_ENABLED=1 -DUPCXX_MPSC_QUEUE_ATOMIC=1 -mp -O0 -g -c /home/data2/upcnightly/dirac/EX-dirac-ibv-pgi_1910/work/dbg/upcxx/src/future/core.cpp -o /home/data2/upcnightly/dirac/EX-dirac-ibv-pgi_1910/work/dbg/upcxx/.nobs/art/1956549d518c3a6f685f3e9112dcf3da7cda68f9.core.cpp.o
"/home/data2/upcnightly/dirac/EX-dirac-ibv-pgi_1910/work/dbg/upcxx/src/future/c
          ore.cpp", line 26: warning: offsetof applied to non-POD types is
          nonstandard
    offsetof(future_header_promise<>, pro_meta),
    ^

I will enter a new issue for this later.

[john bachan]

I birthed this idea in a meeting: we could supply a upcxx::tuple<T...> type that permits the following:

template<int i, typename ...T>
upcxx::global_ptr<?> upcxx::tuple_member(upcxx::global_ptr<upcxx::tuple<T...>> gp)

// usage
global_ptr<upcxx::tuple<int,int>> gp = ...;
global_ptr<int> m1 = upcxx::tuple_member<0>(gp);
global_ptr<int> m2 = upcxx::tuple_member<1>(gp);

[Dan Bonachea]

Strawman specification:

The three variants below are currently implemented in Implementation PR #150.

upcxx_memberof

// Macro:
global_ptr<field_type> upcxx_memberof(global_ptr<T> gp, field-designator FIELD)

Preconditions: gp is a pointer to a (possibly uninitialized) object of type T. T must be a standard-layout type. The expression offsetof(T, FIELD) must be valid in the calling context.

Semantics: Evaluates to a global pointer referencing the specified field of the object referenced by gp.

Progress level: none

upcxx_memberof_unsafe

// Macro:
global_ptr<field_type> upcxx_memberof_unsafe(global_ptr<T> gp, field-designator FIELD)

Preconditions: gp is a pointer to a (possibly uninitialized) object of type T. The expression offsetof(T, FIELD) must be valid in the calling context~\footnote{This implies either T is a standard-layout type, or (C++17) is conditionally supported by the compiler for use in offsetof().).

Semantics: Evaluates to a global pointer referencing the specified field of the object referenced by gp.

Advice to users: In the event that T is not a standard-layout type, upcxx_memberof_unsafe relies on compiler-dependent behavior and may generate an error or undefined result.

Progress level: none

upcxx_memberof_general

// Macro:
future<global_ptr<field_type>> upcxx_memberof_general(global_ptr<T> gp, field-designator FIELD)

Preconditions: gp is a pointer to a (possibly uninitialized) object of type T. FIELD is a field designator specifying a subobject of that object with type field_type. Given a valid T* lp referencing the target object, the expression lp->FIELD must be valid in the calling context.

Semantics: Computes a global pointer referencing the specified field of the object referenced by gp, using the most efficient mechanism available. If the result is determined using purely local information, then the progress level is none and the result is a readied future. Otherwise, the progress level is internal and the resulting future will be readied during a subsequent user-level progress for the calling persona.

Progress level: none or internal

Discussion:

The limitations of C++ make it impossible to portably provide a communication-free memberof macro that works for all user types. The three variations above satisfy distinct use cases:

1. upcxx_memberof is the preferred choice for standard layout types, because it's guaranteed to always work, without communication or future overheads.
2. For types that diverge in only minor ways from standard layout, upcxx_memberof_unsafe provides the same functionality for users who want to "live dangerously" (hopefully because they know their compiler conditionally supports this type for offsetof).
3. Finally, upcxx_memberof_general is the most general solution that works for any type, but always adds future overheads and if the object is remote and not standard layout then it additionally requires communication (and user-level progress on both sides).

[Dan Bonachea]

This was discussed in the 2020-02-12 meeting .

We decided to keep the upcxx_memberof_unsafe variant unspecified for now.

I will iterate on spec and implementation, still targeting this release if possible.

[Dan Bonachea]

Proposed specification in pull request #30

[Dan Bonachea]

Specify upcxx_memberof(_general)

See Issue #151 for discussion.

→ <<cset e7e492fba7c7>>

[Dan Bonachea]

Specified in pull request #30, implemented in implementation PR #150.