Function pointer shipping

[john bachan]

Currently, the implementation serializes function pointers using a translation mechanism (as offsets from some fixed function, say main()). This happens anytime it is evident the type T being serialized is a function pointer/reference (takes the from Ret(*)(Arg...) or Ret(&)(Arg...) for some typename Ret, Arg...). This has gotchas, because function pointers are considered trivial types that according to the spec should be serialized blindly byte-wise. To address this, we can revert function pointers back to trivial serialization and add a type that wraps a function pointer and but serializes with translation.

Proposed type is specialization of global_ptr<T> where T is a function (not a function pointer or reference). This fits well since global_ptr to function is otherwise useless to the user and acts analogously to the C++ pointer (*)-type.

// specialize global_ptr to function as our translated function pointer
// nice guarantee: global_ptr is TriviallySerializable
template<typename Ret, typename ...Arg>
class global_ptr<Ret(Arg...)> {
  std::intptr_t delta_main_; // byte offset from &::main, or some other reference point
public:
  global_ptr(Ret(*fn)(Arg...)) {
    delta_main_ = fn - &main;
  }

  Ret operator()(Arg ...a) const {
    return (delta_main_ + &main)(std::forward<Arg>(a)...);
  }

  using function_t = Ret(Arg...);
  operator funciton_t*() { return delta_main_ + &main; }

  friend bool operator==();
  friend bool operator!=(); // and more pointer-like stuff...
};

[Dan Bonachea]

I like the idea of exposing our function translation mechanism for explicit use by users, so they can conveniently perform function translation on function pointers embedded in their data as necessary. The concept of a "portable function pointer" is very nice.

Presumably you'd apply some magic so this mechanism would continue to be automatically applied to the function argument to rpc and rpc_ff, so that naive users can reap the benefits with no change to syntax?

Given this proposal embodies function translation as a first-class concept, we should probably think about what's required to support the corner cases where our current hack falls down - namely dynamic libraries and multiple independent code segments. I think the fix here might be as simple as allowing a template specialization to specify the "basis" function to be something in the same code segment (ie something other than main, which can remain the default).

Minor comment on the proposed implementation: You are probably missing some casts on the arithmetic operations, to ensure correct operation.

[Dan Bonachea]

Some discussion of this issue in the implementation PR

[Dan Bonachea]

This topic was discussed in the 2/20/18 meeting , and we resolved that a public feature meeting this need could be useful, but we don't yet understand all the design issues and thus it probably won't be ready in time for the March release.

We tentatively agreed that the eventual feature should not publicly specialize global_ptr<>, since portable function pointers do not behave like normal global pointers in two important ways: (1) they don't support pointer arithmetic, (2) they have no well-defined concept of affinity. They may also eventually need an additional template parameter to accommodate a user-defined basis. Given these important conceptual differences, global function pointers merit their own dedicated type abstraction (assuming we decide to expose one at all).

[Dan Bonachea]

This issue was triaged at the 2018-06-13 Pagoda meeting and assigned a new milestone/priority.

[Dan Bonachea]

Non-trivial Serialization support is slated to appear in the Mar 2020 release.

[BrianS]

I’ve been struggling with device function pointers for some versions of nvcc and C++. trying to use device pointers in variadic functions and the C++ compiler sort of losing it’s type-ness. This has lead to me using odd cuda runtime features to manually map function pointers from host to device. It is all so ugly under the hood. I wonder if this is a better abstraction style for my Proto effort.

[Dan Bonachea]

This was discussed in the 2020-02-12 meeting and deferred to next release milestone.

We identified that one important stakeholder may have a use case for RPC to functions in a dynamic library. If so, this might justify a milestone-level feature upgrade to support this use case, and the capability described in this issue would likely form a part of that solution. It might even be a an appropriate topic for a working group draft in an outreach quarter.

[Dan Bonachea]

Mass roll-over of open issues to next release milestone

[Colin MacLean]

There are some issues with having an interface that’s written to accept all runtime function pointers. Using offset from main or offset from library (ie, function pointers from dlsym) only handles a subset of potential function pointers. One of the main reasons for dlopen/dlsym function loading is to have a plugin interface where polymorphism is used to do different things using a common function call signature. Such a symbol would have the same name but would reside in a different location within the object. Another problem is that the library may not be loaded on all nodes, as would be the case in a heterogeneous system where opening a library tries to access hardware not available on all nodes. Or the library may be different, as is the case with SYCL FPGA libraries which each contain a set of kernels to be loaded together and it is common to want different FPGAs to do different stages of the data flow. C++ doesn’t guarantee that a function even exist within an object. Although such function pointers aren’t of practical concern today, there is a good chance they will be in the future. One of the papers that generated quite a bit of excitement at the Prague C++ committee meeting was Hal Finkel’s proposal for C++ JIT, p1609. While this paper is still early in the committee process, it is a good bet that some sort of C++ JIT is a feature we need to watch out for and shouldn’t back ourselves into a corner regarding forward compatibility. Facebook and Google already do their own C++ JIT, so there is a good amount of industry support for standardizing it, and there are significant HPC use cases such as implementing a high performance domain-specific language in C++. JITed C++ would create different temporary objects compiled and cached on each node. Therefore, I would be opposed to a generic function pointer interface that relies on such implementation-defined behavior. Such functionality is useful, but I think the naming and interface should be more explicit. To avoid UB, a generic global_ptr of function type needs to only be guaranteed to be valid on the rank that originally created it.

Implementation-defined behavior such as calculating the address from anchor points could be supported by giving global_ptr<F> a function pointer of type F*, a void* anchor pointer to be recomputed internally using linker trickery, and the owning rank. A nullptr anchor would indicate that the function pointer was a full pointer rather than a difference and local to the owning rank. A non-null anchor would indicate computing the address based upon linker details. Or such a pointer could have its own class. My vote would be to give it its own class, as the pointer with affinity generic case is something that global_ptr can already handle, global_ptrs are assumed to have affinity, and the ability to recompute a pointer based upon known anchors is a different sort of functionality.

[Dan Bonachea]

Mass roll-over of open issues to next release milestone

[Dan Bonachea]

Mass roll-over of open issues to next release milestone

[Dan Bonachea]

The 2022.3.0 release introuced CCS support, solving the specific problem of cross-code-segment RPC calls.

This idea still remains as a potential future enhancement, which would allow users to store function pointers in the shared heap and leverage our relocation infrastructure. However with the deployment of CCS we currently have no stakeholder-driven motivation for such an enhancement.