Provide contrib code that implements the C++ Allocator concept for the shared heap

[Amir Kamil]

Implementing the Allocator concept would allow STL and other C++ code to allocate memory in the registered segment.

[Amir Kamil]

So this should be as simple as the following:

template<typename T>
struct allocator {
  using value_type = T;
  allocator() {}
  template<typename U>
  allocator(const allocator<U> &other) {}
  [[nodiscard]] T* allocate(std::size_t n) const {
    if (T* result = upcxx::allocate<T>(n).local()) {
      return result;
    }
    throw std::bad_alloc();
  }
  void deallocate(T* ptr, std::size_t n) const {
    upcxx::deallocate(ptr);
  }
};

template<typename T, typename U>
bool operator==(const allocator<T>& lhs, const allocator<U>& rhs) {
  return true;
}

template<typename T, typename U>
bool operator!=(const allocator<T>& lhs, const allocator<U>& rhs) {
  return false;
}

[BrianS]

this would be done in namespace upcxx? so all our allocators evaluate as equal and type conversion is trivial?

[Amir Kamil]

Yes. This is compliant with the Allocator concept and essentially the same as std::allocator.

[BrianS]

This has worked out really nice for me in my preliminary testing

struct particle_t {
  float x, y, z, vx, vy, vz, charge;
  uint64_t id;
};

typedef std::vector<particle_t,upcxx::allocator<particle_t>> uvector;

A few gotchas for me were having any uvector objects that get destructed after finalize(), but that is precisely what we expect to happen.

[Dan Bonachea]

Brian encountered a problem when using a statically-scoped vector with this custom shared allocator, because destruction happens after upcxx::finalize() and causes an exit-time crash in upcxx::deallocate

If we ever deploy the custom shared allocator as part of upcxx it should probably keep a library "epoch" counter in each object and skip deallocation of objects for expired epochs. The global epoch counter would advance each time an "outermost" upcxx::finalize() tears down the library and shared segment.

[Mathias Jacquelin]

Dan, just so I understand clearly, we are not spilling any memory because the shared segment has been teared down by the upcxx::finalize() call. Is this correct ?

Best,

Mathias Jacquelin Research Scientist Lawrence Berkeley National Laboratory [e-mail] 1-510-495-2605

[Amir Kamil]

I think we fundamentally cannot support statically-scoped containers that use this allocator. In the case of non-TriviallyDestructible element type, the container will invoke the destructor of each element, resulting in UB if the memory has been released. While this doesn't effect element types such as doubles, I'm not sure we should be making it easier to do something that is a conceptual error even if we can make it work in practice.

[Paul Hargrove]

It occurs to me that there may also me a use for constructing C++ objects in GPU memory. So, the interaction with Memory Kinds should be considered.

[Dan Bonachea]

This issue was triaged at the 2018-06-13 Pagoda meeting and assigned a new milestone/priority.

It was noted that satisfying this use case doesn't necessarily require any spec changes at all, as the required glue could be provided as an external "add-on" utility header that doesn't touch any upcxx internals.

[Dan Bonachea]

The consensus seems to be that this is not a spec issue at all, nor does it require access to UPC++ internals. This is a task that someone should provide an appropriate header in a contrib dir to encapsulate Amir's code above for ease of re-use.

Removing from milestone, but leaving it open until someone does this.

[Dan Bonachea]

This has been added to upcxx-extras, a forthcoming optional add-on package for UPC++.