Expose selected parts of the upcxx serialization interface

In the Pagoda meeting 2/24/2021, it was discussed that parts of the serialization interface implemented internally could be exposed for use, and possible re-use, of the serialized byte stream.

As a proof of concept, I’ve prototyped an implementation extending the upcxx::detail::serialization_reader and upcxx::detail::serialization_writer implementations and, at least in my tests, it seems to be working properly. (see https://bitbucket.org/berkeleylab/upcxx-utils/src/ScatterBlockAllocator/include/upcxx_utils/memory-allocators/ScatterBlockAllocator.h )

So this proposal is to discuss what can easily be exposed from the upcxx side to support something like this.

Dan commented on slack:

design a much narrower interface that takes an object and produces the 
serialized representation in a contiguous buffer (and vice-versa), 
which is something that can be implemented using the internal interface 
and should be supportable long-term.

Rob commented:

For my purposes all I needed from upcxx::detail was a generic (and 
inheritable) Writer and Reader -- I don't think that I need anything 
beyond constructors and what the specs specify must be their public 
member functions.  I ended up extending the detail 
serialization_writer<true> to dynamically manage the 
temporary buffer to which it was writing and then eventualy 
rput the contents.

I’ll start by proposing that upcxx offer the following classes to the user in its API to allow the user to construct and use the Writer and Reader templated class API mentioned in the specs easily, and thereby leverage all the work on supporting upcxx_serialization structs and UPCXX_SERIALZED_* macros

template<typename Allocator=std::allocator<char>>
class generic_writer {
public:
  generic_writer();

  // the underlying byte stream 
  // from Allocator owned and managed by this generic_writer
  // data is guaranteed to be a single and contiguous allocation
  char const *data() const;

  // the bytes used in the buffer
  size_t size() const;

  // (optional) ensures there are at least more_bytes allocated and available
  // in the byte stream for the next set of writes
  void reserve(size_t more_bytes);

  template <typename T> void write(T const &x);
  template <typename Iter> std::size_t write_sequence(Iter beg, Iter end, std::size_t n = -1);

};

class generic_reader {
public:
   generic_reader(char const *byte_stream);

   // returns the current position in the byte_stream
   char const *data() const;

   template<typename T> T read();
   template<typename T> T* read_into(char *storage);
   template<typename T> T* read_sequence_into(char *storage, size_t num_items);

   // (optional) skips n elements of T
   // if possible, the implementation may avoid explicit deserialization
   template<typename T> void skip(size_t n);
};

‌

Comments (9)