envvar-tunable RPC eager/rendezvous threshold

[Dan Bonachea]

In today's SPMV meeting, we identified a need to allow the eager/rendzevous threshold used internally by the UPC++ RPC data transfer (gasnet::am_size_rdzv_cutover) to be tunable by the end-user, up to the GASNet AM MaxMedium.

We should really add an envvar to turn this voodoo knob (probably not for the September release).

[Mathias Jacquelin]

I have tried this yesterday on Cori Haswell and improvements are... non existent. Also, the variant in which a LPC is issued from within the RPC is slower than doing the flops directly inside the RPC.

I will forward the plots by email.

[john bachan]

Mathias, can you try this in combination with maxing out the max-medium allowed by gasnet? To do this, put GASNET_CONFIGURE_ARGS=--with-areis-max-medium=64896 in your environment while building upcxx.

[Mathias Jacquelin]

OK thanks, I will give it a try today. I will also try to remove the progress call before the diagonal spmv call.

Best,

Mathias Jacquelin Research Scientist Lawrence Berkeley National Laboratory [e-mail] 1-510-495-2605

[Dan Bonachea]

Spelling correction:

GASNET_CONFIGURE_ARGS=--with-aries-max-medium=64896

this will allow you to change the gasnet::am_size_rdzv_cutover up to anywhere close to that size, although note that maxxing the size near 64k may not be optimal. There is presumably a local maxima for some value of the threshold corresponding to a cross-over where rendezvous begins to be profitable (but probably something higher than the hard-coded 512 byte default). My guess would be in the few KB range, but it depends on the exact overhead of the additional messages.

[Dan Bonachea]

I've been working on tuning enhancements to the RPC implementation that will include resolving this issue.

This is not yet a PR, but some preliminary results were shown at the 2020-11-19 Pagoda meeting

CC: @Steven Hofmeyr @Rob Egan

[Dan Bonachea]

Proposed resolution now in PR 301

[Dan Bonachea]

issue #164: Add UPCXX_RPC_EAGER_THRESHOLD knob

This new knob controls the eager/rendezvous threshold for the RPC protocol.

Resolves issue #164

→ <<cset b13c27b9c471>>