Invalid GASNet call while deserializing a global ptr

[Rob Egan]

So I can only reproduce this error on cori with the upcxx/nightly but will next try with a development version on my desktop. This is triggered in the unit test of upcxx_utils' global_shared_ptr which is attempting to implement an interface similar to std::shared_ptr but with a global_ptr on the local_team… It basically is a global_ptr< pair< atomic<int64> , global_ptr<T> >, where the atomic is incremented on construction and serialization and decremented on destruction and the global pointers are freed when the count == 0. It has been working fine since 2020.03 up until I tried it on nightly, and haven’t been able to figure out what is breaking.

https://bitbucket.org/berkeleylab/upcxx-utils/src/SharedGlobalPtrSerialization/include/upcxx_utils/shared_global_ptr.hpp

and

https://bitbucket.org/berkeleylab/upcxx-utils/src/SharedGlobalPtrSerialization/test/test_shared_global_ptr.cpp

commit 2f8220bd5a9650a044d1d7fe5e894e6f8e627b8d (sorry I couldn’t get the links to hit the latest commit).

I ran on 2 knl nodes with 136 ranks, but it also fails on 1 node. All the ranks throw this error and here is trace proc10

*** Details for bug reporting (proc 10): config=RELEASE=2020.10.3,SPEC=1.16,PTR=64bit,debug,SEQ,timers_native,membars_native,atomics_native,atomic32_native,atomic64_native compiler=GNU/8.3.0 sys=x86_64
-cnl-linux-gnu


[10] /usr/bin/gdb -nx -batch -x /tmp/gasnet_dQXI8Z '/global/cscratch1/sd/regan/mhm2-builds/upcxx-utils/build-debug-nightly/test/test_shared_global_ptr' 241334
[10] [Thread debugging using libthread_db enabled]
[10] Using host libthread_db library "/lib64/libthread_db.so.1".
[10] 0x00002aaaac7fe2ba in waitpid () from /lib64/libc.so.6
[10] To enable execution of this file add
[10]    add-auto-load-safe-path /opt/gcc/10.1.0/snos/lib64/libstdc++.so.6.0.28-gdb.py
[10] line to your configuration file "/global/homes/r/regan/.gdbinit".
[10] To completely disable this security protection add
[10]    set auto-load safe-path /
[10] line to your configuration file "/global/homes/r/regan/.gdbinit".
[10] For more information about this security protection see the
[10] "Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
[10]    info "(gdb)Auto-loading safe path"
[10] #0  0x00002aaaac7fe2ba in waitpid () from /lib64/libc.so.6
[10] #1  0x00002aaaac77b86f in do_system () from /lib64/libc.so.6
[10] #2  0x0000000020143c1b in gasneti_system_redirected (cmd=0x4054a580 <cmd> "/usr/bin/gdb -nx -batch -x /tmp/gasnet_dQXI8Z '/global/cscratch1/sd/regan/mhm2-builds/upcxx-utils/build-debug-nightly/test/test_shared_global_ptr' 241334", stdout_fd=8) at /global/cscratch1/sd/hargrove/upcxx-nightly-cori_knl-gnu/bld/upcxx_install/berkeleylab-upcxx-mk-develop/bld/GASNet-develop/gasnet_tools.c:1276
[10] #3  0x0000000020144395 in gasneti_bt_gdb (fd=8) at /global/cscratch1/sd/hargrove/upcxx-nightly-cori_knl-gnu/bld/upcxx_install/berkeleylab-upcxx-mk-develop/bld/GASNet-develop/gasnet_tools.c:1532
[10] #4  0x0000000020144bdd in gasneti_print_backtrace (fd=2) at /global/cscratch1/sd/hargrove/upcxx-nightly-cori_knl-gnu/bld/upcxx_install/berkeleylab-upcxx-mk-develop/bld/GASNet-develop/gasnet_tools.
c:1810
[10] #5  0x00000000201451d9 in _gasneti_print_backtrace_ifenabled (fd=2) at /global/cscratch1/sd/hargrove/upcxx-nightly-cori_knl-gnu/bld/upcxx_install/berkeleylab-upcxx-mk-develop/bld/GASNet-develop/ga
snet_tools.c:1943
[10] #6  0x0000000020142c96 in gasneti_error_abort () at /global/cscratch1/sd/hargrove/upcxx-nightly-cori_knl-gnu/bld/upcxx_install/berkeleylab-upcxx-mk-develop/bld/GASNet-develop/gasnet_tools.c:764
[10] #7  0x0000000020142e41 in _gasneti_fatalerror (msg=0x204a3208 "Invalid GASNet call (communication injection or poll) while executing a Request handler") at /global/cscratch1/sd/hargrove/upcxx-nigh
tly-cori_knl-gnu/bld/upcxx_install/berkeleylab-upcxx-mk-develop/bld/GASNet-develop/gasnet_tools.c:800
[10] #8  0x000000002034b3c6 in gasneti_check_inject (for_reply=0) at /global/cscratch1/sd/hargrove/upcxx-nightly-cori_knl-gnu/bld/upcxx_install/berkeleylab-upcxx-mk-develop/bld/GASNet-develop/gasnet_in
ternal.c:375
[10] #9  0x00000000203b00a0 in gasneti_Segment_QueryBound (tm=0xffffffffbfa9537f, rank=11, owneraddr_p=0x7fffffff3098, localaddr_p=0x0, size_p=0x7fffffff3090) at /global/cscratch1/sd/hargrove/upcxx-nig
htly-cori_knl-gnu/bld/upcxx_install/berkeleylab-upcxx-mk-develop/bld/GASNet-develop/gasnet_mmap.c:2315
[10] #10 0x0000000020065c86 in upcxx::backend::validate_global_ptr (allow_null=true, rank=11, raw_ptr=0x2aaaf7e003e0, heap_idx=0, KindSet=upcxx::memory_kind::host, T_align=8, T_name=0x2041b920 <typeinf
o name for std::pair<std::atomic<long>, upcxx::global_ptr<int, (upcxx::memory_kind)1> >> "St4pairISt6atomicIlEN5upcxx10global_ptrIiLNS2_11memory_kindE1EEEE", short_context=0x2040b770 <upcxx::global_ptr
<std::pair<std::atomic<long>, upcxx::global_ptr<int, (upcxx::memory_kind)1> > const, (upcxx::memory_kind)1>::is_local() const::__func__> "is_local", context=0x2040b6a0 <upcxx::global_ptr<std::pair<std:
:atomic<long>, upcxx::global_ptr<int, (upcxx::memory_kind)1> > const, (upcxx::memory_kind)1>::is_local() const::__PRETTY_FUNCTION__> "bool upcxx::global_ptr<const T, KindSet>::is_local() const [with T
= std::pair<std::atomic<long int>, upcxx::global_ptr<int, (upcxx::memory_kind)1> >; upcxx::memory_kind KindSet = (upcxx::memory_kind)"...) at /global/cscratch1/sd/hargrove/upcxx-nightly-cori_knl-gnu/bl
d/upcxx_install/be[10] rkeleylab-upcxx-mk-develop/src/backend/gasnet/runtime.cpp:1416
[10] #11 0x00000000200293f2 in upcxx::global_ptr<std::pair<std::atomic<long>, upcxx::global_ptr<int, (upcxx::memory_kind)1> > const, (upcxx::memory_kind)1>::check (this=0x7fffffff36b0, allow_null=true,
 short_context=0x2040b770 <upcxx::global_ptr<std::pair<std::atomic<long>, upcxx::global_ptr<int, (upcxx::memory_kind)1> > const, (upcxx::memory_kind)1>::is_local() const::__func__> "is_local", context=
0x2040b6a0 <upcxx::global_ptr<std::pair<std::atomic<long>, upcxx::global_ptr<int, (upcxx::memory_kind)1> > const, (upcxx::memory_kind)1>::is_local() const::__PRETTY_FUNCTION__> "bool upcxx::global_ptr<
const T, KindSet>::is_local() const [with T = std::pair<std::atomic<long int>, upcxx::global_ptr<int, (upcxx::memory_kind)1> >; upcxx::memory_kind KindSet = (upcxx::memory_kind)"...) at /usr/common/ftg
/upcxx/nightly/craype-2.6.2/knl/gnu/PrgEnv-gnu-6.0.5-8.3.0-2021.01.16/upcxx.debug.gasnet_seq.aries/include/upcxx/global_ptr.hpp:94
[10] #12 0x0000000020024bff in upcxx::global_ptr<std::pair<std::atomic<long>, upcxx::global_ptr<int, (upcxx::memory_kind)1> > const, (upcxx::memory_kind)1>::is_local (this=0x7fffffff36b0) at /usr/commo
n/ftg/upcxx/nightly/craype-2.6.2/knl/gnu/PrgEnv-gnu-6.0.5-8.3.0-2021.01.16/upcxx.debug.gasnet_seq.aries/include/upcxx/global_ptr.hpp:142
[10] #13 0x0000000020043215 in upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1>::upcxx_serialization::deserialize<upcxx::detail::serialization_reader> (reader=..., storage=0x7fffffff3790) at
/global/homes/r/regan/workspace/mhm2/upcxx-utils/include/upcxx_utils/shared_global_ptr.hpp:247
[10] #14 0x0000000020043cdf in upcxx::detail::serialization_reader::read_into<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1>&&, true, upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind
)1> > (this=0x7fffffff3930, raw=0x7fffffff3790) at /usr/common/ftg/upcxx/nightly/craype-2.6.2/knl/gnu/PrgEnv-gnu-6.0.5-8.3.0-2021.01.16/upcxx.debug.gasnet_seq.aries/include/upcxx/serialization.hpp:648
[10] #15 0x00000000200434a3 in upcxx::detail::serialization_tuple<std::tuple<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1>&&>, 0, 1>::deserialize_each<std::tuple<upcxx_utils::shared_global
_ptr<int, (upcxx::memory_kind)1> >, upcxx::detail::serialization_reader> (r=..., spot=0x7fffffff3918) at /usr/common/ftg/upcxx/nightly/craype-2.6.2/knl/gnu/PrgEnv-gnu-6.0.5-8.3.0-2021.01.16/upcxx.debug
.gasnet_seq.aries/include/upcxx/serialization.hpp:1552
[10] #16 0x000000002004305b in upcxx::serialization<std::tuple<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1>&&> >::deserialize<upcxx::detail::serialization_reader> (r=..., spot=0x7fffffff3
918) at /usr/common/ftg/upcxx/nightly/craype-2.6.2/knl/gnu/PrgEnv-gnu-6.0.5-8.3.0-2021.01.16/upcxx.debug.gasnet_seq.aries/include/upcxx/serialization.hpp:1605
[10] #17 0x0000000020042a8b in upcxx::detail::serialization_reader::read_into<std::tuple<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1>&&>, true, std::tuple<upcxx_utils::shared_global_ptr<i
nt, (upcxx::memory_kind)1> > > (this=0x7fffffff3930, raw=0x7fffffff3918) at /usr/common/ftg/upcxx/nightly/craype-2.6.2/knl/gnu/PrgEnv-gnu-6.0.5-8.3.0-2021.01.16/upcxx.debug.gasnet_seq.aries/include/upc
xx/serialization.hpp:648
[10] #18 0x0000000020041e6c in upcxx::detail::deserialized_bound_function_storage<upcxx::backend::send_awaken_lpc<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1>, upcxx_utils::shared_global_
ptr<int, (upcxx::memory_kind)1>&&>(int, upcxx::detail::lpc_dormant<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >*, std::tuple<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1>&&
>&&)::{lambda(std::tuple<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >&&)#1}, std::tuple<std::tuple<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1>&&> >, upcxx::detail::index_
sequence<0> >::deserialized_bound_function_storage(upcxx::detail::serialization_reader&) (this=0x7fffffff3910, r=...) at /usr/common/ftg/upcxx/nightly/craype-2.6.2/knl/gnu/PrgEnv-gnu-6.0.5-8.3.0-2021.0
1.16/upcxx.debug.gasnet_seq.aries/include/upcxx/bind.hpp:157
[10] #19 0x000000002004146b in upcxx::detail::deserialized_bound_function_base<upcxx::backend::send_awaken_lpc<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1>, upcxx_utils::shared_global_ptr
<int, (upcxx::memory_kind)1>&&>(int, upcxx::detail::lpc_dormant<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >*, std::tuple<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1>&&>&&
)::{lambda(std::tuple<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >&&)#1}, std::tuple<std::tuple<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1>&&> >, upcxx::detail::index_seq
uence<0>, true>::deserialized_bound_function_base(upcxx::detail::serialization_reader&) (this=0x7fffffff3910, r=...) at /usr/common/ftg/upcxx/nightly/craype-2.6.2/knl/gnu/PrgEnv-gnu-6.0.5-8.3.0-2021.01
.16/upcxx.debug.gasnet_seq.aries/include/upcxx/bind.hpp:206
[10] #20 0x000000002003ffcd in upcxx::detail::deserialized_bound_function<upcxx::backend::send_awaken_lpc<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1>, upcxx_utils::shared_global_ptr<int,
 (upcxx::memory_kind)1>&&>(int, upcxx::detail::lpc_dormant<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >*, std::tuple<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1>&&>&&)::{l
ambda(std::tuple<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >&&)#1}, std::tuple<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1>&&> >::deserialized_bound_function(upcxx::detai
l::serialization_reader&) (this=0x7fffffff3910, r=...) at /usr/common/ftg/upcxx/nightly/craype-2.6.2/knl/gnu/PrgEnv-gnu-6.0.5-8.3.0-2021.01.16/upcxx.debug.gasnet_seq.aries/include/upcxx/bind.hpp:287
[10] #21 0x000000002003edde in upcxx::serialization<upcxx::bound_function<upcxx::backend::send_awaken_lpc<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1>, upcxx_utils::shared_global_ptr<int,
 (upcxx::memory_kind)1>&&>(int, upcxx::detail::lpc_dormant<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >*, std::tuple<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1>&&>&&)::{l
ambda(std::tuple<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >&&)#1}, std::tuple<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1>&&> > >::deserialize<upcxx::detail::serializati
on_reader>(upcxx::detail::serialization_reader&, void*) (r=..., spot=0x7fffffff3910) at /usr/common/ftg/upcxx/nightly/craype-2.6.2/knl/gnu/PrgEnv-gnu-6.0.5-8.3.0-2021.01.16/upcxx.debug.gasnet_seq.aries
/include/upcxx/bind.hpp:368
[10] #22 0x000000002003d42d in upcxx::detail::command<upcxx::detail::lpc_base*>::the_executor<upcxx::bound_function<upcxx::backend::send_awaken_lpc<upcxx_utils::shared_global_ptr<int, (upcxx::memory_ki
nd)1>, upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1>&&>(int, upcxx::detail::lpc_dormant<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >*, std::tuple<upcxx_utils::shared_global
_ptr<int, (upcxx::memory_kind)1>&&>&&)::{lambda(std::tuple<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >&&)#1}, std::tuple<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1>&&> >
, &upcxx::backend::gasnet::rpc_as_lpc::reader_of, &(void upcxx::backend::gasnet::cleanup<false, true>(upcxx::detail::lpc_base*))>(upcxx::detail::lpc_base*) (a#0=0x7fffffff39a0) at /usr/common/ftg/upcxx
/nightly/craype-2.6.2/knl/gnu/PrgEnv-gnu-6.0.5-8.3.0-2021.01.16/upcxx.debug.gasnet_seq.aries/include/upcxx/command.hpp:67
[10] #23 0x0000000020068045 in (anonymous namespace)::am_eager_restricted (buf=0x2aaaae860078, buf_size=32, buf_align=8) at /global/cscratch1/sd/hargrove/upcxx-nightly-cori_knl-gnu/bld/upcxx_install/be
rkeleylab-upcxx-mk-develop/src/backend/gasnet/runtime.cpp:2121
[10] #24 0x00000000201663c8 in gasneti_AMPSHM_service_incoming_msg (vnet=0x4056b4e0, isReq=1) at /global/cscratch1/sd/hargrove/upcxx-nightly-cori_knl-gnu/bld/upcxx_install/berkeleylab-upcxx-mk-develop/
bld/GASNet-develop/gasnet_pshm.c:1208
[10] #25 0x000000002016802e in gasneti_AMPSHMPoll (repliesOnly=0) at /global/cscratch1/sd/hargrove/upcxx-nightly-cori_knl-gnu/bld/upcxx_install/berkeleylab-upcxx-mk-develop/bld/GASNet-develop/gasnet_ps
hm.c:1245
[10] #26 0x00000000200e0ad9 in gasnetc_AMPoll () at /global/cscratch1/sd/hargrove/upcxx-nightly-cori_knl-gnu/bld/upcxx_install/berkeleylab-upcxx-mk-develop/bld/GASNet-develop/aries-conduit/gasnet_core.
c:1239
[10] #27 0x0000000020056c18 in _gasneti_AMPoll () at /global/cscratch1/sd/hargrove/upcxx-nightly-cori_knl-gnu/bld/upcxx_install/berkeleylab-upcxx-mk-develop/bld/GASNet-develop/gasnet_help.h:1210
[10] #28 0x0000000020056fb1 in _gasnet_AMPoll () at /global/cscratch1/sd/hargrove/upcxx-nightly-cori_knl-gnu/bld/upcxx_install/berkeleylab-upcxx-mk-develop/bld/GASNet-develop/gasnet_help.h:1343
[10] #29 0x0000000020067e13 in upcxx::progress (level=upcxx::progress_level::user) at /global/cscratch1/sd/hargrove/upcxx-nightly-cori_knl-gnu/bld/upcxx_install/berkeleylab-upcxx-mk-develop/src/backend
/gasnet/runtime.cpp:2045
[10] #30 0x0000000020009f46 in upcxx::detail::future_wait_upcxx_progress_user::operator() (this=0x7fffffff46eb) at /usr/common/ftg/upcxx/nightly/craype-2.6.2/knl/gnu/PrgEnv-gnu-6.0.5-8.3.0-2021.01.16/u
pcxx.debug.gasnet_seq.aries/include/upcxx/future/future1.hpp:53
[10] #31 0x0000000020020855 in upcxx::future1<upcxx::detail::future_kind_shref<upcxx::detail::future_header_ops_general, false>, upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >::wait<-1, u
pcxx::detail::future_wait_upcxx_progress_user>(upcxx::detail::future_wait_upcxx_progress_user&&) && (this=0x7fffffff46d8, progress=...) at /usr/common/ftg/upcxx/nightly/craype-2.6.2/knl/gnu/PrgEnv-gnu-
6.0.5-8.3.0-2021.01.16/upcxx.debug.gasnet_seq.aries/include/upcxx/future/future1.hpp:327
[10] #32 0x000000002000ecac in test_shared_global_ptr (argc=1, argv=0x7fffffff4c68) at /global/homes/r/regan/workspace/mhm2/upcxx-utils/test/test_shared_global_ptr.cpp:133
[10] #33 0x00000000200097a6 in main (argc=1, argv=0x7fffffff4c68) at /global/homes/r/regan/workspace/mhm2/upcxx-utils/test/main_shell.cpp:17
[10] [Inferior 1 (process 241334) detached]

‌

[Rob Egan]

The good news is that I cannot replicate it using the latest version from the mk-development branch on my single machine: 2635ac137feff9036fe24ca31092112145dc7922

‌

On dirac I can reproduce it on both 1 and 2 nodes with the nightly version, but not the next latest upcxx/2020.11.0/auto. Here is the trace for 2 ranks on a single node

Currently Loaded Modulefiles:
  1) clang/11.0.0              3) upcxx/nightly/auto        5) gcc/10.2.0                7) PrgEnv/gnu/10.2.0
  2) core/Dirac56              4) cmake/newest              6) mpi/openmpi4-gcc-10.2.0



GASNET_BACKTRACE=1 upcxx-run -n 2 -N 1 test/test_shared_global_ptr 


  Found upcxx_utils version 0.3.6.v0.3.5-98-g2f8220b
Opened debug log file:per_rank/00000000/00000000/test_global_shared_ptr-1610947154.dbg
*** FATAL ERROR (proc 1): in gasneti_check_inject() at b-upcxx-mk-develop/bld/GASNet-develop/gasnet_internal.c:375: Invalid GASNet call (communication injection or poll) while executing a Request handler
*** Details for bug reporting (proc 1): config=RELEASE=2020.10.3,SPEC=1.16,PTR=64bit,debug,SEQ,timers_native,membars_native,atomics_native,atomic32_native,atomic64_native compiler=GNU/10.2.0 sys=x86_64-unknown-linux-gnu
[1] Invoking GDB for backtrace...
[1] /usr/local/pkg/gdb/newest/bin/gdb -nx -batch -x /tmp/gasnet_GJ6k7v '/home/pagoda1/regan/workspace/upcxx-utils/build-debug/test/test_shared_global_ptr' 25450
[1] [Thread debugging using libthread_db enabled]
[1] Using host libthread_db library "/usr/lib64/libthread_db.so.1".
[1] 0x00007fdb587f6a3c in waitpid () from /usr/lib64/libc.so.6
[1] To enable execution of this file add
[1]     add-auto-load-safe-path /usr/local/pkg/gcc/10.2.0/lib64/libstdc++.so.6.0.28-gdb.py
[1] line to your configuration file "/home/pagoda1/regan/.gdbinit".
[1] To completely disable this security protection add
[1]     set auto-load safe-path /
[1] line to your configuration file "/home/pagoda1/regan/.gdbinit".
[1] For more information about this security protection see the
[1] "Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
[1]     info "(gdb)Auto-loading safe path"
[1] #0  0x00007fdb587f6a3c in waitpid () from /usr/lib64/libc.so.6
[1] #1  0x00007fdb58774de2 in do_system () from /usr/lib64/libc.so.6
[1] #2  0x000000000056200f in gasneti_system_redirected (cmd=0xbb7c40 <cmd> "/usr/local/pkg/gdb/newest/bin/gdb -nx -batch -x /tmp/gasnet_GJ6k7v '/home/pagoda1/regan/workspace/upcxx-utils/build-debug/test/test_shared_global_ptr' 25450", stdout_fd=7) at /tmp/upcxx-nightly-dirac-gcc/bld/upcxx_install/berkeleylab-upcxx-mk-develop/bld/GASNet-develop/gasnet_tools.c:1276
[1] #3  0x0000000000562927 in gasneti_bt_gdb (fd=7) at /tmp/upcxx-nightly-dirac-gcc/bld/upcxx_install/berkeleylab-upcxx-mk-develop/bld/GASNet-develop/gasnet_tools.c:1532
[1] #4  0x0000000000563129 in gasneti_print_backtrace (fd=2) at /tmp/upcxx-nightly-dirac-gcc/bld/upcxx_install/berkeleylab-upcxx-mk-develop/bld/GASNet-develop/gasnet_tools.c:1810
[1] #5  0x0000000000563719 in _gasneti_print_backtrace_ifenabled (fd=2) at /tmp/upcxx-nightly-dirac-gcc/bld/upcxx_install/berkeleylab-upcxx-mk-develop/bld/GASNet-develop/gasnet_tools.c:1943
[1] #6  0x0000000000561103 in gasneti_error_abort () at /tmp/upcxx-nightly-dirac-gcc/bld/upcxx_install/berkeleylab-upcxx-mk-develop/bld/GASNet-develop/gasnet_tools.c:764
[1] #7  0x00000000005612a4 in _gasneti_fatalerror (msg=0x8f14e0 "Invalid GASNet call (communication injection or poll) while executing a Request handler") at /tmp/upcxx-nightly-dirac-gcc/bld/upcxx_install/berkeleylab-upcxx-mk-develop/bld/GASNet-develop/gasnet_tools.c:800
[1] #8  0x0000000000759221 in gasneti_check_inject (for_reply=0) at /tmp/upcxx-nightly-dirac-gcc/bld/upcxx_install/berkeleylab-upcxx-mk-develop/bld/GASNet-develop/gasnet_internal.c:375
[1] #9  0x00000000007c23b9 in gasneti_Segment_QueryBound (tm=0xfffffffffd9f76ff, rank=0, owneraddr_p=0x7ffc81df6268, localaddr_p=0x0, size_p=0x7ffc81df6260) at /tmp/upcxx-nightly-dirac-gcc/bld/upcxx_install/berkeleylab-upcxx-mk-develop/bld/GASNet-develop/gasnet_mmap.c:2315
[1] #10 0x000000000045e712 in upcxx::backend::validate_global_ptr (allow_null=true, rank=0, raw_ptr=0x7fdb4e2963e0, heap_idx=0, KindSet=upcxx::memory_kind::host, T_align=8, T_name=0x861d80 <typeinfo name for std::pair<std::atomic<long>, upcxx::global_ptr<int, (upcxx::memory_kind)1> >> "St4pairISt6atomicIlEN5upcxx10global_ptrIiLNS2_11memory_kindE1EEEE", short_context=0x853989 "is_local", context=0x8538b8 "bool upcxx::global_ptr<const T, KindSet>::is_local() const [with T = std::pair<std::atomic<long int>, upcxx::global_ptr<int, upcxx::memory_kind::host> >; upcxx::memory_kind KindSet = upcxx::memory_kin"...) at /tmp/upcxx-nightly-dirac-gcc/bld/upcxx_install/berkeleylab-upcxx-mk-develop/src/backend/gasnet/runtime.cpp:1416
[1] #11 0x0000000000426cba in upcxx::global_ptr<std::pair<std::atomic<long>, upcxx::global_ptr<int, (upcxx::memory_kind)1> > const, (upcxx::memory_kind)1>::check (this=0x7ffc81df6820, allow_null=true, short_context=0x853989 "is_local", context=0x8538b8 "bool upcxx::global_ptr<const T, KindSet>::is_local() const [with T = std::pair<std::atomic<long int>, upcxx::global_ptr<int, upcxx::memory_kind::host> >; upcxx::memory_kind KindSet = upcxx::memory_kin"...) at /usr/local/pkg/upcxx-dirac/gcc-10.2.0/nightly-2021.01.16/upcxx.debug.gasnet_seq.ibv/include/upcxx/global_ptr.hpp:94
[1] #12 0x0000000000422d62 in upcxx::global_ptr<std::pair<std::atomic<long>, upcxx::global_ptr<int, (upcxx::memory_kind)1> > const, (upcxx::memory_kind)1>::is_local (this=0x7ffc81df6820) at /usr/local/pkg/upcxx-dirac/gcc-10.2.0/nightly-2021.01.16/upcxx.debug.gasnet_seq.ibv/include/upcxx/global_ptr.hpp:142
[1] #13 0x000000000043e7b2 in upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1>::upcxx_serialization::deserialize<upcxx::detail::serialization_reader> (reader=..., storage=0x7ffc81df6900) at /
home/pagoda1/regan/workspace/upcxx-utils/include/upcxx_utils/shared_global_ptr.hpp:247
[1] #14 0x000000000043f1c8 in upcxx::detail::serialization_reader::read_into<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1>&&, true, upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)
1> > (this=0x7ffc81df6aa0, raw=0x7ffc81df6900) at /usr/local/pkg/upcxx-dirac/gcc-10.2.0/nightly-2021.01.16/upcxx.debug.gasnet_seq.ibv/include/upcxx/serialization.hpp:648
[1] #15 0x000000000043ea06 in upcxx::detail::serialization_tuple<std::tuple<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1>&&>, 0, 1>::deserialize_each<std::tuple<upcxx_utils::shared_global_
ptr<int, (upcxx::memory_kind)1> >, upcxx::detail::serialization_reader> (r=..., spot=0x7ffc81df6a88) at /usr/local/pkg/upcxx-dirac/gcc-10.2.0/nightly-2021.01.16/upcxx.debug.gasnet_seq.ibv/include/upcxx
/serialization.hpp:1552
[1] #16 0x000000000043e60d in upcxx::serialization<std::tuple<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1>&&> >::deserialize<upcxx::detail::serialization_reader> (r=..., spot=0x7ffc81df6a
88) at /usr/local/pkg/upcxx-dirac/gcc-10.2.0/nightly-2021.01.16/upcxx.debug.gasnet_seq.ibv/include/upcxx/serialization.hpp:1605
[1] #17 0x000000000043e098 in upcxx::detail::serialization_reader::read_into<std::tuple<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1>&&>, true, std::tuple<upcxx_utils::shared_global_ptr<in
t, (upcxx::memory_kind)1> > > (this=0x7ffc81df6aa0, raw=0x7ffc81df6a88) at /usr/local/pkg/upcxx-dirac/gcc-10.2.0/nightly-2021.01.16/upcxx.debug.gasnet_seq.ibv/include/upcxx/serialization.hpp:648
[1] #18 0x000000000043d48e in upcxx::detail::deserialized_bound_function_storage<void upcxx::backend::send_awaken_lpc<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1>, upcxx_utils::shared_glo
bal_ptr<int, (upcxx::memory_kind)1>&&>(int, upcxx::detail::lpc_dormant<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >*, std::tuple<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)
1>&&>&&)::{lambda(std::tuple<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >&&)#1}, std::tuple<std::tuple<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1>&&> >, upcxx::detail::in
dex_sequence<0> >::deserialized_bound_function_storage(upcxx::detail::serialization_reader&) (this=0x7ffc81df6a80, r=...) at /usr/local/pkg/upcxx-dirac/gcc-10.2.0/nightly-2021.01.16/upcxx.debug.gasnet_
seq.ibv/include/upcxx/bind.hpp:157
[1] #19 0x000000000043cabd in upcxx::detail::deserialized_bound_function_base<void upcxx::backend::send_awaken_lpc<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1>, upcxx_utils::shared_global
_ptr<int, (upcxx::memory_kind)1>&&>(int, upcxx::detail::lpc_dormant<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >*, std::tuple<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1>&
&>&&)::{lambda(std::tuple<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >&&)#1}, std::tuple<std::tuple<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1>&&> >, upcxx::detail::index
_sequence<0>, true>::deserialized_bound_function_base(upcxx::detail::serialization_reader&) (this=0x7ffc81df6a80, r=...) at /usr/local/pkg/upcxx-dirac/gcc-10.2.0/nightly-2021.01.16/upcxx.debug.gasnet_s
eq.ibv/include/upcxx/bind.hpp:206
[1] #20 0x000000000043b691 in upcxx::detail::deserialized_bound_function<void upcxx::backend::send_awaken_lpc<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1>, upcxx_utils::shared_global_ptr<
int, (upcxx::memory_kind)1>&&>(int, upcxx::detail::lpc_dormant<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >*, std::tuple<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1>&&>&&)
::{lambda(std::tuple<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >&&)#1}, std::tuple<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1>&&> >::deserialized_bound_function(upcxx::d
etail::serialization_reader&) (this=0x7ffc81df6a80, r=...) at /usr/local/pkg/upcxx-dirac/gcc-10.2.0/nightly-2021.01.16/upcxx.debug.gasnet_seq.ibv/include/upcxx/bind.hpp:287
[1] #21 0x000000000043a460 in upcxx::serialization<upcxx::bound_function<void upcxx::backend::send_awaken_lpc<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1>, upcxx_utils::shared_global_ptr<
int, (upcxx::memory_kind)1>&&>(int, upcxx::detail::lpc_dormant<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >*, std::tuple<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1>&&>&&)
::{lambda(std::tuple<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >&&)#1}, std::tuple<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1>&&> > >::deserialize<upcxx::detail::seriali
zation_reader>(upcxx::detail::serialization_reader&, void*) (r=..., spot=0x7ffc81df6a80) at /usr/local/pkg/upcxx-dirac/gcc-10.2.0/nightly-2021.01.16/upcxx.debug.gasnet_seq.ibv/include/upcxx/bind.hpp:36
8
[1] #22 0x0000000000438bd2 in upcxx::detail::command<upcxx::detail::lpc_base*>::the_executor<upcxx::bound_function<void upcxx::backend::send_awaken_lpc<upcxx_utils::shared_global_ptr<int, (upcxx::memor
y_kind)1>, upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1>&&>(int, upcxx::detail::lpc_dormant<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >*, std::tuple<upcxx_utils::shared_gl
obal_ptr<int, (upcxx::memory_kind)1>&&>&&)::{lambda(std::tuple<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >&&)#1}, std::tuple<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1>&
&> >, &upcxx::backend::gasnet::rpc_as_lpc::reader_of, &(void upcxx::backend::gasnet::cleanup<false, true>(upcxx::detail::lpc_base*))>(upcxx::detail::lpc_base*) (a#0=0x7ffc81df6b10) at /usr/local/pkg/up
cxx-dirac/gcc-10.2.0/nightly-2021.01.16/upcxx.debug.gasnet_seq.ibv/include/upcxx/command.hpp:67
[1] #23 0x0000000000460a2d in (anonymous namespace)::am_eager_restricted (buf=0x7fdb56829078, buf_size=32, buf_align=8) at /tmp/upcxx-nightly-dirac-gcc/bld/upcxx_install/berkeleylab-upcxx-mk-develop/sr
c/backend/gasnet/runtime.cpp:2121
[1] #24 0x0000000000583ef9 in gasneti_AMPSHM_service_incoming_msg (vnet=0x2606dd0, isReq=1) at /tmp/upcxx-nightly-dirac-gcc/bld/upcxx_install/berkeleylab-upcxx-mk-develop/bld/GASNet-develop/gasnet_pshm
.c:1208
[1] #25 0x0000000000585a40 in gasneti_AMPSHMPoll (repliesOnly=0) at /tmp/upcxx-nightly-dirac-gcc/bld/upcxx_install/berkeleylab-upcxx-mk-develop/bld/GASNet-develop/gasnet_pshm.c:1245
[1] #26 0x0000000000520305 in gasnetc_poll_rcv_all (ep=0x260bb60, limit=16) at /tmp/upcxx-nightly-dirac-gcc/bld/upcxx_install/berkeleylab-upcxx-mk-develop/bld/GASNet-develop/ibv-conduit/gasnet_core_snd
rcv.c:1103
[1] #27 0x0000000000520338 in gasnetc_do_poll (poll_rcv=1, poll_snd=0) at /tmp/upcxx-nightly-dirac-gcc/bld/upcxx_install/berkeleylab-upcxx-mk-develop/bld/GASNet-develop/ibv-conduit/gasnet_core_sndrcv.c
:1110
[1] #28 0x00000000004e4f95 in gasnetc_AMRequestMediumM (tm=0xfffffffffd9f76ff, rank=0, handler=248 '\370', source_addr=0x7ffc81df7600, nbytes=32, lc_opt=0x1, flags=0, numargs=1) at /tmp/upcxx-nightly-d
irac-gcc/bld/upcxx_install/berkeleylab-upcxx-mk-develop/bld/GASNet-develop/ibv-conduit/gasnet_core.c:5118
[1] #29 0x000000000045f533 in upcxx::backend::gasnet::send_am_eager_restricted (recipient=0, buf=0x7ffc81df7600, buf_size=32, buf_align=8, npam_nonce=0) at /tmp/upcxx-nightly-dirac-gcc/bld/upcxx_instal
l/berkeleylab-upcxx-mk-develop/src/backend/gasnet/runtime.cpp:1497
[1] #30 0x0000000000433b02 in upcxx::backend::send_awaken_lpc<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1>, upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1>&&> (recipient=0, lpc
=0x266e690, vals=...) at /usr/local/pkg/upcxx-dirac/gcc-10.2.0/nightly-2021.01.16/upcxx.debug.gasnet_seq.ibv/include/upcxx/backend/gasnet/runtime.hpp:618
[1] #31 0x000000000043231c in upcxx::detail::rpc_recipient_after<upcxx::detail::lpc_dormant<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >*>::operator()<upcxx_utils::shared_global_ptr<int
, (upcxx::memory_kind)1> > (this=0x2670880) at /usr/local/pkg/upcxx-dirac/gcc-10.2.0/nightly-2021.01.16/upcxx.debug.gasnet_seq.ibv/include/upcxx/rpc.hpp:264
[1] #32 0x0000000000430777 in upcxx::detail::apply_tupled_as_future_dispatch<upcxx::detail::rpc_recipient_after<upcxx::detail::lpc_dormant<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >*>
&&, std::tuple<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1>&&>&&, upcxx::detail::index_sequence<0>, void>::operator() (this=0x7ffc81df78f7, fn=..., args=...) at /usr/local/pkg/upcxx-dirac
/gcc-10.2.0/nightly-2021.01.16/upcxx.debug.gasnet_seq.ibv/include/upcxx/future/apply.hpp:32
[1] #33 0x000000000042e55f in upcxx::detail::apply_futured_as_future_help<upcxx::detail::rpc_recipient_after<upcxx::detail::lpc_dormant<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >*>&&,
 upcxx::future1<upcxx::detail::future_kind_result, upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >&&, upcxx::future1<upcxx::detail::future_kind_result, upcxx_utils::shared_global_ptr<int,
(upcxx::memory_kind)1> > >::operator() (this=0x7ffc81df794f, fn=..., arg=...) at /usr/local/pkg/upcxx-dirac/gcc-10.2.0/nightly-2021.01.16/upcxx.debug.gasnet_seq.ibv/include/upcxx/future/apply.hpp:160
[1] #34 0x000000000042c33a in upcxx::detail::future_then<upcxx::future1<upcxx::detail::future_kind_result, upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >, upcxx::detail::rpc_recipient_aft
er<upcxx::detail::lpc_dormant<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >*>, true, upcxx::future1<upcxx::detail::future_kind_result>, true>::operator()<upcxx::future1<upcxx::detail::fu
ture_kind_result, upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >, upcxx::detail::rpc_recipient_after<upcxx::detail::lpc_dormant<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1>
>*> > (this=0x7ffc81df799f, arg=..., fn=...) at /usr/local/pkg/upcxx-dirac/gcc-10.2.0/nightly-2021.01.16/upcxx.debug.gasnet_seq.ibv/include/upcxx/future/then.hpp:184
[1] #35 0x00000000004194fb in upcxx::detail::future_composite_fn<upcxx::detail::bound_function_applicator<test_shared_global_ptr(int, char**)::<lambda(DSGP&, upcxx::intrank_t)>, upcxx::dist_object<upcx
x_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >&, int>, upcxx::detail::rpc_recipient_after<upcxx::detail::lpc_dormant<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >*> >::operator
()<test_shared_global_ptr(int, char**)::<lambda(DSGP&, upcxx::intrank_t)>, upcxx::dist_object<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >&, int>(void) (this=0x2670878) at /usr/local/pk
g/upcxx-dirac/gcc-10.2.0/nightly-2021.01.16/upcxx.debug.gasnet_seq.ibv/include/upcxx/future/impl_then_lazy.hpp:255
[1] #36 0x00000000004196a9 in upcxx::detail::apply_variadic_as_future_dispatch<upcxx::detail::future_composite_fn<upcxx::detail::bound_function_applicator<test_shared_global_ptr(int, char**)::<lambda(D
SGP&, upcxx::intrank_t)>, upcxx::dist_object<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >&, int>, upcxx::detail::rpc_recipient_after<upcxx::detail::lpc_dormant<upcxx_utils::shared_globa
l_ptr<int, (upcxx::memory_kind)1> >*> >&&, std::tuple<test_shared_global_ptr(int, char**)::<lambda(DSGP&, upcxx::intrank_t)>&&, upcxx::dist_object<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kin
d)1> >&, int&&>, upcxx::future1<upcxx::detail::future_kind_result> >::operator()(upcxx::detail::future_composite_fn<upcxx::detail::bound_function_applicator<test_shared_global_ptr(int, char**)::<lambda
(DSGP&, upcxx::intrank_t)>, upcxx::dist_object<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >&, int>, upcxx::detail::rpc_recipient_after<upcxx::detail::lpc_dormant<upcxx_utils::shared_glo
bal_ptr<int, (upcx[1] x::memory_kind)1> >*> > &&, struct {...} &&, upcxx::dist_object<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> > &, int &&) (this=0x7ffc81df7a5f, fn=..., arg#0=..., ar
g#1=..., arg#2=@0x266e428: 0) at /usr/local/pkg/upcxx-dirac/gcc-10.2.0/nightly-2021.01.16/upcxx.debug.gasnet_seq.ibv/include/upcxx/future/apply.hpp:108
[1] #37 0x0000000000419288 in upcxx::detail::future_composite_fn<upcxx::detail::future_composite_fn<upcxx::detail::bound_function_applicator<test_shared_global_ptr(int, char**)::<lambda(DSGP&, upcxx::i
ntrank_t)>, upcxx::dist_object<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >&, int>, upcxx::detail::rpc_recipient_after<upcxx::detail::lpc_dormant<upcxx_utils::shared_global_ptr<int, (up
cxx::memory_kind)1> >*> >, upcxx::detail::command<upcxx::detail::lpc_base*>::after_execute<upcxx::detail::deserialized_bound_function<const upcxx::detail::rpc_internal<upcxx::completions<upcxx::future_
cx<upcxx::operation_cx_event, (upcxx::progress_level)1> >, test_shared_global_ptr(int, char**)::<lambda(DSGP&, upcxx::intrank_t)>&&, {upcxx::dist_object<upcxx_utils::shared_global_ptr<int, (upcxx::memo
ry_kind)1> >&, int&&}>::<lambda(upcxx::deserialized_type_t<upcxx::bound_function<const test_shared_global_ptr(int, char**)::<lambda(DSGP&, upcxx::intrank_t)>&, upcxx::dist_object<upcxx_utils::shared_gl
obal_ptr<int, (upc[1] xx::memory_kind)1> >&, int const&> >&&)>&, const upcxx::bound_function<const test_shared_global_ptr(int, char**)::<lambda(DSGP&, upcxx::intrank_t)>&, upcxx::dist_object<upcxx_util
s::shared_global_ptr<int, (upcxx::memory_kind)1> >&, int const&>&>, true, upcxx::backend::gasnet::rpc_as_lpc::cleanup<true, false> > >::operator()<test_shared_global_ptr(int, char**)::<lambda(DSGP&, up
cxx::intrank_t)>, upcxx::dist_object<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >&, int>(void) (this=0x2670878) at /usr/local/pkg/upcxx-dirac/gcc-10.2.0/nightly-2021.01.16/upcxx.debug.g
asnet_seq.ibv/include/upcxx/future/impl_then_lazy.hpp:256
[1] #38 0x0000000000418e58 in upcxx::detail::apply_tupled_as_future_dispatch<upcxx::detail::future_composite_fn<upcxx::detail::future_composite_fn<upcxx::detail::bound_function_applicator<test_shared_g
lobal_ptr(int, char**)::<lambda(DSGP&, upcxx::intrank_t)>, upcxx::dist_object<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >&, int>, upcxx::detail::rpc_recipient_after<upcxx::detail::lpc_
dormant<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >*> >, upcxx::detail::command<upcxx::detail::lpc_base*>::after_execute<upcxx::detail::deserialized_bound_function<const upcxx::detail:
:rpc_internal<upcxx::completions<upcxx::future_cx<upcxx::operation_cx_event, (upcxx::progress_level)1> >, test_shared_global_ptr(int, char**)::<lambda(DSGP&, upcxx::intrank_t)>&&, {upcxx::dist_object<u
pcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >&, int&&}>::<lambda(upcxx::deserialized_type_t<upcxx::bound_function<const test_shared_global_ptr(int, char**)::<lambda(DSGP&, upcxx::intrank_
t)>&, upcxx::dist_[1] object<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >&, int const&> >&&)>&, const upcxx::bound_function<const test_shared_global_ptr(int, char**)::<lambda(DSGP&, upc
xx::intrank_t)>&, upcxx::dist_object<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >&, int const&>&>, true, upcxx::backend::gasnet::rpc_as_lpc::cleanup<true, false> > >&&, std::tuple<test_
shared_global_ptr(int, char**)::<lambda(DSGP&, upcxx::intrank_t)>&&, upcxx::dist_object<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >&, int&&>&&, upcxx::detail::index_sequence<0, 1, 2>,
upcxx::future1<upcxx::detail::future_kind_result> >::operator()(upcxx::detail::future_composite_fn<upcxx::detail::future_composite_fn<upcxx::detail::bound_function_applicator<test_shared_global_ptr(int
, char**)::<lambda(DSGP&, upcxx::intrank_t)>, upcxx::dist_object<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >&, int>, upcxx::detail::rpc_recipient_after<upcxx::detail::lpc_dormant<upcxx
_utils::shared_global_ptr<int, (upcx[1] x::memory_kind)1> >*> >, upcxx::detail::command<upcxx::detail::lpc_base*>::after_execute<upcxx::detail::deserialized_bound_function<const upcxx::detail::rpc_inte
rnal<upcxx::completions<upcxx::future_cx<upcxx::operation_cx_event, (upcxx::progress_level)1> >, test_shared_global_ptr(int, char**)::<lambda(DSGP&, upcxx::intrank_t)>&&, {upcxx::dist_object<upcxx_util
s::shared_global_ptr<int, (upcxx::memory_kind)1> >&, int&&}>::<lambda(upcxx::deserialized_type_t<upcxx::bound_function<const test_shared_global_ptr(int, char**)::<lambda(DSGP&, upcxx::intrank_t)>&, upc
xx::dist_object<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >&, int const&> >&&)>&, const upcxx::bound_function<const test_shared_global_ptr(int, char**)::<lambda(DSGP&, upcxx::intrank_t
)>&, upcxx::dist_object<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >&, int const&>&>, true, upcxx::backend::gasnet::rpc_as_lpc::cleanup<true, false> > > &&, std::tuple<test_shared_globa
l_ptr(int, char**)::<lambda(DSGP&, upcxx::intrank_t)>&[1] &, upcxx::dist_object<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >&, int&&> &&) (this=0x7ffc81df7aff, fn=..., args=...) at /usr
/local/pkg/upcxx-dirac/gcc-10.2.0/nightly-2021.01.16/upcxx.debug.gasnet_seq.ibv/include/upcxx/future/apply.hpp:97
[1] #39 0x0000000000418c65 in upcxx::detail::apply_futured_as_future_help<upcxx::detail::future_composite_fn<upcxx::detail::future_composite_fn<upcxx::detail::bound_function_applicator<test_shared_glob
al_ptr(int, char**)::<lambda(DSGP&, upcxx::intrank_t)>, upcxx::dist_object<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >&, int>, upcxx::detail::rpc_recipient_after<upcxx::detail::lpc_dor
mant<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >*> >, upcxx::detail::command<upcxx::detail::lpc_base*>::after_execute<upcxx::detail::deserialized_bound_function<const upcxx::detail::rp
c_internal<upcxx::completions<upcxx::future_cx<upcxx::operation_cx_event, (upcxx::progress_level)1> >, test_shared_global_ptr(int, char**)::<lambda(DSGP&, upcxx::intrank_t)>&&, {upcxx::dist_object<upcx
x_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >&, int&&}>::<lambda(upcxx::deserialized_type_t<upcxx::bound_function<const test_shared_global_ptr(int, char**)::<lambda(DSGP&, upcxx::intrank_t)>
&, upcxx::dist_obj[1] ect<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >&, int const&> >&&)>&, const upcxx::bound_function<const test_shared_global_ptr(int, char**)::<lambda(DSGP&, upcxx:
:intrank_t)>&, upcxx::dist_object<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >&, int const&>&>, true, upcxx::backend::gasnet::rpc_as_lpc::cleanup<true, false> > >&&, upcxx::future1<upcx
x::detail::future_kind_when_all<upcxx::future1<upcxx::detail::future_kind_result, test_shared_global_ptr(int, char**)::<lambda(DSGP&, upcxx::intrank_t)>&&>, upcxx::future1<upcxx::detail::future_kind_sh
ref<upcxx::detail::future_header_ops_general, false>, upcxx::dist_object<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >&>, upcxx::future1<upcxx::detail::future_kind_result, int&&> >, test
_shared_global_ptr(int, char**)::<lambda(DSGP&, upcxx::intrank_t)>&&, upcxx::dist_object<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >&, int&&>&&, upcxx::future1<upcxx::detail::future_ki
nd_when_all<upcxx::future1<upcxx::de[1] tail::future_kind_result, test_shared_global_ptr(int, char**)::<lambda(DSGP&, upcxx::intrank_t)>&&>, upcxx::future1<upcxx::detail::future_kind_shref<upcxx::detai
l::future_header_ops_general, false>, upcxx::dist_object<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >&>, upcxx::future1<upcxx::detail::future_kind_result, int&&> >, test_shared_global_p
tr(int, char**)::<lambda(DSGP&, upcxx::intrank_t)>&&, upcxx::dist_object<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >&, int&&> >::operator()(upcxx::detail::future_composite_fn<upcxx::de
tail::future_composite_fn<upcxx::detail::bound_function_applicator<test_shared_global_ptr(int, char**)::<lambda(DSGP&, upcxx::intrank_t)>, upcxx::dist_object<upcxx_utils::shared_global_ptr<int, (upcxx:
:memory_kind)1> >&, int>, upcxx::detail::rpc_recipient_after<upcxx::detail::lpc_dormant<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >*> >, upcxx::detail::command<upcxx::detail::lpc_base*
>::after_execute<upcxx::detail::deserialized_bound_fun[1] ction<const upcxx::detail::rpc_internal<upcxx::completions<upcxx::future_cx<upcxx::operation_cx_event, (upcxx::progress_level)1> >, test_shared
_global_ptr(int, char**)::<lambda(DSGP&, upcxx::intrank_t)>&&, {upcxx::dist_object<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >&, int&&}>::<lambda(upcxx::deserialized_type_t<upcxx::boun
d_function<const test_shared_global_ptr(int, char**)::<lambda(DSGP&, upcxx::intrank_t)>&, upcxx::dist_object<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >&, int const&> >&&)>&, const upc
xx::bound_function<const test_shared_global_ptr(int, char**)::<lambda(DSGP&, upcxx::intrank_t)>&, upcxx::dist_object<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >&, int const&>&>, true,
upcxx::backend::gasnet::rpc_as_lpc::cleanup<true, false> > > &&, upcxx::detail::future_dependency<upcxx::future1<upcxx::detail::future_kind_when_all<upcxx::future1<upcxx::detail::future_kind_result, te
st_shared_global_ptr(int, char**)::<lambda(DSGP&, upcxx::intrank_t)>&&>,[1]  upcxx::future1<upcxx::detail::future_kind_shref<upcxx::detail::future_header_ops_general, false>, upcxx::dist_object<upcxx_u
tils::shared_global_ptr<int, (upcxx::memory_kind)1> >&>, upcxx::future1<upcxx::detail::future_kind_result, int&&> >, test_shared_global_ptr(int, char**)::<lambda(DSGP&, upcxx::intrank_t)>&&, upcxx::dis
t_object<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >&, int&&> > &&) (this=0x7ffc81df7b57, fn=..., arg=...) at /usr/local/pkg/upcxx-dirac/gcc-10.2.0/nightly-2021.01.16/upcxx.debug.gasne
t_seq.ibv/include/upcxx/future/apply.hpp:173
[1] #40 0x0000000000418930 in upcxx::detail::future_body_then<upcxx::future1<upcxx::detail::future_kind_when_all<upcxx::future1<upcxx::detail::future_kind_result, test_shared_global_ptr(int, char**)::<
lambda(DSGP&, upcxx::intrank_t)>&&>, upcxx::future1<upcxx::detail::future_kind_shref<upcxx::detail::future_header_ops_general, false>, upcxx::dist_object<upcxx_utils::shared_global_ptr<int, (upcxx::mem
ory_kind)1> >&>, upcxx::future1<upcxx::detail::future_kind_result, int&&> >, test_shared_global_ptr(int, char**)::<lambda(DSGP&, upcxx::intrank_t)>&&, upcxx::dist_object<upcxx_utils::shared_global_ptr<
int, (upcxx::memory_kind)1> >&, int&&>, upcxx::detail::future_composite_fn<upcxx::detail::future_composite_fn<upcxx::detail::bound_function_applicator<test_shared_global_ptr(int, char**)::<lambda(DSGP&
, upcxx::intrank_t)>, upcxx::dist_object<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >&, int>, upcxx::detail::rpc_recipient_after<upcxx::detail::lpc_dormant<upcxx_utils::shared_global_pt
r<int, (upcxx::mem[1] ory_kind)1> >*> >, upcxx::detail::command<upcxx::detail::lpc_base*>::after_execute<upcxx::detail::deserialized_bound_function<const upcxx::detail::rpc_internal<upcxx::completions<
upcxx::future_cx<upcxx::operation_cx_event, (upcxx::progress_level)1> >, test_shared_global_ptr(int, char**)::<lambda(DSGP&, upcxx::intrank_t)>&&, {upcxx::dist_object<upcxx_utils::shared_global_ptr<int
, (upcxx::memory_kind)1> >&, int&&}>::<lambda(upcxx::deserialized_type_t<upcxx::bound_function<const test_shared_global_ptr(int, char**)::<lambda(DSGP&, upcxx::intrank_t)>&, upcxx::dist_object<upcxx_ut
ils::shared_global_ptr<int, (upcxx::memory_kind)1> >&, int const&> >&&)>&, const upcxx::bound_function<const test_shared_global_ptr(int, char**)::<lambda(DSGP&, upcxx::intrank_t)>&, upcxx::dist_object<
upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >&, int const&>&>, true, upcxx::backend::gasnet::rpc_as_lpc::cleanup<true, false> > > >::leave_active(upcxx::detail::future_header_dependent *
) (this=0x2670840, hdr=0x2670810) at[1]  /usr/local/pkg/upcxx-dirac/gcc-10.2.0/nightly-2021.01.16/upcxx.debug.gasnet_seq.ibv/include/upcxx/future/then.hpp:68
[1] #41 0x000000000047c4d7 in upcxx::detail::future_header_dependent::entered_active (this=0x2670810) at /tmp/upcxx-nightly-dirac-gcc/bld/upcxx_install/berkeleylab-upcxx-mk-develop/src/future/core.cpp:
57
[1] #42 0x0000000000416841 in upcxx::detail::future_body_then<upcxx::future1<upcxx::detail::future_kind_when_all<upcxx::future1<upcxx::detail::future_kind_result, test_shared_global_ptr(int, char**)::<
lambda(DSGP&, upcxx::intrank_t)>&&>, upcxx::future1<upcxx::detail::future_kind_shref<upcxx::detail::future_header_ops_general, false>, upcxx::dist_object<upcxx_utils::shared_global_ptr<int, (upcxx::mem
ory_kind)1> >&>, upcxx::future1<upcxx::detail::future_kind_result, int&&> >, test_shared_global_ptr(int, char**)::<lambda(DSGP&, upcxx::intrank_t)>&&, upcxx::dist_object<upcxx_utils::shared_global_ptr<
int, (upcxx::memory_kind)1> >&, int&&>, upcxx::detail::future_composite_fn<upcxx::detail::future_composite_fn<upcxx::detail::bound_function_applicator<test_shared_global_ptr(int, char**)::<lambda(DSGP&
, upcxx::intrank_t)>, upcxx::dist_object<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >&, int>, upcxx::detail::rpc_recipient_after<upcxx::detail::lpc_dormant<upcxx_utils::shared_global_pt
r<int, (upcxx::mem[1] ory_kind)1> >*> >, upcxx::detail::command<upcxx::detail::lpc_base*>::after_execute<upcxx::detail::deserialized_bound_function<const upcxx::detail::rpc_internal<upcxx::completions<
upcxx::future_cx<upcxx::operation_cx_event, (upcxx::progress_level)1> >, test_shared_global_ptr(int, char**)::<lambda(DSGP&, upcxx::intrank_t)>&&, {upcxx::dist_object<upcxx_utils::shared_global_ptr<int
, (upcxx::memory_kind)1> >&, int&&}>::<lambda(upcxx::deserialized_type_t<upcxx::bound_function<const test_shared_global_ptr(int, char**)::<lambda(DSGP&, upcxx::intrank_t)>&, upcxx::dist_object<upcxx_ut
ils::shared_global_ptr<int, (upcxx::memory_kind)1> >&, int const&> >&&)>&, const upcxx::bound_function<const test_shared_global_ptr(int, char**)::<lambda(DSGP&, upcxx::intrank_t)>&, upcxx::dist_object<
upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >&, int const&>&>, true, upcxx::backend::gasnet::rpc_as_lpc::cleanup<true, false> > > >::make_header<upcxx::future1<upcxx::detail::future_kind
_when_all<upcxx::future1<upcxx::deta[1] il::future_kind_result, test_shared_global_ptr(int, char**)::<lambda(DSGP&, upcxx::intrank_t)>&&>, upcxx::future1<upcxx::detail::future_kind_shref<upcxx::detail:
:future_header_ops_general, false>, upcxx::dist_object<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >&>, upcxx::future1<upcxx::detail::future_kind_result, int&&> >, test_shared_global_ptr
(int, char**)::<lambda(DSGP&, upcxx::intrank_t)>&&, upcxx::dist_object<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >&, int&&>, upcxx::detail::future_composite_fn<upcxx::detail::future_co
mposite_fn<upcxx::detail::bound_function_applicator<test_shared_global_ptr(int, char**)::<lambda(DSGP&, upcxx::intrank_t)>, upcxx::dist_object<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1>
 >&, int>, upcxx::detail::rpc_recipient_after<upcxx::detail::lpc_dormant<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >*> >, upcxx::detail::command<upcxx::detail::lpc_base*>::after_execut
e<upcxx::detail::deserialized_bound_function<const upc[1] xx::detail::rpc_internal<upcxx::completions<upcxx::future_cx<upcxx::operation_cx_event, (upcxx::progress_level)1> >, test_shared_global_ptr(int
, char**)::<lambda(DSGP&, upcxx::intrank_t)>&&, {upcxx::dist_object<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >&, int&&}>::<lambda(upcxx::deserialized_type_t<upcxx::bound_function<cons
t test_shared_global_ptr(int, char**)::<lambda(DSGP&, upcxx::intrank_t)>&, upcxx::dist_object<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >&, int const&> >&&)>&, const upcxx::bound_funct
ion<const test_shared_global_ptr(int, char**)::<lambda(DSGP&, upcxx::intrank_t)>&, upcxx::dist_object<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >&, int const&>&>, true, upcxx::backend:
:gasnet::rpc_as_lpc::cleanup<true, false> > > >(upcxx::future1<upcxx::detail::future_kind_when_all<upcxx::future1<upcxx::detail::future_kind_result, test_shared_global_ptr(int, char**)::<lambda(DSGP&,
upcxx::intrank_t)>&&>, upcxx::future1<upcxx::detail::future_kind_shref<u[1] pcxx::detail::future_header_ops_general, false>, upcxx::dist_object<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1
> >&>, upcxx::future1<upcxx::detail::future_kind_result, int&&> >, test_shared_global_ptr(int, char**)::<lambda(DSGP&, upcxx::intrank_t)>&&, upcxx::dist_object<upcxx_utils::shared_global_ptr<int, (upcx
x::memory_kind)1> >&, int&&> &&, upcxx::detail::future_composite_fn<upcxx::detail::future_composite_fn<upcxx::detail::bound_function_applicator<test_shared_global_ptr(int, char**)::<lambda(DSGP&, upcxx
::intrank_t)>, upcxx::dist_object<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >&, int>, upcxx::detail::rpc_recipient_after<upcxx::detail::lpc_dormant<upcxx_utils::shared_global_ptr<int,
(upcxx::memory_kind)1> >*> >, upcxx::detail::command<upcxx::detail::lpc_base*>::after_execute<upcxx::detail::deserialized_bound_function<const upcxx::detail::rpc_internal<upcxx::completions<upcxx::futu
re_cx<upcxx::operation_cx_event, (upcxx::progress_level)1> >, test_shared_global_ptr(int, [1] char**)::<lambda(DSGP&, upcxx::intrank_t)>&&, {upcxx::dist_object<upcxx_utils::shared_global_ptr<int, (upcx
x::memory_kind)1> >&, int&&}>::<lambda(upcxx::deserialized_type_t<upcxx::bound_function<const test_shared_global_ptr(int, char**)::<lambda(DSGP&, upcxx::intrank_t)>&, upcxx::dist_object<upcxx_utils::sh
ared_global_ptr<int, (upcxx::memory_kind)1> >&, int const&> >&&)>&, const upcxx::bound_function<const test_shared_global_ptr(int, char**)::<lambda(DSGP&, upcxx::intrank_t)>&, upcxx::dist_object<upcxx_u
tils::shared_global_ptr<int, (upcxx::memory_kind)1> >&, int const&>&>, true, upcxx::backend::gasnet::rpc_as_lpc::cleanup<true, false> > > &&) (arg=..., fn=...) at /usr/local/pkg/upcxx-dirac/gcc-10.2.0/
nightly-2021.01.16/upcxx.debug.gasnet_seq.ibv/include/upcxx/future/then.hpp:95
[1] #43 0x0000000000415c3f in upcxx::detail::future_impl_then_lazy<upcxx::future1<upcxx::detail::future_kind_when_all<upcxx::future1<upcxx::detail::future_kind_result, test_shared_global_ptr(int, char*
*)::<lambda(DSGP&, upcxx::intrank_t)>&&>, upcxx::future1<upcxx::detail::future_kind_shref<upcxx::detail::future_header_ops_general, false>, upcxx::dist_object<upcxx_utils::shared_global_ptr<int, (upcxx
::memory_kind)1> >&>, upcxx::future1<upcxx::detail::future_kind_result, int&&> >, test_shared_global_ptr(int, char**)::<lambda(DSGP&, upcxx::intrank_t)>&&, upcxx::dist_object<upcxx_utils::shared_global
_ptr<int, (upcxx::memory_kind)1> >&, int&&>, upcxx::detail::future_composite_fn<upcxx::detail::future_composite_fn<upcxx::detail::bound_function_applicator<test_shared_global_ptr(int, char**)::<lambda(
DSGP&, upcxx::intrank_t)>, upcxx::dist_object<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >&, int>, upcxx::detail::rpc_recipient_after<upcxx::detail::lpc_dormant<upcxx_utils::shared_glob
al_ptr<int, (upcxx[1] ::memory_kind)1> >*> >, upcxx::detail::command<upcxx::detail::lpc_base*>::after_execute<upcxx::detail::deserialized_bound_function<const upcxx::detail::rpc_internal<upcxx::complet
ions<upcxx::future_cx<upcxx::operation_cx_event, (upcxx::progress_level)1> >, test_shared_global_ptr(int, char**)::<lambda(DSGP&, upcxx::intrank_t)>&&, {upcxx::dist_object<upcxx_utils::shared_global_pt
r<int, (upcxx::memory_kind)1> >&, int&&}>::<lambda(upcxx::deserialized_type_t<upcxx::bound_function<const test_shared_global_ptr(int, char**)::<lambda(DSGP&, upcxx::intrank_t)>&, upcxx::dist_object<upc
xx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >&, int const&> >&&)>&, const upcxx::bound_function<const test_shared_global_ptr(int, char**)::<lambda(DSGP&, upcxx::intrank_t)>&, upcxx::dist_ob
ject<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >&, int const&>&>, true, upcxx::backend::gasnet::rpc_as_lpc::cleanup<true, false> > > >::steal_header(void) (this=0x7ffc81df7cc0) at /usr
/local/pkg/upcxx-dirac/gcc-10.2.0/ni[1] ghtly-2021.01.16/upcxx.debug.gasnet_seq.ibv/include/upcxx/future/impl_then_lazy.hpp:123
[1] #44 0x0000000000414d49 in upcxx::detail::future_then<upcxx::future1<upcxx::detail::future_kind_then_lazy<upcxx::future1<upcxx::detail::future_kind_when_all<upcxx::future1<upcxx::detail::future_kind
_result, test_shared_global_ptr(int, char**)::<lambda(DSGP&, upcxx::intrank_t)>&&>, upcxx::future1<upcxx::detail::future_kind_shref<upcxx::detail::future_header_ops_general, false>, upcxx::dist_object<
upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >&>, upcxx::future1<upcxx::detail::future_kind_result, int&&> >, test_shared_global_ptr(int, char**)::<lambda(DSGP&, upcxx::intrank_t)>&&, upc
xx::dist_object<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >&, int&&>, upcxx::detail::future_composite_fn<upcxx::detail::bound_function_applicator<test_shared_global_ptr(int, char**)::<
lambda(DSGP&, upcxx::intrank_t)>, upcxx::dist_object<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >&, int>, upcxx::detail::rpc_recipient_after<upcxx::detail::lpc_dormant<upcxx_utils::shar
ed_global_ptr<int,[1]  (upcxx::memory_kind)1> >*> > > >, upcxx::detail::command<upcxx::detail::lpc_base*>::after_execute<upcxx::detail::deserialized_bound_function<const upcxx::detail::rpc_internal<upc
xx::completions<upcxx::future_cx<upcxx::operation_cx_event, (upcxx::progress_level)1> >, test_shared_global_ptr(int, char**)::<lambda(DSGP&, upcxx::intrank_t)>&&, {upcxx::dist_object<upcxx_utils::share
d_global_ptr<int, (upcxx::memory_kind)1> >&, int&&}>::<lambda(upcxx::deserialized_type_t<upcxx::bound_function<const test_shared_global_ptr(int, char**)::<lambda(DSGP&, upcxx::intrank_t)>&, upcxx::dist
_object<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >&, int const&> >&&)>&, const upcxx::bound_function<const test_shared_global_ptr(int, char**)::<lambda(DSGP&, upcxx::intrank_t)>&, upc
xx::dist_object<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >&, int const&>&>, true, upcxx::backend::gasnet::rpc_as_lpc::cleanup<true, false> >, false, upcxx::future1<upcxx::detail::futu
re_kind_result>, false>::operator()<[1] upcxx::future1<upcxx::detail::future_kind_then_lazy<upcxx::future1<upcxx::detail::future_kind_when_all<upcxx::future1<upcxx::detail::future_kind_result, test_sha
red_global_ptr(int, char**)::<lambda(DSGP&, upcxx::intrank_t)>&&>, upcxx::future1<upcxx::detail::future_kind_shref<upcxx::detail::future_header_ops_general, false>, upcxx::dist_object<upcxx_utils::shar
ed_global_ptr<int, (upcxx::memory_kind)1> >&>, upcxx::future1<upcxx::detail::future_kind_result, int&&> >, test_shared_global_ptr(int, char**)::<lambda(DSGP&, upcxx::intrank_t)>&&, upcxx::dist_object<u
pcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >&, int&&>, upcxx::detail::future_composite_fn<upcxx::detail::bound_function_applicator<test_shared_global_ptr(int, char**)::<lambda(DSGP&, upc
xx::intrank_t)>, upcxx::dist_object<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >&, int>, upcxx::detail::rpc_recipient_after<upcxx::detail::lpc_dormant<upcxx_utils::shared_global_ptr<int
, (upcxx::memory_kind)1> >*> > > >, upcxx::detail::com[1] mand<upcxx::detail::lpc_base*>::after_execute<upcxx::detail::deserialized_bound_function<const upcxx::detail::rpc_internal<upcxx::completions<u
pcxx::future_cx<upcxx::operation_cx_event, (upcxx::progress_level)1> >, test_shared_global_ptr(int, char**)::<lambda(DSGP&, upcxx::intrank_t)>&&, {upcxx::dist_object<upcxx_utils::shared_global_ptr<int,
 (upcxx::memory_kind)1> >&, int&&}>::<lambda(upcxx::deserialized_type_t<upcxx::bound_function<const test_shared_global_ptr(int, char**)::<lambda(DSGP&, upcxx::intrank_t)>&, upcxx::dist_object<upcxx_uti
ls::shared_global_ptr<int, (upcxx::memory_kind)1> >&, int const&> >&&)>&, const upcxx::bound_function<const test_shared_global_ptr(int, char**)::<lambda(DSGP&, upcxx::intrank_t)>&, upcxx::dist_object<u
pcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >&, int const&>&>, true, upcxx::backend::gasnet::rpc_as_lpc::cleanup<true, false> > >(upcxx::future1<upcxx::detail::future_kind_then_lazy<upcxx
::future1<upcxx::detail::future_kind_when_all<upcxx::future1<upcxx::deta[1] il::future_kind_result, test_shared_global_ptr(int, char**)::<lambda(DSGP&, upcxx::intrank_t)>&&>, upcxx::future1<upcxx::deta
il::future_kind_shref<upcxx::detail::future_header_ops_general, false>, upcxx::dist_object<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >&>, upcxx::future1<upcxx::detail::future_kind_resu
lt, int&&> >, test_shared_global_ptr(int, char**)::<lambda(DSGP&, upcxx::intrank_t)>&&, upcxx::dist_object<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >&, int&&>, upcxx::detail::future_c
omposite_fn<upcxx::detail::bound_function_applicator<test_shared_global_ptr(int, char**)::<lambda(DSGP&, upcxx::intrank_t)>, upcxx::dist_object<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1
> >&, int>, upcxx::detail::rpc_recipient_after<upcxx::detail::lpc_dormant<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >*> > > > &&, upcxx::detail::command<upcxx::detail::lpc_base*>::afte
r_execute<upcxx::detail::deserialized_bound_function<const upcxx::detail::rpc_internal<upc[1] xx::completions<upcxx::future_cx<upcxx::operation_cx_event, (upcxx::progress_level)1> >, test_shared_global
_ptr(int, char**)::<lambda(DSGP&, upcxx::intrank_t)>&&, {upcxx::dist_object<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >&, int&&}>::<lambda(upcxx::deserialized_type_t<upcxx::bound_funct
ion<const test_shared_global_ptr(int, char**)::<lambda(DSGP&, upcxx::intrank_t)>&, upcxx::dist_object<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >&, int const&> >&&)>&, const upcxx::bou
nd_function<const test_shared_global_ptr(int, char**)::<lambda(DSGP&, upcxx::intrank_t)>&, upcxx::dist_object<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >&, int const&>&>, true, upcxx::
backend::gasnet::rpc_as_lpc::cleanup<true, false> > &&) (this=0x7ffc81df7d5f, arg1=..., fn1=...) at /usr/local/pkg/upcxx-dirac/gcc-10.2.0/nightly-2021.01.16/upcxx.debug.gasnet_seq.ibv/include/upcxx/fut
ure/then.hpp:166
[1] #45 0x0000000000413f1a in upcxx::future1<upcxx::detail::future_kind_then_lazy<upcxx::future1<upcxx::detail::future_kind_when_all<upcxx::future1<upcxx::detail::future_kind_result, test_shared_global
_ptr(int, char**)::<lambda(DSGP&, upcxx::intrank_t)>&&>, upcxx::future1<upcxx::detail::future_kind_shref<upcxx::detail::future_header_ops_general, false>, upcxx::dist_object<upcxx_utils::shared_global_
ptr<int, (upcxx::memory_kind)1> >&>, upcxx::future1<upcxx::detail::future_kind_result, int&&> >, test_shared_global_ptr(int, char**)::<lambda(DSGP&, upcxx::intrank_t)>&&, upcxx::dist_object<upcxx_utils
::shared_global_ptr<int, (upcxx::memory_kind)1> >&, int&&>, upcxx::detail::future_composite_fn<upcxx::detail::bound_function_applicator<test_shared_global_ptr(int, char**)::<lambda(DSGP&, upcxx::intran
k_t)>, upcxx::dist_object<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >&, int>, upcxx::detail::rpc_recipient_after<upcxx::detail::lpc_dormant<upcxx_utils::shared_global_ptr<int, (upcxx::
memory_kind)1> >*>[1]  > > >::then<upcxx::detail::command<upcxx::detail::lpc_base*>::after_execute<upcxx::detail::deserialized_bound_function<const upcxx::detail::rpc_internal<upcxx::completions<upcxx:
:future_cx<upcxx::operation_cx_event, (upcxx::progress_level)1> >, test_shared_global_ptr(int, char**)::<lambda(DSGP&, upcxx::intrank_t)>&&, {upcxx::dist_object<upcxx_utils::shared_global_ptr<int, (upc
xx::memory_kind)1> >&, int&&}>::<lambda(upcxx::deserialized_type_t<upcxx::bound_function<const test_shared_global_ptr(int, char**)::<lambda(DSGP&, upcxx::intrank_t)>&, upcxx::dist_object<upcxx_utils::s
hared_global_ptr<int, (upcxx::memory_kind)1> >&, int const&> >&&)>&, const upcxx::bound_function<const test_shared_global_ptr(int, char**)::<lambda(DSGP&, upcxx::intrank_t)>&, upcxx::dist_object<upcxx_
utils::shared_global_ptr<int, (upcxx::memory_kind)1> >&, int const&>&>, true, upcxx::backend::gasnet::rpc_as_lpc::cleanup<true, false> > >(upcxx::detail::command<upcxx::detail::lpc_base*>::after_execut
e<upcxx::detail::deserialized_bound_[1] function<const upcxx::detail::rpc_internal<upcxx::completions<upcxx::future_cx<upcxx::operation_cx_event, (upcxx::progress_level)1> >, test_shared_global_ptr(int
, char**)::<lambda(DSGP&, upcxx::intrank_t)>&&, {upcxx::dist_object<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >&, int&&}>::<lambda(upcxx::deserialized_type_t<upcxx::bound_function<cons
t test_shared_global_ptr(int, char**)::<lambda(DSGP&, upcxx::intrank_t)>&, upcxx::dist_object<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >&, int const&> >&&)>&, const upcxx::bound_funct
ion<const test_shared_global_ptr(int, char**)::<lambda(DSGP&, upcxx::intrank_t)>&, upcxx::dist_object<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >&, int const&>&>, true, upcxx::backend:
:gasnet::rpc_as_lpc::cleanup<true, false> > &&) (this=0x7ffc81df7da0, fn=...) at /usr/local/pkg/upcxx-dirac/gcc-10.2.0/nightly-2021.01.16/upcxx.debug.gasnet_seq.ibv/include/upcxx/future/future1.hpp:270
[1] #46 0x000000000041372b in upcxx::detail::command<upcxx::detail::lpc_base*>::the_executor<upcxx::bound_function<const upcxx::detail::rpc_internal<upcxx::completions<upcxx::future_cx<upcxx::operation
_cx_event, (upcxx::progress_level)1> >, test_shared_global_ptr(int, char**)::<lambda(DSGP&, upcxx::intrank_t)>&&, {upcxx::dist_object<upcxx_utils::shared_global_ptr<int, (upcxx::memory_kind)1> >&, int&
&}>::<lambda(upcxx::deserialized_type_t<upcxx::bound_function<const test_shared_global_ptr(int, char**)::<lambda(DSGP&, upcxx::intrank_t)>&, upcxx::dist_object<upcxx_utils::shared_global_ptr<int, (upcx
x::memory_kind)1> >&, int const&> >&&)>&, const upcxx::bound_function<const test_shared_global_ptr(int, char**)::<lambda(DSGP&, upcxx::intrank_t)>&, upcxx::dist_object<upcxx_utils::shared_global_ptr<in
t, (upcxx::memory_kind)1> >&, int const&>&>, upcxx::backend::gasnet::rpc_as_lpc::reader_of, upcxx::backend::gasnet::rpc_as_lpc::cleanup<true, false> >(upcxx::detail::lpc_base *) (a#0=0x26707e0) at /usr
/local/pkg/upcxx-d[1] irac/gcc-10.2.0/nightly-2021.01.16/upcxx.debug.gasnet_seq.ibv/include/upcxx/command.hpp:74
[1] #47 0x000000000046d299 in upcxx::detail::lpc_inbox<(upcxx::detail::intru_queue_safety)0>::burst(int)::{lambda(upcxx::detail::lpc_base*)#1}::operator()(upcxx::detail::lpc_base*) const (this=0x7ffc81
df7eff, m=0x26707e0) at /tmp/upcxx-nightly-dirac-gcc/bld/upcxx_install/berkeleylab-upcxx-mk-develop/bld/upcxx.assert1.optlev0.dbgsym1.gasnet_seq.ibv/include/upcxx/lpc.hpp:117
[1] #48 0x000000000046f8be in upcxx::detail::intru_queue<upcxx::detail::lpc_base, (upcxx::detail::intru_queue_safety)0, &upcxx::detail::lpc_base::intruder>::burst_something<upcxx::detail::lpc_inbox<(up
cxx::detail::intru_queue_safety)0>::burst(int)::{lambda(upcxx::detail::lpc_base*)#1}>(int, upcxx::detail::lpc_inbox<(upcxx::detail::intru_queue_safety)0>::burst(int)::{lambda(upcxx::detail::lpc_base*)#
1}&&, upcxx::detail::lpc_base*) (this=0xbb17d8 <upcxx::backend::master+56>, max_n=100, fn=..., head1=0x26707e0) at /tmp/upcxx-nightly-dirac-gcc/bld/upcxx_install/berkeleylab-upcxx-mk-develop/bld/upcxx.
assert1.optlev0.dbgsym1.gasnet_seq.ibv/include/upcxx/intru_queue.hpp:172
[1] #49 0x000000000046eb53 in upcxx::detail::intru_queue<upcxx::detail::lpc_base, (upcxx::detail::intru_queue_safety)0, &upcxx::detail::lpc_base::intruder>::burst<upcxx::detail::lpc_inbox<(upcxx::detai
l::intru_queue_safety)0>::burst(int)::{lambda(upcxx::detail::lpc_base*)#1}>(int, upcxx::detail::lpc_inbox<(upcxx::detail::intru_queue_safety)0>::burst(int)::{lambda(upcxx::detail::lpc_base*)#1}&&) (thi
s=0xbb17d8 <upcxx::backend::master+56>, max_n=100, fn=...) at /tmp/upcxx-nightly-dirac-gcc/bld/upcxx_install/berkeleylab-upcxx-mk-develop/bld/upcxx.assert1.optlev0.dbgsym1.gasnet_seq.ibv/include/upcxx/
intru_queue.hpp:157
*** Caught a fatal signal (proc 1): SIGABRT(6)
*** Details for bug reporting (proc 1): config=RELEASE=2020.10.3,SPEC=1.16,PTR=64bit,debug,SEQ,timers_native,membars_native,atomics_native,atomic32_native,atomic64_native compiler=GNU/10.2.0 sys=x86_64-unknown-linux-gnu
*** Caught a signal (proc 0): SIGTERM(15)

‌

[Dan Bonachea]

This looks like a conflict between a debug check that @Paul Hargrove recently added in the GASNet bleeding-edge development branch and the UPC++ runtime. This GASNet change should not yet appear in the stable branches of GASNet or in the mk-develop branch, but it looks like both cori installs are currently pinned to the bleeding-edge develop which is affected.

We definitely need to resolve this before the next GASNet stable advance.

[Dan Bonachea]

Here is a simple reproducer:

#include <upcxx/upcxx.hpp>
#include <iostream>
#include <assert.h>

using namespace upcxx;

struct A {
  global_ptr<int> g;
  bool local;
  A(global_ptr<int> gp) {
   this->g = gp;
   this->local = gp.is_local();
  }
  UPCXX_SERIALIZED_VALUES(g);
};

int main() {
  upcxx::init();

  if (upcxx::rank_me()) {
    auto f = upcxx::rpc(0,[]() {
      global_ptr<int> gp = upcxx::new_<int>();
      A a(gp);
      return a;
    });
    A const &a = f.wait_reference();
    assert(a.g);
  }

  upcxx::barrier();
  if (!upcxx::rank_me()) std::cout<<"SUCCESS"<<std::endl;
  upcxx::finalize();
  return 0;
}

The root cause here is the eager path for returning the upcxx::rpc() return value to the initiator is performing deserialization in AM handler context. global_ptr<T> itself is TriviallySerializable, but other user-supplied deserialization code operating on global_ptr<T> values activates the (DEBUG mode only) global_ptr<T> validity checks, which violate the AM handler context restrictions (because I did not consider this case when writing it).

The likely solution is to disable the checking (or at least the problematic part) in this context.

[Dan Bonachea]

Proposed solution in PR 309

[Dan Bonachea]

issue #440: Deploy a temporary solution for 'Invalid GASNet call' in deserialization

Add debug-mode only TLS to track the problematic AM handler context and skip the portion of global_ptr checking that issues the prohibited call. The solution notably preserves global_ptr segment bounds checking for same-process affinity, and degrades correctly in NDEBUG mode.

This will eventually be replaced with a better solution once GASNet segment query support is expanded.

Fixes issue #440.

→ <<cset b244cddc9a42>>

[Dan Bonachea]

@Rob Egan : cori's upcxx/nightly module has already been updated to "pre-date" this defect, and starting tonight the upcxx/bleeding-edge module will include the fix that I've just merged.

[Dan Bonachea]

PR 330 deploys a permanent and more robust solution to this problem, using the new gex_EP_QueryBoundSegmentNB() call in the forthcoming GASNet release.