intermittent lpc-stress-par-{debug,opt}-ofi hangs on Polaris (SS-10)
Issue #611
new
GitLab CI has revealed intermittent but reapeatable timeouts of lpc-stress-par-{debug,opt}-ofi on Polaris (SS-10). Examples:
- https://socks.lbl.gov/Pagoda/upcxx/-/jobs/21094
- https://socks.lbl.gov/Pagoda/upcxx/-/jobs/21093
- https://socks.lbl.gov/Pagoda/upcxx/-/jobs/21091
- https://socks.lbl.gov/Pagoda/upcxx/-/jobs/21088
- https://socks.lbl.gov/Pagoda/upcxx/-/jobs/21082
To date the problem has NOT been seen for smp-conduit or SEQ threadmode runs, despite many repeated CI runs (including all those above) that include all combinations of {seq,par}-{debug,opt}-{smp,ofi}.
Below are the thread stacks for the process that timed out in this job using the default test parameters (iters = 10000 threads = 10 WORDS = 64), the other processes are stuck in the exit barrier (which is the ONLY inter-process communication in this test).
[1] Id Target Id Frame
[1] * 1 Thread 0x15469756f300 (LWP 33112) "test-lpc-stress" 0x00001546991e862f in wait4 () from /lib64/libc.so.6
[1] 2 Thread 0x15469725e700 (LWP 33113) "test-lpc-stress" 0x00001546992120a9 in poll () from /lib64/libc.so.6
[1] 3 Thread 0x15467c2ed700 (LWP 33116) "test-lpc-stress" 0x000015469bb109c5 in pthread_spin_lock () from /lib64/libpthread.so.0
[1] 4 Thread 0x15467c0ec700 (LWP 33118) "test-lpc-stress" 0x000015469bb109c5 in pthread_spin_lock () from /lib64/libpthread.so.0
[1] 5 Thread 0x15467beeb700 (LWP 33121) "test-lpc-stress" 0x000015469bb109c5 in pthread_spin_lock () from /lib64/libpthread.so.0
[1] 6 Thread 0x15467bae9700 (LWP 33125) "test-lpc-stress" 0x000015469bb109c5 in pthread_spin_lock () from /lib64/libpthread.so.0
[1] 7 Thread 0x15467b8e8700 (LWP 33126) "test-lpc-stress" 0x000015469bb109c5 in pthread_spin_lock () from /lib64/libpthread.so.0
[1] 8 Thread 0x15467b6e7700 (LWP 33128) "test-lpc-stress" 0x00007fff970a2b62 in clock_gettime ()
[1] 9 Thread 0x15467b2e5700 (LWP 33133) "test-lpc-stress" 0x000015469bb109c5 in pthread_spin_lock () from /lib64/libpthread.so.0
[1] 10 Thread 0x15467b0e4700 (LWP 33134) "test-lpc-stress" 0x000015469bb109c5 in pthread_spin_lock () from /lib64/libpthread.so.0
[1]
[1] Thread 10 (Thread 0x15467b0e4700 (LWP 33134) "test-lpc-stress"):
[1] #0 0x000015469bb109c5 in pthread_spin_lock () from /lib64/libpthread.so.0
[1] #1 0x000015469b87b35e in ofi_cq_progress (cq=0xf35020) at prov/util/src/util_cq.c:595
[1] #2 0x000015469b87a68e in ofi_cq_readfrom (cq_fid=0xf35020, buf=<optimized out>, count=64, src_addr=0x0) at prov/util/src/util_cq.c:247
[1] #3 0x000000000052c396 in fi_cq_read (cq=0xf35020, buf=0x15467b0e2200, count=64) at /opt/cray/libfabric/1.11.0.4.125/include/rdma/fi_eq.h:394
[1] #4 0x000000000053a962 in gasnetc_ofi_tx_poll_one (cqfd=0xf35020) at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/berkeleylab-gasnet-512c17e80297/ofi-conduit/gasnet_ofi.c:2028
[1] #5 0x000000000053ad27 in gasnetc_ofi_tx_poll () at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/berkeleylab-gasnet-512c17e80297/ofi-conduit/gasnet_ofi.c:2071
[1] #6 0x000000000053b4ff in gasnetc_ofi_poll () at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/berkeleylab-gasnet-512c17e80297/ofi-conduit/gasnet_ofi.c:2241
[1] #7 0x00000000004e0c82 in gasnetc_AMPoll () at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/berkeleylab-gasnet-512c17e80297/ofi-conduit/gasnet_core.c:650
[1] #8 0x000000000040fa3c in _gasneti_AMPoll () at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/berkeleylab-gasnet-512c17e80297/gasnet_help.h:1298
[1] #9 0x000000000040fc26 in _gasnet_AMPoll () at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/berkeleylab-gasnet-512c17e80297/gasnet_help.h:1511
[1] #10 0x000000000042481e in do_progress<(upcxx::progress_level)1> () at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/src/backend/gasnet/runtime.cpp:2192
[1] #11 0x00000000004227af in upcxx::detail::progress_user () at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/src/backend/gasnet/runtime.cpp:2252
[1] #12 0x0000000000408977 in upcxx::progress (level=upcxx::progress_level::user) at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/upcxx.assert1.optlev0.dbgsym1.gasnet_par.ofi/include/upcxx/backend.hpp:34
[1] #13 0x0000000000408635 in upcxx::detail::future_wait_upcxx_progress_user::operator() (this=0x15467b0e2faf) at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/upcxx.assert1.optlev0.dbgsym1.gasnet_par.ofi/include/upcxx/future/future1.hpp:53
[1] #14 0x000000000040952c in upcxx::detail::future1<upcxx::detail::future_kind_shref<upcxx::detail::future_header_ops_general, false>, msg>::wait_reference<-1>() const & (this=0x15467b0e2fe8) at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/upcxx.assert1.optlev0.dbgsym1.gasnet_par.ofi/include/upcxx/future/future1.hpp:416
[1] #15 0x0000000000407892 in operator() (__closure=0x1240708) at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/test/lpc-stress.cpp:109
[1] #16 0x0000000000407f3a in std::__invoke_impl<void, main(int, char**)::<lambda()> >(std::__invoke_other, struct {...} &&) (__f=...) at /opt/cray/pe/gcc/11.2.0/snos/include/g++/bits/invoke.h:61
[1] #17 0x0000000000407efd in std::__invoke<main(int, char**)::<lambda()> >(struct {...} &&) (__fn=...) at /opt/cray/pe/gcc/11.2.0/snos/include/g++/bits/invoke.h:96
[1] #18 0x0000000000407eaa in std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > >::_M_invoke<0>(std::_Index_tuple<0>) (this=0x1240708) at /opt/cray/pe/gcc/11.2.0/snos/include/g++/bits/std_thread.h:253
[1] #19 0x0000000000407e7e in std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > >::operator()(void) (this=0x1240708) at /opt/cray/pe/gcc/11.2.0/snos/include/g++/bits/std_thread.h:260
[1] #20 0x0000000000407e62 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > > >::_M_run(void) (this=0x1240700) at /opt/cray/pe/gcc/11.2.0/snos/include/g++/bits/std_thread.h:211
[1] #21 0x000015469973cf44 in std::execute_native_thread_routine (__p=0x1240700) at ../../../../../cpe-gcc-11.2.0-202108140355.9bf1fd589a5c1/libstdc++-v3/src/c++11/thread.cc:82
[1] #22 0x000015469bb086ea in start_thread () from /lib64/libpthread.so.0
[1] #23 0x000015469921e94f in clone () from /lib64/libc.so.6
[1]
[1] Thread 9 (Thread 0x15467b2e5700 (LWP 33133) "test-lpc-stress"):
[1] #0 0x000015469bb109c5 in pthread_spin_lock () from /lib64/libpthread.so.0
[1] #1 0x000015469b8ca009 in ofi_ep_lock_acquire (ep=0xf33be0) at ./include/ofi_util.h:323
[1] #2 rxm_ep_progress (util_ep=0xf33be0) at prov/rxm/src/rxm_cq.c:1887
[1] #3 0x000015469b87b37d in ofi_cq_progress (cq=0xf35020) at prov/util/src/util_cq.c:599
[1] #4 0x000015469b87a68e in ofi_cq_readfrom (cq_fid=0xf35020, buf=<optimized out>, count=64, src_addr=0x0) at prov/util/src/util_cq.c:247
[1] #5 0x000000000052c396 in fi_cq_read (cq=0xf35020, buf=0x15467b2e3200, count=64) at /opt/cray/libfabric/1.11.0.4.125/include/rdma/fi_eq.h:394
[1] #6 0x000000000053a962 in gasnetc_ofi_tx_poll_one (cqfd=0xf35020) at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/berkeleylab-gasnet-512c17e80297/ofi-conduit/gasnet_ofi.c:2028
[1] #7 0x000000000053ad27 in gasnetc_ofi_tx_poll () at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/berkeleylab-gasnet-512c17e80297/ofi-conduit/gasnet_ofi.c:2071
[1] #8 0x000000000053b4ff in gasnetc_ofi_poll () at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/berkeleylab-gasnet-512c17e80297/ofi-conduit/gasnet_ofi.c:2241
[1] #9 0x00000000004e0c82 in gasnetc_AMPoll () at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/berkeleylab-gasnet-512c17e80297/ofi-conduit/gasnet_core.c:650
[1] #10 0x000000000040fa3c in _gasneti_AMPoll () at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/berkeleylab-gasnet-512c17e80297/gasnet_help.h:1298
[1] #11 0x000000000040fc26 in _gasnet_AMPoll () at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/berkeleylab-gasnet-512c17e80297/gasnet_help.h:1511
[1] #12 0x000000000042481e in do_progress<(upcxx::progress_level)1> () at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/src/backend/gasnet/runtime.cpp:2192
[1] #13 0x00000000004227af in upcxx::detail::progress_user () at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/src/backend/gasnet/runtime.cpp:2252
[1] #14 0x0000000000408977 in upcxx::progress (level=upcxx::progress_level::user) at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/upcxx.assert1.optlev0.dbgsym1.gasnet_par.ofi/include/upcxx/backend.hpp:34
[1] #15 0x0000000000408635 in upcxx::detail::future_wait_upcxx_progress_user::operator() (this=0x15467b2e3faf) at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/upcxx.assert1.optlev0.dbgsym1.gasnet_par.ofi/include/upcxx/future/future1.hpp:53
[1] #16 0x000000000040952c in upcxx::detail::future1<upcxx::detail::future_kind_shref<upcxx::detail::future_header_ops_general, false>, msg>::wait_reference<-1>() const & (this=0x15467b2e3fe8) at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/upcxx.assert1.optlev0.dbgsym1.gasnet_par.ofi/include/upcxx/future/future1.hpp:416
[1] #17 0x0000000000407892 in operator() (__closure=0x11dd438) at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/test/lpc-stress.cpp:109
[1] #18 0x0000000000407f3a in std::__invoke_impl<void, main(int, char**)::<lambda()> >(std::__invoke_other, struct {...} &&) (__f=...) at /opt/cray/pe/gcc/11.2.0/snos/include/g++/bits/invoke.h:61
[1] #19 0x0000000000407efd in std::__invoke<main(int, char**)::<lambda()> >(struct {...} &&) (__fn=...) at /opt/cray/pe/gcc/11.2.0/snos/include/g++/bits/invoke.h:96
[1] #20 0x0000000000407eaa in std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > >::_M_invoke<0>(std::_Index_tuple<0>) (this=0x11dd438) at /opt/cray/pe/gcc/11.2.0/snos/include/g++/bits/std_thread.h:253
[1] #21 0x0000000000407e7e in std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > >::operator()(void) (this=0x11dd438) at /opt/cray/pe/gcc/11.2.0/snos/include/g++/bits/std_thread.h:260
[1] #22 0x0000000000407e62 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > > >::_M_run(void) (this=0x11dd430) at /opt/cray/pe/gcc/11.2.0/snos/include/g++/bits/std_thread.h:211
[1] #23 0x000015469973cf44 in std::execute_native_thread_routine (__p=0x11dd430) at ../../../../../cpe-gcc-11.2.0-202108140355.9bf1fd589a5c1/libstdc++-v3/src/c++11/thread.cc:82
[1] #24 0x000015469bb086ea in start_thread () from /lib64/libpthread.so.0
[1] #25 0x000015469921e94f in clone () from /lib64/libc.so.6
[1]
[1] Thread 8 (Thread 0x15467b6e7700 (LWP 33128) "test-lpc-stress"):
[1] #0 0x00007fff970a2b62 in clock_gettime ()
[1] #1 0x00001546991e2a60 in clock_gettime@GLIBC_2.2.5 () from /lib64/libc.so.6
[1] #2 0x000015469b8688c1 in ofi_gettime_ns () at src/common.c:242
[1] #3 0x000015469b8688e9 in ofi_gettime_us () at src/common.c:248
[1] #4 0x000015469b8c9ecd in rxm_ep_do_progress (util_ep=util_ep@entry=0xf33be0) at prov/rxm/src/rxm_cq.c:1867
[1] #5 0x000015469b8ca011 in rxm_ep_progress (util_ep=0xf33be0) at prov/rxm/src/rxm_cq.c:1888
[1] #6 0x000015469b87b37d in ofi_cq_progress (cq=0xf34b50) at prov/util/src/util_cq.c:599
[1] #7 0x000015469b87a68e in ofi_cq_readfrom (cq_fid=0xf34b50, buf=<optimized out>, count=1, src_addr=0x0) at prov/util/src/util_cq.c:247
[1] #8 0x000000000052c396 in fi_cq_read (cq=0xf34b50, buf=0x15467b6e5b40, count=1) at /opt/cray/libfabric/1.11.0.4.125/include/rdma/fi_eq.h:394
[1] #9 0x000000000053ae82 in gasnetc_ofi_am_recv_poll (is_request=1) at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/berkeleylab-gasnet-512c17e80297/ofi-conduit/gasnet_ofi.c:2117
[1] #10 0x000000000053b509 in gasnetc_ofi_poll () at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/berkeleylab-gasnet-512c17e80297/ofi-conduit/gasnet_ofi.c:2242
[1] #11 0x00000000004e0c82 in gasnetc_AMPoll () at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/berkeleylab-gasnet-512c17e80297/ofi-conduit/gasnet_core.c:650
[1] #12 0x000000000040fa3c in _gasneti_AMPoll () at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/berkeleylab-gasnet-512c17e80297/gasnet_help.h:1298
[1] #13 0x000000000040fc26 in _gasnet_AMPoll () at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/berkeleylab-gasnet-512c17e80297/gasnet_help.h:1511
[1] #14 0x000000000042481e in do_progress<(upcxx::progress_level)1> () at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/src/backend/gasnet/runtime.cpp:2192
[1] #15 0x00000000004227af in upcxx::detail::progress_user () at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/src/backend/gasnet/runtime.cpp:2252
[1] #16 0x0000000000408977 in upcxx::progress (level=upcxx::progress_level::user) at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/upcxx.assert1.optlev0.dbgsym1.gasnet_par.ofi/include/upcxx/backend.hpp:34
[1] #17 0x0000000000408635 in upcxx::detail::future_wait_upcxx_progress_user::operator() (this=0x15467b6e5faf) at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/upcxx.assert1.optlev0.dbgsym1.gasnet_par.ofi/include/upcxx/future/future1.hpp:53
[1] #18 0x000000000040952c in upcxx::detail::future1<upcxx::detail::future_kind_shref<upcxx::detail::future_header_ops_general, false>, msg>::wait_reference<-1>() const & (this=0x15467b6e5fe8) at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/upcxx.assert1.optlev0.dbgsym1.gasnet_par.ofi/include/upcxx/future/future1.hpp:416
[1] #19 0x0000000000407892 in operator() (__closure=0x1240638) at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/test/lpc-stress.cpp:109
[1] #20 0x0000000000407f3a in std::__invoke_impl<void, main(int, char**)::<lambda()> >(std::__invoke_other, struct {...} &&) (__f=...) at /opt/cray/pe/gcc/11.2.0/snos/include/g++/bits/invoke.h:61
[1] #21 0x0000000000407efd in std::__invoke<main(int, char**)::<lambda()> >(struct {...} &&) (__fn=...) at /opt/cray/pe/gcc/11.2.0/snos/include/g++/bits/invoke.h:96
[1] #22 0x0000000000407eaa in std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > >::_M_invoke<0>(std::_Index_tuple<0>) (this=0x1240638) at /opt/cray/pe/gcc/11.2.0/snos/include/g++/bits/std_thread.h:253
[1] #23 0x0000000000407e7e in std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > >::operator()(void) (this=0x1240638) at /opt/cray/pe/gcc/11.2.0/snos/include/g++/bits/std_thread.h:260
[1] #24 0x0000000000407e62 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > > >::_M_run(void) (this=0x1240630) at /opt/cray/pe/gcc/11.2.0/snos/include/g++/bits/std_thread.h:211
[1] #25 0x000015469973cf44 in std::execute_native_thread_routine (__p=0x1240630) at ../../../../../cpe-gcc-11.2.0-202108140355.9bf1fd589a5c1/libstdc++-v3/src/c++11/thread.cc:82
[1] #26 0x000015469bb086ea in start_thread () from /lib64/libpthread.so.0
[1] #27 0x000015469921e94f in clone () from /lib64/libc.so.6
[1]
[1] Thread 7 (Thread 0x15467b8e8700 (LWP 33126) "test-lpc-stress"):
[1] #0 0x000015469bb109c5 in pthread_spin_lock () from /lib64/libpthread.so.0
[1] #1 0x000015469b87b35e in ofi_cq_progress (cq=0xf35020) at prov/util/src/util_cq.c:595
[1] #2 0x000015469b87a68e in ofi_cq_readfrom (cq_fid=0xf35020, buf=<optimized out>, count=64, src_addr=0x0) at prov/util/src/util_cq.c:247
[1] #3 0x000000000052c396 in fi_cq_read (cq=0xf35020, buf=0x15467b8e6200, count=64) at /opt/cray/libfabric/1.11.0.4.125/include/rdma/fi_eq.h:394
[1] #4 0x000000000053a962 in gasnetc_ofi_tx_poll_one (cqfd=0xf35020) at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/berkeleylab-gasnet-512c17e80297/ofi-conduit/gasnet_ofi.c:2028
[1] #5 0x000000000053ad27 in gasnetc_ofi_tx_poll () at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/berkeleylab-gasnet-512c17e80297/ofi-conduit/gasnet_ofi.c:2071
[1] #6 0x000000000053b4ff in gasnetc_ofi_poll () at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/berkeleylab-gasnet-512c17e80297/ofi-conduit/gasnet_ofi.c:2241
[1] #7 0x00000000004e0c82 in gasnetc_AMPoll () at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/berkeleylab-gasnet-512c17e80297/ofi-conduit/gasnet_core.c:650
[1] #8 0x000000000040fa3c in _gasneti_AMPoll () at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/berkeleylab-gasnet-512c17e80297/gasnet_help.h:1298
[1] #9 0x000000000040fc26 in _gasnet_AMPoll () at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/berkeleylab-gasnet-512c17e80297/gasnet_help.h:1511
[1] #10 0x000000000042481e in do_progress<(upcxx::progress_level)1> () at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/src/backend/gasnet/runtime.cpp:2192
[1] #11 0x00000000004227af in upcxx::detail::progress_user () at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/src/backend/gasnet/runtime.cpp:2252
[1] #12 0x0000000000408977 in upcxx::progress (level=upcxx::progress_level::user) at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/upcxx.assert1.optlev0.dbgsym1.gasnet_par.ofi/include/upcxx/backend.hpp:34
[1] #13 0x0000000000408635 in upcxx::detail::future_wait_upcxx_progress_user::operator() (this=0x15467b8e6faf) at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/upcxx.assert1.optlev0.dbgsym1.gasnet_par.ofi/include/upcxx/future/future1.hpp:53
[1] #14 0x000000000040952c in upcxx::detail::future1<upcxx::detail::future_kind_shref<upcxx::detail::future_header_ops_general, false>, msg>::wait_reference<-1>() const & (this=0x15467b8e6fe8) at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/upcxx.assert1.optlev0.dbgsym1.gasnet_par.ofi/include/upcxx/future/future1.hpp:416
[1] #15 0x0000000000407892 in operator() (__closure=0x1240488) at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/test/lpc-stress.cpp:109
[1] #16 0x0000000000407f3a in std::__invoke_impl<void, main(int, char**)::<lambda()> >(std::__invoke_other, struct {...} &&) (__f=...) at /opt/cray/pe/gcc/11.2.0/snos/include/g++/bits/invoke.h:61
[1] #17 0x0000000000407efd in std::__invoke<main(int, char**)::<lambda()> >(struct {...} &&) (__fn=...) at /opt/cray/pe/gcc/11.2.0/snos/include/g++/bits/invoke.h:96
[1] #18 0x0000000000407eaa in std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > >::_M_invoke<0>(std::_Index_tuple<0>) (this=0x1240488) at /opt/cray/pe/gcc/11.2.0/snos/include/g++/bits/std_thread.h:253
[1] #19 0x0000000000407e7e in std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > >::operator()(void) (this=0x1240488) at /opt/cray/pe/gcc/11.2.0/snos/include/g++/bits/std_thread.h:260
[1] #20 0x0000000000407e62 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > > >::_M_run(void) (this=0x1240480) at /opt/cray/pe/gcc/11.2.0/snos/include/g++/bits/std_thread.h:211
[1] #21 0x000015469973cf44 in std::execute_native_thread_routine (__p=0x1240480) at ../../../../../cpe-gcc-11.2.0-202108140355.9bf1fd589a5c1/libstdc++-v3/src/c++11/thread.cc:82
[1] #22 0x000015469bb086ea in start_thread () from /lib64/libpthread.so.0
[1] #23 0x000015469921e94f in clone () from /lib64/libc.so.6
[1]
[1] Thread 6 (Thread 0x15467bae9700 (LWP 33125) "test-lpc-stress"):
[1] #0 0x000015469bb109c5 in pthread_spin_lock () from /lib64/libpthread.so.0
[1] #1 0x000015469b87b35e in ofi_cq_progress (cq=0xf35020) at prov/util/src/util_cq.c:595
[1] #2 0x000015469b87a68e in ofi_cq_readfrom (cq_fid=0xf35020, buf=<optimized out>, count=64, src_addr=0x0) at prov/util/src/util_cq.c:247
[1] #3 0x000000000052c396 in fi_cq_read (cq=0xf35020, buf=0x15467bae7200, count=64) at /opt/cray/libfabric/1.11.0.4.125/include/rdma/fi_eq.h:394
[1] #4 0x000000000053a962 in gasnetc_ofi_tx_poll_one (cqfd=0xf35020) at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/berkeleylab-gasnet-512c17e80297/ofi-conduit/gasnet_ofi.c:2028
[1] #5 0x000000000053ad27 in gasnetc_ofi_tx_poll () at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/berkeleylab-gasnet-512c17e80297/ofi-conduit/gasnet_ofi.c:2071
[1] #6 0x000000000053b4ff in gasnetc_ofi_poll () at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/berkeleylab-gasnet-512c17e80297/ofi-conduit/gasnet_ofi.c:2241
[1] #7 0x00000000004e0c82 in gasnetc_AMPoll () at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/berkeleylab-gasnet-512c17e80297/ofi-conduit/gasnet_core.c:650
[1] #8 0x000000000040fa3c in _gasneti_AMPoll () at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/berkeleylab-gasnet-512c17e80297/gasnet_help.h:1298
[1] #9 0x000000000040fc26 in _gasnet_AMPoll () at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/berkeleylab-gasnet-512c17e80297/gasnet_help.h:1511
[1] #10 0x000000000042481e in do_progress<(upcxx::progress_level)1> () at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/src/backend/gasnet/runtime.cpp:2192
[1] #11 0x00000000004227af in upcxx::detail::progress_user () at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/src/backend/gasnet/runtime.cpp:2252
[1] #12 0x0000000000408977 in upcxx::progress (level=upcxx::progress_level::user) at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/upcxx.assert1.optlev0.dbgsym1.gasnet_par.ofi/include/upcxx/backend.hpp:34
[1] #13 0x0000000000408635 in upcxx::detail::future_wait_upcxx_progress_user::operator() (this=0x15467bae7faf) at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/upcxx.assert1.optlev0.dbgsym1.gasnet_par.ofi/include/upcxx/future/future1.hpp:53
[1] #14 0x000000000040952c in upcxx::detail::future1<upcxx::detail::future_kind_shref<upcxx::detail::future_header_ops_general, false>, msg>::wait_reference<-1>() const & (this=0x15467bae7fe8) at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/upcxx.assert1.optlev0.dbgsym1.gasnet_par.ofi/include/upcxx/future/future1.hpp:416
[1] #15 0x0000000000407892 in operator() (__closure=0x12402d8) at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/test/lpc-stress.cpp:109
[1] #16 0x0000000000407f3a in std::__invoke_impl<void, main(int, char**)::<lambda()> >(std::__invoke_other, struct {...} &&) (__f=...) at /opt/cray/pe/gcc/11.2.0/snos/include/g++/bits/invoke.h:61
[1] #17 0x0000000000407efd in std::__invoke<main(int, char**)::<lambda()> >(struct {...} &&) (__fn=...) at /opt/cray/pe/gcc/11.2.0/snos/include/g++/bits/invoke.h:96
[1] #18 0x0000000000407eaa in std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > >::_M_invoke<0>(std::_Index_tuple<0>) (this=0x12402d8) at /opt/cray/pe/gcc/11.2.0/snos/include/g++/bits/std_thread.h:253
[1] #19 0x0000000000407e7e in std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > >::operator()(void) (this=0x12402d8) at /opt/cray/pe/gcc/11.2.0/snos/include/g++/bits/std_thread.h:260
[1] #20 0x0000000000407e62 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > > >::_M_run(void) (this=0x12402d0) at /opt/cray/pe/gcc/11.2.0/snos/include/g++/bits/std_thread.h:211
[1] #21 0x000015469973cf44 in std::execute_native_thread_routine (__p=0x12402d0) at ../../../../../cpe-gcc-11.2.0-202108140355.9bf1fd589a5c1/libstdc++-v3/src/c++11/thread.cc:82
[1] #22 0x000015469bb086ea in start_thread () from /lib64/libpthread.so.0
[1] #23 0x000015469921e94f in clone () from /lib64/libc.so.6
[1]
[1] Thread 5 (Thread 0x15467beeb700 (LWP 33121) "test-lpc-stress"):
[1] #0 0x000015469bb109c5 in pthread_spin_lock () from /lib64/libpthread.so.0
[1] #1 0x000015469b87b35e in ofi_cq_progress (cq=0xf35020) at prov/util/src/util_cq.c:595
[1] #2 0x000015469b87a68e in ofi_cq_readfrom (cq_fid=0xf35020, buf=<optimized out>, count=64, src_addr=0x0) at prov/util/src/util_cq.c:247
[1] #3 0x000000000052c396 in fi_cq_read (cq=0xf35020, buf=0x15467bee9200, count=64) at /opt/cray/libfabric/1.11.0.4.125/include/rdma/fi_eq.h:394
[1] #4 0x000000000053a962 in gasnetc_ofi_tx_poll_one (cqfd=0xf35020) at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/berkeleylab-gasnet-512c17e80297/ofi-conduit/gasnet_ofi.c:2028
[1] #5 0x000000000053ad27 in gasnetc_ofi_tx_poll () at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/berkeleylab-gasnet-512c17e80297/ofi-conduit/gasnet_ofi.c:2071
[1] #6 0x000000000053b4ff in gasnetc_ofi_poll () at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/berkeleylab-gasnet-512c17e80297/ofi-conduit/gasnet_ofi.c:2241
[1] #7 0x00000000004e0c82 in gasnetc_AMPoll () at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/berkeleylab-gasnet-512c17e80297/ofi-conduit/gasnet_core.c:650
[1] #8 0x000000000040fa3c in _gasneti_AMPoll () at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/berkeleylab-gasnet-512c17e80297/gasnet_help.h:1298
[1] #9 0x000000000040fc26 in _gasnet_AMPoll () at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/berkeleylab-gasnet-512c17e80297/gasnet_help.h:1511
[1] #10 0x000000000042481e in do_progress<(upcxx::progress_level)1> () at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/src/backend/gasnet/runtime.cpp:2192
[1] #11 0x00000000004227af in upcxx::detail::progress_user () at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/src/backend/gasnet/runtime.cpp:2252
[1] #12 0x0000000000408977 in upcxx::progress (level=upcxx::progress_level::user) at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/upcxx.assert1.optlev0.dbgsym1.gasnet_par.ofi/include/upcxx/backend.hpp:34
[1] #13 0x0000000000408635 in upcxx::detail::future_wait_upcxx_progress_user::operator() (this=0x15467bee9faf) at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/upcxx.assert1.optlev0.dbgsym1.gasnet_par.ofi/include/upcxx/future/future1.hpp:53
[1] #14 0x000000000040952c in upcxx::detail::future1<upcxx::detail::future_kind_shref<upcxx::detail::future_header_ops_general, false>, msg>::wait_reference<-1>() const & (this=0x15467bee9fe8) at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/upcxx.assert1.optlev0.dbgsym1.gasnet_par.ofi/include/upcxx/future/future1.hpp:416
[1] #15 0x0000000000407892 in operator() (__closure=0x123ff98) at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/test/lpc-stress.cpp:109
[1] #16 0x0000000000407f3a in std::__invoke_impl<void, main(int, char**)::<lambda()> >(std::__invoke_other, struct {...} &&) (__f=...) at /opt/cray/pe/gcc/11.2.0/snos/include/g++/bits/invoke.h:61
[1] #17 0x0000000000407efd in std::__invoke<main(int, char**)::<lambda()> >(struct {...} &&) (__fn=...) at /opt/cray/pe/gcc/11.2.0/snos/include/g++/bits/invoke.h:96
[1] #18 0x0000000000407eaa in std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > >::_M_invoke<0>(std::_Index_tuple<0>) (this=0x123ff98) at /opt/cray/pe/gcc/11.2.0/snos/include/g++/bits/std_thread.h:253
[1] #19 0x0000000000407e7e in std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > >::operator()(void) (this=0x123ff98) at /opt/cray/pe/gcc/11.2.0/snos/include/g++/bits/std_thread.h:260
[1] #20 0x0000000000407e62 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > > >::_M_run(void) (this=0x123ff90) at /opt/cray/pe/gcc/11.2.0/snos/include/g++/bits/std_thread.h:211
[1] #21 0x000015469973cf44 in std::execute_native_thread_routine (__p=0x123ff90) at ../../../../../cpe-gcc-11.2.0-202108140355.9bf1fd589a5c1/libstdc++-v3/src/c++11/thread.cc:82
[1] #22 0x000015469bb086ea in start_thread () from /lib64/libpthread.so.0
[1] #23 0x000015469921e94f in clone () from /lib64/libc.so.6
[1]
[1] Thread 4 (Thread 0x15467c0ec700 (LWP 33118) "test-lpc-stress"):
[1] #0 0x000015469bb109c5 in pthread_spin_lock () from /lib64/libpthread.so.0
[1] #1 0x000015469b87b35e in ofi_cq_progress (cq=0xf35020) at prov/util/src/util_cq.c:595
[1] #2 0x000015469b87a68e in ofi_cq_readfrom (cq_fid=0xf35020, buf=<optimized out>, count=64, src_addr=0x0) at prov/util/src/util_cq.c:247
[1] #3 0x000000000052c396 in fi_cq_read (cq=0xf35020, buf=0x15467c0ea200, count=64) at /opt/cray/libfabric/1.11.0.4.125/include/rdma/fi_eq.h:394
[1] #4 0x000000000053a962 in gasnetc_ofi_tx_poll_one (cqfd=0xf35020) at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/berkeleylab-gasnet-512c17e80297/ofi-conduit/gasnet_ofi.c:2028
[1] #5 0x000000000053ad27 in gasnetc_ofi_tx_poll () at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/berkeleylab-gasnet-512c17e80297/ofi-conduit/gasnet_ofi.c:2071
[1] #6 0x000000000053b4ff in gasnetc_ofi_poll () at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/berkeleylab-gasnet-512c17e80297/ofi-conduit/gasnet_ofi.c:2241
[1] #7 0x00000000004e0c82 in gasnetc_AMPoll () at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/berkeleylab-gasnet-512c17e80297/ofi-conduit/gasnet_core.c:650
[1] #8 0x000000000040fa3c in _gasneti_AMPoll () at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/berkeleylab-gasnet-512c17e80297/gasnet_help.h:1298
[1] #9 0x000000000040fc26 in _gasnet_AMPoll () at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/berkeleylab-gasnet-512c17e80297/gasnet_help.h:1511
[1] #10 0x000000000042481e in do_progress<(upcxx::progress_level)1> () at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/src/backend/gasnet/runtime.cpp:2192
[1] #11 0x00000000004227af in upcxx::detail::progress_user () at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/src/backend/gasnet/runtime.cpp:2252
[1] #12 0x0000000000408977 in upcxx::progress (level=upcxx::progress_level::user) at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/upcxx.assert1.optlev0.dbgsym1.gasnet_par.ofi/include/upcxx/backend.hpp:34
[1] #13 0x0000000000408635 in upcxx::detail::future_wait_upcxx_progress_user::operator() (this=0x15467c0eafaf) at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/upcxx.assert1.optlev0.dbgsym1.gasnet_par.ofi/include/upcxx/future/future1.hpp:53
[1] #14 0x000000000040952c in upcxx::detail::future1<upcxx::detail::future_kind_shref<upcxx::detail::future_header_ops_general, false>, msg>::wait_reference<-1>() const & (this=0x15467c0eafe8) at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/upcxx.assert1.optlev0.dbgsym1.gasnet_par.ofi/include/upcxx/future/future1.hpp:416
[1] #15 0x0000000000407892 in operator() (__closure=0x123fde8) at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/test/lpc-stress.cpp:109
[1] #16 0x0000000000407f3a in std::__invoke_impl<void, main(int, char**)::<lambda()> >(std::__invoke_other, struct {...} &&) (__f=...) at /opt/cray/pe/gcc/11.2.0/snos/include/g++/bits/invoke.h:61
[1] #17 0x0000000000407efd in std::__invoke<main(int, char**)::<lambda()> >(struct {...} &&) (__fn=...) at /opt/cray/pe/gcc/11.2.0/snos/include/g++/bits/invoke.h:96
[1] #18 0x0000000000407eaa in std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > >::_M_invoke<0>(std::_Index_tuple<0>) (this=0x123fde8) at /opt/cray/pe/gcc/11.2.0/snos/include/g++/bits/std_thread.h:253
[1] #19 0x0000000000407e7e in std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > >::operator()(void) (this=0x123fde8) at /opt/cray/pe/gcc/11.2.0/snos/include/g++/bits/std_thread.h:260
[1] #20 0x0000000000407e62 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > > >::_M_run(void) (this=0x123fde0) at /opt/cray/pe/gcc/11.2.0/snos/include/g++/bits/std_thread.h:211
[1] #21 0x000015469973cf44 in std::execute_native_thread_routine (__p=0x123fde0) at ../../../../../cpe-gcc-11.2.0-202108140355.9bf1fd589a5c1/libstdc++-v3/src/c++11/thread.cc:82
[1] #22 0x000015469bb086ea in start_thread () from /lib64/libpthread.so.0
[1] #23 0x000015469921e94f in clone () from /lib64/libc.so.6
[1]
[1] Thread 3 (Thread 0x15467c2ed700 (LWP 33116) "test-lpc-stress"):
[1] #0 0x000015469bb109c5 in pthread_spin_lock () from /lib64/libpthread.so.0
[1] #1 0x000015469b87b35e in ofi_cq_progress (cq=0xf35020) at prov/util/src/util_cq.c:595
[1] #2 0x000015469b87a68e in ofi_cq_readfrom (cq_fid=0xf35020, buf=<optimized out>, count=64, src_addr=0x0) at prov/util/src/util_cq.c:247
[1] #3 0x000000000052c396 in fi_cq_read (cq=0xf35020, buf=0x15467c2eb200, count=64) at /opt/cray/libfabric/1.11.0.4.125/include/rdma/fi_eq.h:394
[1] #4 0x000000000053a962 in gasnetc_ofi_tx_poll_one (cqfd=0xf35020) at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/berkeleylab-gasnet-512c17e80297/ofi-conduit/gasnet_ofi.c:2028
[1] #5 0x000000000053ad27 in gasnetc_ofi_tx_poll () at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/berkeleylab-gasnet-512c17e80297/ofi-conduit/gasnet_ofi.c:2071
[1] #6 0x000000000053b4ff in gasnetc_ofi_poll () at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/berkeleylab-gasnet-512c17e80297/ofi-conduit/gasnet_ofi.c:2241
[1] #7 0x00000000004e0c82 in gasnetc_AMPoll () at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/berkeleylab-gasnet-512c17e80297/ofi-conduit/gasnet_core.c:650
[1] #8 0x000000000040fa3c in _gasneti_AMPoll () at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/berkeleylab-gasnet-512c17e80297/gasnet_help.h:1298
[1] #9 0x000000000040fc26 in _gasnet_AMPoll () at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/berkeleylab-gasnet-512c17e80297/gasnet_help.h:1511
[1] #10 0x000000000042481e in do_progress<(upcxx::progress_level)1> () at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/src/backend/gasnet/runtime.cpp:2192
[1] #11 0x00000000004227af in upcxx::detail::progress_user () at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/src/backend/gasnet/runtime.cpp:2252
[1] #12 0x0000000000408977 in upcxx::progress (level=upcxx::progress_level::user) at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/upcxx.assert1.optlev0.dbgsym1.gasnet_par.ofi/include/upcxx/backend.hpp:34
[1] #13 0x0000000000408635 in upcxx::detail::future_wait_upcxx_progress_user::operator() (this=0x15467c2ebfaf) at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/upcxx.assert1.optlev0.dbgsym1.gasnet_par.ofi/include/upcxx/future/future1.hpp:53
[1] #14 0x000000000040952c in upcxx::detail::future1<upcxx::detail::future_kind_shref<upcxx::detail::future_header_ops_general, false>, msg>::wait_reference<-1>() const & (this=0x15467c2ebfe8) at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/upcxx.assert1.optlev0.dbgsym1.gasnet_par.ofi/include/upcxx/future/future1.hpp:416
[1] #15 0x0000000000407892 in operator() (__closure=0xf35008) at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/test/lpc-stress.cpp:109
[1] #16 0x0000000000407f3a in std::__invoke_impl<void, main(int, char**)::<lambda()> >(std::__invoke_other, struct {...} &&) (__f=...) at /opt/cray/pe/gcc/11.2.0/snos/include/g++/bits/invoke.h:61
[1] #17 0x0000000000407efd in std::__invoke<main(int, char**)::<lambda()> >(struct {...} &&) (__fn=...) at /opt/cray/pe/gcc/11.2.0/snos/include/g++/bits/invoke.h:96
[1] #18 0x0000000000407eaa in std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > >::_M_invoke<0>(std::_Index_tuple<0>) (this=0xf35008) at /opt/cray/pe/gcc/11.2.0/snos/include/g++/bits/std_thread.h:253
[1] #19 0x0000000000407e7e in std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > >::operator()(void) (this=0xf35008) at /opt/cray/pe/gcc/11.2.0/snos/include/g++/bits/std_thread.h:260
[1] #20 0x0000000000407e62 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > > >::_M_run(void) (this=0xf35000) at /opt/cray/pe/gcc/11.2.0/snos/include/g++/bits/std_thread.h:211
[1] #21 0x000015469973cf44 in std::execute_native_thread_routine (__p=0xf35000) at ../../../../../cpe-gcc-11.2.0-202108140355.9bf1fd589a5c1/libstdc++-v3/src/c++11/thread.cc:82
[1] #22 0x000015469bb086ea in start_thread () from /lib64/libpthread.so.0
[1] #23 0x000015469921e94f in clone () from /lib64/libc.so.6
[1]
[1] Thread 2 (Thread 0x15469725e700 (LWP 33113) "test-lpc-stress"):
[1] #0 0x00001546992120a9 in poll () from /lib64/libc.so.6
[1] #1 0x000015469b882cfc in poll (__timeout=-1, __nfds=1, __fds=0x15469725d2e8) at /usr/include/bits/poll2.h:46
[1] #2 ofi_uffd_handler (arg=<optimized out>) at prov/util/src/util_mem_monitor.c:312
[1] #3 0x000015469bb086ea in start_thread () from /lib64/libpthread.so.0
[1] #4 0x000015469921e94f in clone () from /lib64/libc.so.6
[1]
[1] Thread 1 (Thread 0x15469756f300 (LWP 33112) "test-lpc-stress"):
[1] #0 0x00001546991e862f in wait4 () from /lib64/libc.so.6
[1] #1 0x000015469915fb57 in do_system () from /lib64/libc.so.6
[1] #2 0x00000000005560da in gasneti_system_redirected (cmd=0x9c1d00 <cmd> "/usr/bin/gdb -nx -batch -x /var/tmp/pbs.537549.polaris-pbs-01.hsn.cm.polaris.alcf.anl.gov/gasnet_l7AwD0 '/lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/./test-lpc-stre"..., stdout_fd=22) at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/berkeleylab-gasnet-512c17e80297/gasnet_tools.c:1679
[1] #3 0x000000000055678c in gasneti_bt_gdb (fd=22) at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/berkeleylab-gasnet-512c17e80297/gasnet_tools.c:1944
[1] #4 0x0000000000556f82 in gasneti_print_backtrace (fd=2) at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/berkeleylab-gasnet-512c17e80297/gasnet_tools.c:2228
[1] #5 0x00000000005558d0 in gasneti_ondemandHandler (sig=15) at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/berkeleylab-gasnet-512c17e80297/gasnet_tools.c:1238
[1] #6 <signal handler called>
[1] #7 0x000015469bb109c5 in pthread_spin_lock () from /lib64/libpthread.so.0
[1] #8 0x000015469b87b35e in ofi_cq_progress (cq=0xf35020) at prov/util/src/util_cq.c:595
[1] #9 0x000015469b87a68e in ofi_cq_readfrom (cq_fid=0xf35020, buf=<optimized out>, count=64, src_addr=0x0) at prov/util/src/util_cq.c:247
[1] #10 0x000000000052c396 in fi_cq_read (cq=0xf35020, buf=0x7fff97079fd0, count=64) at /opt/cray/libfabric/1.11.0.4.125/include/rdma/fi_eq.h:394
[1] #11 0x000000000053a962 in gasnetc_ofi_tx_poll_one (cqfd=0xf35020) at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/berkeleylab-gasnet-512c17e80297/ofi-conduit/gasnet_ofi.c:2028
[1] #12 0x000000000053ad27 in gasnetc_ofi_tx_poll () at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/berkeleylab-gasnet-512c17e80297/ofi-conduit/gasnet_ofi.c:2071
[1] #13 0x000000000053b4ff in gasnetc_ofi_poll () at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/berkeleylab-gasnet-512c17e80297/ofi-conduit/gasnet_ofi.c:2241
[1] #14 0x00000000004e0c82 in gasnetc_AMPoll () at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/berkeleylab-gasnet-512c17e80297/ofi-conduit/gasnet_core.c:650
[1] #15 0x000000000040fa3c in _gasneti_AMPoll () at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/berkeleylab-gasnet-512c17e80297/gasnet_help.h:1298
[1] #16 0x000000000040fc26 in _gasnet_AMPoll () at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/berkeleylab-gasnet-512c17e80297/gasnet_help.h:1511
[1] #17 0x000000000042481e in do_progress<(upcxx::progress_level)1> () at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/src/backend/gasnet/runtime.cpp:2192
[1] #18 0x00000000004227af in upcxx::detail::progress_user () at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/src/backend/gasnet/runtime.cpp:2252
[1] #19 0x0000000000408977 in upcxx::progress (level=upcxx::progress_level::user) at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/bld/upcxx.assert1.optlev0.dbgsym1.gasnet_par.ofi/include/upcxx/backend.hpp:34
[1] #20 0x0000000000407af1 in main (argc=1, argv=0x7fff9707ae78) at /lus/eagle/projects/CSC250STPM17/phargrov/gitlab-builds/M2Xa3QmJ/0/Pagoda/upcxx/test/lpc-stress.cpp:119
The stacks seem to indicate that 9 user threads (master + 8 workers) are deep inside ofi (verbs provider) CQ polling in (as part of normal GASNet AM polling during the test's progress calls). Two other worker threads have apparently completed their test and exited. All remaining thread stacks include fi_cq_read() -> ofi_cq_progress(), and all but one are blocked on a libfabric pthread_spin_lock(). The remaining thread is inside further stack frame rxm_ep_do_progress() and appears to possibly have encountered an FI_EAGAIN from the lower-layer fi_cq_read() call.
This snapshot of the stacks does not provide sufficient evidence to disambiguate whether the process is hung inside libfabric, deadlocked for some other reason, or just running very slowly for some unknown reason. However the PAR mode of this test notably differs from the SEQ mode in that SEQ mode only polls GASNet from the primordial thread, whereas PAR mode polls GASNet concurrently from all 11 threads. The smp mode of this test obviously excludes libfabric. This suggests that concurrent CQ polling on libfabric verbs provider may be the causal factor here.
Marking this low priority, as it seems likely to indicate a defect specific to the (ancient) HPE fork of libfabric 1.11 verbs provider, which GASNet only supports on the Slingshot-10 hardware that is being phased out.
Comments (1)
-
reporter
- Log in to comment
- Assignee
- –
- Type
- Priority
- Status
- new
- Component
- GASNet integration
- Milestone
- –
- Version
- Development Branch
- Votes
- 0
- Watchers
- 1
Another similar set of thread stacks below from this run