persona-example deadlocks when --with-mpsc-queue=biglock

[Dan Bonachea]

CI reveals that persona-example from the programmer's guide is hanging on multiple platforms when the runtime is built using the UPCXX_LPC_INBOX=locked mode. For example, here are the debug-mode hangs on Linux and macOS.

The hangs date back to at least 2019-07-23, but possibly much longer - until that date this test was running overcommitted with an overly high iteration count in CI, so it's not possible to disambiguate run timeouts due to overcommit from genuine deadlock.

The same test built with the default UPCXX_LPC_INBOX=lockfree does not appear to exhibit the problem.

Demonstration on dirac, using current develop afea793 and the default gcc/9.1.0 and openmpi4:

$ rm -Rf .nobs/ ; env CXX=mpicxx UPCXX_LPC_INBOX=locked ./install ./inst-lpc-locked
[...]
UPC++ successfully installed
$ cd example/prog-guide
$ gmake UPCXX_INSTALL=../../inst-lpc-locked UPCXX_NETWORK=smp clean persona-example               rm -f dmap-test dmap-async-insert-test dmap-async-find-test dmap-quiescence-test dmap-atomics-test dmap-conjoined-test dmap-promises-test rb1d-av hello-world view-histogram1 view-histogram2 view-matrix-tasks non-contig-example team_simple team_advanced persona-example-rputs persona-example
rm -f h-d h-d-remote
../../inst-lpc-locked/bin/upcxx persona-example.cpp -g -pthread -o persona-example
$ ../../inst-lpc-locked/bin/upcxx-run -np 1 ./persona-example 1000
Running persona-example on 1 procs with 1000 iterations.
[hangs forever]

Smaller iteration counts will sometimes complete, but the behavior is highly non-linear, which is indicative of a low-probability deadlock.

An on-demand backtrace of the deadlocked state in the run above is shown below. It reveals the primordial thread at the expected join at persona-example.cpp:90, all ten worker threads are waiting in the progress loop at persona-example.cpp:84, and the progress thread in the progress loop at persona-example.cpp:59. This behavior is consistent with one or more LPCs being "lost" and never delivered to the progress thread persona. Attaching a debugger shows the value of lpc_count in the progress thread at 800, which means that only 200 of the 1000 LPCs were delivered before the deadlock occurred. The backtrace also shows two threads simultaneously attempting to unlock the same std::mutex at 0x9b2de0, which should never happen and might be a clue to the problem. This suspicious behavior was confirmed in a separate run in a debugger with the process stopped.

[0] [Thread debugging using libthread_db enabled]
[0] Using host libthread_db library "/lib64/libthread_db.so.1".
[0] 0x00007f0ca8356a89 in waitpid () from /lib64/libc.so.6
[0] To enable execution of this file add
[0]     add-auto-load-safe-path /usr/local/pkg/gcc/9.1.0/lib64/libstdc++.so.6.0.26-gdb.py
[0] line to your configuration file "/home/pcp1/bonachea/.gdbinit".
[0] To completely disable this security protection add
[0]     set auto-load safe-path /
[0] line to your configuration file "/home/pcp1/bonachea/.gdbinit".
[0] For more information about this security protection see the
[0] "Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
[0]     info "(gdb)Auto-loading safe path"
[0]   Id   Target Id         Frame 
[0] * 1    Thread 0x7f0ca9890840 (LWP 11314) "persona-example" 0x00007f0ca8356a89 in waitpid () from /lib64/libc.so.6
[0]   2    Thread 0x7f0c9eea5700 (LWP 11315) "persona-example" 0x00007f0ca948958a in __lll_unlock_wake () from /lib64/libpthread.so.0
[0]   3    Thread 0x7f0c9e6a4700 (LWP 11316) "persona-example" 0x00007f0ca948958a in __lll_unlock_wake () from /lib64/libpthread.so.0
[0]   4    Thread 0x7f0c9dea3700 (LWP 11317) "persona-example" 0x00007f0ca9484d60 in pthread_mutex_lock () from /lib64/libpthread.so.0
[0]   5    Thread 0x7f0c9d6a2700 (LWP 11318) "persona-example" 0x00007f0ca94894ed in __lll_lock_wait () from /lib64/libpthread.so.0
[0]   6    Thread 0x7f0c9cea1700 (LWP 11319) "persona-example" 0x00007f0ca9484d60 in pthread_mutex_lock () from /lib64/libpthread.so.0
[0]   7    Thread 0x7f0c9c6a0700 (LWP 11320) "persona-example" 0x00007f0ca94894ed in __lll_lock_wait () from /lib64/libpthread.so.0
[0]   8    Thread 0x7f0c9be9f700 (LWP 11321) "persona-example" 0x00007f0ca94894ed in __lll_lock_wait () from /lib64/libpthread.so.0
[0]   9    Thread 0x7f0c93fff700 (LWP 11322) "persona-example" 0x00007f0ca94894ed in __lll_lock_wait () from /lib64/libpthread.so.0
[0]   10   Thread 0x7f0c9b69e700 (LWP 11323) "persona-example" 0x00007f0ca94894ed in __lll_lock_wait () from /lib64/libpthread.so.0
[0]   11   Thread 0x7f0c9ae9d700 (LWP 11324) "persona-example" 0x00007f0ca94894ed in __lll_lock_wait () from /lib64/libpthread.so.0
[0]   12   Thread 0x7f0c9a69c700 (LWP 11325) "persona-example" 0x00007f0ca94894ed in __lll_lock_wait () from /lib64/libpthread.so.0
[0] 
[0] Thread 12 (Thread 0x7f0c9a69c700 (LWP 11325)):
[0] #0  0x00007f0ca94894ed in __lll_lock_wait () from /lib64/libpthread.so.0
[0] #1  0x00007f0ca9484e01 in _L_lock_1093 () from /lib64/libpthread.so.0
[0] #2  0x00007f0ca9484da2 in pthread_mutex_lock () from /lib64/libpthread.so.0
[0] #3  0x0000000000662a74 in _gasneti_memcheck_one (curloc=0x6dfcc8 "/home/pcp1/bonachea/UPC/upcxx/inst-lpc-locked/gasnet.debug/include/gasnet_help.h:930") at /home/pcp1/bonachea/UPC/upcxx/.nobs/art/852c9b835201c2c8a76dedec10c1863fad479e6b/GASNet-stable/gasnet_internal.c:1990
[0] #4  0x0000000000442cbb in _gasneti_AMPoll () at /home/pcp1/bonachea/UPC/upcxx/inst-lpc-locked/gasnet.debug/include/gasnet_help.h:930
[0] #5  0x0000000000442ea0 in _gasnet_AMPoll () at /home/pcp1/bonachea/UPC/upcxx/inst-lpc-locked/gasnet.debug/include/gasnet_help.h:1063
[0] #6  0x000000000044e8d8 in upcxx::progress (level=upcxx::progress_level::user) at /home/pcp1/bonachea/UPC/upcxx/src/backend/gasnet/runtime.cpp:1441
[0] #7  0x0000000000405a4d in <lambda()>::operator()(void) const (__closure=0x2765458) at persona-example.cpp:84
[0] #8  0x000000000040aa81 in std::__invoke_impl<void, main(int, char**)::<lambda()> >(std::__invoke_other, <lambda()> &&) (__f=...) at /usr/local/pkg/gcc/9.1.0/include/c++/9.1.0/bits/invoke.h:60
[0] #9  0x000000000040a545 in std::__invoke<main(int, char**)::<lambda()> >(<lambda()> &&) (__fn=...) at /usr/local/pkg/gcc/9.1.0/include/c++/9.1.0/bits/invoke.h:95
[0] #10 0x000000000040a15a in std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > >::_M_invoke<0>(std::_Index_tuple<0>) (this=0x2765458) at /usr/local/pkg/gcc/9.1.0/include/c++/9.1.0/thread:244
[0] #11 0x0000000000409f60 in std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > >::operator()(void) (this=0x2765458) at /usr/local/pkg/gcc/9.1.0/include/c++/9.1.0/thread:251
[0] #12 0x0000000000409e1c in std::thread::_State_impl<std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > > >::_M_run(void) (this=0x2765450) at /usr/local/pkg/gcc/9.1.0/include/c++/9.1.0/thread:195
[0] #13 0x00007f0ca8c468b0 in execute_native_thread_routine () at ../../../../../libstdc++-v3/src/c++11/thread.cc:80
[0] #14 0x00007f0ca9482dd5 in start_thread () from /lib64/libpthread.so.0
[0] #15 0x00007f0ca838fead in clone () from /lib64/libc.so.6
[0] 
[0] Thread 11 (Thread 0x7f0c9ae9d700 (LWP 11324)):
[0] #0  0x00007f0ca94894ed in __lll_lock_wait () from /lib64/libpthread.so.0
[0] #1  0x00007f0ca9484e01 in _L_lock_1093 () from /lib64/libpthread.so.0
[0] #2  0x00007f0ca9484da2 in pthread_mutex_lock () from /lib64/libpthread.so.0
[0] #3  0x0000000000662a74 in _gasneti_memcheck_one (curloc=0x6dfcc8 "/home/pcp1/bonachea/UPC/upcxx/inst-lpc-locked/gasnet.debug/include/gasnet_help.h:930") at /home/pcp1/bonachea/UPC/upcxx/.nobs/art/852c9b835201c2c8a76dedec10c1863fad479e6b/GASNet-stable/gasnet_internal.c:1990
[0] #4  0x0000000000442cbb in _gasneti_AMPoll () at /home/pcp1/bonachea/UPC/upcxx/inst-lpc-locked/gasnet.debug/include/gasnet_help.h:930
[0] #5  0x0000000000442ea0 in _gasnet_AMPoll () at /home/pcp1/bonachea/UPC/upcxx/inst-lpc-locked/gasnet.debug/include/gasnet_help.h:1063
[0] #6  0x000000000044e8d8 in upcxx::progress (level=upcxx::progress_level::user) at /home/pcp1/bonachea/UPC/upcxx/src/backend/gasnet/runtime.cpp:1441
[0] #7  0x0000000000405a4d in <lambda()>::operator()(void) const (__closure=0x2765198) at persona-example.cpp:84
[0] #8  0x000000000040aa81 in std::__invoke_impl<void, main(int, char**)::<lambda()> >(std::__invoke_other, <lambda()> &&) (__f=...) at /usr/local/pkg/gcc/9.1.0/include/c++/9.1.0/bits/invoke.h:60
[0] #9  0x000000000040a545 in std::__invoke<main(int, char**)::<lambda()> >(<lambda()> &&) (__fn=...) at /usr/local/pkg/gcc/9.1.0/include/c++/9.1.0/bits/invoke.h:95
[0] #10 0x000000000040a15a in std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > >::_M_invoke<0>(std::_Index_tuple<0>) (this=0x2765198) at /usr/local/pkg/gcc/9.1.0/include/c++/9.1.0/thread:244
[0] #11 0x0000000000409f60 in std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > >::operator()(void) (this=0x2765198) at /usr/local/pkg/gcc/9.1.0/include/c++/9.1.0/thread:251
[0] #12 0x0000000000409e1c in std::thread::_State_impl<std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > > >::_M_run(void) (this=0x2765190) at /usr/local/pkg/gcc/9.1.0/include/c++/9.1.0/thread:195
[0] #13 0x00007f0ca8c468b0 in execute_native_thread_routine () at ../../../../../libstdc++-v3/src/c++11/thread.cc:80
[0] #14 0x00007f0ca9482dd5 in start_thread () from /lib64/libpthread.so.0
[0] #15 0x00007f0ca838fead in clone () from /lib64/libc.so.6
[0] 
[0] Thread 10 (Thread 0x7f0c9b69e700 (LWP 11323)):
[0] #0  0x00007f0ca94894ed in __lll_lock_wait () from /lib64/libpthread.so.0
[0] #1  0x00007f0ca9484e01 in _L_lock_1093 () from /lib64/libpthread.so.0
[0] #2  0x00007f0ca9484da2 in pthread_mutex_lock () from /lib64/libpthread.so.0
[0] #3  0x0000000000662a74 in _gasneti_memcheck_one (curloc=0x6dfcc8 "/home/pcp1/bonachea/UPC/upcxx/inst-lpc-locked/gasnet.debug/include/gasnet_help.h:930") at /home/pcp1/bonachea/UPC/upcxx/.nobs/art/852c9b835201c2c8a76dedec10c1863fad479e6b/GASNet-stable/gasnet_internal.c:1990
[0] #4  0x0000000000442cbb in _gasneti_AMPoll () at /home/pcp1/bonachea/UPC/upcxx/inst-lpc-locked/gasnet.debug/include/gasnet_help.h:930
[0] #5  0x0000000000442ea0 in _gasnet_AMPoll () at /home/pcp1/bonachea/UPC/upcxx/inst-lpc-locked/gasnet.debug/include/gasnet_help.h:1063
[0] #6  0x000000000044e8d8 in upcxx::progress (level=upcxx::progress_level::user) at /home/pcp1/bonachea/UPC/upcxx/src/backend/gasnet/runtime.cpp:1441
[0] #7  0x0000000000405a4d in <lambda()>::operator()(void) const (__closure=0x2764ed8) at persona-example.cpp:84
[0] #8  0x000000000040aa81 in std::__invoke_impl<void, main(int, char**)::<lambda()> >(std::__invoke_other, <lambda()> &&) (__f=...) at /usr/local/pkg/gcc/9.1.0/include/c++/9.1.0/bits/invoke.h:60
[0] #9  0x000000000040a545 in std::__invoke<main(int, char**)::<lambda()> >(<lambda()> &&) (__fn=...) at /usr/local/pkg/gcc/9.1.0/include/c++/9.1.0/bits/invoke.h:95
[0] #10 0x000000000040a15a in std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > >::_M_invoke<0>(std::_Index_tuple<0>) (this=0x2764ed8) at /usr/local/pkg/gcc/9.1.0/include/c++/9.1.0/thread:244
[0] #11 0x0000000000409f60 in std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > >::operator()(void) (this=0x2764ed8) at /usr/local/pkg/gcc/9.1.0/include/c++/9.1.0/thread:251
[0] #12 0x0000000000409e1c in std::thread::_State_impl<std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > > >::_M_run(void) (this=0x2764ed0) at /usr/local/pkg/gcc/9.1.0/include/c++/9.1.0/thread:195
[0] #13 0x00007f0ca8c468b0 in execute_native_thread_routine () at ../../../../../libstdc++-v3/src/c++11/thread.cc:80
[0] #14 0x00007f0ca9482dd5 in start_thread () from /lib64/libpthread.so.0
[0] #15 0x00007f0ca838fead in clone () from /lib64/libc.so.6
[0] 
[0] Thread 9 (Thread 0x7f0c93fff700 (LWP 11322)):
[0] #0  0x00007f0ca94894ed in __lll_lock_wait () from /lib64/libpthread.so.0
[0] #1  0x00007f0ca9484e01 in _L_lock_1093 () from /lib64/libpthread.so.0
[0] #2  0x00007f0ca9484da2 in pthread_mutex_lock () from /lib64/libpthread.so.0
[0] #3  0x0000000000662a74 in _gasneti_memcheck_one (curloc=0x6dfcc8 "/home/pcp1/bonachea/UPC/upcxx/inst-lpc-locked/gasnet.debug/include/gasnet_help.h:930") at /home/pcp1/bonachea/UPC/upcxx/.nobs/art/852c9b835201c2c8a76dedec10c1863fad479e6b/GASNet-stable/gasnet_internal.c:1990
[0] #4  0x0000000000442cbb in _gasneti_AMPoll () at /home/pcp1/bonachea/UPC/upcxx/inst-lpc-locked/gasnet.debug/include/gasnet_help.h:930
[0] #5  0x0000000000442ea0 in _gasnet_AMPoll () at /home/pcp1/bonachea/UPC/upcxx/inst-lpc-locked/gasnet.debug/include/gasnet_help.h:1063
[0] #6  0x000000000044e8d8 in upcxx::progress (level=upcxx::progress_level::user) at /home/pcp1/bonachea/UPC/upcxx/src/backend/gasnet/runtime.cpp:1441
[0] #7  0x0000000000405a4d in <lambda()>::operator()(void) const (__closure=0x2764c18) at persona-example.cpp:84
[0] #8  0x000000000040aa81 in std::__invoke_impl<void, main(int, char**)::<lambda()> >(std::__invoke_other, <lambda()> &&) (__f=...) at /usr/local/pkg/gcc/9.1.0/include/c++/9.1.0/bits/invoke.h:60
[0] #9  0x000000000040a545 in std::__invoke<main(int, char**)::<lambda()> >(<lambda()> &&) (__fn=...) at /usr/local/pkg/gcc/9.1.0/include/c++/9.1.0/bits/invoke.h:95
[0] #10 0x000000000040a15a in std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > >::_M_invoke<0>(std::_Index_tuple<0>) (this=0x2764c18) at /usr/local/pkg/gcc/9.1.0/include/c++/9.1.0/thread:244
[0] #11 0x0000000000409f60 in std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > >::operator()(void) (this=0x2764c18) at /usr/local/pkg/gcc/9.1.0/include/c++/9.1.0/thread:251
[0] #12 0x0000000000409e1c in std::thread::_State_impl<std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > > >::_M_run(void) (this=0x2764c10) at /usr/local/pkg/gcc/9.1.0/include/c++/9.1.0/thread:195
[0] #13 0x00007f0ca8c468b0 in execute_native_thread_routine () at ../../../../../libstdc++-v3/src/c++11/thread.cc:80
[0] #14 0x00007f0ca9482dd5 in start_thread () from /lib64/libpthread.so.0
[0] #15 0x00007f0ca838fead in clone () from /lib64/libc.so.6
[0] 
[0] Thread 8 (Thread 0x7f0c9be9f700 (LWP 11321)):
[0] #0  0x00007f0ca94894ed in __lll_lock_wait () from /lib64/libpthread.so.0
[0] #1  0x00007f0ca9484e01 in _L_lock_1093 () from /lib64/libpthread.so.0
[0] #2  0x00007f0ca9484da2 in pthread_mutex_lock () from /lib64/libpthread.so.0
[0] #3  0x0000000000662a74 in _gasneti_memcheck_one (curloc=0x6dfcc8 "/home/pcp1/bonachea/UPC/upcxx/inst-lpc-locked/gasnet.debug/include/gasnet_help.h:930") at /home/pcp1/bonachea/UPC/upcxx/.nobs/art/852c9b835201c2c8a76dedec10c1863fad479e6b/GASNet-stable/gasnet_internal.c:1990
[0] #4  0x0000000000442cbb in _gasneti_AMPoll () at /home/pcp1/bonachea/UPC/upcxx/inst-lpc-locked/gasnet.debug/include/gasnet_help.h:930
[0] #5  0x0000000000442ea0 in _gasnet_AMPoll () at /home/pcp1/bonachea/UPC/upcxx/inst-lpc-locked/gasnet.debug/include/gasnet_help.h:1063
[0] #6  0x000000000044e8d8 in upcxx::progress (level=upcxx::progress_level::user) at /home/pcp1/bonachea/UPC/upcxx/src/backend/gasnet/runtime.cpp:1441
[0] #7  0x0000000000405a4d in <lambda()>::operator()(void) const (__closure=0x27286a8) at persona-example.cpp:84
[0] #8  0x000000000040aa81 in std::__invoke_impl<void, main(int, char**)::<lambda()> >(std::__invoke_other, <lambda()> &&) (__f=...) at /usr/local/pkg/gcc/9.1.0/include/c++/9.1.0/bits/invoke.h:60
[0] #9  0x000000000040a545 in std::__invoke<main(int, char**)::<lambda()> >(<lambda()> &&) (__fn=...) at /usr/local/pkg/gcc/9.1.0/include/c++/9.1.0/bits/invoke.h:95
[0] #10 0x000000000040a15a in std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > >::_M_invoke<0>(std::_Index_tuple<0>) (this=0x27286a8) at /usr/local/pkg/gcc/9.1.0/include/c++/9.1.0/thread:244
[0] #11 0x0000000000409f60 in std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > >::operator()(void) (this=0x27286a8) at /usr/local/pkg/gcc/9.1.0/include/c++/9.1.0/thread:251
[0] #12 0x0000000000409e1c in std::thread::_State_impl<std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > > >::_M_run(void) (this=0x27286a0) at /usr/local/pkg/gcc/9.1.0/include/c++/9.1.0/thread:195
[0] #13 0x00007f0ca8c468b0 in execute_native_thread_routine () at ../../../../../libstdc++-v3/src/c++11/thread.cc:80
[0] #14 0x00007f0ca9482dd5 in start_thread () from /lib64/libpthread.so.0
[0] #15 0x00007f0ca838fead in clone () from /lib64/libc.so.6
[0] 
[0] Thread 7 (Thread 0x7f0c9c6a0700 (LWP 11320)):
[0] #0  0x00007f0ca94894ed in __lll_lock_wait () from /lib64/libpthread.so.0
[0] #1  0x00007f0ca9484e01 in _L_lock_1093 () from /lib64/libpthread.so.0
[0] #2  0x00007f0ca9484da2 in pthread_mutex_lock () from /lib64/libpthread.so.0
[0] #3  0x0000000000662a74 in _gasneti_memcheck_one (curloc=0x6dfcc8 "/home/pcp1/bonachea/UPC/upcxx/inst-lpc-locked/gasnet.debug/include/gasnet_help.h:930") at /home/pcp1/bonachea/UPC/upcxx/.nobs/art/852c9b835201c2c8a76dedec10c1863fad479e6b/GASNet-stable/gasnet_internal.c:1990
[0] #4  0x0000000000442cbb in _gasneti_AMPoll () at /home/pcp1/bonachea/UPC/upcxx/inst-lpc-locked/gasnet.debug/include/gasnet_help.h:930
[0] #5  0x0000000000442ea0 in _gasnet_AMPoll () at /home/pcp1/bonachea/UPC/upcxx/inst-lpc-locked/gasnet.debug/include/gasnet_help.h:1063
[0] #6  0x000000000044e8d8 in upcxx::progress (level=upcxx::progress_level::user) at /home/pcp1/bonachea/UPC/upcxx/src/backend/gasnet/runtime.cpp:1441
[0] #7  0x0000000000405a4d in <lambda()>::operator()(void) const (__closure=0x2754d08) at persona-example.cpp:84
[0] #8  0x000000000040aa81 in std::__invoke_impl<void, main(int, char**)::<lambda()> >(std::__invoke_other, <lambda()> &&) (__f=...) at /usr/local/pkg/gcc/9.1.0/include/c++/9.1.0/bits/invoke.h:60
[0] #9  0x000000000040a545 in std::__invoke<main(int, char**)::<lambda()> >(<lambda()> &&) (__fn=...) at /usr/local/pkg/gcc/9.1.0/include/c++/9.1.0/bits/invoke.h:95
[0] #10 0x000000000040a15a in std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > >::_M_invoke<0>(std::_Index_tuple<0>) (this=0x2754d08) at /usr/local/pkg/gcc/9.1.0/include/c++/9.1.0/thread:244
[0] #11 0x0000000000409f60 in std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > >::operator()(void) (this=0x2754d08) at /usr/local/pkg/gcc/9.1.0/include/c++/9.1.0/thread:251
[0] #12 0x0000000000409e1c in std::thread::_State_impl<std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > > >::_M_run(void) (this=0x2754d00) at /usr/local/pkg/gcc/9.1.0/include/c++/9.1.0/thread:195
[0] #13 0x00007f0ca8c468b0 in execute_native_thread_routine () at ../../../../../libstdc++-v3/src/c++11/thread.cc:80
[0] #14 0x00007f0ca9482dd5 in start_thread () from /lib64/libpthread.so.0
[0] #15 0x00007f0ca838fead in clone () from /lib64/libc.so.6
[0] 
[0] Thread 6 (Thread 0x7f0c9cea1700 (LWP 11319)):
[0] #0  0x00007f0ca9484d60 in pthread_mutex_lock () from /lib64/libpthread.so.0
[0] #1  0x0000000000662a74 in _gasneti_memcheck_one (curloc=0x6dfcc8 "/home/pcp1/bonachea/UPC/upcxx/inst-lpc-locked/gasnet.debug/include/gasnet_help.h:930") at /home/pcp1/bonachea/UPC/upcxx/.nobs/art/852c9b835201c2c8a76dedec10c1863fad479e6b/GASNet-stable/gasnet_internal.c:1990
[0] #2  0x0000000000442cbb in _gasneti_AMPoll () at /home/pcp1/bonachea/UPC/upcxx/inst-lpc-locked/gasnet.debug/include/gasnet_help.h:930
[0] #3  0x0000000000442ea0 in _gasnet_AMPoll () at /home/pcp1/bonachea/UPC/upcxx/inst-lpc-locked/gasnet.debug/include/gasnet_help.h:1063
[0] #4  0x000000000044e8d8 in upcxx::progress (level=upcxx::progress_level::user) at /home/pcp1/bonachea/UPC/upcxx/src/backend/gasnet/runtime.cpp:1441
[0] #5  0x0000000000405a4d in <lambda()>::operator()(void) const (__closure=0x2748c68) at persona-example.cpp:84
[0] #6  0x000000000040aa81 in std::__invoke_impl<void, main(int, char**)::<lambda()> >(std::__invoke_other, <lambda()> &&) (__f=...) at /usr/local/pkg/gcc/9.1.0/include/c++/9.1.0/bits/invoke.h:60
[0] #7  0x000000000040a545 in std::__invoke<main(int, char**)::<lambda()> >(<lambda()> &&) (__fn=...) at /usr/local/pkg/gcc/9.1.0/include/c++/9.1.0/bits/invoke.h:95
[0] #8  0x000000000040a15a in std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > >::_M_invoke<0>(std::_Index_tuple<0>) (this=0x2748c68) at /usr/local/pkg/gcc/9.1.0/include/c++/9.1.0/thread:244
[0] #9  0x0000000000409f60 in std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > >::operator()(void) (this=0x2748c68) at /usr/local/pkg/gcc/9.1.0/include/c++/9.1.0/thread:251
[0] #10 0x0000000000409e1c in std::thread::_State_impl<std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > > >::_M_run(void) (this=0x2748c60) at /usr/local/pkg/gcc/9.1.0/include/c++/9.1.0/thread:195
[0] #11 0x00007f0ca8c468b0 in execute_native_thread_routine () at ../../../../../libstdc++-v3/src/c++11/thread.cc:80
[0] #12 0x00007f0ca9482dd5 in start_thread () from /lib64/libpthread.so.0
[0] #13 0x00007f0ca838fead in clone () from /lib64/libc.so.6
[0] 
[0] Thread 5 (Thread 0x7f0c9d6a2700 (LWP 11318)):
[0] #0  0x00007f0ca94894ed in __lll_lock_wait () from /lib64/libpthread.so.0
[0] #1  0x00007f0ca9484dcb in _L_lock_883 () from /lib64/libpthread.so.0
[0] #2  0x00007f0ca9484c98 in pthread_mutex_lock () from /lib64/libpthread.so.0
[0] #3  0x000000000040c111 in __gthread_mutex_lock (__mutex=0x9b2de0 <upcxx::detail::intru_queue<upcxx::detail::lpc_base, (upcxx::detail::intru_queue_safety)1, &upcxx::detail::lpc_base::intruder>::the_lock_>) at /usr/local/pkg/gcc/9.1.0/include/c++/9.1.0/x86_64-pc-linux-gnu/bits/gthr-default.h:749
[0] #4  0x000000000040c2ca in std::mutex::lock (this=0x9b2de0 <upcxx::detail::intru_queue<upcxx::detail::lpc_base, (upcxx::detail::intru_queue_safety)1, &upcxx::detail::lpc_base::intruder>::the_lock_>) at /usr/local/pkg/gcc/9.1.0/include/c++/9.1.0/bits/std_mutex.h:100
[0] #5  0x000000000040eda8 in std::lock_guard<std::mutex>::lock_guard (this=0x7f0c9d6a1a88, __m=...) at /usr/local/pkg/gcc/9.1.0/include/c++/9.1.0/bits/std_mutex.h:159
[0] #6  0x0000000000454095 in upcxx::detail::intru_queue<upcxx::detail::lpc_base, (upcxx::detail::intru_queue_safety)1, &upcxx::detail::lpc_base::intruder>::burst<upcxx::detail::lpc_inbox<(upcxx::detail::intru_queue_safety)1>::burst(int)::{lambda(upcxx::detail::lpc_base*)#1}>(int, upcxx::detail::lpc_inbox<(upcxx::detail::intru_queue_safety)1>::burst(int)::{lambda(upcxx::detail::lpc_base*)#1}&&) (this=0x7f0c9d6a2620, max_n=100, fn=...) at /home/pcp1/bonachea/UPC/upcxx/.nobs/art/b60cc4bea850969f25f9ba76c05bc5e46b83acb8/upcxx/intru_queue.hpp:423
[0] #7  0x00000000004532f4 in upcxx::detail::lpc_inbox<(upcxx::detail::intru_queue_safety)1>::burst (this=0x7f0c9d6a2620, max_n=100) at /home/pcp1/bonachea/UPC/upcxx/.nobs/art/b60cc4bea850969f25f9ba76c05bc5e46b83acb8/upcxx/lpc.hpp:78
[0] #8  0x00000000004528bb in upcxx::detail::persona_tls::burst_internal (this=0x7f0c9d6a2610, p=...) at /home/pcp1/bonachea/UPC/upcxx/.nobs/art/b60cc4bea850969f25f9ba76c05bc5e46b83acb8/upcxx/persona.hpp:719
[0] #9  0x000000000044e7e8 in upcxx::<lambda(upcxx::persona&)>::operator()(upcxx::persona &) const (__closure=0x7f0c9d6a1bf0, p=...) at /home/pcp1/bonachea/UPC/upcxx/src/backend/gasnet/runtime.cpp:1456
[0] #10 0x000000000044f48a in upcxx::detail::persona_tls::foreach_active_as_top<upcxx::progress(upcxx::progress_level)::<lambda(upcxx::persona&)> >(upcxx::<lambda(upcxx::persona&)> &&) (this=0x7f0c9d6a2610, fn=...) at /home/pcp1/bonachea/UPC/upcxx/.nobs/art/b60cc4bea850969f25f9ba76c05bc5e46b83acb8/upcxx/persona.hpp:700
[0] #11 0x000000000044e90a in upcxx::progress (level=upcxx::progress_level::user) at /home/pcp1/bonachea/UPC/upcxx/src/backend/gasnet/runtime.cpp:1446
[0] #12 0x0000000000405a4d in <lambda()>::operator()(void) const (__closure=0x2748d28) at persona-example.cpp:84
[0] #13 0x000000000040aa81 in std::__invoke_impl<void, main(int, char**)::<lambda()> >(std::__invoke_other, <lambda()> &&) (__f=...) at /usr/local/pkg/gcc/9.1.0/include/c++/9.1.0/bits/invoke.h:60
[0] #14 0x000000000040a545 in std::__invoke<main(int, char**)::<lambda()> >(<lambda()> &&) (__fn=...) at /usr/local/pkg/gcc/9.1.0/include/c++/9.1.0/bits/invoke.h:95
[0] #15 0x000000000040a15a in std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > >::_M_invoke<0>(std::_Index_tuple<0>) (this=0x2748d28) at /usr/local/pkg/gcc/9.1.0/include/c++/9.1.0/thread:244
[0] #16 0x0000000000409f60 in std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > >::operator()(void) (this=0x2748d28) at /usr/local/pkg/gcc/9.1.0/include/c++/9.1.0/thread:251
[0] #17 0x0000000000409e1c in std::thread::_State_impl<std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > > >::_M_run(void) (this=0x2748d20) at /usr/local/pkg/gcc/9.1.0/include/c++/9.1.0/thread:195
[0] #18 0x00007f0ca8c468b0 in execute_native_thread_routine () at ../../../../../libstdc++-v3/src/c++11/thread.cc:80
[0] #19 0x00007f0ca9482dd5 in start_thread () from /lib64/libpthread.so.0
[0] #20 0x00007f0ca838fead in clone () from /lib64/libc.so.6
[0] 
[0] Thread 4 (Thread 0x7f0c9dea3700 (LWP 11317)):
[0] #0  0x00007f0ca9484d60 in pthread_mutex_lock () from /lib64/libpthread.so.0
[0] #1  0x0000000000662a74 in _gasneti_memcheck_one (curloc=0x6dfcc8 "/home/pcp1/bonachea/UPC/upcxx/inst-lpc-locked/gasnet.debug/include/gasnet_help.h:930") at /home/pcp1/bonachea/UPC/upcxx/.nobs/art/852c9b835201c2c8a76dedec10c1863fad479e6b/GASNet-stable/gasnet_internal.c:1990
[0] #2  0x0000000000442cbb in _gasneti_AMPoll () at /home/pcp1/bonachea/UPC/upcxx/inst-lpc-locked/gasnet.debug/include/gasnet_help.h:930
[0] #3  0x0000000000442ea0 in _gasnet_AMPoll () at /home/pcp1/bonachea/UPC/upcxx/inst-lpc-locked/gasnet.debug/include/gasnet_help.h:1063
[0] #4  0x000000000044e8d8 in upcxx::progress (level=upcxx::progress_level::user) at /home/pcp1/bonachea/UPC/upcxx/src/backend/gasnet/runtime.cpp:1441
[0] #5  0x0000000000405a4d in <lambda()>::operator()(void) const (__closure=0x2728ab8) at persona-example.cpp:84
[0] #6  0x000000000040aa81 in std::__invoke_impl<void, main(int, char**)::<lambda()> >(std::__invoke_other, <lambda()> &&) (__f=...) at /usr/local/pkg/gcc/9.1.0/include/c++/9.1.0/bits/invoke.h:60
[0] #7  0x000000000040a545 in std::__invoke<main(int, char**)::<lambda()> >(<lambda()> &&) (__fn=...) at /usr/local/pkg/gcc/9.1.0/include/c++/9.1.0/bits/invoke.h:95
[0] #8  0x000000000040a15a in std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > >::_M_invoke<0>(std::_Index_tuple<0>) (this=0x2728ab8) at /usr/local/pkg/gcc/9.1.0/include/c++/9.1.0/thread:244
[0] #9  0x0000000000409f60 in std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > >::operator()(void) (this=0x2728ab8) at /usr/local/pkg/gcc/9.1.0/include/c++/9.1.0/thread:251
[0] #10 0x0000000000409e1c in std::thread::_State_impl<std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > > >::_M_run(void) (this=0x2728ab0) at /usr/local/pkg/gcc/9.1.0/include/c++/9.1.0/thread:195
[0] #11 0x00007f0ca8c468b0 in execute_native_thread_routine () at ../../../../../libstdc++-v3/src/c++11/thread.cc:80
[0] #12 0x00007f0ca9482dd5 in start_thread () from /lib64/libpthread.so.0
[0] #13 0x00007f0ca838fead in clone () from /lib64/libc.so.6
[0] 
[0] Thread 3 (Thread 0x7f0c9e6a4700 (LWP 11316)):
[0] #0  0x00007f0ca948958a in __lll_unlock_wake () from /lib64/libpthread.so.0
[0] #1  0x00007f0ca9485ece in _L_unlock_738 () from /lib64/libpthread.so.0
[0] #2  0x00007f0ca9485e40 in pthread_mutex_unlock () from /lib64/libpthread.so.0
[0] #3  0x000000000040c140 in __gthread_mutex_unlock (__mutex=0x9b2de0 <upcxx::detail::intru_queue<upcxx::detail::lpc_base, (upcxx::detail::intru_queue_safety)1, &upcxx::detail::lpc_base::intruder>::the_lock_>) at /usr/local/pkg/gcc/9.1.0/include/c++/9.1.0/x86_64-pc-linux-gnu/bits/gthr-default.h:779
[0] #4  0x000000000040c2f8 in std::mutex::unlock (this=0x9b2de0 <upcxx::detail::intru_queue<upcxx::detail::lpc_base, (upcxx::detail::intru_queue_safety)1, &upcxx::detail::lpc_base::intruder>::the_lock_>) at /usr/local/pkg/gcc/9.1.0/include/c++/9.1.0/bits/std_mutex.h:118
[0] #5  0x000000000040edc7 in std::lock_guard<std::mutex>::~lock_guard (this=0x7f0c9e6a3a88, __in_chrg=<optimized out>) at /usr/local/pkg/gcc/9.1.0/include/c++/9.1.0/bits/std_mutex.h:165
[0] #6  0x00000000004540d3 in upcxx::detail::intru_queue<upcxx::detail::lpc_base, (upcxx::detail::intru_queue_safety)1, &upcxx::detail::lpc_base::intruder>::burst<upcxx::detail::lpc_inbox<(upcxx::detail::intru_queue_safety)1>::burst(int)::{lambda(upcxx::detail::lpc_base*)#1}>(int, upcxx::detail::lpc_inbox<(upcxx::detail::intru_queue_safety)1>::burst(int)::{lambda(upcxx::detail::lpc_base*)#1}&&) (this=0x7f0c9e6a4630, max_n=100, fn=...) at /home/pcp1/bonachea/UPC/upcxx/.nobs/art/b60cc4bea850969f25f9ba76c05bc5e46b83acb8/upcxx/intru_queue.hpp:423
[0] #7  0x00000000004532f4 in upcxx::detail::lpc_inbox<(upcxx::detail::intru_queue_safety)1>::burst (this=0x7f0c9e6a4630, max_n=100) at /home/pcp1/bonachea/UPC/upcxx/.nobs/art/b60cc4bea850969f25f9ba76c05bc5e46b83acb8/upcxx/lpc.hpp:78
[0] #8  0x000000000045292f in upcxx::detail::persona_tls::burst_user (this=0x7f0c9e6a4610, p=...) at /home/pcp1/bonachea/UPC/upcxx/.nobs/art/b60cc4bea850969f25f9ba76c05bc5e46b83acb8/upcxx/persona.hpp:740
[0] #9  0x000000000044e839 in upcxx::<lambda(upcxx::persona&)>::operator()(upcxx::persona &) const (__closure=0x7f0c9e6a3bf0, p=...) at /home/pcp1/bonachea/UPC/upcxx/src/backend/gasnet/runtime.cpp:1460
[0] #10 0x000000000044f48a in upcxx::detail::persona_tls::foreach_active_as_top<upcxx::progress(upcxx::progress_level)::<lambda(upcxx::persona&)> >(upcxx::<lambda(upcxx::persona&)> &&) (this=0x7f0c9e6a4610, fn=...) at /home/pcp1/bonachea/UPC/upcxx/.nobs/art/b60cc4bea850969f25f9ba76c05bc5e46b83acb8/upcxx/persona.hpp:700
[0] #11 0x000000000044e90a in upcxx::progress (level=upcxx::progress_level::user) at /home/pcp1/bonachea/UPC/upcxx/src/backend/gasnet/runtime.cpp:1446
[0] #12 0x0000000000405a4d in <lambda()>::operator()(void) const (__closure=0x2728908) at persona-example.cpp:84
[0] #13 0x000000000040aa81 in std::__invoke_impl<void, main(int, char**)::<lambda()> >(std::__invoke_other, <lambda()> &&) (__f=...) at /usr/local/pkg/gcc/9.1.0/include/c++/9.1.0/bits/invoke.h:60
[0] #14 0x000000000040a545 in std::__invoke<main(int, char**)::<lambda()> >(<lambda()> &&) (__fn=...) at /usr/local/pkg/gcc/9.1.0/include/c++/9.1.0/bits/invoke.h:95
[0] #15 0x000000000040a15a in std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > >::_M_invoke<0>(std::_Index_tuple<0>) (this=0x2728908) at /usr/local/pkg/gcc/9.1.0/include/c++/9.1.0/thread:244
[0] #16 0x0000000000409f60 in std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > >::operator()(void) (this=0x2728908) at /usr/local/pkg/gcc/9.1.0/include/c++/9.1.0/thread:251
[0] #17 0x0000000000409e1c in std::thread::_State_impl<std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > > >::_M_run(void) (this=0x2728900) at /usr/local/pkg/gcc/9.1.0/include/c++/9.1.0/thread:195
[0] #18 0x00007f0ca8c468b0 in execute_native_thread_routine () at ../../../../../libstdc++-v3/src/c++11/thread.cc:80
[0] #19 0x00007f0ca9482dd5 in start_thread () from /lib64/libpthread.so.0
[0] #20 0x00007f0ca838fead in clone () from /lib64/libc.so.6
[0] 
[0] Thread 2 (Thread 0x7f0c9eea5700 (LWP 11315)):
[0] #0  0x00007f0ca948958a in __lll_unlock_wake () from /lib64/libpthread.so.0
[0] #1  0x00007f0ca9485ece in _L_unlock_738 () from /lib64/libpthread.so.0
[0] #2  0x00007f0ca9485e40 in pthread_mutex_unlock () from /lib64/libpthread.so.0
[0] #3  0x000000000040c140 in __gthread_mutex_unlock (__mutex=0x9b2de0 <upcxx::detail::intru_queue<upcxx::detail::lpc_base, (upcxx::detail::intru_queue_safety)1, &upcxx::detail::lpc_base::intruder>::the_lock_>) at /usr/local/pkg/gcc/9.1.0/include/c++/9.1.0/x86_64-pc-linux-gnu/bits/gthr-default.h:779
[0] #4  0x000000000040c2f8 in std::mutex::unlock (this=0x9b2de0 <upcxx::detail::intru_queue<upcxx::detail::lpc_base, (upcxx::detail::intru_queue_safety)1, &upcxx::detail::lpc_base::intruder>::the_lock_>) at /usr/local/pkg/gcc/9.1.0/include/c++/9.1.0/bits/std_mutex.h:118
[0] #5  0x000000000040edc7 in std::lock_guard<std::mutex>::~lock_guard (this=0x7f0c9eea4b08, __in_chrg=<optimized out>) at /usr/local/pkg/gcc/9.1.0/include/c++/9.1.0/bits/std_mutex.h:165
[0] #6  0x00000000004540d3 in upcxx::detail::intru_queue<upcxx::detail::lpc_base, (upcxx::detail::intru_queue_safety)1, &upcxx::detail::lpc_base::intruder>::burst<upcxx::detail::lpc_inbox<(upcxx::detail::intru_queue_safety)1>::burst(int)::{lambda(upcxx::detail::lpc_base*)#1}>(int, upcxx::detail::lpc_inbox<(upcxx::detail::intru_queue_safety)1>::burst(int)::{lambda(upcxx::detail::lpc_base*)#1}&&) (this=0x9b2f58 <upcxx::backend::master+24>, max_n=100, fn=...) at /home/pcp1/bonachea/UPC/upcxx/.nobs/art/b60cc4bea850969f25f9ba76c05bc5e46b83acb8/upcxx/intru_queue.hpp:423
[0] #7  0x00000000004532f4 in upcxx::detail::lpc_inbox<(upcxx::detail::intru_queue_safety)1>::burst (this=0x9b2f58 <upcxx::backend::master+24>, max_n=100) at /home/pcp1/bonachea/UPC/upcxx/.nobs/art/b60cc4bea850969f25f9ba76c05bc5e46b83acb8/upcxx/lpc.hpp:78
[0] #8  0x000000000045292f in upcxx::detail::persona_tls::burst_user (this=0x7f0c9eea5610, p=...) at /home/pcp1/bonachea/UPC/upcxx/.nobs/art/b60cc4bea850969f25f9ba76c05bc5e46b83acb8/upcxx/persona.hpp:740
[0] #9  0x000000000044e839 in upcxx::<lambda(upcxx::persona&)>::operator()(upcxx::persona &) const (__closure=0x7f0c9eea4c70, p=...) at /home/pcp1/bonachea/UPC/upcxx/src/backend/gasnet/runtime.cpp:1460
[0] #10 0x000000000044f48a in upcxx::detail::persona_tls::foreach_active_as_top<upcxx::progress(upcxx::progress_level)::<lambda(upcxx::persona&)> >(upcxx::<lambda(upcxx::persona&)> &&) (this=0x7f0c9eea5610, fn=...) at /home/pcp1/bonachea/UPC/upcxx/.nobs/art/b60cc4bea850969f25f9ba76c05bc5e46b83acb8/upcxx/persona.hpp:700
[0] #11 0x000000000044e90a in upcxx::progress (level=upcxx::progress_level::user) at /home/pcp1/bonachea/UPC/upcxx/src/backend/gasnet/runtime.cpp:1446
[0] #12 0x00000000004056f7 in <lambda()>::operator()(void) const (__closure=0x2748c38) at persona-example.cpp:59
[0] #13 0x000000000040aacc in std::__invoke_impl<void, main(int, char**)::<lambda()> >(std::__invoke_other, <lambda()> &&) (__f=...) at /usr/local/pkg/gcc/9.1.0/include/c++/9.1.0/bits/invoke.h:60
[0] #14 0x000000000040a598 in std::__invoke<main(int, char**)::<lambda()> >(<lambda()> &&) (__fn=...) at /usr/local/pkg/gcc/9.1.0/include/c++/9.1.0/bits/invoke.h:95
[0] #15 0x000000000040a186 in std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > >::_M_invoke<0>(std::_Index_tuple<0>) (this=0x2748c38) at /usr/local/pkg/gcc/9.1.0/include/c++/9.1.0/thread:244
[0] #16 0x0000000000409f7a in std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > >::operator()(void) (this=0x2748c38) at /usr/local/pkg/gcc/9.1.0/include/c++/9.1.0/thread:251
[0] #17 0x0000000000409e3c in std::thread::_State_impl<std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > > >::_M_run(void) (this=0x2748c30) at /usr/local/pkg/gcc/9.1.0/include/c++/9.1.0/thread:195
[0] #18 0x00007f0ca8c468b0 in execute_native_thread_routine () at ../../../../../libstdc++-v3/src/c++11/thread.cc:80
[0] #19 0x00007f0ca9482dd5 in start_thread () from /lib64/libpthread.so.0
[0] #20 0x00007f0ca838fead in clone () from /lib64/libc.so.6
[0] 
[0] Thread 1 (Thread 0x7f0ca9890840 (LWP 11314)):
[0] #0  0x00007f0ca8356a89 in waitpid () from /lib64/libc.so.6
[0] #1  0x00007f0ca82d4de2 in do_system () from /lib64/libc.so.6
[0] #2  0x00007f0ca82d5191 in system () from /lib64/libc.so.6
[0] #3  0x00000000004b436d in gasneti_system_redirected (cmd=0x9baca0 <cmd> "/usr/local/pkg/gdb/newest/bin/gdb -nx -batch -x /tmp/gasnet_FaWQfy '/home/pcp1/bonachea/UPC/upcxx/example/prog-guide/./persona-example' 11314", stdout_fd=3) at /home/pcp1/bonachea/UPC/upcxx/.nobs/art/852c9b835201c2c8a76dedec10c1863fad479e6b/GASNet-stable/gasnet_tools.c:1271
[0] #4  0x00000000004b4dcb in gasneti_bt_gdb (fd=3) at /home/pcp1/bonachea/UPC/upcxx/.nobs/art/852c9b835201c2c8a76dedec10c1863fad479e6b/GASNet-stable/gasnet_tools.c:1518
[0] #5  0x00000000004b5645 in gasneti_print_backtrace (fd=2) at /home/pcp1/bonachea/UPC/upcxx/.nobs/art/852c9b835201c2c8a76dedec10c1863fad479e6b/GASNet-stable/gasnet_tools.c:1793
[0] #6  0x00000000004b4022 in gasneti_ondemandHandler (sig=10) at /home/pcp1/bonachea/UPC/upcxx/.nobs/art/852c9b835201c2c8a76dedec10c1863fad479e6b/GASNet-stable/gasnet_tools.c:1147
[0] #7  <signal handler called>
[0] #8  0x00007f0ca9483f45 in pthread_join () from /lib64/libpthread.so.0
[0] #9  0x00007f0ca8c46ac3 in __gthread_join (__value_ptr=0x0, __threadid=<optimized out>) at /home/phargrov/gcc-9.1.0/bld/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:669
[0] #10 std::thread::join (this=0x7ffe8e947700) at ../../../../../libstdc++-v3/src/c++11/thread.cc:110
[0] #11 0x0000000000405dec in main (argc=2, argv=0x7ffe8e947a08) at persona-example.cpp:90

[Dan Bonachea]

Bulk roll-over of unresolved issues to next milestone

[Dan Bonachea]

This was discussed in the 2020-02-12 meeting and deferred to next release milestone.

It was noted this might eventually be "fixed" by removing this deliberately undocumented configuration option.

[Dan Bonachea]

Mass roll-over of open issues to next release milestone

[Dan Bonachea]

This was a bug in the biglock implementation - proposed fix in pull request #312

[Dan Bonachea]

fix issue #245: persona-example deadlocks when --with-mpsc-queue=biglock

We have an undocumented --with-mpsc-queue=biglock option that enables a low-performance version of our thread-safe LPC queues, which utilizes a "one big lock" approach where queue access is controlled via mutex. The idea was to have a version that was so obviously correct that it could serve as a sanity check against the far more subtle default lock-free LPC queue implementation that is based on atomic operations.

Unfortunately, there was a serious logic error in the biglock implementation of intru_queue<T, intru_queue_safety::mpsc>::burst(max_n, fn) that caused it to SILENTLY DISCARD LPCs from the queue whenever it was called with a queue containing more than max_n entries (default 100). This was causing deadlocks in persona-example whenever sending more than about 200 LPCs, because some were very likely to get "dropped".

Apply the simplest possible fix which is to ignore the value of max_n and always drain (a snapshot of) the entire queue when called.

Actually implementing a max_n limit is possible but could be prohibitively expensive under the current implementation approach; given the current queuing operators, the cost of deferment via re-enqueue would scale linearly with the length of the deferred suffix of elements (raising the time to process the entire queue to quadratic in its length). It would also compromise the FIFO fairness of the queue.

→ <<cset eaff89f59132>>

[Dan Bonachea]

Merge pull request #312 into develop

• issue245-mpsc-biglock-deadlock: Update ChangeLog fix issue #245: persona-example deadlocks when --with-mpsc-queue=biglock intru_queue.hpp: Note a design limitation

→ <<cset 3b68690f1d0f>>

[Dan Bonachea]

Merge develop for 2021.3.0 release

• develop: Bump version to 2021.3.0 for release Add spec 2021.3.0 PDF Update GASNet default URL ChangeLog updates INSTALL.md updates Tweak issue462 test for slightly stronger coverage. Fix issue 462 -- restore DefaultConstructibility of dist_id. Also fix stream-insertion operator. Add test. Update ChangeLog test/copy: issue #241: Add missing CUDA device synchronization test/copy: add error checking to cuda calls test/copy: Update tests to abide by the strengthened requirement copy: Revert issue #423 workaround Update UPCXX_ASSERT_MASTER_IFSEQ to enforce the strengthened requirement issue 423: prohibit SEQ-mode communication using non-master personas configure: allow nvcc setting to contain args Minor optimization to upcxx::current_persona() Update ChangeLog issue #455: don't force synchronous source_cx for rput(remote_cx) Add an rput completions coverage test Fix a think-o in UPCXX_ASSERT_ALWAYS_MASTER() ChangeLog: updates for recent ibv-conduit work Add issue 460 to the change log. Implement detail::invoke_result as an abstraction over std::result_of and std::invoke_result (issue 460). fixup: Resolve issue 430: cannot disable the de... tweak UPCXX_INTERNAL_ONLY to include "detail" in the symbol name Update ChangeLog Resolve issue 430: cannot disable the default network Update INSTALL.md list of (un)supported platforms Update ChangeLog Move upcxx::os_env to upcxx::experimental::os_env Move upcxx::say to upcxx::experimental::say Bump development version to 2020.11.11 Add ChangeLog entry for issue #459. Add ChangeLog entries for issues 25 and 276. Move destroy_heap, restore_heap, op_min, and op_max to experimental namespace. Add ChangeLog entry. Move notrivial reduce and broadcast to experimental namespace (issue 459). Move non-fast reduce ops to experimental. Fix for CI failure with Clang floor. Add detail:internal_only parameter to future1(impl_type) constructor. Update documentation to reflect move of future1 into detail. Add detail::internal_only parameter to future1::then_lazy(). Remove obsolete wait.hpp. Move assert_failed() and fatal_error() into detail namespace. Move view_default_iterator into detail namespace. Rename helper class templates. Move serialization_align_max to detail namespace. Rename operation_cx_as[internal]_future to operation_cx_as[_internal]_future_t. Move apply[_tupled]_as_future into detail namespace. Move future1 and future_is_trivially_ready into detail namespace. Remove long-obsolete reflection.hpp. Move global_fnptr into detail namespace. Move completions into detail namespace. Move digest into detail namespace. Move storage_size and friends into detail namespace. Move bind and helpers into the detail namespace. Protect two more persona_scope members. Proctect persona members. Protect persona_scope members (issue #276). update global_ptr::check() to use gex_EP_QueryBoundSegmentNB Add OFI as an unofficial conduit configure: enable GASNet memory kinds as needed rpc-ctor-trace: add coverage for VIS rput(as_rpc) VIS rput: ensure synchronous serialization of remote_cx rpc-ctor-trace: Add coverage for device memory and GEX MK copy: ensure synchronous serialization of remote_cx for reference PUT case copy: ensure synchronous serialization of remote_cx for MK PUT case copy: ensure synchronous serialization of remote_cx for GET cases Implement backend::prepare_deferred_am_master() rpc-ctor-trace: update expected counts Remove unnecessary move/copy of internal lambda completions deferred in the cuda queue rpc-ctor-trace: expand to exercise upcxx::copy Add a verbose-ctor-trace test lpc-ctor-trace: strengthen as_lpc coverage lpc-ctor-trace: expand to exercise upcxx::copy Remove extraneous qualifiers from friend declarations. Move UPCXX_INTERNAL_ONLY macro to backend_fwd.hpp. Add internal accessor for raw pointer of a global_ptr. Make members of team_id, dist_id, and global_fnptr private and add friend declarations. Wrap global_ptr members with a macro to signify that they are intended to be private. Address issue #276. Refactor deserialized move constructibility check to make it liklier to trip the static assertions. Add assertions to rpc/rpc_ff/as_rpc that deserialized arguments/return values are MoveConstructible. Add tests. Fix a major bug introduced a few commits ago in rpc-ctor-trace. Add a few more test cases. Move utility code to utility.hpp. Modify the text of an error message. Add static_asserts for movability/copyability of rpc/rpc_ff/as_rpc rvalue arguments. Add tests. Fix rpc-ctor-trace test to avoid violating movability requirements in rpc. issue427b: fix a harmless warning rpc-ctor-trace: remove unnecessary barriers Optimize computation of element size with alignment. Updates to documentation and test case. Fix some small issues from the last commit. Account for alignment constraints when computing space per element in serialization of a sequence. Fixes #427. Update Apple compiler detection utility.hpp: clarify a comment Update copy to use bound function instead of serialized completion. Remove obsolete declarations. Move event binding of completions back into completions_state to take advantage of the event-matching logic. Reorganize code and add comments. Rearchitect implementation of bind_event to work on completions in addition to completions_state. Allows reduction of copies/moves. Update documentation for bind_event() and remove requirement that it be called on an rvalue. Refactoring in cx_remote_dispatch to reduce future chaining. Rename a type for clarity. Undo obsolete changes introduced by PR#263. Add some comments. Excise serialization of completions in favor of directly serializing remote bound functions. Use bind_event() in rget and vis. Update ChangeLog Avoid other possible instances of issue #421 Fix issue #421: PGI optimizer problem with upcxx::copy() Add issue421c: upgraded version of issue421 Partially revert "bld/tests.mak: Ban upcxx::copy tests on PGI, avoiding issue 421" Revert "issue 421: Disable use of upcxx::copy with PGI" fixup^2: Support more delimiters in NETWORKS Add change log entry for PR 323. Update static assertion messages to reflect feedback on PR 323. Add static_asserts to detect use of new and new_array with array types. fixup: Support more delimiters in NETWORKS Support more delimiters in NETWORKS ChangeLog: document issue 422 as fixed configure: gasnet/version.git for auto-viv tarball configure: automate bootstrap of GASNet-EX sources configure: relax assumption on GASNet tarball Add issue #408 as fixed to change log. Resolve issue 446 configure: recognize new Intel compilers Add build/run commands to test-results logging Preserve return value of apply_as_future_then_lazy in all three cases. Consolidate apply_as_future_then and apply_as_future_then_lazy. Move logic for avoiding future overheads in apply_as_future().then[lazy] to apply.hpp. Make use of it in other contexts. Rearchitect rpc reply to sidestep future machinery when possible; reduces moves. Restore check for copyability in lpc_dormant::awaken. Deserialize into temporary storage and copy from there when there are multiple completions. Modify lpc_dormant::awaken to deserialize arguments separately for each completion. Fixes #408. test/util.hpp: Add a backwards compatilbity hack lpc-stress: strengthen the test to check LPC return path Add an lpc-stress test Bump development version to 2020.11.9 Update lpc-ctor-trace for issue #450 Update ChangeLog Fix issue #450: upcxx::lpc callback return of rvalue reference not decayed as specified Add issue450 test Allow building view in seq threadmode Add threaded SEQ testing (issue 451) bld/tests.mak: fix harmless typos in comments Update ChangeLog Add issue447 test Fix issue #447: REGRESSION: bulk upcxx::rput with l-value completions fixup: Add some alignment assertions Add some alignment assertions Relocate UPCXX_MPSC_QUEUE[FOO] defn to header fixup: issue #440: Deploy a temporary solution for 'Invalid GASNet call' in deserialization Update ChangeLog fix issue #245: persona-example deadlocks when --with-mpsc-queue=biglock intru_queue.hpp: Note a design limitation Bump development version to 2020.11.7 Remove PROTOTYPE BRANCH notices src/cuda.cpp: Fix a harmless warning from PGI 20.4 bld/tests.mak: Ban upcxx::copy tests on PGI, avoiding issue 421 Update ChangeLog issue 421: Disable use of upcxx::copy with PGI Revert "INSTALL.md: Prohibit PGI compilers for memory_kinds branch" Revert "Prohibit PGI on the memory_kinds branch" re-enable test/regression/issue336.cpp for dev-check test/shared-seg-query: fix a harmless pedantic sign-compare warning Bump patch number to 2020.11.5 Update ChangeLog Add shared segment query test issue #382: Expose shared heap usage at runtime build-devel.md: update ALCF server URL issue440.cpp: Fix a harmless -pedantic warning Update ChangeLog issue #440: Deploy a temporary solution for 'Invalid GASNet call' in deserialization strengthen the issue440.cpp test Add issue440 test Add GitLab CI information to developer's docs issue #438: Report degenerate local_team in upcxx-run -vv verbose output Bump copyright year mpi-hybrid.md: Update udp-conduit recommended CSPAWN settings fixup: CI testing: add probe for OpenMP support Move GitLab CI and dev-ci scripts to a new repo Update ChangeLog configure: default to 8kb max-medium on aries-conduit dev-check: work-arounds for known issues CI testing: add probe for OpenMP support GitLab CI: renames to get junit/xml reporting GitLab CI: enable junit/xml report generation Update ChangeLog Add issue #432 test issue #432: Some upcxx::copy cases do not discharge() properly aries: cap default UPCXX_RPC_EAGER_THRESHOLD to 8kb Bump patch number to 2020.11.3 Update ChangeLog runtime.cpp: ensure max_medium calc accounts for GEX_FLAG_AM_PREPARE_LEAST_ALLOC rma_put_then_am_master_protocol(): Fix gex_AM_MaxRequestMedium arg count UPCXX_RPC_EAGER_THRESHOLD tweaks Update spelling for GASNET_NATIVE_NP_ALLOC_REQ_MEDIUM bench/misc_perf: expand rpc tests to cover the internal code paths Statically optimize am_send_buffer for NPAM Repair am_send_buffer tiny buffer optimization Add NPAM support for Eager RPC injection Add post-configure check for conduit native NPAM Add UPCXX_RPC_EAGER_THRESHOLD_LOCAL to control local threshold Upgrade prepare_am with local threshold and NPAM prereqs issue #261: tweak gasnet::rma_put_then_am_master runtime.hpp: Remove dead code that is no longer maintained RPC: Hoist TM/rank translation overheads rpc.hpp: "Invert" tm/no-tm overloads of rpc/rpc_ff bench/misc_perf: Add test of rput-then-rpc Change UPCXX_RPC_EAGER_THRESHOLD default to the maximum value issue #164: Add UPCXX_RPC_EAGER_THRESHOLD knob Add an RPC microbenchmark Update ChangeLog move remaining upcxx code in to separate header file Move UPC++ dl_malloc defines into separate header file Complete dlmalloc function list Name shift UPC++'s dlmalloc symbols Fix dev-check when configured using --without-cuda tests.mak: Add bench directory to dev-check Add issue428 test Update ChangeLog issue #428: Regression in rpc(team,rank,..,view) overload resolution leads to confusing type errors ChangeLog: spell-check issue #420: Suppress harmless GASNet debug-mode configure warning GitLab CI / CI scripts: de-version macOS bits Remove the deprecated top-level install script GitLab CI: Resume testing of Theta PrgEnv-cray Resolve issue #425 copy-cover: Fix harmless sign-compare warnings from clang 11 cuda.hpp: Fix a harmless warning from clang 11 -Wall Revert "Set GASNet 2020.11.0 tarball URL" Bump version to 2020.11.1 for mk-develop Bump package version for 2020.11.0 release Set GASNet 2020.11.0 tarball URL Merge 2020.3.2 section into ChangeLog Bump development version to 2020.10.1 Add PDFs for 2020.11.0 spec and 2020.10.0 guide INSTALL.md: Expand instructions for GDR memory kinds INSTALL.md: Prohibit PGI compilers for memory_kinds branch Add prototype branch notices to major documents Update ChangeLog syncx3 updated examples from guide re-sync updated examples from guide sync updated examples from guide Change configuration of private segment tests Add support for using private segment for host array Fix order of arguments being passed Add host to host transfers Use promises rather than futures in flood benchmark Local GPU and remote host tests Prohibit PGI on the memory_kinds branch Makefile: Fix Git version stamp detection copy-cover: Exercise remote-only completion events copy-cover: Add source completion validation copy-cover: sweep buf sizes fixupx2: Add new copy-cover test fixup: Implement UPCXX_BUG4148_WORKAROUND fixup: Add new copy-cover test Implement UPCXX_BUG4148_WORKAROUND and RC-only opt for puts copy.hpp: factor initiator variable upcxx::copy: Tune use_mk for no RC fixup: issue #221: Fix upcxx::copy() mishandling of private memory arguments GitLab CI: PGI testers opt-in only (issue #421) Upgrade global_ptr::check() to bounds-check device pointers Change device_allocator to throw upcxx::bad_segment_alloc device_allocator construct: prohibit empty active segments Add bad-segment-alloc test to validate device_allocator exn behavior device_allocator: fix undocumented device memory allocation Fix delete of array in view-accumulate.cpp to delete[] example/prog-guide: apply updates from guide repo cuda_microbenchmark: correct windowing logic Less specific GitLab tags for Xcode runners. correct an assert typo appearing in multiple functions Increase max_heaps in the reference implementation to 33 Avoid issue 423 for upcxx::copy() Deploy GPUDirect transfers in upcxx::copy Construct GEX EP/MK/Segment objects Implement heap_idx recycling Add negative compilation tests copy: assert additional preconditions test/regression/issue405: fix a violated copy() precondition upcxx::copy()/detail::rma_copy_local() use memcpy copy-cover: exploit fix to issue421 Add new copy-cover test issue #221: Fix upcxx::copy() mishandling of private memory arguments Change default --with-gasnet URL to memory_kinds nightly snapshot Split issue 421 test into two, one with lambdas and another with an asymmetric fuction object. Fix issue #421. Add issue421 Ban PGI 20.7+ due to known bugs fixup: Bump required GEX spec version to 0.10 and remove cruft test/copy: update for new collective requirement on device_allocator construction Add collective device_allocator creation Add a high-level coverage test for memory kinds fix an unreported bug in cuda_device move constructor fix unreported bug in device_allocator template inheritance Improve heap_state safety Probe for GASNET_MAXEPS Rework internal memory kinds objects global_ptr: Rename device_ field to heap_idx_

→ <<cset efd61603c031>>