Stop using C++ thread_local: uts_{threads,hybrid,omp_ranks}/lpc_barrier crash on (High)Sierra w/ g++ 7.2.0

Issue #49 resolved

Dan Bonachea created an issue 2017-09-22

The new uts_threads and uts_hybrid tests are crashing in automated testing on Mac OS X sierra with homebrew g++ 7.2.0. The problem did not manifest with the builds using the Xcode clang.

The crash includes the following stderr:

terminate called after throwing an instance of 'std::system_error'
  what():  Invalid argument

Full output here, including a backtrace for the second test which uses GASNet:

uts_threads failure on Sierra

uts_hybrid failure on Sierra

Marking this as minor, since it could be a bug in the test, or possibly something specific to our non-default compiler on Sierra.

Comments (37)

Dan Bonachea reporter

These failures continue on the same platform.

Now also same failure for new lpc_barrier test

Here is a crash stack for lpc_barrier:

* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGABRT
  * frame #0: 0x00007fff98750d42 libsystem_kernel.dylib`__pthread_kill + 10
    frame #1: 0x00007fff9883e457 libsystem_pthread.dylib`pthread_kill + 90
    frame #2: 0x00007fff986b6420 libsystem_c.dylib`abort + 129
    frame #3: 0x00000001000c42c4 libstdc++.6.dylib`__gnu_cxx::__verbose_terminate_handler() + 276
    frame #4: 0x00000001000c1c28 libstdc++.6.dylib`__cxxabiv1::__terminate(void (*)()) + 8
    frame #5: 0x00000001000c1c73 libstdc++.6.dylib`std::terminate() + 19
    frame #6: 0x00000001000c1def libstdc++.6.dylib`__cxa_throw + 111
    frame #7: 0x000000010015e55a libstdc++.6.dylib`std::__throw_system_error(int) + 122
    frame #8: 0x0000000100000cfb lpc_barrier`std::mutex::lock(this=0x0000000100700238) at std_mutex.h:107
    frame #9: 0x0000000100001523 lpc_barrier`upcxx::lpc_inbox_base<2, true>::lock_guard::lock_guard(this=0x00007fff5fbff0e8, m=0x0000000100700238) at lpc_inbox.hpp:66
    frame #10: 0x00000001000014cb lpc_barrier`upcxx::lpc_inbox_base<2, true>::locked_scope(this=0x0000000100700218) at lpc_inbox.hpp:79
    frame #11: 0x00000001000044fd lpc_barrier`lpc_inbox<2, true>::send<lpc_barrier(this=0x0000000100700218, q=1, fn=0x00007fff5fbff130)::<lambda()> >(int, <lambda()> &) const at lpc_inbox.hpp:136
    frame #12: 0x0000000100003f91 lpc_barrier`persona::lpc_ff<lpc_barrier()::<lambda()> >(this=0x00000001007001d8, fn=(__epoch = 0, __bit = 1))>) const at persona.hpp:99
    frame #13: 0x00000001000032db lpc_barrier`lpc_barrier() at lpc_barrier.cpp:52
    frame #14: 0x0000000100003531 lpc_barrier`thread_main() at lpc_barrier.cpp:87
    frame #15: 0x0000000100003dab lpc_barrier`operator(__closure=0x00007fff5fbff500, me=0) at lpc_barrier.cpp:156
    frame #16: 0x0000000100003e2e lpc_barrier`main at lpc_barrier.cpp:164
    frame #17: 0x00007fff98622235 libdyld.dylib`start + 1

  thread #2
    frame #0: 0x00007fff98869de1 libunwind.dylib`libunwind::CFI_Parser<libunwind::LocalAddressSpace>::parseCIE(libunwind::LocalAddressSpace&, unsigned long long, libunwind::CFI_Parser<libunwind::LocalAddressSpace>::CIE_Info*) + 723
    frame #1: 0x00007fff9886995b libunwind.dylib`libunwind::CFI_Parser<libunwind::LocalAddressSpace>::findFDE(libunwind::LocalAddressSpace&, unsigned long long, unsigned long long, unsigned int, unsigned long long, libunwind::CFI_Parser<libunwind::LocalAddressSpace>::FDE_Info*, libunwind::CFI_Parser<libunwind::LocalAddressSpace>::CIE_Info*) + 155
    frame #2: 0x00007fff9886979d libunwind.dylib`libunwind::UnwindCursor<libunwind::LocalAddressSpace, libunwind::Registers_x86_64>::getInfoFromDwarfSection(unsigned long long, unsigned long long, unsigned long long, unsigned int, unsigned int) + 227
    frame #3: 0x00007fff98865a9f libunwind.dylib`libunwind::UnwindCursor<libunwind::LocalAddressSpace, libunwind::Registers_x86_64>::setInfoBasedOnIPRegister(bool) + 235
    frame #4: 0x00007fff988660e5 libunwind.dylib`libunwind::UnwindCursor<libunwind::LocalAddressSpace, libunwind::Registers_x86_64>::step() + 133
    frame #5: 0x00007fff9886586f libunwind.dylib`_Unwind_RaiseException + 175
    frame #6: 0x00000001000c1de2 libstdc++.6.dylib`__cxa_throw + 98
    frame #7: 0x000000010015e55a libstdc++.6.dylib`std::__throw_system_error(int) + 122
    frame #8: 0x0000000100000cfb lpc_barrier`std::mutex::lock(this=0x0000000100500778) at std_mutex.h:107
    frame #9: 0x0000000100001523 lpc_barrier`upcxx::lpc_inbox_base<2, true>::lock_guard::lock_guard(this=0x0000700006a27a18, m=0x0000000100500778) at lpc_inbox.hpp:66
    frame #10: 0x00000001000014cb lpc_barrier`upcxx::lpc_inbox_base<2, true>::locked_scope(this=0x0000000100500758) at lpc_inbox.hpp:79
    frame #11: 0x00000001000044fd lpc_barrier`lpc_inbox<2, true>::send<lpc_barrier(this=0x0000000100500758, q=1, fn=0x0000700006a27a60)::<lambda()> >(int, <lambda()> &) const at lpc_inbox.hpp:136
    frame #12: 0x0000000100003f91 lpc_barrier`persona::lpc_ff<lpc_barrier()::<lambda()> >(this=0x0000000100500718, fn=(__epoch = 0, __bit = 1))>) const at persona.hpp:99
    frame #13: 0x00000001000032db lpc_barrier`lpc_barrier() at lpc_barrier.cpp:52
    frame #14: 0x0000000100003531 lpc_barrier`thread_main() at lpc_barrier.cpp:87
    frame #15: 0x0000000100003dab lpc_barrier`operator(__closure=0x00000001005001e0, me=1) at lpc_barrier.cpp:156
    frame #16: 0x0000000100004ba8 lpc_barrier`::__invoke_impl<void, main()::<lambda(int)>, int>((null)=__invoke_other @ 0x0000700006a27e00, __f=0x00000001005001e0, (null)=<unavailable>, __args#0=0x00000001005001d8)> &, int &) at invoke.h:60
    frame #17: 0x0000000100004300 lpc_barrier`::__invoke<main()::<lambda(int)>, int>(__fn=0x00000001005001e0, (null)=<unavailable>, __args#0=0x00000001005001d8)> &, int &) at invoke.h:95
    frame #18: 0x0000000100005c79 lpc_barrier`thread::_Invoker<std::tuple<main()::<lambda(int)>, int> >::_M_invoke<0, 1>(this=0x00000001005001d8, (null)=_Index_tuple<0, 1> @ 0x0000700006a27e80) const at thread:234
    frame #19: 0x0000000100005b03 lpc_barrier`thread::_Invoker<std::tuple<main()::<lambda(int)>, int> >::operator(this=0x00000001005001d8)() const at thread:243
    frame #20: 0x0000000100005a00 lpc_barrier`thread::_State_impl<std::thread::_Invoker<std::tuple<main()::<lambda(int)>, int> > >::_M_run(this=0x00000001005001d0) const at thread:186
    frame #21: 0x000000010015e62f libstdc++.6.dylib`execute_native_thread_routine + 15
    frame #22: 0x00007fff9883b93b libsystem_pthread.dylib`_pthread_body + 180
    frame #23: 0x00007fff9883b887 libsystem_pthread.dylib`_pthread_start + 286
    frame #24: 0x00007fff9883b08d libsystem_pthread.dylib`thread_start + 13

  thread #3
    frame #0: 0x00007fff98750d42 libsystem_kernel.dylib`__pthread_kill + 10
    frame #1: 0x00007fff9883e457 libsystem_pthread.dylib`pthread_kill + 90
    frame #2: 0x00007fff986b6420 libsystem_c.dylib`abort + 129
    frame #3: 0x00000001000c428c libstdc++.6.dylib`__gnu_cxx::__verbose_terminate_handler() + 220
    frame #4: 0x00000001000c1c28 libstdc++.6.dylib`__cxxabiv1::__terminate(void (*)()) + 8
    frame #5: 0x00000001000c1c73 libstdc++.6.dylib`std::terminate() + 19
    frame #6: 0x000000010015e65e libstdc++.6.dylib`execute_native_thread_routine + 62
    frame #7: 0x00007fff9883b93b libsystem_pthread.dylib`_pthread_body + 180
    frame #8: 0x00007fff9883b887 libsystem_pthread.dylib`_pthread_start + 286
    frame #9: 0x00007fff9883b08d libsystem_pthread.dylib`thread_start + 13

  thread #4
    frame #0: 0x00007fff98750d42 libsystem_kernel.dylib`__pthread_kill + 10
    frame #1: 0x00007fff9883e457 libsystem_pthread.dylib`pthread_kill + 90
    frame #2: 0x00007fff986b6420 libsystem_c.dylib`abort + 129
    frame #3: 0x00000001000c428c libstdc++.6.dylib`__gnu_cxx::__verbose_terminate_handler() + 220
    frame #4: 0x00000001000c1c28 libstdc++.6.dylib`__cxxabiv1::__terminate(void (*)()) + 8
    frame #5: 0x00000001000c1c73 libstdc++.6.dylib`std::terminate() + 19
    frame #6: 0x000000010015e65e libstdc++.6.dylib`execute_native_thread_routine + 62
    frame #7: 0x00007fff9883b93b libsystem_pthread.dylib`_pthread_body + 180
    frame #8: 0x00007fff9883b887 libsystem_pthread.dylib`_pthread_start + 286
    frame #9: 0x00007fff9883b08d libsystem_pthread.dylib`thread_start + 13

  thread #5
    frame #0: 0x00007fff98750d42 libsystem_kernel.dylib`__pthread_kill + 10
    frame #1: 0x00007fff9883e457 libsystem_pthread.dylib`pthread_kill + 90
    frame #2: 0x00007fff986b6420 libsystem_c.dylib`abort + 129
    frame #3: 0x00000001000c428c libstdc++.6.dylib`__gnu_cxx::__verbose_terminate_handler() + 220
    frame #4: 0x00000001000c1c28 libstdc++.6.dylib`__cxxabiv1::__terminate(void (*)()) + 8
    frame #5: 0x00000001000c1c73 libstdc++.6.dylib`std::terminate() + 19
    frame #6: 0x000000010015e65e libstdc++.6.dylib`execute_native_thread_routine + 62
    frame #7: 0x00007fff9883b93b libsystem_pthread.dylib`_pthread_body + 180
    frame #8: 0x00007fff9883b887 libsystem_pthread.dylib`_pthread_start + 286
    frame #9: 0x00007fff9883b08d libsystem_pthread.dylib`thread_start + 13

  thread #6
    frame #0: 0x00007fff98750d42 libsystem_kernel.dylib`__pthread_kill + 10
    frame #1: 0x00007fff9883e457 libsystem_pthread.dylib`pthread_kill + 90
    frame #2: 0x00007fff986b6420 libsystem_c.dylib`abort + 129
    frame #3: 0x00000001000c428c libstdc++.6.dylib`__gnu_cxx::__verbose_terminate_handler() + 220
    frame #4: 0x00000001000c1c28 libstdc++.6.dylib`__cxxabiv1::__terminate(void (*)()) + 8
    frame #5: 0x00000001000c1c73 libstdc++.6.dylib`std::terminate() + 19
    frame #6: 0x000000010015e65e libstdc++.6.dylib`execute_native_thread_routine + 62
    frame #7: 0x00007fff9883b93b libsystem_pthread.dylib`_pthread_body + 180
    frame #8: 0x00007fff9883b887 libsystem_pthread.dylib`_pthread_start + 286
    frame #9: 0x00007fff9883b08d libsystem_pthread.dylib`thread_start + 13

  thread #7
    frame #0: 0x00007fff98750d42 libsystem_kernel.dylib`__pthread_kill + 10
    frame #1: 0x00007fff9883e457 libsystem_pthread.dylib`pthread_kill + 90
    frame #2: 0x00007fff986b6420 libsystem_c.dylib`abort + 129
    frame #3: 0x00000001000c428c libstdc++.6.dylib`__gnu_cxx::__verbose_terminate_handler() + 220
    frame #4: 0x00000001000c1c28 libstdc++.6.dylib`__cxxabiv1::__terminate(void (*)()) + 8
    frame #5: 0x00000001000c1c73 libstdc++.6.dylib`std::terminate() + 19
    frame #6: 0x000000010015e65e libstdc++.6.dylib`execute_native_thread_routine + 62
    frame #7: 0x00007fff9883b93b libsystem_pthread.dylib`_pthread_body + 180
    frame #8: 0x00007fff9883b887 libsystem_pthread.dylib`_pthread_start + 286
    frame #9: 0x00007fff9883b08d libsystem_pthread.dylib`thread_start + 13

  thread #8
    frame #0: 0x00007fff98750d42 libsystem_kernel.dylib`__pthread_kill + 10
    frame #1: 0x00007fff9883e457 libsystem_pthread.dylib`pthread_kill + 90
    frame #2: 0x00007fff986b6420 libsystem_c.dylib`abort + 129
    frame #3: 0x00000001000c428c libstdc++.6.dylib`__gnu_cxx::__verbose_terminate_handler() + 220
    frame #4: 0x00000001000c1c28 libstdc++.6.dylib`__cxxabiv1::__terminate(void (*)()) + 8
    frame #5: 0x00000001000c1c73 libstdc++.6.dylib`std::terminate() + 19
    frame #6: 0x000000010015e65e libstdc++.6.dylib`execute_native_thread_routine + 62
    frame #7: 0x00007fff9883b93b libsystem_pthread.dylib`_pthread_body + 180
    frame #8: 0x00007fff9883b887 libsystem_pthread.dylib`_pthread_start + 286
    frame #9: 0x00007fff9883b08d libsystem_pthread.dylib`thread_start + 13

  thread #9
    frame #0: 0x00007fff98750d42 libsystem_kernel.dylib`__pthread_kill + 10
    frame #1: 0x00007fff9883e457 libsystem_pthread.dylib`pthread_kill + 90
    frame #2: 0x00007fff986b6420 libsystem_c.dylib`abort + 129
    frame #3: 0x00000001000c428c libstdc++.6.dylib`__gnu_cxx::__verbose_terminate_handler() + 220
    frame #4: 0x00000001000c1c28 libstdc++.6.dylib`__cxxabiv1::__terminate(void (*)()) + 8
    frame #5: 0x00000001000c1c73 libstdc++.6.dylib`std::terminate() + 19
    frame #6: 0x000000010015e65e libstdc++.6.dylib`execute_native_thread_routine + 62
    frame #7: 0x00007fff9883b93b libsystem_pthread.dylib`_pthread_body + 180
    frame #8: 0x00007fff9883b887 libsystem_pthread.dylib`_pthread_start + 286
    frame #9: 0x00007fff9883b08d libsystem_pthread.dylib`thread_start + 13

  thread #10
    frame #0: 0x00007fff98750d42 libsystem_kernel.dylib`__pthread_kill + 10
    frame #1: 0x00007fff9883e457 libsystem_pthread.dylib`pthread_kill + 90
    frame #2: 0x00007fff986b6420 libsystem_c.dylib`abort + 129
    frame #3: 0x00000001000c428c libstdc++.6.dylib`__gnu_cxx::__verbose_terminate_handler() + 220
    frame #4: 0x00000001000c1c28 libstdc++.6.dylib`__cxxabiv1::__terminate(void (*)()) + 8
    frame #5: 0x00000001000c1c73 libstdc++.6.dylib`std::terminate() + 19
    frame #6: 0x000000010015e65e libstdc++.6.dylib`execute_native_thread_routine + 62
    frame #7: 0x00007fff9883b93b libsystem_pthread.dylib`_pthread_body + 180
    frame #8: 0x00007fff9883b887 libsystem_pthread.dylib`_pthread_start + 286
    frame #9: 0x00007fff9883b08d libsystem_pthread.dylib`thread_start + 13

and uts_threads:

  thread #1, queue = 'com.apple.main-thread'
    frame #0: 0x00007fff98750f5e libsystem_kernel.dylib`__semwait_signal_nocancel + 10
    frame #1: 0x00007fff9865e904 libsystem_c.dylib`nanosleep$NOCANCEL + 188
    frame #2: 0x00007fff98687e9e libsystem_c.dylib`usleep$NOCANCEL + 54
    frame #3: 0x00007fff986b642a libsystem_c.dylib`abort + 139
    frame #4: 0x00000001000ca2c4 libstdc++.6.dylib`__gnu_cxx::__verbose_terminate_handler() + 276
    frame #5: 0x00000001000c7c28 libstdc++.6.dylib`__cxxabiv1::__terminate(void (*)()) + 8
    frame #6: 0x00000001000c7c73 libstdc++.6.dylib`std::terminate() + 19
    frame #7: 0x00000001000c7def libstdc++.6.dylib`__cxa_throw + 111
    frame #8: 0x000000010016455a libstdc++.6.dylib`std::__throw_system_error(int) + 122
    frame #9: 0x00000001000009cb uts_threads`std::mutex::lock(this=0x0000000100402518) at std_mutex.h:107
    frame #10: 0x00000001000011f3 uts_threads`upcxx::lpc_inbox_base<2, true>::lock_guard::lock_guard(this=0x00007fff5fbff228, m=0x0000000100402518) at lpc_inbox.hpp:66
    frame #11: 0x000000010000119b uts_threads`upcxx::lpc_inbox_base<2, true>::locked_scope(this=0x00000001004024f8) at lpc_inbox.hpp:79
    frame #12: 0x0000000100005abe uts_threads`lpc_inbox<2, true>::send<uts_parallel(this=0x00000001004024f8, q=1, fn=0x00007fff5fbff2a0)::<lambda()> >(int, <lambda()> &) const at lpc_inbox.hpp:136
    frame #13: 0x0000000100004c5f uts_threads`persona::lpc_ff<uts_parallel(uint64_t&, upcxx::digest&)::<lambda()> >(this=0x00000001004024b8, fn=<lambda()> @ 0x00007fff5fbff2a0)>) const at persona.hpp:99
    frame #14: 0x0000000100004326 uts_threads`::send<uts_parallel(uint64_t&, upcxx::digest&)::<lambda()> >(vrank=6, msg=<lambda()> @ 0x00007fff5fbff2f0)>) at vranks_threads.hpp:21
    frame #15: 0x00000001000035dd uts_threads`uts_parallel(par_node_n=0x00007fff5fbff3b8, par_hash=0x00007fff5fbff3a0) at uts.cpp:141
    frame #16: 0x0000000100002f6d uts_threads`operator(__closure=0x00007fff5fbff4c0, vrank_me1=0, vrank_n1=10) at uts.cpp:51
    frame #17: 0x0000000100003f4e uts_threads`operator(__closure=0x00007fff5fbff430, vrank_me=0) at vranks_threads.hpp:54
    frame #18: 0x0000000100004093 uts_threads`::spawn<main()::<lambda(int, int)> >(fn=<lambda(int, int)> @ 0x00007fff5fbff4c0)>) at vranks_threads.hpp:60
    frame #19: 0x0000000100003063 uts_threads`main at uts.cpp:43
    frame #20: 0x00007fff98622235 libdyld.dylib`start + 1

* thread #2, stop reason = signal SIGABRT
  * frame #0: 0x00007fff98750d42 libsystem_kernel.dylib`__pthread_kill + 10
    frame #1: 0x00007fff9883e457 libsystem_pthread.dylib`pthread_kill + 90
    frame #2: 0x00007fff986b6420 libsystem_c.dylib`abort + 129
    frame #3: 0x00000001000ca28c libstdc++.6.dylib`__gnu_cxx::__verbose_terminate_handler() + 220
    frame #4: 0x00000001000c7c28 libstdc++.6.dylib`__cxxabiv1::__terminate(void (*)()) + 8
    frame #5: 0x00000001000c7c73 libstdc++.6.dylib`std::terminate() + 19
    frame #6: 0x000000010016465e libstdc++.6.dylib`execute_native_thread_routine + 62
    frame #7: 0x00007fff9883b93b libsystem_pthread.dylib`_pthread_body + 180
    frame #8: 0x00007fff9883b887 libsystem_pthread.dylib`_pthread_start + 286
    frame #9: 0x00007fff9883b08d libsystem_pthread.dylib`thread_start + 13

  thread #3
    frame #0: 0x00007fff98750d42 libsystem_kernel.dylib`__pthread_kill + 10
    frame #1: 0x00007fff9883e457 libsystem_pthread.dylib`pthread_kill + 90
    frame #2: 0x00007fff986b6420 libsystem_c.dylib`abort + 129
    frame #3: 0x00000001000ca28c libstdc++.6.dylib`__gnu_cxx::__verbose_terminate_handler() + 220
    frame #4: 0x00000001000c7c28 libstdc++.6.dylib`__cxxabiv1::__terminate(void (*)()) + 8
    frame #5: 0x00000001000c7c73 libstdc++.6.dylib`std::terminate() + 19
    frame #6: 0x000000010016465e libstdc++.6.dylib`execute_native_thread_routine + 62
    frame #7: 0x00007fff9883b93b libsystem_pthread.dylib`_pthread_body + 180
    frame #8: 0x00007fff9883b887 libsystem_pthread.dylib`_pthread_start + 286
    frame #9: 0x00007fff9883b08d libsystem_pthread.dylib`thread_start + 13

  thread #4
    frame #0: 0x00007fff9874943a libsystem_kernel.dylib`swtch_pri + 10
    frame #1: 0x00007fff9883c479 libsystem_pthread.dylib`sched_yield + 11
    frame #2: 0x0000000100002e57 uts_threads`vranks::progress() at vranks_threads.hpp:35
    frame #3: 0x0000000100003c16 uts_threads`qd_progress(local_quiescence=true) at uts.cpp:258
    frame #4: 0x0000000100003615 uts_threads`uts_parallel(par_node_n=0x000070000f43fd68, par_hash=0x000070000f43fd50) at uts.cpp:150
    frame #5: 0x0000000100002f6d uts_threads`operator(__closure=0x00007fff5fbff4c0, vrank_me1=2, vrank_n1=10) at uts.cpp:51
    frame #6: 0x0000000100003f4e uts_threads`operator(__closure=0x0000000100402940, vrank_me=2) at vranks_threads.hpp:54
    frame #7: 0x000000010000550c uts_threads`::__invoke_impl<void, vranks::spawn(Fn) [with Fn = main()::<lambda(int, int)>]::<lambda(int)>, int>((null)=__invoke_other @ 0x000070000f43fe00, __f=0x0000000100402940, (null)=<unavailable>, __args#0=0x0000000100402938)> &, int &) at invoke.h:60
    frame #8: 0x0000000100004891 uts_threads`::__invoke<vranks::spawn(Fn) [with Fn = main()::<lambda(int, int)>]::<lambda(int)>, int>(__fn=0x0000000100402940, (null)=<unavailable>, __args#0=0x0000000100402938)> &, int &) at invoke.h:95
    frame #9: 0x000000010000771f uts_threads`thread::_Invoker<std::tuple<vranks::spawn(Fn) [with Fn = main()::<lambda(int, int)>]::<lambda(int)>, int> >::_M_invoke<0, 1>(this=0x0000000100402938, (null)=_Index_tuple<0, 1> @ 0x000070000f43fe80) const at thread:234
    frame #10: 0x00000001000076d5 uts_threads`thread::_Invoker<std::tuple<vranks::spawn(Fn) [with Fn = main()::<lambda(int, int)>]::<lambda(int)>, int> >::operator(this=0x0000000100402938)() const at thread:243
    frame #11: 0x0000000100007670 uts_threads`thread::_State_impl<std::thread::_Invoker<std::tuple<vranks::spawn(Fn) [with Fn = main()::<lambda(int, int)>]::<lambda(int)>, int> > >::_M_run(this=0x0000000100402930) const at thread:186
    frame #12: 0x000000010016462f libstdc++.6.dylib`execute_native_thread_routine + 15
    frame #13: 0x00007fff9883b93b libsystem_pthread.dylib`_pthread_body + 180
    frame #14: 0x00007fff9883b887 libsystem_pthread.dylib`_pthread_start + 286
    frame #15: 0x00007fff9883b08d libsystem_pthread.dylib`thread_start + 13

  thread #5
    frame #0: 0x00007fff98750d42 libsystem_kernel.dylib`__pthread_kill + 10
    frame #1: 0x00007fff9883e457 libsystem_pthread.dylib`pthread_kill + 90
    frame #2: 0x00007fff986b6420 libsystem_c.dylib`abort + 129
    frame #3: 0x00000001000ca28c libstdc++.6.dylib`__gnu_cxx::__verbose_terminate_handler() + 220
    frame #4: 0x00000001000c7c28 libstdc++.6.dylib`__cxxabiv1::__terminate(void (*)()) + 8
    frame #5: 0x00000001000c7c73 libstdc++.6.dylib`std::terminate() + 19
    frame #6: 0x000000010016465e libstdc++.6.dylib`execute_native_thread_routine + 62
    frame #7: 0x00007fff9883b93b libsystem_pthread.dylib`_pthread_body + 180
    frame #8: 0x00007fff9883b887 libsystem_pthread.dylib`_pthread_start + 286
    frame #9: 0x00007fff9883b08d libsystem_pthread.dylib`thread_start + 13

  thread #6
    frame #0: 0x00007fff9874943a libsystem_kernel.dylib`swtch_pri + 10
    frame #1: 0x00007fff9883c479 libsystem_pthread.dylib`sched_yield + 11
    frame #2: 0x0000000100002e57 uts_threads`vranks::progress() at vranks_threads.hpp:35
    frame #3: 0x0000000100003c16 uts_threads`qd_progress(local_quiescence=true) at uts.cpp:258
    frame #4: 0x0000000100003615 uts_threads`uts_parallel(par_node_n=0x000070000f545d68, par_hash=0x000070000f545d50) at uts.cpp:150
    frame #5: 0x0000000100002f6d uts_threads`operator(__closure=0x00007fff5fbff4c0, vrank_me1=4, vrank_n1=10) at uts.cpp:51
    frame #6: 0x0000000100003f4e uts_threads`operator(__closure=0x00000001004020a0, vrank_me=4) at vranks_threads.hpp:54
    frame #7: 0x000000010000550c uts_threads`::__invoke_impl<void, vranks::spawn(Fn) [with Fn = main()::<lambda(int, int)>]::<lambda(int)>, int>((null)=__invoke_other @ 0x000070000f545e00, __f=0x00000001004020a0, (null)=<unavailable>, __args#0=0x0000000100402098)> &, int &) at invoke.h:60
    frame #8: 0x0000000100004891 uts_threads`::__invoke<vranks::spawn(Fn) [with Fn = main()::<lambda(int, int)>]::<lambda(int)>, int>(__fn=0x00000001004020a0, (null)=<unavailable>, __args#0=0x0000000100402098)> &, int &) at invoke.h:95
    frame #9: 0x000000010000771f uts_threads`thread::_Invoker<std::tuple<vranks::spawn(Fn) [with Fn = main()::<lambda(int, int)>]::<lambda(int)>, int> >::_M_invoke<0, 1>(this=0x0000000100402098, (null)=_Index_tuple<0, 1> @ 0x000070000f545e80) const at thread:234
    frame #10: 0x00000001000076d5 uts_threads`thread::_Invoker<std::tuple<vranks::spawn(Fn) [with Fn = main()::<lambda(int, int)>]::<lambda(int)>, int> >::operator(this=0x0000000100402098)() const at thread:243
    frame #11: 0x0000000100007670 uts_threads`thread::_State_impl<std::thread::_Invoker<std::tuple<vranks::spawn(Fn) [with Fn = main()::<lambda(int, int)>]::<lambda(int)>, int> > >::_M_run(this=0x0000000100402090) const at thread:186
    frame #12: 0x000000010016462f libstdc++.6.dylib`execute_native_thread_routine + 15
    frame #13: 0x00007fff9883b93b libsystem_pthread.dylib`_pthread_body + 180
    frame #14: 0x00007fff9883b887 libsystem_pthread.dylib`_pthread_start + 286
    frame #15: 0x00007fff9883b08d libsystem_pthread.dylib`thread_start + 13

  thread #7
    frame #0: 0x00007fff9886678a libunwind.dylib`libunwind::LocalAddressSpace::getEncodedP(unsigned long long&, unsigned long long, unsigned char) + 246
    frame #1: 0x00007fff98869dc0 libunwind.dylib`libunwind::CFI_Parser<libunwind::LocalAddressSpace>::parseCIE(libunwind::LocalAddressSpace&, unsigned long long, libunwind::CFI_Parser<libunwind::LocalAddressSpace>::CIE_Info*) + 690
    frame #2: 0x00007fff9886995b libunwind.dylib`libunwind::CFI_Parser<libunwind::LocalAddressSpace>::findFDE(libunwind::LocalAddressSpace&, unsigned long long, unsigned long long, unsigned int, unsigned long long, libunwind::CFI_Parser<libunwind::LocalAddressSpace>::FDE_Info*, libunwind::CFI_Parser<libunwind::LocalAddressSpace>::CIE_Info*) + 155
    frame #3: 0x00007fff9886979d libunwind.dylib`libunwind::UnwindCursor<libunwind::LocalAddressSpace, libunwind::Registers_x86_64>::getInfoFromDwarfSection(unsigned long long, unsigned long long, unsigned long long, unsigned int, unsigned int) + 227
    frame #4: 0x00007fff98865a9f libunwind.dylib`libunwind::UnwindCursor<libunwind::LocalAddressSpace, libunwind::Registers_x86_64>::setInfoBasedOnIPRegister(bool) + 235
    frame #5: 0x00007fff988660e5 libunwind.dylib`libunwind::UnwindCursor<libunwind::LocalAddressSpace, libunwind::Registers_x86_64>::step() + 133
    frame #6: 0x00007fff9886586f libunwind.dylib`_Unwind_RaiseException + 175
    frame #7: 0x00000001000c7de2 libstdc++.6.dylib`__cxa_throw + 98
    frame #8: 0x000000010016455a libstdc++.6.dylib`std::__throw_system_error(int) + 122
    frame #9: 0x00000001000009cb uts_threads`std::mutex::lock(this=0x0000000100500618) at std_mutex.h:107
    frame #10: 0x00000001000011f3 uts_threads`upcxx::lpc_inbox_base<2, true>::lock_guard::lock_guard(this=0x000070000f5c8b28, m=0x0000000100500618) at lpc_inbox.hpp:66
    frame #11: 0x000000010000119b uts_threads`upcxx::lpc_inbox_base<2, true>::locked_scope(this=0x00000001005005f8) at lpc_inbox.hpp:79
    frame #12: 0x0000000100005f5c uts_threads`lpc_inbox<2, true>::send<qd_reduce(this=0x00000001005005f8, q=1, fn=0x000070000f5c8ba0)::<lambda()> >(int, <lambda()> &) const at lpc_inbox.hpp:136
    frame #13: 0x0000000100004da7 uts_threads`persona::lpc_ff<qd_reduce(uint64_t, uint64_t, int)::<lambda()> >(this=0x00000001005005b8, fn=<lambda()> @ 0x000070000f5c8ba0)>) const at persona.hpp:99
    frame #14: 0x000000010000442c uts_threads`::send<qd_reduce(uint64_t, uint64_t, int)::<lambda()> >(vrank=4, msg=<lambda()> @ 0x000070000f5c8bf0)>) at vranks_threads.hpp:21
    frame #15: 0x0000000100003bf6 uts_threads`qd_reduce(send_n=0, recv_n=0, from=5) at uts.cpp:250
    frame #16: 0x0000000100003c9d uts_threads`qd_progress(local_quiescence=true) at uts.cpp:264
    frame #17: 0x0000000100003615 uts_threads`uts_parallel(par_node_n=0x000070000f5c8d68, par_hash=0x000070000f5c8d50) at uts.cpp:150
    frame #18: 0x0000000100002f6d uts_threads`operator(__closure=0x00007fff5fbff4c0, vrank_me1=5, vrank_n1=10) at uts.cpp:51
    frame #19: 0x0000000100003f4e uts_threads`operator(__closure=0x00000001004020d0, vrank_me=5) at vranks_threads.hpp:54
    frame #20: 0x000000010000550c uts_threads`::__invoke_impl<void, vranks::spawn(Fn) [with Fn = main()::<lambda(int, int)>]::<lambda(int)>, int>((null)=__invoke_other @ 0x000070000f5c8e00, __f=0x00000001004020d0, (null)=<unavailable>, __args#0=0x00000001004020c8)> &, int &) at invoke.h:60
    frame #21: 0x0000000100004891 uts_threads`::__invoke<vranks::spawn(Fn) [with Fn = main()::<lambda(int, int)>]::<lambda(int)>, int>(__fn=0x00000001004020d0, (null)=<unavailable>, __args#0=0x00000001004020c8)> &, int &) at invoke.h:95
    frame #22: 0x000000010000771f uts_threads`thread::_Invoker<std::tuple<vranks::spawn(Fn) [with Fn = main()::<lambda(int, int)>]::<lambda(int)>, int> >::_M_invoke<0, 1>(this=0x00000001004020c8, (null)=_Index_tuple<0, 1> @ 0x000070000f5c8e80) const at thread:234
    frame #23: 0x00000001000076d5 uts_threads`thread::_Invoker<std::tuple<vranks::spawn(Fn) [with Fn = main()::<lambda(int, int)>]::<lambda(int)>, int> >::operator(this=0x00000001004020c8)() const at thread:243
    frame #24: 0x0000000100007670 uts_threads`thread::_State_impl<std::thread::_Invoker<std::tuple<vranks::spawn(Fn) [with Fn = main()::<lambda(int, int)>]::<lambda(int)>, int> > >::_M_run(this=0x00000001004020c0) const at thread:186
    frame #25: 0x000000010016462f libstdc++.6.dylib`execute_native_thread_routine + 15
    frame #26: 0x00007fff9883b93b libsystem_pthread.dylib`_pthread_body + 180
    frame #27: 0x00007fff9883b887 libsystem_pthread.dylib`_pthread_start + 286
    frame #28: 0x00007fff9883b08d libsystem_pthread.dylib`thread_start + 13

  thread #8
    frame #0: 0x00007fff9874943a libsystem_kernel.dylib`swtch_pri + 10
    frame #1: 0x00007fff9883c479 libsystem_pthread.dylib`sched_yield + 11
    frame #2: 0x0000000100002e57 uts_threads`vranks::progress() at vranks_threads.hpp:35
    frame #3: 0x0000000100003c16 uts_threads`qd_progress(local_quiescence=true) at uts.cpp:258
    frame #4: 0x0000000100003615 uts_threads`uts_parallel(par_node_n=0x000070000f64bd68, par_hash=0x000070000f64bd50) at uts.cpp:150
    frame #5: 0x0000000100002f6d uts_threads`operator(__closure=0x00007fff5fbff4c0, vrank_me1=6, vrank_n1=10) at uts.cpp:51
    frame #6: 0x0000000100003f4e uts_threads`operator(__closure=0x0000000100402100, vrank_me=6) at vranks_threads.hpp:54
    frame #7: 0x000000010000550c uts_threads`::__invoke_impl<void, vranks::spawn(Fn) [with Fn = main()::<lambda(int, int)>]::<lambda(int)>, int>((null)=__invoke_other @ 0x000070000f64be00, __f=0x0000000100402100, (null)=<unavailable>, __args#0=0x00000001004020f8)> &, int &) at invoke.h:60
    frame #8: 0x0000000100004891 uts_threads`::__invoke<vranks::spawn(Fn) [with Fn = main()::<lambda(int, int)>]::<lambda(int)>, int>(__fn=0x0000000100402100, (null)=<unavailable>, __args#0=0x00000001004020f8)> &, int &) at invoke.h:95
    frame #9: 0x000000010000771f uts_threads`thread::_Invoker<std::tuple<vranks::spawn(Fn) [with Fn = main()::<lambda(int, int)>]::<lambda(int)>, int> >::_M_invoke<0, 1>(this=0x00000001004020f8, (null)=_Index_tuple<0, 1> @ 0x000070000f64be80) const at thread:234
    frame #10: 0x00000001000076d5 uts_threads`thread::_Invoker<std::tuple<vranks::spawn(Fn) [with Fn = main()::<lambda(int, int)>]::<lambda(int)>, int> >::operator(this=0x00000001004020f8)() const at thread:243
    frame #11: 0x0000000100007670 uts_threads`thread::_State_impl<std::thread::_Invoker<std::tuple<vranks::spawn(Fn) [with Fn = main()::<lambda(int, int)>]::<lambda(int)>, int> > >::_M_run(this=0x00000001004020f0) const at thread:186
    frame #12: 0x000000010016462f libstdc++.6.dylib`execute_native_thread_routine + 15
    frame #13: 0x00007fff9883b93b libsystem_pthread.dylib`_pthread_body + 180
    frame #14: 0x00007fff9883b887 libsystem_pthread.dylib`_pthread_start + 286
    frame #15: 0x00007fff9883b08d libsystem_pthread.dylib`thread_start + 13

  thread #9
    frame #0: 0x00007fff98750d42 libsystem_kernel.dylib`__pthread_kill + 10
    frame #1: 0x00007fff9883e457 libsystem_pthread.dylib`pthread_kill + 90
    frame #2: 0x00007fff986b6420 libsystem_c.dylib`abort + 129
    frame #3: 0x00000001000ca28c libstdc++.6.dylib`__gnu_cxx::__verbose_terminate_handler() + 220
    frame #4: 0x00000001000c7c28 libstdc++.6.dylib`__cxxabiv1::__terminate(void (*)()) + 8
    frame #5: 0x00000001000c7c73 libstdc++.6.dylib`std::terminate() + 19
    frame #6: 0x000000010016465e libstdc++.6.dylib`execute_native_thread_routine + 62
    frame #7: 0x00007fff9883b93b libsystem_pthread.dylib`_pthread_body + 180
    frame #8: 0x00007fff9883b887 libsystem_pthread.dylib`_pthread_start + 286
    frame #9: 0x00007fff9883b08d libsystem_pthread.dylib`thread_start + 13

  thread #10
    frame #0: 0x00007fff9874943a libsystem_kernel.dylib`swtch_pri + 10
    frame #1: 0x00007fff9883c479 libsystem_pthread.dylib`sched_yield + 11
    frame #2: 0x0000000100002e57 uts_threads`vranks::progress() at vranks_threads.hpp:35
    frame #3: 0x0000000100003c16 uts_threads`qd_progress(local_quiescence=true) at uts.cpp:258
    frame #4: 0x0000000100003615 uts_threads`uts_parallel(par_node_n=0x000070000f751d68, par_hash=0x000070000f751d50) at uts.cpp:150
    frame #5: 0x0000000100002f6d uts_threads`operator(__closure=0x00007fff5fbff4c0, vrank_me1=8, vrank_n1=10) at uts.cpp:51
    frame #6: 0x0000000100003f4e uts_threads`operator(__closure=0x0000000100402180, vrank_me=8) at vranks_threads.hpp:54
    frame #7: 0x000000010000550c uts_threads`::__invoke_impl<void, vranks::spawn(Fn) [with Fn = main()::<lambda(int, int)>]::<lambda(int)>, int>((null)=__invoke_other @ 0x000070000f751e00, __f=0x0000000100402180, (null)=<unavailable>, __args#0=0x0000000100402178)> &, int &) at invoke.h:60
    frame #8: 0x0000000100004891 uts_threads`::__invoke<vranks::spawn(Fn) [with Fn = main()::<lambda(int, int)>]::<lambda(int)>, int>(__fn=0x0000000100402180, (null)=<unavailable>, __args#0=0x0000000100402178)> &, int &) at invoke.h:95
    frame #9: 0x000000010000771f uts_threads`thread::_Invoker<std::tuple<vranks::spawn(Fn) [with Fn = main()::<lambda(int, int)>]::<lambda(int)>, int> >::_M_invoke<0, 1>(this=0x0000000100402178, (null)=_Index_tuple<0, 1> @ 0x000070000f751e80) const at thread:234
    frame #10: 0x00000001000076d5 uts_threads`thread::_Invoker<std::tuple<vranks::spawn(Fn) [with Fn = main()::<lambda(int, int)>]::<lambda(int)>, int> >::operator(this=0x0000000100402178)() const at thread:243
    frame #11: 0x0000000100007670 uts_threads`thread::_State_impl<std::thread::_Invoker<std::tuple<vranks::spawn(Fn) [with Fn = main()::<lambda(int, int)>]::<lambda(int)>, int> > >::_M_run(this=0x0000000100402170) const at thread:186
    frame #12: 0x000000010016462f libstdc++.6.dylib`execute_native_thread_routine + 15
    frame #13: 0x00007fff9883b93b libsystem_pthread.dylib`_pthread_body + 180
    frame #14: 0x00007fff9883b887 libsystem_pthread.dylib`_pthread_start + 286
    frame #15: 0x00007fff9883b08d libsystem_pthread.dylib`thread_start + 13

and uts_hybrid:

[0] * thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
[0]   * frame #0: 0x00007fff9874943a libsystem_kernel.dylib`swtch_pri + 10
[0]     frame #1: 0x00007fff9883c479 libsystem_pthread.dylib`sched_yield + 11
[0]     frame #2: 0x0000000102420892 uts_hybrid`gasneti_system_redirected_coprocess(cmd="/usr/bin/lldb -p 41882 -o 'bt all' -o quit", stdout_fd=3) at gasnet_tools.c:1025
[0]     frame #3: 0x0000000102420c50 uts_hybrid`gasneti_bt_lldb(fd=3) at gasnet_tools.c:1135
[0]     frame #4: 0x000000010242129f uts_hybrid`gasneti_print_backtrace(fd=2) at gasnet_tools.c:1470
[0]     frame #5: 0x0000000102421825 uts_hybrid`_gasneti_print_backtrace_ifenabled(fd=2) at gasnet_tools.c:1561
[0]     frame #6: 0x000000010256e554 uts_hybrid`gasneti_defaultSignalHandler(sig=6) at gasnet_internal.c:1422
[0]     frame #7: 0x00007fff98831b3a libsystem_platform.dylib`_sigtramp + 26
[0]     frame #8: 0x00007fff98750d43 libsystem_kernel.dylib`__pthread_kill + 11
[0]     frame #9: 0x00007fff9883e457 libsystem_pthread.dylib`pthread_kill + 90
[0]     frame #10: 0x00007fff986b6420 libsystem_c.dylib`abort + 129
[0]     frame #11: 0x00000001027612c4 libstdc++.6.dylib`__gnu_cxx::__verbose_terminate_handler() + 276
[0]     frame #12: 0x000000010275ec28 libstdc++.6.dylib`__cxxabiv1::__terminate(void (*)()) + 8
[0]     frame #13: 0x000000010275ec73 libstdc++.6.dylib`std::terminate() + 19
[0]     frame #14: 0x000000010275edef libstdc++.6.dylib`__cxa_throw + 111
[0]     frame #15: 0x00000001027fb55a libstdc++.6.dylib`std::__throw_system_error(int) + 122
[0]     frame #16: 0x00000001023be15f uts_hybrid`std::mutex::lock(this=0x00007fbe38c02f28) at std_mutex.h:107
[0]     frame #17: 0x00000001023e3c03 uts_hybrid`upcxx::lpc_inbox_base<2, true>::lock_guard::lock_guard(this=0x00007fff5d843148, m=0x00007fbe38c02f28) at lpc_inbox.hpp:66
[0]     frame #18: 0x00000001023be5e1 uts_hybrid`upcxx::lpc_inbox_base<2, true>::locked_scope(this=0x00007fbe38c02f08) at lpc_inbox.hpp:79
[0]     frame #19: 0x00000001023d3ede uts_hybrid`lpc_inbox<2, true>::send<uts_parallel(this=0x00007fbe38c02f08, q=1, fn=0x00007fff5d8431c0)::<lambda()> >(int, <lambda()> &) const at lpc_inbox.hpp:136
[0]     frame #20: 0x00000001023d2f45 uts_hybrid`persona::lpc_ff<uts_parallel(uint64_t&, upcxx::digest&)::<lambda()> >(this=0x00007fbe38c02ec8, fn=<lambda()> @ 0x00007fff5d8431c0)>) const at persona.hpp:99
[0]     frame #21: 0x00000001023d2469 uts_hybrid`operator(__closure=0x00007fff5d843210) at vranks_hybrid.hpp:33
[0]     frame #22: 0x00000001023d6435 uts_hybrid`commanding<vranks::send(int, Fn) [with Fn = uts_parallel(uint64_t&, upcxx::digest&)::<lambda()>]::<lambda()> >::execute(r=0x00007fff5d8432c0) at command.hpp:46
[0]     frame #23: 0x00000001023d5f42 uts_hybrid`::command_executor<vranks::send(int, Fn) [with Fn = uts_parallel(uint64_t&, upcxx::digest&)::<lambda()>]::<lambda()> >(r=0x00007fff5d8432c0) at command.hpp:70
[0]     frame #24: 0x00000001023d8883 uts_hybrid`upcxx::command_execute(r=0x00007fff5d8432c0) at command.hpp:76
[0]     frame #25: 0x00000001023bf26e uts_hybrid`upcxx::backend::gasnet::rpc_inbox::burst(this=0x000000010262dfa0, burst_n=100) at rpc_inbox.cpp:15
[0]     frame #26: 0x00000001023e1d7e uts_hybrid`operator(__closure=0x00007fff5d843398) at runtime.cpp:367
[0]     frame #27: 0x00000001023e2d8b uts_hybrid`::persona_as_top<upcxx::progress(upcxx::progress_level)::<lambda()> >(p=0x00000001026367a0, fn=0x00007fff5d843398)> &) at persona.hpp:321
[0]     frame #28: 0x00000001023e1e75 uts_hybrid`upcxx::progress(level=user) at runtime.cpp:365
[0]     frame #29: 0x00000001023d0f5e uts_hybrid`vranks::progress() at vranks_hybrid.hpp:38
[0]     frame #30: 0x00000001023d1d08 uts_hybrid`qd_progress(local_quiescence=true) at uts.cpp:258
[0]     frame #31: 0x00000001023d1707 uts_hybrid`uts_parallel(par_node_n=0x00007fff5d8434b8, par_hash=0x00007fff5d8434a0) at uts.cpp:150
[0]     frame #32: 0x00000001023d105f uts_hybrid`operator(__closure=0x00007fff5d8435e0, vrank_me1=0, vrank_n1=4) at uts.cpp:51
[0]     frame #33: 0x00000001023d2012 uts_hybrid`operator(__closure=0x00007fff5d843550, thread_me=0) at vranks_hybrid.hpp:58
[0]     frame #34: 0x00000001023d21de uts_hybrid`::spawn<main()::<lambda(int, int)> >(fn=<lambda(int, int)> @ 0x00007fff5d8435e0)>) at vranks_hybrid.hpp:71
[0]     frame #35: 0x00000001023d1155 uts_hybrid`main at uts.cpp:43
[0]     frame #36: 0x00007fff98622235 libdyld.dylib`start + 1
[0] 
[0]   thread #2
[0]     frame #0: 0x00007fff98750c22 libsystem_kernel.dylib`__psynch_mutexwait + 10
[0]     frame #1: 0x00007fff9883bdfa libsystem_pthread.dylib`_pthread_mutex_lock_wait + 100
[0]     frame #2: 0x000000010257211e uts_hybrid`_gasneti_memcheck_one(curloc="/usr/local/upc/testers64/EX-sierra-smp-gcc-pshm/runtime/inst/dbg/include/gasnet_help.h:683") at gasnet_internal.c:2726
[0]     frame #3: 0x00000001023db52b uts_hybrid`::_gasneti_AMPoll() at gasnet_help.h:683
[0]     frame #4: 0x00000001023db7de uts_hybrid`::_gasnet_AMPoll() at gasnet_help.h:800
[0]     frame #5: 0x00000001023e1e4d uts_hybrid`upcxx::progress(level=user) at runtime.cpp:355
[0]     frame #6: 0x00000001023d0f5e uts_hybrid`vranks::progress() at vranks_hybrid.hpp:38
[0]     frame #7: 0x00000001023d1d08 uts_hybrid`qd_progress(local_quiescence=true) at uts.cpp:258
[0]     frame #8: 0x00000001023d1707 uts_hybrid`uts_parallel(par_node_n=0x000070000db68d48, par_hash=0x000070000db68d30) at uts.cpp:150
[0]     frame #9: 0x00000001023d105f uts_hybrid`operator(__closure=0x00007fff5d8435e0, vrank_me1=1, vrank_n1=4) at uts.cpp:51
[0]     frame #10: 0x00000001023d2012 uts_hybrid`operator(__closure=0x00007fbe38c02630, thread_me=1) at vranks_hybrid.hpp:58
[0]     frame #11: 0x00000001023d392c uts_hybrid`::__invoke_impl<void, vranks::spawn(Fn) [with Fn = main()::<lambda(int, int)>]::<lambda(int)>, int>((null)=__invoke_other @ 0x000070000db68e00, __f=0x00007fbe38c02630, (null)=<unavailable>, __args#0=0x00007fbe38c02628)> &, int &) at invoke.h:60
[0]     frame #12: 0x00000001023d2b69 uts_hybrid`::__invoke<vranks::spawn(Fn) [with Fn = main()::<lambda(int, int)>]::<lambda(int)>, int>(__fn=0x00007fbe38c02630, (null)=<unavailable>, __args#0=0x00007fbe38c02628)> &, int &) at invoke.h:95
[0]     frame #13: 0x00000001023d8011 uts_hybrid`thread::_Invoker<std::tuple<vranks::spawn(Fn) [with Fn = main()::<lambda(int, int)>]::<lambda(int)>, int> >::_M_invoke<0, 1>(this=0x00007fbe38c02628, (null)=_Index_tuple<0, 1> @ 0x000070000db68e80) const at thread:234
[0]     frame #14: 0x00000001023d7fc7 uts_hybrid`thread::_Invoker<std::tuple<vranks::spawn(Fn) [with Fn = main()::<lambda(int, int)>]::<lambda(int)>, int> >::operator(this=0x00007fbe38c02628)() const at thread:243
[0]     frame #15: 0x00000001023d7f62 uts_hybrid`thread::_State_impl<std::thread::_Invoker<std::tuple<vranks::spawn(Fn) [with Fn = main()::<lambda(int, int)>]::<lambda(int)>, int> > >::_M_run(this=0x00007fbe38c02620) const at thread:186
[0]     frame #16: 0x00000001027fb62f libstdc++.6.dylib`execute_native_thread_routine + 15
[0]     frame #17: 0x00007fff9883b93b libsystem_pthread.dylib`_pthread_body + 180
[0]     frame #18: 0x00007fff9883b887 libsystem_pthread.dylib`_pthread_start + 286
[0]     frame #19: 0x00007fff9883b08d libsystem_pthread.dylib`thread_start + 13
[0] 
[0]   thread #3
[0]     frame #0: 0x00007fff98750c22 libsystem_kernel.dylib`__psynch_mutexwait + 10
[0]     frame #1: 0x00007fff9883bdfa libsystem_pthread.dylib`_pthread_mutex_lock_wait + 100
[0]     frame #2: 0x000000010257211e uts_hybrid`_gasneti_memcheck_one(curloc="/usr/local/upc/testers64/EX-sierra-smp-gcc-pshm/runtime/inst/dbg/include/gasnet_help.h:683") at gasnet_internal.c:2726
[0]     frame #3: 0x00000001023db52b uts_hybrid`::_gasneti_AMPoll() at gasnet_help.h:683
[0]     frame #4: 0x00000001023db7de uts_hybrid`::_gasnet_AMPoll() at gasnet_help.h:800
[0]     frame #5: 0x00000001023e1e4d uts_hybrid`upcxx::progress(level=user) at runtime.cpp:355
[0]     frame #6: 0x00000001023d0f5e uts_hybrid`vranks::progress() at vranks_hybrid.hpp:38
[0]     frame #7: 0x00000001023d1d08 uts_hybrid`qd_progress(local_quiescence=true) at uts.cpp:258
[0]     frame #8: 0x00000001023d1707 uts_hybrid`uts_parallel(par_node_n=0x000070000dbebd48, par_hash=0x000070000dbebd30) at uts.cpp:150
[0]     frame #9: 0x00000001023d105f uts_hybrid`operator(__closure=0x00007fff5d8435e0, vrank_me1=2, vrank_n1=4) at uts.cpp:51
[0]     frame #10: 0x00000001023d2012 uts_hybrid`operator(__closure=0x00007fbe38c03360, thread_me=2) at vranks_hybrid.hpp:58
[0]     frame #11: 0x00000001023d392c uts_hybrid`::__invoke_impl<void, vranks::spawn(Fn) [with Fn = main()::<lambda(int, int)>]::<lambda(int)>, int>((null)=__invoke_other @ 0x000070000dbebe00, __f=0x00007fbe38c03360, (null)=<unavailable>, __args#0=0x00007fbe38c03358)> &, int &) at invoke.h:60
[0]     frame #12: 0x00000001023d2b69 uts_hybrid`::__invoke<vranks::spawn(Fn) [with Fn = main()::<lambda(int, int)>]::<lambda(int)>, int>(__fn=0x00007fbe38c03360, (null)=<unavailable>, __args#0=0x00007fbe38c03358)> &, int &) at invoke.h:95
[0]     frame #13: 0x00000001023d8011 uts_hybrid`thread::_Invoker<std::tuple<vranks::spawn(Fn) [with Fn = main()::<lambda(int, int)>]::<lambda(int)>, int> >::_M_invoke<0, 1>(this=0x00007fbe38c03358, (null)=_Index_tuple<0, 1> @ 0x000070000dbebe80) const at thread:234
[0]     frame #14: 0x00000001023d7fc7 uts_hybrid`thread::_Invoker<std::tuple<vranks::spawn(Fn) [with Fn = main()::<lambda(int, int)>]::<lambda(int)>, int> >::operator(this=0x00007fbe38c03358)() const at thread:243
[0]     frame #15: 0x00000001023d7f62 uts_hybrid`thread::_State_impl<std::thread::_Invoker<std::tuple<vranks::spawn(Fn) [with Fn = main()::<lambda(int, int)>]::<lambda(int)>, int> > >::_M_run(this=0x00007fbe38c03350) const at thread:186
[0]     frame #16: 0x00000001027fb62f libstdc++.6.dylib`execute_native_thread_routine + 15
[0]     frame #17: 0x00007fff9883b93b libsystem_pthread.dylib`_pthread_body + 180
[0]     frame #18: 0x00007fff9883b887 libsystem_pthread.dylib`_pthread_start + 286
[0]     frame #19: 0x00007fff9883b08d libsystem_pthread.dylib`thread_start + 13
[0] 
[0]   thread #4
[0]     frame #0: 0x00000001029f1730 libgcc_s.1.dylib`__emutls_get_address
[0]     frame #1: 0x00000001023be1d4 uts_hybrid`_ZTWN5upcxx6detail14tl_top_personaE + 16
[0]     frame #2: 0x00000001023e2dae uts_hybrid`::persona_foreach_active<upcxx::progress(upcxx::progress_level)::<lambda(upcxx::persona&)> >(body=0x000070000dc6ec30)> &) at persona.hpp:307
[0]     frame #3: 0x00000001023e1e91 uts_hybrid`upcxx::progress(level=user) at runtime.cpp:371
[0]     frame #4: 0x00000001023d0f5e uts_hybrid`vranks::progress() at vranks_hybrid.hpp:38
[0]     frame #5: 0x00000001023d1d08 uts_hybrid`qd_progress(local_quiescence=true) at uts.cpp:258
[0]     frame #6: 0x00000001023d1707 uts_hybrid`uts_parallel(par_node_n=0x000070000dc6ed48, par_hash=0x000070000dc6ed30) at uts.cpp:150
[0]     frame #7: 0x00000001023d105f uts_hybrid`operator(__closure=0x00007fff5d8435e0, vrank_me1=3, vrank_n1=4) at uts.cpp:51
[0]     frame #8: 0x00000001023d2012 uts_hybrid`operator(__closure=0x00007fbe38c03390, thread_me=3) at vranks_hybrid.hpp:58
[0]     frame #9: 0x00000001023d392c uts_hybrid`::__invoke_impl<void, vranks::spawn(Fn) [with Fn = main()::<lambda(int, int)>]::<lambda(int)>, int>((null)=__invoke_other @ 0x000070000dc6ee00, __f=0x00007fbe38c03390, (null)=<unavailable>, __args#0=0x00007fbe38c03388)> &, int &) at invoke.h:60
[0]     frame #10: 0x00000001023d2b69 uts_hybrid`::__invoke<vranks::spawn(Fn) [with Fn = main()::<lambda(int, int)>]::<lambda(int)>, int>(__fn=0x00007fbe38c03390, (null)=<unavailable>, __args#0=0x00007fbe38c03388)> &, int &) at invoke.h:95
[0]     frame #11: 0x00000001023d8011 uts_hybrid`thread::_Invoker<std::tuple<vranks::spawn(Fn) [with Fn = main()::<lambda(int, int)>]::<lambda(int)>, int> >::_M_invoke<0, 1>(this=0x00007fbe38c03388, (null)=_Index_tuple<0, 1> @ 0x000070000dc6ee80) const at thread:234
[0]     frame #12: 0x00000001023d7fc7 uts_hybrid`thread::_Invoker<std::tuple<vranks::spawn(Fn) [with Fn = main()::<lambda(int, int)>]::<lambda(int)>, int> >::operator(this=0x00007fbe38c03388)() const at thread:243
[0]     frame #13: 0x00000001023d7f62 uts_hybrid`thread::_State_impl<std::thread::_Invoker<std::tuple<vranks::spawn(Fn) [with Fn = main()::<lambda(int, int)>]::<lambda(int)>, int> > >::_M_run(this=0x00007fbe38c03380) const at thread:186
[0]     frame #14: 0x00000001027fb62f libstdc++.6.dylib`execute_native_thread_routine + 15
[0]     frame #15: 0x00007fff9883b93b libsystem_pthread.dylib`_pthread_body + 180
[0]     frame #16: 0x00007fff9883b887 libsystem_pthread.dylib`_pthread_start + 286
[0]     frame #17: 0x00007fff9883b08d libsystem_pthread.dylib`thread_start + 13

Seems all are dropping dead inside a upcxx std::mutex::lock.

2017-09-26T17:17:01+00:00

Former user Account Deleted
I have a feeling this is thanks to -lpthread as opposed to doing whatever the correct thing is.
- 2017-09-26T17:21:45+00:00

Dan Bonachea reporter

I have a feeling this is thanks to -lpthread as opposed to doing whatever the correct thing is.

Good idea - however when I manually remove -lpthread and add -D_REENTRANT (the "correct thing" for Darwin), I still get what looks like the same crash from uts_threads.

FWIW GASNet threaded tests pass fine using this compiler, although they don't use C++ std::mutex.

{ihill ~/UPC/upcxx} rm -Rf .nobs ; env DBGSYM=1 OPTLEV=0 CC="gcc-7 -D_REENTRANT" CXX="g++-7 -D_REENTRANT" nobs exe test/uts/uts_threads.cpp               
g++-7 -D_REENTRANT -std=c++11 -D_GNU_SOURCE=1 -I/Users/bonachea/UPC/upcxx/.nobs/art/6b2d3ca0e48dbdaa5214ab78e87a9321d0b268fb -MM -MT x /Users/bonachea/UPC/upcxx/test/uts/uts_threads.cpp

g++-7 -D_REENTRANT -std=c++11 -D_GNU_SOURCE=1 -I/Users/bonachea/UPC/upcxx/.nobs/art/6b2d3ca0e48dbdaa5214ab78e87a9321d0b268fb -O0 -g -Wall -c /Users/bonachea/UPC/upcxx/test/uts/uts_threads.cpp -o /Users/bonachea/UPC/upcxx/.nobs/art/7a6c42ef3f8e7582b9b47a769cc93a0c67dbc5a9.uts_threads.cpp.o

g++-7 -D_REENTRANT -std=c++11 -D_GNU_SOURCE=1 -I/Users/bonachea/UPC/upcxx/.nobs/art/003cce40553652379c63db173a73d73f9c08046b -MM -MT x /Users/bonachea/UPC/upcxx/src/persona.cpp

g++-7 -D_REENTRANT -std=c++11 -D_GNU_SOURCE=1 -I/Users/bonachea/UPC/upcxx/.nobs/art/003cce40553652379c63db173a73d73f9c08046b -MM -MT x /Users/bonachea/UPC/upcxx/src/future/core.cpp

g++-7 -D_REENTRANT -std=c++11 -D_GNU_SOURCE=1 -I/Users/bonachea/UPC/upcxx/.nobs/art/003cce40553652379c63db173a73d73f9c08046b -MM -MT x /Users/bonachea/UPC/upcxx/src/diagnostic.cpp

g++-7 -D_REENTRANT -std=c++11 -D_GNU_SOURCE=1 -I/Users/bonachea/UPC/upcxx/.nobs/art/003cce40553652379c63db173a73d73f9c08046b -MM -MT x /Users/bonachea/UPC/upcxx/src/lpc_inbox.cpp

g++-7 -D_REENTRANT -std=c++11 -D_GNU_SOURCE=1 -I/Users/bonachea/UPC/upcxx/.nobs/art/003cce40553652379c63db173a73d73f9c08046b -MM -MT x /Users/bonachea/UPC/upcxx/src/digest.cpp

g++-7 -D_REENTRANT -std=c++11 -D_GNU_SOURCE=1 -I/Users/bonachea/UPC/upcxx/.nobs/art/003cce40553652379c63db173a73d73f9c08046b -O0 -g -Wall -c /Users/bonachea/UPC/upcxx/src/future/core.cpp -o /Users/bonachea/UPC/upcxx/.nobs/art/79ceed1b592c532a1953505487fa81c1b01aaecf.core.cpp.o

g++-7 -D_REENTRANT -std=c++11 -D_GNU_SOURCE=1 -I/Users/bonachea/UPC/upcxx/.nobs/art/003cce40553652379c63db173a73d73f9c08046b -O0 -g -Wall -c /Users/bonachea/UPC/upcxx/src/persona.cpp -o /Users/bonachea/UPC/upcxx/.nobs/art/8ceeec9dbf07666314d3c810421fc7ca0caf5fad.persona.cpp.o

g++-7 -D_REENTRANT -std=c++11 -D_GNU_SOURCE=1 -I/Users/bonachea/UPC/upcxx/.nobs/art/003cce40553652379c63db173a73d73f9c08046b -O0 -g -Wall -c /Users/bonachea/UPC/upcxx/src/diagnostic.cpp -o /Users/bonachea/UPC/upcxx/.nobs/art/f98885a94f6b11c757785556c9c8f534a642e250.diagnostic.cpp.o

g++-7 -D_REENTRANT -std=c++11 -D_GNU_SOURCE=1 -I/Users/bonachea/UPC/upcxx/.nobs/art/003cce40553652379c63db173a73d73f9c08046b -O0 -g -Wall -c /Users/bonachea/UPC/upcxx/src/digest.cpp -o /Users/bonachea/UPC/upcxx/.nobs/art/e93b948a9526c5c9a7037e4fd457b02937e2ef67.digest.cpp.o

g++-7 -D_REENTRANT -std=c++11 -D_GNU_SOURCE=1 -I/Users/bonachea/UPC/upcxx/.nobs/art/003cce40553652379c63db173a73d73f9c08046b -O0 -g -Wall -c /Users/bonachea/UPC/upcxx/src/lpc_inbox.cpp -o /Users/bonachea/UPC/upcxx/.nobs/art/1ea3731099ed8edd05bdec902fcc0dc79e2d7596.lpc_inbox.cpp.o

g++-7 -D_REENTRANT -o /Users/bonachea/UPC/upcxx/.nobs/art/50a9b6e533efb7900baddf6a6405c015dfd0d352.x /Users/bonachea/UPC/upcxx/.nobs/art/79ceed1b592c532a1953505487fa81c1b01aaecf.core.cpp.o /Users/bonachea/UPC/upcxx/.nobs/art/f98885a94f6b11c757785556c9c8f534a642e250.diagnostic.cpp.o /Users/bonachea/UPC/upcxx/.nobs/art/8ceeec9dbf07666314d3c810421fc7ca0caf5fad.persona.cpp.o /Users/bonachea/UPC/upcxx/.nobs/art/e93b948a9526c5c9a7037e4fd457b02937e2ef67.digest.cpp.o /Users/bonachea/UPC/upcxx/.nobs/art/7a6c42ef3f8e7582b9b47a769cc93a0c67dbc5a9.uts_threads.cpp.o /Users/bonachea/UPC/upcxx/.nobs/art/1ea3731099ed8edd05bdec902fcc0dc79e2d7596.lpc_inbox.cpp.o -lpthread

/Users/bonachea/UPC/upcxx/.nobs/art/50a9b6e533efb7900baddf6a6405c015dfd0d352.x{ihill ~/UPC/upcxx} 
{ihill ~/UPC/upcxx} g++-7 -D_REENTRANT -o uts_threads /Users/bonachea/UPC/upcxx/.nobs/art/79ceed1b592c532a1953505487fa81c1b01aaecf.core.cpp.o /Users/bonachea/UPC/upcxx/.nobs/art/f98885a94f6b11c757785556c9c8f534a642e250.diagnostic.cpp.o /Users/bonachea/UPC/upcxx/.nobs/art/8ceeec9dbf07666314d3c810421fc7ca0caf5fad.persona.cpp.o /Users/bonachea/UPC/upcxx/.nobs/art/e93b948a9526c5c9a7037e4fd457b02937e2ef67.digest.cpp.o /Users/bonachea/UPC/upcxx/.nobs/art/7a6c42ef3f8e7582b9b47a769cc93a0c67dbc5a9.uts_threads.cpp.o /Users/bonachea/UPC/upcxx/.nobs/art/1ea3731099ed8edd05bdec902fcc0dc79e2d7596.lpc_inbox.cpp.o 
{ihill ~/UPC/upcxx} ./uts_threads 
Using default UTS_WIDTH=100
terminate called after throwing an instance of 'std::system_error'
  what():  terminate called recursively
Invalid argumentAbort
{ihill ~/UPC/upcxx} otool -L uts_threads
uts_threads:
        /usr/local/opt/gcc/lib/gcc/7/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.24.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1238.60.2)
        /usr/local/lib/gcc/7/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)

2017-09-26T17:44:51+00:00

Dan Bonachea reporter
Note the backtrace shows libsystem_pthread.dylib which does not appear in the otool output, so it seems possible the executable is dlopen()ing the a libpthread which is not compatible with the g+++-7.2 install.

CC: @PHHargrove
- 2017-09-26T22:38:54+00:00

Paul Hargrove

Dan and I have verified that gcc is not providing its own libpthreads.
So, the difference between the backtrace and otool output that Dan observed is not an indication of a problem.

With gcc-7.2.0 I saw the same failure as Dan reported.

I have installed gcc-6 and gcc-5 on the "sierra" system we have shared access to.

With gcc-6.4.0 I also see the same failure.

With gcc-5.4.0 I see a completely new failure mode (assertion failure):

sierra:upcxx phargrov$ git describe --always
4dc6b14
sierra:upcxx phargrov$ export DBGSYM=1 OPTLEV=0 CC="gcc-5" CXX="g++-5"
sierra:upcxx phargrov$ nobs run test/uts/uts_threads.cpp
g++-5 -std=c++11 -D_GNU_SOURCE=1 -I/Users/phargrov/upcxx/.nobs/art/944cf6b97bee949d7acb942ecfe0e15c67f2d6f5 -MM -MT x /Users/phargrov/upcxx/test/uts/uts_threads.cpp

g++-5 -std=c++11 -D_GNU_SOURCE=1 -I/Users/phargrov/upcxx/.nobs/art/944cf6b97bee949d7acb942ecfe0e15c67f2d6f5 -O0 -g -Wall -c /Users/phargrov/upcxx/test/uts/uts_threads.cpp -o /Users/phargrov/upcxx/.nobs/art/a3da3ea4c587e6ba12dd9997b2c227abe1ae9f9f.uts_threads.cpp.o

g++-5 -std=c++11 -D_GNU_SOURCE=1 -I/Users/phargrov/upcxx/.nobs/art/d82abdf8840aa7a18c561a62501b68454bb80d47 -MM -MT x /Users/phargrov/upcxx/src/persona.cpp

g++-5 -std=c++11 -D_GNU_SOURCE=1 -I/Users/phargrov/upcxx/.nobs/art/d82abdf8840aa7a18c561a62501b68454bb80d47 -MM -MT x /Users/phargrov/upcxx/src/future/core.cpp

g++-5 -std=c++11 -D_GNU_SOURCE=1 -I/Users/phargrov/upcxx/.nobs/art/d82abdf8840aa7a18c561a62501b68454bb80d47 -MM -MT x /Users/phargrov/upcxx/src/diagnostic.cpp

g++-5 -std=c++11 -D_GNU_SOURCE=1 -I/Users/phargrov/upcxx/.nobs/art/d82abdf8840aa7a18c561a62501b68454bb80d47 -MM -MT x /Users/phargrov/upcxx/src/lpc_inbox.cpp

g++-5 -std=c++11 -D_GNU_SOURCE=1 -I/Users/phargrov/upcxx/.nobs/art/d82abdf8840aa7a18c561a62501b68454bb80d47 -MM -MT x /Users/phargrov/upcxx/src/digest.cpp

g++-5 -std=c++11 -D_GNU_SOURCE=1 -I/Users/phargrov/upcxx/.nobs/art/d82abdf8840aa7a18c561a62501b68454bb80d47 -O0 -g -Wall -c /Users/phargrov/upcxx/src/future/core.cpp -o /Users/phargrov/upcxx/.nobs/art/bdbe989d50667c7ae22f53d6eb81783fefa89e9a.core.cpp.o

g++-5 -std=c++11 -D_GNU_SOURCE=1 -I/Users/phargrov/upcxx/.nobs/art/d82abdf8840aa7a18c561a62501b68454bb80d47 -O0 -g -Wall -c /Users/phargrov/upcxx/src/persona.cpp -o /Users/phargrov/upcxx/.nobs/art/846b7db26693a3cdcb9ed4256d43c4639c2d35e1.persona.cpp.o

g++-5 -std=c++11 -D_GNU_SOURCE=1 -I/Users/phargrov/upcxx/.nobs/art/d82abdf8840aa7a18c561a62501b68454bb80d47 -O0 -g -Wall -c /Users/phargrov/upcxx/src/diagnostic.cpp -o /Users/phargrov/upcxx/.nobs/art/065296032c91cb752ade035c7f1b3c2db15ee733.diagnostic.cpp.o

g++-5 -std=c++11 -D_GNU_SOURCE=1 -I/Users/phargrov/upcxx/.nobs/art/d82abdf8840aa7a18c561a62501b68454bb80d47 -O0 -g -Wall -c /Users/phargrov/upcxx/src/lpc_inbox.cpp -o /Users/phargrov/upcxx/.nobs/art/6472a41ec43fb2e18ca0f2902eabcee2308fd051.lpc_inbox.cpp.o

g++-5 -std=c++11 -D_GNU_SOURCE=1 -I/Users/phargrov/upcxx/.nobs/art/d82abdf8840aa7a18c561a62501b68454bb80d47 -O0 -g -Wall -c /Users/phargrov/upcxx/src/digest.cpp -o /Users/phargrov/upcxx/.nobs/art/e5bb4a966ee383bd572cde9d48fe428f44a2eb6f.digest.cpp.o

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
g++-5 -std=c++11 -D_GNU_SOURCE=1 -I/Users/phargrov/upcxx/.nobs/art/d82abdf8840aa7a18c561a62501b68454bb80d47 -O0 -g -Wall -c /Users/phargrov/upcxx/src/future/core.cpp -o /Users/phargrov/upcxx/.nobs/art/bdbe989d50667c7ae22f53d6eb81783fefa89e9a.core.cpp.o

/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccXfhb5q.s:13:11: warning: section "__textcoal_nt" is deprecated
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccXfhb5q.s:13:11: note: change section name to "__text"
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
g++-5 -std=c++11 -D_GNU_SOURCE=1 -I/Users/phargrov/upcxx/.nobs/art/d82abdf8840aa7a18c561a62501b68454bb80d47 -O0 -g -Wall -c /Users/phargrov/upcxx/src/persona.cpp -o /Users/phargrov/upcxx/.nobs/art/846b7db26693a3cdcb9ed4256d43c4639c2d35e1.persona.cpp.o

/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccb7rDPx.s:80:11: warning: section "__textcoal_nt" is deprecated
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccb7rDPx.s:80:11: note: change section name to "__text"
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccb7rDPx.s:108:11: warning: section "__textcoal_nt" is deprecated
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccb7rDPx.s:108:11: note: change section name to "__text"
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccb7rDPx.s:170:11: warning: section "__textcoal_nt" is deprecated
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccb7rDPx.s:170:11: note: change section name to "__text"
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccb7rDPx.s:438:11: warning: section "__textcoal_nt" is deprecated
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccb7rDPx.s:438:11: note: change section name to "__text"
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccb7rDPx.s:709:11: warning: section "__textcoal_nt" is deprecated
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccb7rDPx.s:709:11: note: change section name to "__text"
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccb7rDPx.s:847:11: warning: section "__textcoal_nt" is deprecated
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccb7rDPx.s:847:11: note: change section name to "__text"
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccb7rDPx.s:903:11: warning: section "__textcoal_nt" is deprecated
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccb7rDPx.s:903:11: note: change section name to "__text"
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccb7rDPx.s:1003:11: warning: section "__textcoal_nt" is deprecated
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccb7rDPx.s:1003:11: note: change section name to "__text"
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccb7rDPx.s:1103:11: warning: section "__textcoal_nt" is deprecated
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccb7rDPx.s:1103:11: note: change section name to "__text"
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccb7rDPx.s:1205:11: warning: section "__textcoal_nt" is deprecated
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccb7rDPx.s:1205:11: note: change section name to "__text"
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccb7rDPx.s:1269:11: warning: section "__textcoal_nt" is deprecated
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccb7rDPx.s:1269:11: note: change section name to "__text"
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
g++-5 -std=c++11 -D_GNU_SOURCE=1 -I/Users/phargrov/upcxx/.nobs/art/d82abdf8840aa7a18c561a62501b68454bb80d47 -O0 -g -Wall -c /Users/phargrov/upcxx/src/lpc_inbox.cpp -o /Users/phargrov/upcxx/.nobs/art/6472a41ec43fb2e18ca0f2902eabcee2308fd051.lpc_inbox.cpp.o

/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//cc6GbCx6.s:85:11: warning: section "__textcoal_nt" is deprecated
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//cc6GbCx6.s:85:11: note: change section name to "__text"
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//cc6GbCx6.s:743:11: warning: section "__textcoal_nt" is deprecated
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//cc6GbCx6.s:743:11: note: change section name to "__text"
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//cc6GbCx6.s:809:11: warning: section "__textcoal_nt" is deprecated
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//cc6GbCx6.s:809:11: note: change section name to "__text"
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//cc6GbCx6.s:911:11: warning: section "__textcoal_nt" is deprecated
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//cc6GbCx6.s:911:11: note: change section name to "__text"
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
g++-5 -std=c++11 -D_GNU_SOURCE=1 -I/Users/phargrov/upcxx/.nobs/art/d82abdf8840aa7a18c561a62501b68454bb80d47 -O0 -g -Wall -c /Users/phargrov/upcxx/src/diagnostic.cpp -o /Users/phargrov/upcxx/.nobs/art/065296032c91cb752ade035c7f1b3c2db15ee733.diagnostic.cpp.o

/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccKidzHo.s:6:11: warning: section "__textcoal_nt" is deprecated
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccKidzHo.s:6:11: note: change section name to "__text"
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
g++-5 -std=c++11 -D_GNU_SOURCE=1 -I/Users/phargrov/upcxx/.nobs/art/944cf6b97bee949d7acb942ecfe0e15c67f2d6f5 -O0 -g -Wall -c /Users/phargrov/upcxx/test/uts/uts_threads.cpp -o /Users/phargrov/upcxx/.nobs/art/a3da3ea4c587e6ba12dd9997b2c227abe1ae9f9f.uts_threads.cpp.o

/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:183:11: warning: section "__textcoal_nt" is deprecated
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:183:11: note: change section name to "__text"
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:279:11: warning: section "__textcoal_nt" is deprecated
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:279:11: note: change section name to "__text"
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:362:11: warning: section "__textcoal_nt" is deprecated
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:362:11: note: change section name to "__text"
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:461:11: warning: section "__textcoal_nt" is deprecated
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:461:11: note: change section name to "__text"
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:569:11: warning: section "__textcoal_nt" is deprecated
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:569:11: note: change section name to "__text"
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:623:11: warning: section "__textcoal_nt" is deprecated
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:623:11: note: change section name to "__text"
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:766:11: warning: section "__textcoal_nt" is deprecated
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:766:11: note: change section name to "__text"
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:2341:11: warning: section "__textcoal_nt" is deprecated
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:2341:11: note: change section name to "__text"
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:2421:11: warning: section "__textcoal_nt" is deprecated
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:2421:11: note: change section name to "__text"
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:2606:11: warning: section "__textcoal_nt" is deprecated
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:2606:11: note: change section name to "__text"
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:2978:11: warning: section "__textcoal_nt" is deprecated
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:2978:11: note: change section name to "__text"
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:3009:11: warning: section "__textcoal_nt" is deprecated
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:3009:11: note: change section name to "__text"
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:3053:11: warning: section "__textcoal_nt" is deprecated
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:3053:11: note: change section name to "__text"
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:3194:11: warning: section "__textcoal_nt" is deprecated
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:3194:11: note: change section name to "__text"
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:3391:11: warning: section "__textcoal_nt" is deprecated
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:3391:11: note: change section name to "__text"
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:3494:11: warning: section "__textcoal_nt" is deprecated
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:3494:11: note: change section name to "__text"
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:3645:11: warning: section "__textcoal_nt" is deprecated
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:3645:11: note: change section name to "__text"
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:3734:11: warning: section "__textcoal_nt" is deprecated
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:3734:11: note: change section name to "__text"
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:3866:11: warning: section "__textcoal_nt" is deprecated
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:3866:11: note: change section name to "__text"
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:4081:11: warning: section "__textcoal_nt" is deprecated
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:4081:11: note: change section name to "__text"
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:4125:11: warning: section "__textcoal_nt" is deprecated
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:4125:11: note: change section name to "__text"
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:4185:11: warning: section "__textcoal_nt" is deprecated
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:4185:11: note: change section name to "__text"
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:4362:11: warning: section "__textcoal_nt" is deprecated
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:4362:11: note: change section name to "__text"
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:4458:11: warning: section "__textcoal_nt" is deprecated
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:4458:11: note: change section name to "__text"
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:4937:11: warning: section "__textcoal_nt" is deprecated
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:4937:11: note: change section name to "__text"
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:5167:11: warning: section "__textcoal_nt" is deprecated
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:5167:11: note: change section name to "__text"
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:5195:11: warning: section "__textcoal_nt" is deprecated
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:5195:11: note: change section name to "__text"
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:5464:11: warning: section "__textcoal_nt" is deprecated
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:5464:11: note: change section name to "__text"
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:5506:11: warning: section "__textcoal_nt" is deprecated
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:5506:11: note: change section name to "__text"
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:5639:11: warning: section "__textcoal_nt" is deprecated
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:5639:11: note: change section name to "__text"
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:5689:11: warning: section "__textcoal_nt" is deprecated
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:5689:11: note: change section name to "__text"
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:5892:11: warning: section "__textcoal_nt" is deprecated
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:5892:11: note: change section name to "__text"
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:6197:11: warning: section "__textcoal_nt" is deprecated
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:6197:11: note: change section name to "__text"
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:6337:11: warning: section "__textcoal_nt" is deprecated
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:6337:11: note: change section name to "__text"
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:6568:11: warning: section "__textcoal_nt" is deprecated
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:6568:11: note: change section name to "__text"
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:7289:11: warning: section "__textcoal_nt" is deprecated
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:7289:11: note: change section name to "__text"
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:7787:11: warning: section "__textcoal_nt" is deprecated
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:7787:11: note: change section name to "__text"
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:8056:11: warning: section "__textcoal_nt" is deprecated
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:8056:11: note: change section name to "__text"
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:8119:11: warning: section "__textcoal_nt" is deprecated
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:8119:11: note: change section name to "__text"
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:8580:11: warning: section "__textcoal_nt" is deprecated
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:8580:11: note: change section name to "__text"
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:9015:11: warning: section "__textcoal_nt" is deprecated
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:9015:11: note: change section name to "__text"
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:9498:11: warning: section "__textcoal_nt" is deprecated
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:9498:11: note: change section name to "__text"
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:9818:11: warning: section "__textcoal_nt" is deprecated
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:9818:11: note: change section name to "__text"
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:10228:11: warning: section "__textcoal_nt" is deprecated
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:10228:11: note: change section name to "__text"
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:10778:11: warning: section "__textcoal_nt" is deprecated
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:10778:11: note: change section name to "__text"
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:11018:11: warning: section "__textcoal_nt" is deprecated
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:11018:11: note: change section name to "__text"
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:11180:11: warning: section "__textcoal_nt" is deprecated
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:11180:11: note: change section name to "__text"
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:11444:11: warning: section "__textcoal_nt" is deprecated
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:11444:11: note: change section name to "__text"
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:11555:11: warning: section "__textcoal_nt" is deprecated
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:11555:11: note: change section name to "__text"
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:11702:11: warning: section "__const_coal" is deprecated
        .section __DATA,__const_coal,coalesced
                 ^      ~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:11702:11: note: change section name to "__const"
        .section __DATA,__const_coal,coalesced
                 ^      ~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:11772:11: warning: section "__const_coal" is deprecated
        .section __DATA,__const_coal,coalesced
                 ^      ~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:11772:11: note: change section name to "__const"
        .section __DATA,__const_coal,coalesced
                 ^      ~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:11803:11: warning: section "__const_coal" is deprecated
        .section __DATA,__const_coal,coalesced
                 ^      ~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:11803:11: note: change section name to "__const"
        .section __DATA,__const_coal,coalesced
                 ^      ~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:11810:11: warning: section "__const_coal" is deprecated
        .section __TEXT,__const_coal,coalesced
                 ^      ~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:11810:11: note: change section name to "__const"
        .section __TEXT,__const_coal,coalesced
                 ^      ~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:11856:11: warning: section "__const_coal" is deprecated
        .section __DATA,__const_coal,coalesced
                 ^      ~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:11856:11: note: change section name to "__const"
        .section __DATA,__const_coal,coalesced
                 ^      ~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:11863:11: warning: section "__const_coal" is deprecated
        .section __TEXT,__const_coal,coalesced
                 ^      ~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:11863:11: note: change section name to "__const"
        .section __TEXT,__const_coal,coalesced
                 ^      ~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:11879:11: warning: section "__const_coal" is deprecated
        .section __DATA,__const_coal,coalesced
                 ^      ~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:11879:11: note: change section name to "__const"
        .section __DATA,__const_coal,coalesced
                 ^      ~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:11886:11: warning: section "__const_coal" is deprecated
        .section __TEXT,__const_coal,coalesced
                 ^      ~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:11886:11: note: change section name to "__const"
        .section __TEXT,__const_coal,coalesced
                 ^      ~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:11892:11: warning: section "__const_coal" is deprecated
        .section __DATA,__const_coal,coalesced
                 ^      ~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:11892:11: note: change section name to "__const"
        .section __DATA,__const_coal,coalesced
                 ^      ~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:11900:11: warning: section "__const_coal" is deprecated
        .section __TEXT,__const_coal,coalesced
                 ^      ~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:11900:11: note: change section name to "__const"
        .section __TEXT,__const_coal,coalesced
                 ^      ~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:12240:11: warning: section "__textcoal_nt" is deprecated
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:12240:11: note: change section name to "__text"
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:12284:11: warning: section "__textcoal_nt" is deprecated
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:12284:11: note: change section name to "__text"
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:12418:11: warning: section "__textcoal_nt" is deprecated
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:12418:11: note: change section name to "__text"
        .section __TEXT,__textcoal_nt,coalesced,pure_instructions
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:12494:11: warning: section "__const_coal" is deprecated
        .section __DATA,__const_coal,coalesced
                 ^      ~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:12494:11: note: change section name to "__const"
        .section __DATA,__const_coal,coalesced
                 ^      ~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:12501:11: warning: section "__const_coal" is deprecated
        .section __TEXT,__const_coal,coalesced
                 ^      ~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:12501:11: note: change section name to "__const"
        .section __TEXT,__const_coal,coalesced
                 ^      ~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:12670:11: warning: section "__datacoal_nt" is deprecated
        .section __DATA,__datacoal_nt,coalesced
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:12670:11: note: change section name to "__data"
        .section __DATA,__datacoal_nt,coalesced
                 ^      ~~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:12679:11: warning: section "__const_coal" is deprecated
        .section __TEXT,__const_coal,coalesced
                 ^      ~~~~~~~~~~~~
/var/folders/tg/pmhkfgfn1ddfz49g20syzj6c0000gn/T//ccSwENvH.s:12679:11: note: change section name to "__const"
        .section __TEXT,__const_coal,coalesced
                 ^      ~~~~~~~~~~~~

g++-5 -o /Users/phargrov/upcxx/.nobs/art/9de8b4cf1a84bd016970e02083a7c85ea115e3f7.x /Users/phargrov/upcxx/.nobs/art/bdbe989d50667c7ae22f53d6eb81783fefa89e9a.core.cpp.o /Users/phargrov/upcxx/.nobs/art/846b7db26693a3cdcb9ed4256d43c4639c2d35e1.persona.cpp.o /Users/phargrov/upcxx/.nobs/art/6472a41ec43fb2e18ca0f2902eabcee2308fd051.lpc_inbox.cpp.o /Users/phargrov/upcxx/.nobs/art/065296032c91cb752ade035c7f1b3c2db15ee733.diagnostic.cpp.o /Users/phargrov/upcxx/.nobs/art/e5bb4a966ee383bd572cde9d48fe428f44a2eb6f.digest.cpp.o /Users/phargrov/upcxx/.nobs/art/a3da3ea4c587e6ba12dd9997b2c227abe1ae9f9f.uts_threads.cpp.o -lpthread

Using default UTS_WIDTH=100
UPC++ assertion failure [/Users/phargrov/upcxx/.nobs/art/d82abdf8840aa7a18c561a62501b68454bb80d47/upcxx/persona.hpp:233]
UPC++ assertion failure [/Users/phargrov/upcxx/.nobs/art/d82abdf8840aa7a18c561a62501b68454bb80d47/upcxx/persona.hpp:233]
UPC++ assertion failure [/Users/phargrov/upcxx/.nobs/art/d82abdf8840aa7a18c561a62501b68454bb80d47/upcxx/persona.hpp:233]
UPC++ assertion failure [/Users/phargrov/upcxx/.nobs/art/d82abdf8840aa7a18c561a62501b68454bb80d47/upcxx/persona.hpp:233]
UPC++ assertion failure [/Users/phargrov/upcxx/.nobs/art/d82abdf8840aa7a18c561a62501b68454bb80d47/upcxx/persona.hpp:233]
UPC++ assertion failure [/Users/phargrov/upcxx/.nobs/art/d82abdf8840aa7a18c561a62501b68454bb80d47/upcxx/persona.hpp:233]
UPC++ assertion failure [/Users/phargrov/upcxx/.nobs/art/d82abdf8840aa7a18c561a62501b68454bb80d47/upcxx/persona.hpp:233]
UPC++ assertion failure [/Users/phargrov/upcxx/.nobs/art/d82abdf8840aa7a18c561a62501b68454bb80d47/upcxx/persona.hpp:233]
UPC++ assertion failure [/Users/phargrov/upcxx/.nobs/art/d82abdf8840aa7a18c561a62501b68454bb80d47/upcxx/persona.hpp:233]
Tree size: 2823
Tree hash: {2991098530611839359,11274086010336094944}
Test result: SUCCESS
Illegal instruction: 4

This tests fails that way 9-out-of-10 times for me.
The 10th is the same failure seen with gcc-7 and -6.

Note that on my laptop (using fink instead of homebrew for packages) I see the same behavior with gcc-5 and gcc-6 as reported here. However, I do not see the numerous section-related warnings. So, I don't think they are related.

2017-09-26T23:21:32+00:00

Dan Bonachea reporter
The assertion failure above corresponds to the persona_scope destructor:
```
    ~persona_scope() {
      if(persona_ != nullptr) {
        UPCXX_ASSERT(persona_ == detail::tl_top_persona);
```
so that seems the likely place for John to investigate further.
- 2017-09-27T00:16:39+00:00
Dan Bonachea reporter
- changed title to uts_{threads,hybrid}/lpc_barrier crash on (High)Sierra w/ g++ 7.2.0: what(): Invalid argument
Same failures now also observed on High Sierra with gcc:

More recent results
- 2017-09-28T18:48:46+00:00
Paul Hargrove
Re: "Same failures now also observed on High Sierra with gcc".
This is the same "gcc@7" compiler from homebrew as in the original report (but built for newer Xcode and OS).
So, these failures are not a surprise to me.
- 2017-09-28T18:52:31+00:00
Dan Bonachea reporter
- changed milestone to 2018.03.31 release
- 2017-10-03T00:09:52+00:00
Dan Bonachea reporter
- changed milestone to 2017.12.31 release
- 2017-10-09T19:11:16+00:00
Paul Hargrove
- changed title to uts_{threads,hybrid,omp_ranks}/lpc_barrier crash on (High)Sierra w/ g++ 7.2.0: what(): Invalid argument
Also see this failure mode for the new uts_omp_ranks test.
- 2017-10-13T19:05:24+00:00
john bachan
I have reproduced the recursive termination. But does gdb exist on sierra?
```
sierra:upcxx jdbachan$ gdb
-bash: gdb: command not found
```
- 2018-01-09T01:31:15+00:00
Dan Bonachea reporter
@jdbachan you probably want lldb, which should be installed
- 2018-01-09T01:41:45+00:00
BrianS
you can get gdb installed, but High Sierra makes codesigning gdb difficult and so I have switched to lldb as my debugger on that operating system.

If you want an emacs gud for lldb I can provide one. it isn't perfect but it is functional
- 2018-01-09T01:49:07+00:00
john bachan
It looks like this is a platform issue. I have confirmed that non-trivial constructors for thread_local global variables are not being called. There is some discussion about this feature's support here.

Edit: I diagnosed this by placing output to stderr in persona_scope::persona_scope constructor. No output resulted at anytime even though every thread gets a tl_default_persona_scope defined in src/persona.cpp.

In the near term, instead of coding around broken thread_local, I'm hopeful we can find a not-too-limiting set of constraints for our set of supported OS X platforms. But, I know there is already push for abstracting ourselves away from direct reliance on thread_local and this serves as another motivating case for that approach.
- 2018-01-09T18:23:35+00:00
BrianS
XCode 8 and onward says they support thread local, but this was a bug in gcc 7.2 that we are getting this error. Is gcc failing to print the message to stderr ?
- 2018-01-09T20:06:22+00:00
john bachan
Yes, gcc is failing to print my message to stderr for the case of the thread_local default persona_scope. If I declare another persona_scope on the stack, I do see the stderr message corresponding to just that stack variable printed.
- 2018-01-09T20:10:02+00:00

john bachan

I don't think TLS init is a missing feature in our "intented" configuration, I think this is just a misconfiguration somehow. The executable has a function __tls_init containing the following disassembly:

::__tls_init():
    0x1000126b1 <+0>:   pushq  %rbp
    0x1000126b2 <+1>:   movq   %rsp, %rbp
    0x1000126b5 <+4>:   pushq  %r12
    0x1000126b7 <+6>:   pushq  %rbx
    0x1000126b8 <+7>:   leaq   0xd401(%rip), %rdi        ; __emutls_v.__tls_guard
    0x1000126bf <+14>:  callq  0x100013418               ; symbol stub for: __emutls_get_address
    0x1000126c4 <+19>:  movq   %rax, %rcx
    0x1000126c7 <+22>:  movq   %rcx, %rax
    0x1000126ca <+25>:  movq   %rax, %rsi
    0x1000126cd <+28>:  shrq   $0x3, %rsi
    0x1000126d1 <+32>:  movabsq $0x100000000000, %rdx     ; imm = 0x100000000000 
    0x1000126db <+42>:  addq   %rsi, %rdx
    0x1000126de <+45>:  movzbl (%rdx), %edx
    0x1000126e1 <+48>:  testb  %dl, %dl
    0x1000126e3 <+50>:  setne  %sil
    0x1000126e7 <+54>:  movq   %rax, %rdi
    0x1000126ea <+57>:  andl   $0x7, %edi
    0x1000126ed <+60>:  cmpb   %dl, %dil
    0x1000126f0 <+63>:  setge  %dl
    0x1000126f3 <+66>:  andl   %esi, %edx
    0x1000126f5 <+68>:  testb  %dl, %dl
    0x1000126f7 <+70>:  je     0x100012701               ; <+80> at persona.cpp:16
    0x1000126f9 <+72>:  movq   %rax, %rdi
    0x1000126fc <+75>:  callq  0x100013382               ; symbol stub for: __asan_report_load1
    0x100012701 <+80>:  movzbl (%rcx), %eax
    0x100012704 <+83>:  xorl   $0x1, %eax
    0x100012707 <+86>:  testb  %al, %al
    0x100012709 <+88>:  je     0x1000127c6               ; <+277> at persona.cpp:10
    0x10001270f <+94>:  leaq   0xd3aa(%rip), %rdi        ; __emutls_v.__tls_guard
    0x100012716 <+101>: callq  0x100013418               ; symbol stub for: __emutls_get_address
    0x10001271b <+106>: movq   %rax, %rcx
    0x10001271e <+109>: movq   %rcx, %rax
    0x100012721 <+112>: movq   %rax, %rsi
    0x100012724 <+115>: shrq   $0x3, %rsi
    0x100012728 <+119>: movabsq $0x100000000000, %rdx     ; imm = 0x100000000000 
    0x100012732 <+129>: addq   %rsi, %rdx
    0x100012735 <+132>: movzbl (%rdx), %edx
    0x100012738 <+135>: testb  %dl, %dl
    0x10001273a <+137>: setne  %sil
    0x10001273e <+141>: movq   %rax, %rdi
    0x100012741 <+144>: andl   $0x7, %edi
    0x100012744 <+147>: cmpb   %dl, %dil
    0x100012747 <+150>: setge  %dl
    0x10001274a <+153>: andl   %esi, %edx
    0x10001274c <+155>: testb  %dl, %dl
    0x10001274e <+157>: je     0x100012758               ; <+167> at persona.cpp:16
    0x100012750 <+159>: movq   %rax, %rdi
    0x100012753 <+162>: callq  0x1000133a0               ; symbol stub for: __asan_report_store1
    0x100012758 <+167>: movb   $0x1, (%rcx)
    0x10001275b <+170>: leaq   0xd45e(%rip), %rdi        ; __emutls_v._ZN5upcxx6detail18tl_default_personaE
    0x100012762 <+177>: callq  0x100013418               ; symbol stub for: __emutls_get_address
    0x100012767 <+182>: movq   %rax, %rbx
    0x10001276a <+185>: movq   %rbx, %rdi
    0x10001276d <+188>: callq  0x100012e8a               ; symbol stub for: upcxx::persona::persona()
    0x100012772 <+193>: leaq   -0x12779(%rip), %rdx      ; _mh_execute_header
    0x100012779 <+200>: movq   %rbx, %rsi
    0x10001277c <+203>: movq   0x9885(%rip), %rax        ; (void *)0x0000000100006804: upcxx::persona::~persona() at /Users/jdbachan/upcxx/.nobs/art/a0180e082a7ff360d9b0612ef062e18d412383a3/upcxx/persona.hpp:50
    0x100012783 <+210>: movq   %rax, %rdi
    0x100012786 <+213>: callq  0x10001340c               ; symbol stub for: __cxa_thread_atexit
    0x10001278b <+218>: callq  0x100001282               ; _ZTWN5upcxx6detail18tl_default_personaE
    0x100012790 <+223>: movq   %rax, %r12
    0x100012793 <+226>: leaq   0xd3e6(%rip), %rdi        ; __emutls_v._ZN5upcxx6detail24tl_default_persona_scopeE
    0x10001279a <+233>: callq  0x100013418               ; symbol stub for: __emutls_get_address
    0x10001279f <+238>: movq   %rax, %rbx
    0x1000127a2 <+241>: movq   %r12, %rsi
    0x1000127a5 <+244>: movq   %rbx, %rdi
    0x1000127a8 <+247>: callq  0x100012e06               ; symbol stub for: upcxx::persona_scope::persona_scope(upcxx::persona&)
    0x1000127ad <+252>: leaq   -0x127b4(%rip), %rdx      ; _mh_execute_header
    0x1000127b4 <+259>: movq   %rbx, %rsi
    0x1000127b7 <+262>: movq   0x9842(%rip), %rax        ; (void *)0x0000000100001d7a: upcxx::persona_scope::~persona_scope() at /Users/jdbachan/upcxx/.nobs/art/a0180e082a7ff360d9b0612ef062e18d412383a3/upcxx/persona.hpp:231
    0x1000127be <+269>: movq   %rax, %rdi
    0x1000127c1 <+272>: callq  0x10001340c               ; symbol stub for: __cxa_thread_atexit
    0x1000127c6 <+277>: nop    
    0x1000127c7 <+278>: popq   %rbx
    0x1000127c8 <+279>: popq   %r12
    0x1000127ca <+281>: popq   %rbp
    0x1000127cb <+282>: retq

As you can see, my persona_scope() constructor is being called. I set a breakpiont on __tls_init and as expected it is not being called. This seems like some sort of compile-time/runtime interface mismatch.

2018-01-09T23:43:07+00:00

BrianS

I think I have a different error point in uts_thread . I set a breakpoint in std::terminate gcc7.2 and just udp conduit

* thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 1.1
  * frame #0: 0x00000001000bbc60 libstdc++.6.dylib`std::terminate()
    frame #1: 0x00000001000bbdef libstdc++.6.dylib`__cxa_throw + 111
    frame #2: 0x000000010015855a libstdc++.6.dylib`std::__throw_system_error(int) + 122
    frame #3: 0x0000000100003af5 790b13459e01b03bdd6d8d1b643016376b8b4dc4.x`uts_parallel(unsigned long long&, upcxx::digest&) + 661
    frame #4: 0x0000000100004e6c 790b13459e01b03bdd6d8d1b643016376b8b4dc4.x`main + 588
    frame #5: 0x00007fff513ec115 libdyld.dylib`start + 1
    frame #6: 0x00007fff513ec115 libdyld.dylib`start + 1

2018-01-10T00:29:15+00:00

john bachan
@bvstraalen Can you please verify whether __tls_init is being called. If initialization isn't happening than the crash point isn't really very telling, and on my setup is definitely non-deterministic.
- 2018-01-10T00:46:41+00:00
Paul Hargrove
@jbachan who was the previous comment supposed to be directed to?
- 2018-01-10T05:03:29+00:00
john bachan
Whoops. @bvstraalen
- 2018-01-10T05:09:00+00:00
john bachan
Son of a fricking gun. I added -fextern-tls-init to the compile line and got this friendly message back from the compiler:
```
cc1plus: sorry, unimplemented: external TLS initialization functions not supported on this target
```
This platform intentionally does not have what our current implementation needs. But here's the really funny part. I added this to our definition of default_persona, AND NOW THE PROGRAM WORKS:
```
inline persona& default_persona() {
  __tls_init(); // This should be the compiler's job
  return detail::tl_default_persona;
}
```
The compiler has intentionally been cobbled not to support what we want. I checked the disassembly of __tls_init and it guards itself with a check against a thread local boolean so that after successive calls it becomes a no-op. The code is there to handle TLS construction, all they're missing is sprinkling calls to __tls_init before each thread_local use site. I'm a little pissed off.

Ways to move forward:

1) Abandon support of this gcc on mac platform. Insert configure-time check for this error message in the presence of -fextern-tls-init. Then, nicely inform the user their platform has been sabotaged by malicious high-paid engineers using anti-competitive means to increase clang's market share.

2) Give up on C++11 thread_local of non-POD types. Losing object constructors will be a burden, but its not impossible to encode the datastructures which tend to live in TLS to work with only trivial (literal value) construction. This is also slightly faster since it skips the guarded initialization check that __tls_init does. The ones I can think of now are persona_scope and lpc_inbox. Sadly, I think we also lose the destructors for those objects, but pthreads gives us hooks for registering thread local destructors. Ultimately we have to handle TLS like lowly C-programmers.
- 2018-01-10T17:01:57+00:00
BrianS
Do people get around this with pointers to complex objects and their own bool check for their own initialization?
- 2018-01-10T18:42:37+00:00

john bachan

That's one way to do it. A better way (avoids heap allocation) is:

T& my_tls_var() {
 static __thread typename std::aligned_storage<sizeof(T),alignof(T)>::type storage;
 static __thread bool ready = false;
 if(!ready) {
  ready = true;
  ::new(&storage) T;
  // now have to register its destructor, could use pthread API or __cxa_thread_atexit
 }
 return *reinterpret_cast<T*>(&storage);
}

This is essentially what __tls_init is doing. Except the linker collects all the thread local variables' constructors under one bool flag.

2018-01-10T18:48:56+00:00

Dan Bonachea reporter
Ultimately we have to handle TLS like lowly C-programmers.

Another reminder that lack of portability is only one of the major arguments against relying upon bare thread_local (or even __thread).

The other is that compiler-based TLS is fundamentally incompatible with user-level threading libraries, with whom we claim to interoperate.

See our prior discussion on this topic
- 2018-01-10T19:02:46+00:00
john bachan
Issue ~~#92~~ was marked as a duplicate of this issue.
- 2018-01-10T19:39:23+00:00
BrianS
I think we might want to investigate an abstraction to TLS such that a user can opt for compiler TLS but provide TLS support if they have different needs. The only example I had at hand was QThread, but now it seems like we have another use case for TLS abstraction: When compilers don't support it properly.

Can we wrap out TLS objects in a C++ abstraction that accommodates both modes?
- 2018-01-11T02:33:28+00:00
Dan Bonachea reporter
Can we wrap out TLS objects in a C++ abstraction that accommodates both modes?

@bvstraalen : That's exactly what I'm arguing for.

GASNet-tools already provides the runtime with an abstraction documented in the "Thread-specific data" section of README-tools, although it's a C-based interface so could either be wrapped or used for inspiration for a cleaner interface in C++. Note this interface is not yet implemented over QThreads qthread_get_tasklocal, whose interface differs somewhat from pthread_getspecific.
- 2018-01-11T20:29:56+00:00
BrianS
the compiler-based approach certainly has the opportunity to be more efficient, so I guess I need to understand the performance trade offs. Is this in the critical path of execution.
- 2018-01-12T05:12:01+00:00
Dan Bonachea reporter
- changed milestone to 2018.09.30 release
This issue was discussed in the 1/10/18 meeting, and we resolved to move it to the Sept release.

In particular, the vendor compiler on Apple lacks OpenMP support, so we can't continue to ignore g++ on this platform forever.
- 2018-01-12T06:26:15+00:00
Dan Bonachea reporter
@bvstraalen :

the compiler-based approach certainly has the opportunity to be more efficient

I would not assume that. It's possible to get very similar performance with the right design. For example, gasneti_threadkey_get on a compiler with __thread support is a macro that expands to a simple variable reference to a __thread variable.

However that interface does not try to automatically run constructors on thread creation (it behaves like pthread_getspecific and returns NULL for a value the current thread has not yet set). So this likely means a branch on any reference where the thread might not have previously set a value. Alternatively, we could perform the "is this a new thread" branch at each top-level entry to the runtime (amortizing it over internal uses), with code that inits all the per-thread data when a new thread is encountered, and registers a pthread destructor to clean it up (this is essentially the approach used in GASNet).
- 2018-01-12T06:40:22+00:00
john bachan
- assigned issue to
  
  john bachan
- 2018-01-18T21:26:27+00:00
Dan Bonachea reporter
Issue ~~#120~~ was marked as a duplicate of this issue.
- 2018-02-26T22:06:22+00:00
Dan Bonachea reporter
- changed title to Stop using C++ thread_local: uts_{threads,hybrid,omp_ranks}/lpc_barrier crash on (High)Sierra w/ g++ 7.2.0
- marked as blocker
We now have evidence that the use of the C++ thread_local keyword is breaking the UPC++/gnu parallel backend for a user program on Cori, which makes this a high-priority problem to fix.
- 2018-04-17T00:36:02+00:00
BrianS
I think we will have to switch to the gasnet mechanism until we have robust support in C++ for this capability. It is remarkable that people are not burned by this everywhere.
- 2018-04-17T01:49:43+00:00
Dan Bonachea reporter
- changed status to resolved
Nightly results confirm this was fixed by pull request #28 in bf2ebee
- 2018-04-24T21:11:01+00:00
Log in to comment

Assignee: john bachan

Type: bug

Priority: blocker

Status: resolved

Component: Runtime

Milestone: 2018.09.30 release

Version: Development Branch

Votes: 0

Watchers: 3