1. Berkeley Lab Avatar Berkeley Lab
  2. CRD-CLaSS
  3. upcxx
  4. Issues

Including UPC++ in shared libraries

Issue #321 new
Kenneth Lozes created an issue

Hi All,

I would like to include upcxx in a shared library (.so), but it fails to link because of the missing -fPIC flag. I see in the README-tools file for GASNet that the flag -DGASNETI_FORCE_PIC=1 should be set. Would it be possible to add some options in the upcxx install to set necessary flags so it can be included in shared libraries?

Thanks,
Kenneth

‌

Comments (14)

  1. Paul Hargrove

    Kenneth,

    First, I want to make clear that we do not regularly test UPC++ or GASNet built with -fpic or linked into shared libraries. So, what I will describe below is NOT a use-case we are officially supporting today.

    Building the libraries as position-independent-code

    This is actually the easy part. You simply need to include -fpic in the CC and CXX values used when installing UPC++. For instance:

    $ export CC='gcc -fpic' CXX='g++ -fpic'

    or, if using ibv-conduit with mpi-based job launch:

    $ export CC='gcc -fpic' CXX='mpicxx -fpic'

    Building a shared library

    While python probably has specific requirements (not my area), the following is sufficient to construct a .so file containing UPC++ code:

    $ upcxx -g -shared -o libmyupcxx.so foo.cpp bar.cpp

    Not explicit in this recipe is an important requirement: the UPC++ app code should be entirely within this shared library (the foo.cpp and bar.cpp above). This is necessary because RPC in UPC++ names function objects using their address relative to a specific symbol in the library. This creates the requirement that all function objects invoked via RPC must reside in the same executable linker segment as libupcxx. Additionally, since UPC++ and GASNet are static libraries (archives), the shared library will (at least in general) need to include all the UPC++ code simply to ensure that the necessary objects are copied from the static libs into the shared one.

    Some related info on the RPC naming issue appears in https://bitbucket.org/berkeleylab/upcxx-spec/issues/115

    Job launch

    The upcxx-run script is a wrapper around multiple GASNet-level launchers which cover the various networks we support. Therefore, it wants to examine the executable to determine which network. At least the GASNet-level launcher for ibv-conduit (InfiniBand) also needs to examine the executable. For this reason, you are not going to be able to use our standard tools to launch the python interpreter (which lacks the data these launch tools are looking for).

    Since we provide wrappers, the solution for your case is probably just to drop a level lower and use the mechanisms we use under the covers. Use of upcxx-run options -show and/or -v with a static UPC++ hello world executable compiled against the same network can help you discover those lower-level mechanisms. Additionally, we can provide help with specific cases as needed.

  2. Kenneth Lozes reporter

    Paul,

    Thank you for the thorough response. Indeed, it was easy to compile UPC++ with fPIC so it could be included in my shared library. I wrote a simple python app that uses pybind11 to call the upcxx hello world from the shared library. Running the app without the upcxx-run prints:

    Warning: GASNET_PSHM_NODES not specified: running with 1 node
Hello from 0 of 1

    Which is the same as the your hello_upcxx.cpp test, so that’s promising.

    I only just started looking into upcxx-run, but I noticed the he shared library file also contains the ”GASNetConduitName:” So passing that file to the runscript as well may solve the problem of determining the conduit. I still need to look into the case of the ibv-conduit though.

    In case you are interested here is my hello world app.

    upcxx_pybind.cpp

    #include <pybind11/pybind11.h>
#include <upcxx/backend.hpp>

void hello_world(){
  upcxx::init();
  std::cout << "Hello from "<<upcxx::rank_me()<<" of "<<upcxx::rank_n()<<std::endl;
  upcxx::finalize();
}

PYBIND11_MODULE(upcxx_pybind, m) {
    m.def("hello_world", &hello_world, "");
}

    compiled with:

    upcxx -O3 -Wall -shared -std=c++11 -fPIC `python3 -m pybind11 --includes` upcxx_pybind.cpp -o upcxx_pybind `python3-config --extension-suffix`

    test_upcxx.py

    import upcxx_pybind
upcxx_pybind.hello_world()

    ‌

    ‌

  3. Paul Hargrove

    Kenneth,

    I wanted to follow up to inquire if you had success with upcxx-run and its use with ibv-conduit in particular.
    Your observation that the shared library contains the necessary ident strings sounded promising.

  4. Kenneth Lozes reporter

    Paul,

    Yes, it only took some small changes to the run script to have it read the ident string from the shared library instead. Then my pybinded hello world app worked just fine. I have only gotten the wmp conduit to work so far. I cannot get the ibv conduit to work at all. even for standard compilation and tests. My cluster should have infiniband (and the install script says that infiniband is on), but I cannot ge thte run-tests to work with ibv. I am running with “CONDUIT=ibv CXX=mpicxx ./run-tests” as suggestedby the testing doc. Do you have any suggestions on how to solve the problem?

    thanks,

    ‌

    here is the output in run-tests.err

    /packages/7x/openmpi/agave/3.0.3/gcc/7x/normal/bin/mpicxx -std=c++11 -D_GNU_SOURCE=1 -I/home/klozes/upcxx_agave/.nobs/art/9db4ffff379810eb1de11c4ed3def36aadf88ca4 -DNOBS_DISCOVERY -M -MT x /home/klozes/upcxx_agave/test/atomics.cpp

/packages/7x/openmpi/agave/3.0.3/gcc/7x/normal/bin/mpicxx -std=c++11 -D_GNU_SOURCE=1 -I/home/klozes/upcxx_agave/.nobs/art/97e4ddf0259e2d17955ad33e0b011981a9171c5d -DNOBS_DISCOVERY -M -MT x /home/klozes/upcxx_agave/src/atomic.cpp

/packages/7x/openmpi/agave/3.0.3/gcc/7x/normal/bin/mpicxx -std=c++11 -D_GNU_SOURCE=1 -I/home/klozes/upcxx_agave/.nobs/art/97e4ddf0259e2d17955ad33e0b011981a9171c5d -DNOBS_DISCOVERY -M -MT x /home/klozes/upcxx_agave/src/backend/gasnet/runtime.cpp

/packages/7x/openmpi/agave/3.0.3/gcc/7x/normal/bin/mpicxx -std=c++11 -D_GNU_SOURCE=1 -I/home/klozes/upcxx_agave/.nobs/art/97e4ddf0259e2d17955ad33e0b011981a9171c5d -DNOBS_DISCOVERY -M -MT x /home/klozes/upcxx_agave/src/barrier.cpp

/packages/7x/openmpi/agave/3.0.3/gcc/7x/normal/bin/mpicxx -std=c++11 -D_GNU_SOURCE=1 -I/home/klozes/upcxx_agave/.nobs/art/97e4ddf0259e2d17955ad33e0b011981a9171c5d -DNOBS_DISCOVERY -M -MT x /home/klozes/upcxx_agave/src/broadcast.cpp

/packages/7x/openmpi/agave/3.0.3/gcc/7x/normal/bin/mpicxx -std=c++11 -D_GNU_SOURCE=1 -I/home/klozes/upcxx_agave/.nobs/art/97e4ddf0259e2d17955ad33e0b011981a9171c5d -DNOBS_DISCOVERY -M -MT x /home/klozes/upcxx_agave/src/copy.cpp

/packages/7x/openmpi/agave/3.0.3/gcc/7x/normal/bin/mpicxx -std=c++11 -D_GNU_SOURCE=1 -I/home/klozes/upcxx_agave/.nobs/art/97e4ddf0259e2d17955ad33e0b011981a9171c5d -DNOBS_DISCOVERY -M -MT x /home/klozes/upcxx_agave/src/cuda.cpp

/packages/7x/openmpi/agave/3.0.3/gcc/7x/normal/bin/mpicxx -std=c++11 -D_GNU_SOURCE=1 -I/home/klozes/upcxx_agave/.nobs/art/97e4ddf0259e2d17955ad33e0b011981a9171c5d -DNOBS_DISCOVERY -M -MT x /home/klozes/upcxx_agave/src/os_env.cpp

/packages/7x/openmpi/agave/3.0.3/gcc/7x/normal/bin/mpicxx -std=c++11 -D_GNU_SOURCE=1 -I/home/klozes/upcxx_agave/.nobs/art/97e4ddf0259e2d17955ad33e0b011981a9171c5d -DNOBS_DISCOVERY -M -MT x /home/klozes/upcxx_agave/src/reduce.cpp

/packages/7x/openmpi/agave/3.0.3/gcc/7x/normal/bin/mpicxx -std=c++11 -D_GNU_SOURCE=1 -I/home/klozes/upcxx_agave/.nobs/art/97e4ddf0259e2d17955ad33e0b011981a9171c5d -DNOBS_DISCOVERY -M -MT x /home/klozes/upcxx_agave/src/rget.cpp

/packages/7x/openmpi/agave/3.0.3/gcc/7x/normal/bin/mpicxx -std=c++11 -D_GNU_SOURCE=1 -I/home/klozes/upcxx_agave/.nobs/art/97e4ddf0259e2d17955ad33e0b011981a9171c5d -DNOBS_DISCOVERY -M -MT x /home/klozes/upcxx_agave/src/rput.cpp

/packages/7x/openmpi/agave/3.0.3/gcc/7x/normal/bin/mpicxx -std=c++11 -D_GNU_SOURCE=1 -I/home/klozes/upcxx_agave/.nobs/art/97e4ddf0259e2d17955ad33e0b011981a9171c5d -DNOBS_DISCOVERY -M -MT x /home/klozes/upcxx_agave/src/segment_allocator.cpp

/packages/7x/openmpi/agave/3.0.3/gcc/7x/normal/bin/mpicxx -std=c++11 -D_GNU_SOURCE=1 -I/home/klozes/upcxx_agave/.nobs/art/97e4ddf0259e2d17955ad33e0b011981a9171c5d -DNOBS_DISCOVERY -M -MT x /home/klozes/upcxx_agave/src/vis.cpp

/packages/7x/openmpi/agave/3.0.3/gcc/7x/normal/bin/mpicxx -std=c++11 -D_GNU_SOURCE=1 -I/home/klozes/upcxx_agave/.nobs/art/97e4ddf0259e2d17955ad33e0b011981a9171c5d -DNOBS_DISCOVERY -M -MT x /home/klozes/upcxx_agave/src/backend/gasnet/noise_log.cpp

/packages/7x/openmpi/agave/3.0.3/gcc/7x/normal/bin/mpicxx -std=c++11 -D_GNU_SOURCE=1 -I/home/klozes/upcxx_agave/.nobs/art/9db4ffff379810eb1de11c4ed3def36aadf88ca4 -DUPCXX_ASSERT_ENABLED=1 -DUPCXX_BACKEND=1 -DUPCXX_BACKEND_GASNET_SEQ=1 -DUPCXX_MPSC_QUEUE_ATOMIC=1 -D_GNU_SOURCE=1 -DGASNET_SEQ -D_REENTRANT -I/home/klozes/upcxx_agave/.nobs/art/8f69db21b6e4d63289509f1fa71b908ae62bd051/GASNet-2019.9.0 -I/home/klozes/upcxx_agave/.nobs/art/8f69db21b6e4d63289509f1fa71b908ae62bd051/GASNet-2019.9.0/ibv-conduit -I/home/klozes/upcxx_agave/.nobs/art/8f69db21b6e4d63289509f1fa71b908ae62bd051/GASNet-2019.9.0/other -I/home/klozes/upcxx_agave/.nobs/art/8f69db21b6e4d63289509f1fa71b908ae62bd051/GASNet-2019.9.0/other/firehose -I/home/klozes/upcxx_agave/.nobs/art/8f69db21b6e4d63289509f1fa71b908ae62bd051/GASNet-2019.9.0/extended-ref/vis -I/home/klozes/upcxx_agave/.nobs/art/8f69db21b6e4d63289509f1fa71b908ae62bd051/GASNet-2019.9.0/extended-ref/coll -I/home/klozes/upcxx_agave/.nobs/art/8f69db21b6e4d63289509f1fa71b908ae62bd051/GASNet-2019.9.0/extended-ref/ratomic -I/home/klozes/upcxx_agave/.nobs/art/8f69db21b6e4d63289509f1fa71b908ae62bd051/GASNet-2019.9.0/extended-ref -I/home/klozes/upcxx_agave/.nobs/art/18cf07baa7180bf3776f87808fc219dbb83cc5f5 -DNOBS_DISCOVERY -M -MT x /home/klozes/upcxx_agave/test/atomics.cpp

/packages/7x/openmpi/agave/3.0.3/gcc/7x/normal/bin/mpicxx -std=c++11 -D_GNU_SOURCE=1 -I/home/klozes/upcxx_agave/.nobs/art/9db4ffff379810eb1de11c4ed3def36aadf88ca4 -DUPCXX_ASSERT_ENABLED=1 -DUPCXX_BACKEND=1 -DUPCXX_BACKEND_GASNET_SEQ=1 -DUPCXX_MPSC_QUEUE_ATOMIC=1 -D_GNU_SOURCE=1 -DGASNET_SEQ -D_REENTRANT -I/home/klozes/upcxx_agave/.nobs/art/8f69db21b6e4d63289509f1fa71b908ae62bd051/GASNet-2019.9.0 -I/home/klozes/upcxx_agave/.nobs/art/8f69db21b6e4d63289509f1fa71b908ae62bd051/GASNet-2019.9.0/ibv-conduit -I/home/klozes/upcxx_agave/.nobs/art/8f69db21b6e4d63289509f1fa71b908ae62bd051/GASNet-2019.9.0/other -I/home/klozes/upcxx_agave/.nobs/art/8f69db21b6e4d63289509f1fa71b908ae62bd051/GASNet-2019.9.0/other/firehose -I/home/klozes/upcxx_agave/.nobs/art/8f69db21b6e4d63289509f1fa71b908ae62bd051/GASNet-2019.9.0/extended-ref/vis -I/home/klozes/upcxx_agave/.nobs/art/8f69db21b6e4d63289509f1fa71b908ae62bd051/GASNet-2019.9.0/extended-ref/coll -I/home/klozes/upcxx_agave/.nobs/art/8f69db21b6e4d63289509f1fa71b908ae62bd051/GASNet-2019.9.0/extended-ref/ratomic -I/home/klozes/upcxx_agave/.nobs/art/8f69db21b6e4d63289509f1fa71b908ae62bd051/GASNet-2019.9.0/extended-ref -I/home/klozes/upcxx_agave/.nobs/art/18cf07baa7180bf3776f87808fc219dbb83cc5f5 -O0 -g -Wall -g3 -Wno-unused -Wunused-result -Wno-unused-parameter -Wno-address -c /home/klozes/upcxx_agave/test/atomics.cpp -o /home/klozes/upcxx_agave/.nobs/art/ef88b35a1cf7c9f950d58a7eafade4a6fa9a0ffd.atomics.cpp.o

/packages/7x/openmpi/agave/3.0.3/gcc/7x/normal/bin/mpicxx -D_GNU_SOURCE=1 -g3 -Wno-unused -Wunused-result -Wno-unused-parameter -Wno-address -o /home/klozes/upcxx_agave/.nobs/art/fd53c562374c495d8ad05dc7a4e9ad612158a743.x /home/klozes/upcxx_agave/.nobs/art/f0a0de64d7611fe59ad6360409185843dae20ff5.segment_allocator.cpp.o /home/klozes/upcxx_agave/.nobs/art/63881f527d0fb23911834657a914b61558dc98c6.upcxx.cpp.o /home/klozes/upcxx_agave/.nobs/art/a93ae098b208ebb540a0bbb63097f3faa2cb98cb.global_fnptr.cpp.o /home/klozes/upcxx_agave/.nobs/art/2b508796dfcac10ff6a1193a0b680d27c65876ae.serialization.cpp.o /home/klozes/upcxx_agave/.nobs/art/000e8c46eb6a09c2c0656fa1849d18fa56ee2494.upc_link.c.o /home/klozes/upcxx_agave/.nobs/art/a47c6dec264eed3ae3fe2063eb830118e621b861.digest.cpp.o /home/klozes/upcxx_agave/.nobs/art/7facf8428913f25c092f0be397c5854133f3ff7c.runtime.cpp.o /home/klozes/upcxx_agave/.nobs/art/b3561491c72f5374a384f38810b70bc37e3b1342.core.cpp.o /home/klozes/upcxx_agave/.nobs/art/d9670697fe4c303a9684051833439141a6255bf2.dl_malloc.c.o /home/klozes/upcxx_agave/.nobs/art/ef88b35a1cf7c9f950d58a7eafade4a6fa9a0ffd.atomics.cpp.o /home/klozes/upcxx_agave/.nobs/art/64f45054ba66ff7ff52e65694ab62d4a428e252d.rput.cpp.o /home/klozes/upcxx_agave/.nobs/art/b46412435f3280c2a3566e953d9052153a0b3731.persona.cpp.o /home/klozes/upcxx_agave/.nobs/art/d40ff7befa275b5d748895d377653c63d52ab373.rget.cpp.o /home/klozes/upcxx_agave/.nobs/art/187dc0b24c9ac6658910fb36100492ab18bffbb4.copy.cpp.o /home/klozes/upcxx_agave/.nobs/art/1794f3a91321d9977cc30967df8223c47001bc05.broadcast.cpp.o /home/klozes/upcxx_agave/.nobs/art/53c235fecf9d9d88945f59206815daf65d0d9c4d.os_env.cpp.o /home/klozes/upcxx_agave/.nobs/art/7219ecded08c5f68d83326ac01cb32633406f84e.noise_log.cpp.o /home/klozes/upcxx_agave/.nobs/art/9cc90cf6f19906feccd6e3cd02cfde04f3a74e4d.atomic.cpp.o /home/klozes/upcxx_agave/.nobs/art/ff43c8e37648520901a00ddf5cad35b64f11651c.barrier.cpp.o /home/klozes/upcxx_agave/.nobs/art/dc3f484689bcb316cc97c86b9bd37b6f4c4d74c5.vis.cpp.o /home/klozes/upcxx_agave/.nobs/art/844dbdf3dfdfd13261df91b642d158c539499a69.reduce.cpp.o /home/klozes/upcxx_agave/.nobs/art/becdd33d3c71d3c32ae9a355b9de778d05122005.diagnostic.cpp.o /home/klozes/upcxx_agave/.nobs/art/3cfece88b61aba484a3aa214b3fff16de414c675.team.cpp.o /home/klozes/upcxx_agave/.nobs/art/bdad39429f31b5217600b79e344d50436e338576.cuda.cpp.o -L/home/klozes/upcxx_agave/.nobs/art/18cf07baa7180bf3776f87808fc219dbb83cc5f5/ibv-conduit -lgasnet-ibv-seq -libverbs -lpmi2 -lpthread -lrt -L/packages/7x/gcc/gcc-7.2.0/lib/gcc/x86_64-pc-linux-gnu/7.2.0 -lgcc -lm

*** WARNING (agave1.agave.rc.asu.edu:99286): GASNet timer calibration on agave1.agave.rc.asu.edu detected non-linear timer behavior: max_err_tick=0 max_err_wcns=0 ticks_res=24 ref_res=297 lo=2400002228 hi=2399996390. See docs for GASNET_TSC_RATE. Retrying...
*** WARNING (agave1.agave.rc.asu.edu:99288): GASNet timer calibration on agave1.agave.rc.asu.edu detected non-linear timer behavior: max_err_tick=0 max_err_wcns=0 ticks_res=24 ref_res=200 lo=2400002749 hi=2399973518. See docs for GASNET_TSC_RATE. Retrying...
*** TICKS: Debugging information: (agave1.agave.rc.asu.edu:99288): 
 wc1[9]=1485600969584412 wc0[0]=1485600869408397 delta=100176015  lo1[9]=3565571215191555 lo0[0]=3565570974769860 delta=240421695  hi1[9]=3565571215193820 hi0[0]=3565570974767307 delta=240426513  lo=2.399988 hi=2.400046
 wc1[8]=1485600969584412 wc0[1]=1485600869412094 delta=100172318  lo1[8]=3565571215189167 lo0[1]=3565570974777657 delta=240411510  hi1[8]=3565571215191435 hi0[1]=3565570974775095 delta=240416340  lo=2.399974 hi=2.400033
 wc1[7]=1485600969584412 wc0[2]=1485600869414833 delta=100169579  lo1[7]=3565571215186593 lo0[2]=3565570974787581 delta=240399012  hi1[7]=3565571215189053 hi0[2]=3565570974785220 delta=240403833  lo=2.399915 hi=2.399974
 wc1[6]=1485600969582224 wc0[3]=1485600869418308 delta=100163916  lo1[6]=3565571215183890 lo0[3]=3565570974795078 delta=240388812  hi1[6]=3565571215186470 hi0[3]=3565570974792498 delta=240393972  lo=2.399949 hi=2.400011
 wc1[5]=1485600969580034 wc0[4]=1485600869422019 delta=100158015  lo1[5]=3565571215181484 lo0[4]=3565570974802698 delta=240378786  hi1[5]=3565571215183785 hi0[4]=3565570974800154 delta=240383631  lo=2.399990 hi=2.400049
 wc1[4]=1485600969579361 wc0[5]=1485600869425557 delta=100153804  lo1[4]=3565571215178796 lo0[5]=3565570974812730 delta=240366066  hi1[4]=3565571215181391 hi0[5]=3565570974810210 delta=240371181  lo=2.399964 hi=2.400026
 wc1[3]=1485600969577913 wc0[6]=1485600869429314 delta=100148599  lo1[3]=3565571215176402 lo0[6]=3565570974820347 delta=240356055  hi1[3]=3565571215178715 hi0[6]=3565570974817794 delta=240360921  lo=2.399989 hi=2.400048
 wc1[2]=1485600969576839 wc0[7]=1485600869432897 delta=100143942  lo1[2]=3565571215173528 lo0[7]=3565570974830370 delta=240343158  hi1[2]=3565571215176321 hi0[7]=3565570974827844 delta=240348477  lo=2.399972 hi=2.400035
 wc1[1]=1485600969576262 wc0[8]=1485600869436606 delta=100139656  lo1[1]=3565571215170960 lo0[8]=3565570974838008 delta=240332952  hi1[1]=3565571215173450 hi0[8]=3565570974835464 delta=240337986  lo=2.399973 hi=2.400033
 wc1[0]=1485600969576262 wc0[9]=1485600869440326 delta=100135936  lo1[0]=3565571215168449 lo0[9]=3565570974847914 delta=240320535  hi1[0]=3565571215170927 hi0[9]=3565570974845532 delta=240325395  lo=2.399938 hi=2.399997

*** TICKS: Debugging information: (agave1.agave.rc.asu.edu:99286): 
 wc1[9]=1485600969646905 wc0[0]=1485600869400725 delta=100246180  lo1[9]=3565571215340978 lo0[0]=3565570974752994 delta=240587984  hi1[9]=3565571215346922 hi0[0]=3565570974750186 delta=240596736  lo=2.399964 hi=2.400066
 wc1[8]=1485600969644278 wc0[1]=1485600869404824 delta=100239454  lo1[8]=3565571215334674 lo0[1]=3565570974761061 delta=240573613  hi1[8]=3565571215340618 hi0[1]=3565570974758478 delta=240582140  lo=2.399982 hi=2.400082
 wc1[7]=1485600969641661 wc0[2]=1485600869408397 delta=100233264  lo1[7]=3565571215328402 lo0[2]=3565570974768663 delta=240559739  hi1[7]=3565571215334338 hi0[2]=3565570974766206 delta=240568132  lo=2.399992 hi=2.400090
 wc1[6]=1485600969639158 wc0[3]=1485600869412094 delta=100227064  lo1[6]=3565571215322330 lo0[3]=3565570974776481 delta=240545849  hi1[6]=3565571215328170 hi0[3]=3565570974774003 delta=240554167  lo=2.400002 hi=2.400099
 wc1[5]=1485600969639158 wc0[4]=1485600869413739 delta=100225419  lo1[5]=3565571215316266 lo0[4]=3565570974784659 delta=240531607  hi1[5]=3565571215322106 hi0[4]=3565570974782199 delta=240539907  lo=2.399899 hi=2.399996
 wc1[4]=1485600969636115 wc0[5]=1485600869417136 delta=100218979  lo1[4]=3565571215309994 lo0[5]=3565570974792306 delta=240517688  hi1[4]=3565571215316042 hi0[5]=3565570974789849 delta=240526193  lo=2.399914 hi=2.400014
 wc1[3]=1485600969631425 wc0[6]=1485600869420759 delta=100210666  lo1[3]=3565571215303834 lo0[6]=3565570974799689 delta=240504145  hi1[3]=3565571215309786 hi0[6]=3565570974797235 delta=240512551  lo=2.399978 hi=2.400077
 wc1[2]=1485600969628865 wc0[7]=1485600869423257 delta=100205608  lo1[2]=3565571215297346 lo0[7]=3565570974807261 delta=240490085  hi1[2]=3565571215303626 hi0[7]=3565570974804714 delta=240498912  lo=2.399959 hi=2.400062
 wc1[1]=1485600969627474 wc0[8]=1485600869426771 delta=100200703  lo1[1]=3565571215291122 lo0[8]=3565570974814653 delta=240476469  hi1[1]=3565571215297146 hi0[8]=3565570974812202 delta=240484944  lo=2.399941 hi=2.400040
 wc1[0]=1485600969624881 wc0[9]=1485600869430547 delta=100194334  lo1[0]=3565571215284698 lo0[9]=3565570974824901 delta=240459797  hi1[0]=3565571215291042 hi0[9]=3565570974822369 delta=240468673  lo=2.399927 hi=2.400030

*** WARNING (proc 1): GASNet gasnetc_init returning an error code: GASNET_ERR_RESOURCE (Problem with requested resource)
  at /home/klozes/upcxx_agave/.nobs/art/8f69db21b6e4d63289509f1fa71b908ae62bd051/GASNet-2019.9.0/ibv-conduit/gasnet_core.c:2084
  reason: unable to open any HCA ports
*** WARNING (proc 1): GASNet gex_Client_Init_GASNET_201990SEQpshmFASTdebugtracestatsdebugmallocsrclines returning an error code: GASNET_ERR_RESOURCE (Problem with requested resource)
  at /home/klozes/upcxx_agave/.nobs/art/8f69db21b6e4d63289509f1fa71b908ae62bd051/GASNet-2019.9.0/ibv-conduit/gasnet_core.c:2915
*** FATAL ERROR (proc 1): 
//////////////////////////////////////////////////
UPC++ assertion failure:
 rank=0
 file=/home/klozes/upcxx_agave/src/backend/gasnet/runtime.cpp:454

Failed condition: ok == 0

To have UPC++ freeze during these errors so you can attach a debugger, rerun the program with GASNET_FREEZE_ON_ERROR=1 in the environment.
//////////////////////////////////////////////////

[1] Invoking GDB for backtrace...
*** WARNING (proc 3): GASNet gasnetc_init returning an error code: GASNET_ERR_RESOURCE (Problem with requested resource)
  at /home/klozes/upcxx_agave/.nobs/art/8f69db21b6e4d63289509f1fa71b908ae62bd051/GASNet-2019.9.0/ibv-conduit/gasnet_core.c:2084
  reason: unable to open any HCA ports
*** WARNING (proc 3): GASNet gex_Client_Init_GASNET_201990SEQpshmFASTdebugtracestatsdebugmallocsrclines returning an error code: GASNET_ERR_RESOURCE (Problem with requested resource)
  at /home/klozes/upcxx_agave/.nobs/art/8f69db21b6e4d63289509f1fa71b908ae62bd051/GASNet-2019.9.0/ibv-conduit/gasnet_core.c:2915
*** FATAL ERROR (proc 3): 
//////////////////////////////////////////////////
UPC++ assertion failure:
 rank=0
 file=/home/klozes/upcxx_agave/src/backend/gasnet/runtime.cpp:454

Failed condition: ok == 0

To have UPC++ freeze during these errors so you can attach a debugger, rerun the program with GASNET_FREEZE_ON_ERROR=1 in the environment.
//////////////////////////////////////////////////

[3] Invoking GDB for backtrace...
*** WARNING (agave1.agave.rc.asu.edu:99286): GASNet timer calibration on agave1.agave.rc.asu.edu detected non-linear timer behavior: max_err_tick=0 max_err_wcns=0 ticks_res=24 ref_res=297 lo=2399960367 hi=2399951773. See docs for GASNET_TSC_RATE.
*** TICKS: Debugging information: (agave1.agave.rc.asu.edu:99286): 
 wc1[9]=1485601070013318 wc0[0]=1485600969864675 delta=100148643  lo1[9]=3565571456220765 lo0[0]=3565571215867287 delta=240353478  hi1[9]=3565571456223714 hi0[0]=3565571215864524 delta=240359190  lo=2.399960 hi=2.400032
 wc1[8]=1485601070013296 wc0[1]=1485600969867675 delta=100145621  lo1[8]=3565571456218065 lo0[1]=3565571215874454 delta=240343611  hi1[8]=3565571456220630 hi0[1]=3565571215872111 delta=240348519  lo=2.399934 hi=2.399998
 wc1[7]=1485601070013296 wc0[2]=1485600969870562 delta=100142734  lo1[7]=3565571456215491 lo0[2]=3565571215881387 delta=240334104  hi1[7]=3565571456217930 hi0[2]=3565571215879080 delta=240338850  lo=2.399908 hi=2.399970
 wc1[6]=1485601070012320 wc0[3]=1485600969873470 delta=100138850  lo1[6]=3565571456212269 lo0[3]=3565571215888362 delta=240323907  hi1[6]=3565571456215365 hi0[3]=3565571215886055 delta=240329310  lo=2.399899 hi=2.399968
 wc1[5]=1485601070010227 wc0[4]=1485600969876361 delta=100133866  lo1[5]=3565571456208435 lo0[4]=3565571215895307 delta=240313128  hi1[5]=3565571456212167 hi0[4]=3565571215892994 delta=240319173  lo=2.399911 hi=2.399986
 wc1[4]=1485601070009403 wc0[5]=1485600969879254 delta=100130149  lo1[4]=3565571456205921 lo0[5]=3565571215902246 delta=240303675  hi1[4]=3565571456208342 hi0[5]=3565571215899933 delta=240308409  lo=2.399906 hi=2.399968
 wc1[3]=1485601070008243 wc0[6]=1485600969882146 delta=100126097  lo1[3]=3565571456203230 lo0[6]=3565571215909188 delta=240294042  hi1[3]=3565571456205831 hi0[6]=3565571215906878 delta=240298953  lo=2.399907 hi=2.399971
 wc1[2]=1485601070006878 wc0[7]=1485600969885037 delta=100121841  lo1[2]=3565571456200077 lo0[7]=3565571215916124 delta=240283953  hi1[2]=3565571456203080 hi0[7]=3565571215913817 delta=240289263  lo=2.399908 hi=2.399976
 wc1[1]=1485601070006596 wc0[8]=1485600969887927 delta=100118669  lo1[1]=3565571456197425 lo0[8]=3565571215923060 delta=240274365  hi1[1]=3565571456199996 hi0[8]=3565571215920750 delta=240279246  lo=2.399888 hi=2.399952
 wc1[0]=1485601070005395 wc0[9]=1485600969890831 delta=100114564  lo1[0]=3565571456193825 lo0[9]=3565571215930032 delta=240263793  hi1[0]=3565571456197395 hi0[9]=3565571215927725 delta=240269670  lo=2.399881 hi=2.399955

*** WARNING (proc 2): GASNet gasnetc_init returning an error code: GASNET_ERR_RESOURCE (Problem with requested resource)
  at /home/klozes/upcxx_agave/.nobs/art/8f69db21b6e4d63289509f1fa71b908ae62bd051/GASNet-2019.9.0/ibv-conduit/gasnet_core.c:2084
  reason: unable to open any HCA ports
*** WARNING (proc 2): GASNet gex_Client_Init_GASNET_201990SEQpshmFASTdebugtracestatsdebugmallocsrclines returning an error code: GASNET_ERR_RESOURCE (Problem with requested resource)
  at /home/klozes/upcxx_agave/.nobs/art/8f69db21b6e4d63289509f1fa71b908ae62bd051/GASNet-2019.9.0/ibv-conduit/gasnet_core.c:2915
*** FATAL ERROR (proc 2): 
//////////////////////////////////////////////////
UPC++ assertion failure:
 rank=0
 file=/home/klozes/upcxx_agave/src/backend/gasnet/runtime.cpp:454

Failed condition: ok == 0

To have UPC++ freeze during these errors so you can attach a debugger, rerun the program with GASNET_FREEZE_ON_ERROR=1 in the environment.
//////////////////////////////////////////////////

[2] Invoking GDB for backtrace...
*** WARNING (proc 0): GASNet gasnetc_init returning an error code: GASNET_ERR_RESOURCE (Problem with requested resource)
  at /home/klozes/upcxx_agave/.nobs/art/8f69db21b6e4d63289509f1fa71b908ae62bd051/GASNet-2019.9.0/ibv-conduit/gasnet_core.c:2084
  reason: unable to open any HCA ports
*** WARNING (proc 0): GASNet gex_Client_Init_GASNET_201990SEQpshmFASTdebugtracestatsdebugmallocsrclines returning an error code: GASNET_ERR_RESOURCE (Problem with requested resource)
  at /home/klozes/upcxx_agave/.nobs/art/8f69db21b6e4d63289509f1fa71b908ae62bd051/GASNet-2019.9.0/ibv-conduit/gasnet_core.c:2915
*** FATAL ERROR (proc 0): 
//////////////////////////////////////////////////
UPC++ assertion failure:
 rank=0
 file=/home/klozes/upcxx_agave/src/backend/gasnet/runtime.cpp:454

Failed condition: ok == 0

To have UPC++ freeze during these errors so you can attach a debugger, rerun the program with GASNET_FREEZE_ON_ERROR=1 in the environment.
//////////////////////////////////////////////////

[0] Invoking GDB for backtrace...
[1] /usr/bin/gdb -nx -batch -x /tmp/gasnet_e6mef8 '/home/klozes/upcxx_agave/.nobs/art/fd53c562374c495d8ad05dc7a4e9ad612158a743.x' 99287
[1] [New LWP 99294]
[1] [New LWP 99290]
[1] [Thread debugging using libthread_db enabled]
[1] Using host libthread_db library "/lib64/libthread_db.so.1".
[1] 0x00002b5215120e09 in waitpid () from /lib64/libc.so.6
[1] To enable execution of this file add
[1]     add-auto-load-safe-path /packages/7x/gcc/gcc-7.2.0/lib64/libstdc++.so.6.0.24-gdb.py
[1] line to your configuration file "/home/klozes/.gdbinit".
[1] To completely disable this security protection add
[1]     set auto-load safe-path /
[1] line to your configuration file "/home/klozes/.gdbinit".
[1] For more information about this security protection see the
[1] "Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
[1]     info "(gdb)Auto-loading safe path"
[1] #0  0x00002b5215120e09 in waitpid () from /lib64/libc.so.6
[1] #1  0x00002b52150a3cc2 in do_system () from /lib64/libc.so.6
[1] #2  0x00002b52150a4071 in system () from /lib64/libc.so.6
[1] #3  0x000000000059df56 in gasneti_system_redirected (cmd=0xbce5c0 <cmd.6300> "/usr/bin/gdb -nx -batch -x /tmp/gasnet_e6mef8 '/home/klozes/upcxx_agave/.nobs/art/fd53c562374c495d8ad05dc7a4e9ad612158a743.x' 99287", stdout_fd=16) at /home/klozes/upcxx_agave/.nobs/art/8f69db21b6e4d63289509f1fa71b908ae62bd051/GASNet-2019.9.0/gasnet_tools.c:1275
[1] #4  0x000000000059e79e in gasneti_bt_gdb (fd=16) at /home/klozes/upcxx_agave/.nobs/art/8f69db21b6e4d63289509f1fa71b908ae62bd051/GASNet-2019.9.0/gasnet_tools.c:1531
[1] #5  0x000000000059efe1 in gasneti_print_backtrace (fd=2) at /home/klozes/upcxx_agave/.nobs/art/8f69db21b6e4d63289509f1fa71b908ae62bd051/GASNet-2019.9.0/gasnet_tools.c:1806
[1] #6  0x000000000059f5c6 in _gasneti_print_backtrace_ifenabled (fd=2) at /home/klozes/upcxx_agave/.nobs/art/8f69db21b6e4d63289509f1fa71b908ae62bd051/GASNet-2019.9.0/gasnet_tools.c:1938
[1] #7  0x000000000059d025 in gasneti_error_abort () at /home/klozes/upcxx_agave/.nobs/art/8f69db21b6e4d63289509f1fa71b908ae62bd051/GASNet-2019.9.0/gasnet_tools.c:743
[1] #8  0x000000000059d1ac in gasneti_fatalerror (msg=0x8a1645 "\n%s") at /home/klozes/upcxx_agave/.nobs/art/8f69db21b6e4d63289509f1fa71b908ae62bd051/GASNet-2019.9.0/gasnet_tools.c:772
[1] #9  0x00000000004f8610 in upcxx::assert_failed (file=0x87bcc8 "/home/klozes/upcxx_agave/src/backend/gasnet/runtime.cpp", line=454, msg=0x87bd00 "Failed condition: ok == 0") at /home/klozes/upcxx_agave/src/diagnostic.cpp:42
[1] #10 0x000000000042baa2 in upcxx::init () at /home/klozes/upcxx_agave/src/backend/gasnet/runtime.cpp:454
[1] #11 0x0000000000448335 in main (argc=1, argv=0x7ffe9c7a51d8) at /home/klozes/upcxx_agave/test/atomics.cpp:216
[agave1:99287] *** Process received signal ***
[agave1:99287] Signal: Aborted (6)
[agave1:99287] Associated errno: Unknown error 11090 (11090)
[agave1:99287] Signal code:  (352750888)
[agave1:99287] [ 0] /lib64/libc.so.6(+0x35270)[0x2b5215097270]
[agave1:99287] [ 1] /lib64/libc.so.6(gsignal+0x37)[0x2b52150971f7]
[agave1:99287] [ 2] /lib64/libc.so.6(abort+0x148)[0x2b52150988e8]
[agave1:99287] [ 3] /home/klozes/upcxx_agave/.nobs/art/fd53c562374c495d8ad05dc7a4e9ad612158a743.x[0x59d106]
[agave1:99287] [ 4] /home/klozes/upcxx_agave/.nobs/art/fd53c562374c495d8ad05dc7a4e9ad612158a743.x[0x59d1ac]
[agave1:99287] [ 5] /home/klozes/upcxx_agave/.nobs/art/fd53c562374c495d8ad05dc7a4e9ad612158a743.x[0x4f8610]
[agave1:99287] [ 6] /home/klozes/upcxx_agave/.nobs/art/fd53c562374c495d8ad05dc7a4e9ad612158a743.x[0x42baa2]
[agave1:99287] [ 7] /home/klozes/upcxx_agave/.nobs/art/fd53c562374c495d8ad05dc7a4e9ad612158a743.x[0x448335]
[agave1:99287] [ 8] /lib64/libc.so.6(__libc_start_main+0xf5)[0x2b5215083c05]
[agave1:99287] [ 9] /home/klozes/upcxx_agave/.nobs/art/fd53c562374c495d8ad05dc7a4e9ad612158a743.x[0x4068c9]
[agave1:99287] *** End of error message ***
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
[0] /usr/bin/gdb -nx -batch -x /tmp/gasnet_xchJIf '/home/klozes/upcxx_agave/.nobs/art/fd53c562374c495d8ad05dc7a4e9ad612158a743.x' 99286
[0] [New LWP 99296]
[0] [New LWP 99291]
[0] [Thread debugging using libthread_db enabled]
[0] Using host libthread_db library "/lib64/libthread_db.so.1".
[0] 0x00002ad1df147e09 in waitpid () from /lib64/libc.so.6
[0] To enable execution of this file add
[0]     add-auto-load-safe-path /packages/7x/gcc/gcc-7.2.0/lib64/libstdc++.so.6.0.24-gdb.py
[0] line to your configuration file "/home/klozes/.gdbinit".
[0] To completely disable this security protection add
[0]     set auto-load safe-path /
[0] line to your configuration file "/home/klozes/.gdbinit".
[0] For more information about this security protection see the
[0] "Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
[0]     info "(gdb)Auto-loading safe path"
[0] #0  0x00002ad1df147e09 in waitpid () from /lib64/libc.so.6
[0] #1  0x00002ad1df0cacc2 in do_system () from /lib64/libc.so.6
[0] #2  0x00002ad1df0cb071 in system () from /lib64/libc.so.6
[0] #3  0x000000000059df56 in gasneti_system_redirected (cmd=0xbce5c0 <cmd.6300> "/usr/bin/gdb -nx -batch -x /tmp/gasnet_xchJIf '/home/klozes/upcxx_agave/.nobs/art/fd53c562374c495d8ad05dc7a4e9ad612158a743.x' 99286", stdout_fd=16) at /home/klozes/upcxx_agave/.nobs/art/8f69db21b6e4d63289509f1fa71b908ae62bd051/GASNet-2019.9.0/gasnet_tools.c:1275
[0] #4  0x000000000059e79e in gasneti_bt_gdb (fd=16) at /home/klozes/upcxx_agave/.nobs/art/8f69db21b6e4d63289509f1fa71b908ae62bd051/GASNet-2019.9.0/gasnet_tools.c:1531
[0] #5  0x000000000059efe1 in gasneti_print_backtrace (fd=2) at /home/klozes/upcxx_agave/.nobs/art/8f69db21b6e4d63289509f1fa71b908ae62bd051/GASNet-2019.9.0/gasnet_tools.c:1806
[0] #6  0x000000000059f5c6 in _gasneti_print_backtrace_ifenabled (fd=2) at /home/klozes/upcxx_agave/.nobs/art/8f69db21b6e4d63289509f1fa71b908ae62bd051/GASNet-2019.9.0/gasnet_tools.c:1938
[0] #7  0x000000000059d025 in gasneti_error_abort () at /home/klozes/upcxx_agave/.nobs/art/8f69db21b6e4d63289509f1fa71b908ae62bd051/GASNet-2019.9.0/gasnet_tools.c:743
[0] #8  0x000000000059d1ac in gasneti_fatalerror (msg=0x8a1645 "\n%s") at /home/klozes/upcxx_agave/.nobs/art/8f69db21b6e4d63289509f1fa71b908ae62bd051/GASNet-2019.9.0/gasnet_tools.c:772
[0] #9  0x00000000004f8610 in upcxx::assert_failed (file=0x87bcc8 "/home/klozes/upcxx_agave/src/backend/gasnet/runtime.cpp", line=454, msg=0x87bd00 "Failed condition: ok == 0") at /home/klozes/upcxx_agave/src/diagnostic.cpp:42
[0] #10 0x000000000042baa2 in upcxx::init () at /home/klozes/upcxx_agave/src/backend/gasnet/runtime.cpp:454
[0] #11 0x0000000000448335 in main (argc=1, argv=0x7ffefeb2b888) at /home/klozes/upcxx_agave/test/atomics.cpp:216
[agave1:99286] *** Process received signal ***
[agave1:99286] Signal: Aborted (6)
[agave1:99286] Associated errno: Unknown error 10961 (10961)
[agave1:99286] Signal code:  (-553059032)
[agave1:99286] [ 0] /lib64/libc.so.6(+0x35270)[0x2ad1df0be270]
[agave1:99286] [ 1] /lib64/libc.so.6(gsignal+0x37)[0x2ad1df0be1f7]
[agave1:99286] [ 2] /lib64/libc.so.6(abort+0x148)[0x2ad1df0bf8e8]
[agave1:99286] [ 3] /home/klozes/upcxx_agave/.nobs/art/fd53c562374c495d8ad05dc7a4e9ad612158a743.x[0x59d106]
[agave1:99286] [ 4] /home/klozes/upcxx_agave/.nobs/art/fd53c562374c495d8ad05dc7a4e9ad612158a743.x[0x59d1ac]
[agave1:99286] [ 5] /home/klozes/upcxx_agave/.nobs/art/fd53c562374c495d8ad05dc7a4e9ad612158a743.x[0x4f8610]
[agave1:99286] [ 6] /home/klozes/upcxx_agave/.nobs/art/fd53c562374c495d8ad05dc7a4e9ad612158a743.x[0x42baa2]
[agave1:99286] [ 7] /home/klozes/upcxx_agave/.nobs/art/fd53c562374c495d8ad05dc7a4e9ad612158a743.x[0x448335]
[agave1:99286] [ 8] /lib64/libc.so.6(__libc_start_main+0xf5)[0x2ad1df0aac05]
[agave1:99286] [ 9] /home/klozes/upcxx_agave/.nobs/art/fd53c562374c495d8ad05dc7a4e9ad612158a743.x[0x4068c9]
[agave1:99286] *** End of error message ***
[3] /usr/bin/gdb -nx -batch -x /tmp/gasnet_T85r76 '/home/klozes/upcxx_agave/.nobs/art/fd53c562374c495d8ad05dc7a4e9ad612158a743.x' 99289
[3] [New LWP 99297]
[3] [New LWP 99293]
[3] [Thread debugging using libthread_db enabled]
[3] Using host libthread_db library "/lib64/libthread_db.so.1".
[3] 0x00002b567a282e09 in waitpid () from /lib64/libc.so.6
[3] To enable execution of this file add
[3]     add-auto-load-safe-path /packages/7x/gcc/gcc-7.2.0/lib64/libstdc++.so.6.0.24-gdb.py
[3] line to your configuration file "/home/klozes/.gdbinit".
[3] To completely disable this security protection add
[3]     set auto-load safe-path /
[3] line to your configuration file "/home/klozes/.gdbinit".
[3] For more information about this security protection see the
[3] "Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
[3]     info "(gdb)Auto-loading safe path"
[3] #0  0x00002b567a282e09 in waitpid () from /lib64/libc.so.6
[3] #1  0x00002b567a205cc2 in do_system () from /lib64/libc.so.6
[3] #2  0x00002b567a206071 in system () from /lib64/libc.so.6
[3] #3  0x000000000059df56 in gasneti_system_redirected (cmd=0xbce5c0 <cmd.6300> "/usr/bin/gdb -nx -batch -x /tmp/gasnet_T85r76 '/home/klozes/upcxx_agave/.nobs/art/fd53c562374c495d8ad05dc7a4e9ad612158a743.x' 99289", stdout_fd=16) at /home/klozes/upcxx_agave/.nobs/art/8f69db21b6e4d63289509f1fa71b908ae62bd051/GASNet-2019.9.0/gasnet_tools.c:1275
[3] #4  0x000000000059e79e in gasneti_bt_gdb (fd=16) at /home/klozes/upcxx_agave/.nobs/art/8f69db21b6e4d63289509f1fa71b908ae62bd051/GASNet-2019.9.0/gasnet_tools.c:1531
[3] #5  0x000000000059efe1 in gasneti_print_backtrace (fd=2) at /home/klozes/upcxx_agave/.nobs/art/8f69db21b6e4d63289509f1fa71b908ae62bd051/GASNet-2019.9.0/gasnet_tools.c:1806
[3] #6  0x000000000059f5c6 in _gasneti_print_backtrace_ifenabled (fd=2) at /home/klozes/upcxx_agave/.nobs/art/8f69db21b6e4d63289509f1fa71b908ae62bd051/GASNet-2019.9.0/gasnet_tools.c:1938
[3] #7  0x000000000059d025 in gasneti_error_abort () at /home/klozes/upcxx_agave/.nobs/art/8f69db21b6e4d63289509f1fa71b908ae62bd051/GASNet-2019.9.0/gasnet_tools.c:743
[3] #8  0x000000000059d1ac in gasneti_fatalerror (msg=0x8a1645 "\n%s") at /home/klozes/upcxx_agave/.nobs/art/8f69db21b6e4d63289509f1fa71b908ae62bd051/GASNet-2019.9.0/gasnet_tools.c:772
[3] #9  0x00000000004f8610 in upcxx::assert_failed (file=0x87bcc8 "/home/klozes/upcxx_agave/src/backend/gasnet/runtime.cpp", line=454, msg=0x87bd00 "Failed condition: ok == 0") at /home/klozes/upcxx_agave/src/diagnostic.cpp:42
[3] #10 0x000000000042baa2 in upcxx::init () at /home/klozes/upcxx_agave/src/backend/gasnet/runtime.cpp:454
[3] #11 0x0000000000448335 in main (argc=1, argv=0x7fff702a5858) at /home/klozes/upcxx_agave/test/atomics.cpp:216
[agave1:99289] *** Process received signal ***
[agave1:99289] Signal: Aborted (6)
[agave1:99289] Associated errno: Unknown error 11094 (11094)
[agave1:99289] Signal code:  (2048699688)
[agave1:99289] [ 0] /lib64/libc.so.6(+0x35270)[0x2b567a1f9270]
[agave1:99289] [ 1] /lib64/libc.so.6(gsignal+0x37)[0x2b567a1f91f7]
[agave1:99289] [ 2] /lib64/libc.so.6(abort+0x148)[0x2b567a1fa8e8]
[agave1:99289] [ 3] /home/klozes/upcxx_agave/.nobs/art/fd53c562374c495d8ad05dc7a4e9ad612158a743.x[0x59d106]
[agave1:99289] [ 4] /home/klozes/upcxx_agave/.nobs/art/fd53c562374c495d8ad05dc7a4e9ad612158a743.x[0x59d1ac]
[agave1:99289] [ 5] /home/klozes/upcxx_agave/.nobs/art/fd53c562374c495d8ad05dc7a4e9ad612158a743.x[0x4f8610]
[agave1:99289] [ 6] /home/klozes/upcxx_agave/.nobs/art/fd53c562374c495d8ad05dc7a4e9ad612158a743.x[0x42baa2]
[agave1:99289] [ 7] /home/klozes/upcxx_agave/.nobs/art/fd53c562374c495d8ad05dc7a4e9ad612158a743.x[0x448335]
[agave1:99289] [ 8] /lib64/libc.so.6(__libc_start_main+0xf5)[0x2b567a1e5c05]
[agave1:99289] [ 9] /home/klozes/upcxx_agave/.nobs/art/fd53c562374c495d8ad05dc7a4e9ad612158a743.x[0x4068c9]
[agave1:99289] *** End of error message ***
[2] /usr/bin/gdb -nx -batch -x /tmp/gasnet_vkm5Hf '/home/klozes/upcxx_agave/.nobs/art/fd53c562374c495d8ad05dc7a4e9ad612158a743.x' 99288
[2] [New LWP 99295]
[2] [New LWP 99292]
[2] [Thread debugging using libthread_db enabled]
[2] Using host libthread_db library "/lib64/libthread_db.so.1".
[2] 0x00002b24884a0e09 in waitpid () from /lib64/libc.so.6
[2] To enable execution of this file add
[2]     add-auto-load-safe-path /packages/7x/gcc/gcc-7.2.0/lib64/libstdc++.so.6.0.24-gdb.py
[2] line to your configuration file "/home/klozes/.gdbinit".
[2] To completely disable this security protection add
[2]     set auto-load safe-path /
[2] line to your configuration file "/home/klozes/.gdbinit".
[2] For more information about this security protection see the
[2] "Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
[2]     info "(gdb)Auto-loading safe path"
[2] #0  0x00002b24884a0e09 in waitpid () from /lib64/libc.so.6
[2] #1  0x00002b2488423cc2 in do_system () from /lib64/libc.so.6
[2] #2  0x00002b2488424071 in system () from /lib64/libc.so.6
[2] #3  0x000000000059df56 in gasneti_system_redirected (cmd=0xbce5c0 <cmd.6300> "/usr/bin/gdb -nx -batch -x /tmp/gasnet_vkm5Hf '/home/klozes/upcxx_agave/.nobs/art/fd53c562374c495d8ad05dc7a4e9ad612158a743.x' 99288", stdout_fd=16) at /home/klozes/upcxx_agave/.nobs/art/8f69db21b6e4d63289509f1fa71b908ae62bd051/GASNet-2019.9.0/gasnet_tools.c:1275
[2] #4  0x000000000059e79e in gasneti_bt_gdb (fd=16) at /home/klozes/upcxx_agave/.nobs/art/8f69db21b6e4d63289509f1fa71b908ae62bd051/GASNet-2019.9.0/gasnet_tools.c:1531
[2] #5  0x000000000059efe1 in gasneti_print_backtrace (fd=2) at /home/klozes/upcxx_agave/.nobs/art/8f69db21b6e4d63289509f1fa71b908ae62bd051/GASNet-2019.9.0/gasnet_tools.c:1806
[2] #6  0x000000000059f5c6 in _gasneti_print_backtrace_ifenabled (fd=2) at /home/klozes/upcxx_agave/.nobs/art/8f69db21b6e4d63289509f1fa71b908ae62bd051/GASNet-2019.9.0/gasnet_tools.c:1938
[2] #7  0x000000000059d025 in gasneti_error_abort () at /home/klozes/upcxx_agave/.nobs/art/8f69db21b6e4d63289509f1fa71b908ae62bd051/GASNet-2019.9.0/gasnet_tools.c:743
[2] #8  0x000000000059d1ac in gasneti_fatalerror (msg=0x8a1645 "\n%s") at /home/klozes/upcxx_agave/.nobs/art/8f69db21b6e4d63289509f1fa71b908ae62bd051/GASNet-2019.9.0/gasnet_tools.c:772
[2] #9  0x00000000004f8610 in upcxx::assert_failed (file=0x87bcc8 "/home/klozes/upcxx_agave/src/backend/gasnet/runtime.cpp", line=454, msg=0x87bd00 "Failed condition: ok == 0") at /home/klozes/upcxx_agave/src/diagnostic.cpp:42
[2] #10 0x000000000042baa2 in upcxx::init () at /home/klozes/upcxx_agave/src/backend/gasnet/runtime.cpp:454
[2] #11 0x0000000000448335 in main (argc=1, argv=0x7ffdfa130538) at /home/klozes/upcxx_agave/test/atomics.cpp:216
[agave1:99288] *** Process received signal ***
[agave1:99288] Signal: Aborted (6)
[agave1:99288] Associated errno: Unknown error 11044 (11044)
[agave1:99288] Signal code:  (4207913)
[agave1:99288] [ 0] /lib64/libc.so.6(+0x35270)[0x2b2488417270]
[agave1:99288] [ 1] /lib64/libc.so.6(gsignal+0x37)[0x2b24884171f7]
[agave1:99288] [ 2] /lib64/libc.so.6(abort+0x148)[0x2b24884188e8]
[agave1:99288] [ 3] /home/klozes/upcxx_agave/.nobs/art/fd53c562374c495d8ad05dc7a4e9ad612158a743.x[0x59d106]
[agave1:99288] [ 4] /home/klozes/upcxx_agave/.nobs/art/fd53c562374c495d8ad05dc7a4e9ad612158a743.x[0x59d1ac]
[agave1:99288] [ 5] /home/klozes/upcxx_agave/.nobs/art/fd53c562374c495d8ad05dc7a4e9ad612158a743.x[0x4f8610]
[agave1:99288] [ 6] /home/klozes/upcxx_agave/.nobs/art/fd53c562374c495d8ad05dc7a4e9ad612158a743.x[0x42baa2]
[agave1:99288] [ 7] /home/klozes/upcxx_agave/.nobs/art/fd53c562374c495d8ad05dc7a4e9ad612158a743.x[0x448335]
[agave1:99288] [ 8] /lib64/libc.so.6(__libc_start_main+0xf5)[0x2b2488403c05]
[agave1:99288] [ 9] /home/klozes/upcxx_agave/.nobs/art/fd53c562374c495d8ad05dc7a4e9ad612158a743.x[0x4068c9]
[agave1:99288] *** End of error message ***
--------------------------------------------------------------------------
mpirun noticed that process rank 1 with PID 0 on node agave1 exited on signal 6 (Aborted).
--------------------------------------------------------------------------

    ‌

  5. Paul Hargrove

    Kenneth,

    Sorry to hear you are having problems.
    From what I see in the output you provided, there are two distinct issues.

    The first in importance is the (fatal) messages indicating no InfiniBand HCAs (network adapters) were found.
    If I am correct in thinking that this page describes the "Agave" system you are using then these messages are correct: the nodes of Agave are connected by OmniPath and OmniPath is not InfiniBand.

    While there are drivers available for OmniPath to provide ABI-compatibility with InfiniBand, it appears likely that they are not loaded on the Agave nodes. More importantly, however, our experience has show the performance of emulating InfiniBand over OmniPath is poor. Furthermore, we do not currently have a native driver for OmniPath. So, I would recommend against use of this system if possible.

    If the Saguaro cluster described on that page is suitable to your needs, then I suggest you use it rather than Agave. Let me know if that is not possible and I can describe the options available to run over the OmniPath network in Agave.

    The second is an issue with timer calibration on the host agave1.agave.rc.asu.edu.
    If you switch to a different cluster, then hopefully these problems go away. Let me know if is does not, and I can describe options available to address this problem directly.

  6. Kenneth Lozes reporter

    Paul,

    Thankyou for looking into this for me. You are correct that Agave uses OmniPath. Unfortunately, Saguaro cluster is being decommissioned in a couple months and doesn't have the any new enough compilers installed. So I am kind of stuck with Agave.

    I am happy to use another conduit for my code development activities if I can get it working (hopefully the performance is somewhat comparable). On Agave the run-tests script works fine with udp, but installing and trying to run the hello world test manually gives a missing SSH_SERVERS environment variable error.

    As for testing the upc++ in a shared library over InfiniBand, I don't think I will be able to do that. I don't believe I have access to any InfiniBand clusters besides Saguaro.

  7. Paul Hargrove

    Kenneth,

    Your choices on OmniPath are udp-conduit or mpi-conduit.

    Of those two, I suspect mpi-conduit is going to give the better performance, and will be the easiest to setup.
    We don't list it as officially supported, but that is not due to any known defects (rather it is because it almost always slower than the native network). Ideally CONDUIT=mpi CXX=mpicxx ./run-tests should "just work", If not, let us know.

    For completeness: udp-conduit requires a host list.
    This can be provided with upcxx-run -ssh-servers HOSTS ... where HOSTS is a comma-separated list of hostnames (one per process, repeating hostnames if necessary). Alternatively, one can set the SSH_SERVERS environment variable to the same value.

    Regarding ibv-conduit.
    Based on the failure you reported on Agave, I think we can conclude that the job-launch was successful.

  12. Dan Bonachea

    Re-assigning to Paul for build infrastructure work to automate building UPC++ as a shared library. This may initially be a proof-of-concept driven by UPC++ makefile logic.

  15. Log in to comment
Assignee
Type
task
Priority
minor
Status
new
Component
Build/install scripts
Milestone
–
Version
2019.9.0 release
Votes
0
Watchers
2
Jira: the preferred issue tracker for Bitbucket. Join the team!