GASNet FAQ

How do I enable automatic backtracing on my system?

GASNet includes the ability to automatically generate a stack backtrace to the console when a fatal signal or other selected event occurs on any process. This can be handy when tracking down problems in application code, especially in a distributed system. To use this capability:

1. Run with envvar: GASNET_BACKTRACE=1.
2. This works best when gasnet was configured with --enable-debug
3. If you want a particular debugger, set GASNET_BACKTRACE_TYPE=<name>
4. See README for GASNET_BACKTRACE_* and GASNET_FREEZE* variables to control when and how debugger interaction happens.

Automatic backtracing is implemented on many systems by forking a debugger process to attach to the GASNet process. Some OS's include security features that restrict debugger attach, and thus require additional administrative settings to allow automatic generation of backtraces:

• Mac OS X: Mac OS X has some complicated and version-dependent rules governing debugger attach operations. It's likely you will need the following commands:

  1. Global setting: sudo DevToolsSecurity -enable see this doc
  2. Add user to _developer group: sudo /usr/sbin/dseditgroup -o edit -a $USER -t user _developer see this doc
  3. Even with the above, you may still be prompted to enter your user/password to perform an attach using a non-Apple debugger (ie currently only /usr/bin/lldb ships with the Apple signature).
  4. If you are using PGI C (pgcc), you may additionally want to ensure that object files linked into your application are still present on the file system to ensure the best debug symbol reporting (the -Mkeepobj option can help with this).

• OpenBSD:

  • sudo sysctl -w kern.global_ptrace sysctl=1
    While this works-as-advertised in general, we still seem to have a problem w/ our auto-backtrace command
  • LDFLAGS=-rdynamic is needed to get symbols with execinfo
• Linux: there are several security tools that inhibit debugger attaches:
  • YAMA: verify by inspection of /proc/sys/kernel/yama/ptrace_scope.
    • If value is 0 or 1 then GASNet can attach w/ current develop branch (release only handles 0).
    • To change immediately: sudo sysctl -w kernel.yama.ptrace_scope=0
    • To change for future reboots: append kernel.yama.ptrace_scope=0 to /etc/sysctl.d/99-sysctl.conf
  • SELinux: sudo setsebool allow_ptrace on ; setsebool deny_ptrace 0
  • AppArmor: See http://wiki.apparmor.net/index.php/TechnicalDo_Proc_and_ptrace

How do I fix problems with shared libraries?

GASNet itself is compiled as a static library, avoiding many of the pitfalls arising from shared libraries (which include performance degradation and job launch problems). Nonetheless, GASNet applications often linked with other shared libraries. This might include network drivers used by the GASNet implementation, standard libraries used by the C/C++ compiler, or numerical libraries linked by the GASNet client runtime or application code.

In order for an executable using shared libraries to be launched by the OS loader, it must first find all the referenced shared libraries on the file system. This can be especially problematic on a distributed-memory system where the compute nodes might not all mount the same shared file system containing those libraries. The user-facing problem basically boils down to providing the system loader with the correct filesystem paths for finding all the relevant shared libraries. There are several competing mechanisms for doing this (each with pros and cons), and the details vary by OS.

For some guidance on Linux, see this doc

How do I report a problem or bug or feature request regarding GASNet?

The best mechanism for tech support is to file a bug report in the GASNet Bugzilla.

If your issue cannot be posted in a public forum, you can also email gasnet-devel@lbl.gov

How do I correctly microbenchmark GASNet performance?

1. describe tests/test{small,large} and their correct usage
2. mention other relevant GASNet tests
3. mention tests/mpi/* 1 link to excellent HPC benchmarking guide from OpenMPI

What new features are coming in GASNet-EX?

1. Multi-client
2. Multi-endpoint
3. Increased control over buffer lifetimes
4. Collectives and teams
5. VIS

How do I select the right conduit to use?

See "Conduit Status" section of README

How do I build on a cross-compiled system like a Cray XC?

1. ln -s $srcdir/other/contrib/cross-configure-[system] $srcdir
2. invoke $srcdir/cross-configure* as you would configure

How do I debug a GASNet application?

1. configure with --enable-debug
2. GASNET_BACKTRACE=1

What envvars affect the behavior of GASNet?

1. GASNET_VERBOSEENV=1
2. README: "Recognized Environment Variables" section
3. conduit/README

How to a trace communication operations in a GASNet app?

1. GASNET_TRACEFILE=trace.%
2. GASNET_TRACEMASK
3. other/contrib/gasnet_trace trace.*

README section "GASNet tracing & statistical collection" for more details