Wiki
Clone wikiupcxx / docs / system / frontier
Using UPC++ on OLCF Frontier
This document is a continuous work-in-progress, intended to provide up-to-date information on a public install maintained by (or in collaboration with) the UPC++ team. However, systems are constantly changing. So, please report any errors or omissions in the issue tracker.
Typically installs of UPC++ are maintained only for the current default versions of the system-provided environment modules such as for PrgEnv, ROCm and compiler.
This document is not a replacement for the documentation provided by the centers, and assumes general familiarity with the use of the system.
General
Stable installs are available through environment modules. A wrapper is used
to transparently dispatch commands such as upcxx
to an install appropriate to
the currently loaded PrgEnv-{gnu,cray,amd,gnu-amd,cray-amd}
and compiler
(gcc
, cce
, amd
, amd-mixed
) environment modules.
Environment Modules
In order to access the UPC++ installation on Frontier, one must run
$ module load ums ums014
MODULEPATH
before the UPC++ environment
modules will be accessible. We recommend this be done in one's shell startup
files, such as $HOME/.login
or $HOME/.bash_profile
. However, to ensure
compatibility with other OLCF systems sharing the same $HOME
, care should be
taken to do so only if $LMOD_SYSTEM_NAME
equals frontier
.
If not adding the command to one's shell startup files, the module load ...
command will be required once per login shell in which you need a upcxx
environment module.
Environment modules provide two alternative configurations of the UPC++ library:
upcxx-hip
This module supports memory kinds, a UPC++ feature that enables communication to/from GPU memory viaupcxx::copy
onupcxx::global_ptr<T, memory_kind::hip_device>
. When using this module,copy
operations onhip_device
memory leverage ROCm RDMA ("native" memory kinds).upcxx
This module omits support for constructing an activeupcxx::device_allocator<upcxx::hip_device>
object, resulting in a small potential speed-up for applications which do not require a "HIP-aware" build of UPC++.
By default each module above will select the latest recommended version of the
UPC++ library. One can see the installed versions with a command like module
avail upcxx
and optionally explicitly select a particular version with a
command of the form: module load upcxx/20XX.YY.ZZ
.
On Frontier, the UPC++ environment modules select a default network of ofi
.
You can optionally specify this explicitly on the compile line with
upcxx -network=ofi ...
.
Caveats
The installs provided on Frontier utilize the Cray Programming Environment, and
the cc
and CC
compiler wrappers in particular. It is possible to use
upcxx
(or CC
and upcxx-meta
) to link code compiled with the "native
compliers" such as g++
and amdclang++
(provided they match the PrgEnv-*
module). However, direct use of the native compilers to link UPC++ code is not
supported with these installs.
Module name conflicts with E4S SDK
Currently the default MODULEPATH
on Frontier includes center-provided E4S SDK
installs of UPC++ which are not (yet) as well-integrated as the ones described
here. It is currently safe to load upcxx
and upcxx-hip
if one wishes to
use the latest installs described here (the default, and our strong
recommendation). However, module load upcxx/[version]
may resolve to
something different than what one was expecting.
The MODULEPATH
may change each time one loads a PrgEnv-*
module, among
others. This could silently give the E4S SDK installs precedence over the ones
documented here. Consequently, it is advisable to check prior to loading a
upcxx
or upcxx-hip
environment module, as follows. A command such as
module --redirect show upcxx/2023.3.0 | grep -m1 /sw/frontier
will show the full path of the modulefile which would be loaded (without
making changes to ones environment). If the output does not begin with
/sw/frontier/ums/ums014/modulefiles
, then one should repeat module load
ums014
(or simply ml ums014
) to restore the precedence of the installs
provided by the maintainers of UPC++.
Note that these changes to MODULEPATH
are only relevant until you have
loaded a UPC++ environment module.
Job launch
The upcxx-run
utility provided with UPC++ is a relatively simple wrapper,
which in the case of Frontier uses srun
. To have full control over process
placement, thread pinning and GPU allocation, users are advised to consider
launching their UPC++ applications directly with srun
. However, one should
do so only with the upcxx
or upcxx-hip
environment module loaded to ensure
the appropriate environment variable settings.
If you would normally have passed -shared-heap
to upcxx-run
, then it is
particularly important that both UPCXX_SHARED_HEAP_SIZE
and
GASNET_MAX_SEGSIZE
be set accordingly. The values of those and other
potentially relevant environment variables set (or inherited) by upcxx-run
can be listed by adding -show
to your upcxx-run
command (which will print
useful information but not run anything).
Additional information is available in the
Advanced Job Launch
chapter of the UPC++ v1.0 Programmer's Guide.
Each Frontier node contains 64 CPU cores and 4 Slingshot-11 NICs (and 8 GPUs). Currently each UPC++ process can use at most one Slingshot NIC. In order for a job to utilize all four NICs on a Frontier node, all of the following are necessary:
- run at least four processes per node
- ensure each process is bound to distinct CPU cores out of the 64 available
- set environment variables directing each process to use the NIC most appropriate to its core binding
Item 1 is up to the user, and item 2 is the default behavior on Frontier. For item 3, one should ensure the following are set in ones environment:
GASNET_OFI_DEVICE_TYPE=Node
GASNET_OFI_DEVICE_0=cxi2
GASNET_OFI_DEVICE_1=cxi1
GASNET_OFI_DEVICE_2=cxi3
GASNET_OFI_DEVICE_3=cxi0
Those five variables are set by the upcxx
and upcxx-hip
environment
modules, loading one of which is a "best practice" when running a UPC++
application. Care should be taken to ensure they are set even if not
using the environment modules.
Single-node runs
On a system like Frontier, there are multiple complications related to launch of
executables compiled for -network=smp
such that no use of srun
(or simple
wrappers around it) can provide a satisfactory solution in general. Therefore,
we recommend that for single-node (shared memory) application runs on Frontier,
one should compile for the default network (ofi). Note that due to an oddity in
the SLURM configuration, single-node runs on Frontier (and Crusher) need to
include the non-default option srun --network=single_node_vni
when launching
single-node jobs.
It is alternatively acceptable to use
-network=mpi
, such as may be required for some hybrid applications (UPC++ and
MPI in the same executable). However, note that in multi-node runs
-network=mpi
imposes a significant performance penalty.
Batch jobs
By default, batch jobs on Frontier inherit both $PATH
and the $MODULEPATH
from the environment at the time the job is submitted/requested using sbatch
or salloc
. So, no additional steps are needed to use upcxx-run
if a
upcxx
environment module was loaded when sbatch
or salloc
ran.
Interactive example:
NOTE: this example assumes module load ums ums014
has been performed
previously, such as in one's shell startup files.
frontier$ module load upcxx frontier$ upcxx --version UPC++ version 2023.9.0 / gex-2023.9.0-0-g5b1e532 Citing UPC++ in publication? Please see: https://upcxx.lbl.gov/publications Copyright (c) 2023, The Regents of the University of California, through Lawrence Berkeley National Laboratory. https://upcxx.lbl.gov Cray clang version 15.0.0 (324a8e7de6a18594c06a0ee5d8c0eda2109c6ac6) Target: x86_64-unknown-linux-gnu Thread model: posix InstalledDir: /opt/cray/pe/cce/15.0.0/cce-clang/x86_64/share/../bin frontier$ upcxx -O hello-world.cpp -o hello-world.x frontier$ salloc -t 5 --nodes 2 salloc: Granted job allocation 1296496 salloc: Waiting for resource configuration salloc: Nodes frontier[01449-01450] are ready for job frontier$ upcxx-run -n 4 -N 2 ./hello-world.x Hello from 0 of 4 Hello from 1 of 4 Hello from 2 of 4 Hello from 3 of 4
CMake
A UPCXX
CMake package is provided in the UPC++ install on Frontier, as
described in README.md. Thus with the upcxx
environment
module loaded, CMake should "just work".
Known Issues
Correctness problems with intensive communication on HPE Slingshot-11
Currently, there are known issues with the vendor's communications software stack below UPC++ and GASNet-EX which may negatively impact certain communication-intensive UPC++ applications (e.g. those concurrently sending large numbers of RPCs to one or more processes).
Impacts observed have included crashes and hangs of correct UPC++ applications. Or course, either of those failure modes can be the result of other issues. If you believe your application is impacted, please follow the steps below.
- Try running your application on a system with a network other than Slingshot-11 (but not Slingshot-10 which has a similar, but distinct, issue). If the failures persist, then the problem is not the one described here. You should look for defects in your application, or for other defects in UPC++ or external software.
- If you have observed crashes, but not hangs, then try running your
application with
GASNET_OFI_RECEIVE_BUFF_SIZE=recv
in the environment. This disables use of a feature linked to the known source of crashes, but may result in a small reduction in RPC performance. - If you have observed hangs, then try running your application with
all of the following environment variable settings:
GASNET_OFI_RECEIVE_BUFF_SIZE=recv
FI_OFI_RXM_RX_SIZE=8192
FI_CXI_DEFAULT_CQ_SIZE=13107200
FI_MR_CACHE_MONITOR=memhooks
FI_CXI_RX_MATCH_MODE=software
FI_CXI_REQ_BUF_MIN_POSTED=10
FI_CXI_REQ_BUF_SIZE=25165824
These settings will have negative impact on both performance and on memory use. However, in most cases they have been seen to be sufficient to eliminate the problem(s).
If none of the options above resolves crashes or hangs of your communication-intensive UPC++ application, you can seek assistance using the issue tracker.
Information about UPC++ installs on other production systems
Please report any errors or omissions in the issue tracker.
Updated