Support (clang-based) PrgEnv-Cray / CCE 9.0

[Paul Hargrove]

Starting with the 9.0 release of the Cray Compilation Environment (CCE), PrgEnv-Cray may be Clang-based. A such, it may be possible to support in UPC++.

Here is an excerpt (w/ my imperfect effort to preserve formatting) from the Cray Classic C and C++ Reference Manual which tries to explain the situation.

About the Cray Classic C and C++ Reference Manual

The Cray® Classic C and C++ Reference Manual includes reference information for the Cray Classic C and C++ compilers.

Cray Classic C and C++ Reference Manual (9.0.0) S-2179

This version supports the Cray Classic C and C++ compiler, version 9.0.0, released June 20, 2019.

Please note that the title of this guide has changed to Cray Classic C and C++ Reference Manual, to differentiate it from the Clang Compiler User's Guide. Beginning with CCE release 9.0.0, CCE is divided into two modules:

• cce/9.0.0, which includes CrayLibs, the Cray Fortran Compiler and the new Clang (LLVM) C, C++, and UPC compiler
• cce/9.0.0-classic, which includes CrayLibs, the Cray Fortran Compiler, and the legacy Cray Classic C, C++, and UPC Compiler

These modules are mutually exclusive. One or the other can be loaded, but not both at the same time.

Note that man page behavior changes depending on which module is loaded. When the CCE Classic module is loaded, the man craycc(1) and man crayCC(1) commands return the classic Cray C and Cray C++ man pages. When the CCE module is loaded, these commands redirect to the Clang C/C++/UPC man page.

[Dan Bonachea]

FWIW here is the Clang User's Manual.

So what is the UPC compiler in cce/9.0.0? If it's not Cray, does that mean they are actually shipping cupc2c+UPCR? Neither manual seems to have an answer to this question.

I'd also like to understand exactly what parts of the cce/9.0.0 C/C++ toolchain are Cray-proprietary. Ie. are they using the clang optimizer and codegen, or just the clang frontend glued onto Cray's optimizer? The generic clang documentation does not include such info...

Here's a page with some more info on the new clang frontend, which makes it sound like they are just using a clang frontend.

[Paul Hargrove]

I don't know what sorts of Cray changes have been made for optimization and/or code gen.
However, I just finished poking the "whence UPC?" question.

Short version:

Looks like UPC compiler/runtime is probably majority Cray's, though use of Intrepid's code cannot be entirely ruled out.

Full version:

The link line (at bottom of this comment) for cc -hupc hello.upc -v shows very little change between 8.7.9 and 9.0.0. In particular -lpgas-dmap still appears to be the runtime, and there is no sign of UPCR or GASNet.

The front-end appears to be clang-9 -x upc.
There is also a /opt/cray/pe/cce/9.0.0/cce-clang/x86_64/bin/clang-upc, which is a symlink to clang.

Running ident or string -a against clang doesn't turn up any identifying marks.

There is no sign of libupc which is part of Interpid's clang-upc.

The only upc.h in the printed search path is /opt/cray/pe/cce/9.0.0/cce/x86_64/include/craylibs/upc.h, which carries a "Copyright 2013 Cray Inc. All Rights Reserved." legend.
However, that is not conclusive both because the same dir seems to be used by the "Classic" (not-Clang) front-end, and because the compiler may be searching locations not listed.

The help output doesn't look like Intrepid's either:

{hargrove@gert01 ~}$ clang-upc --help 2>&1 | grep -i upc
  -fno-upc-auto-amo       [Cray] Do not automatically use network atomics for remote updates (e.g., +=)
  -fno-upc-buffered-async [Cray] Disable use of the buffered_async pragma
  -fno-upc-pattern        [Cray] Do not aggregate remote accesses within simple communication loops
  -fupc-auto-amo          [Cray] Automatically use network atomics for remote updates (e.g., +=)
  -fupc-buffered-async    [Cray] Enable use of the buffered_async pragma
  -fupc-pattern           [Cray] Aggregate remote accesses within simple communication loops
  -fupc-threads=<value>   [Cray] Set the number of threads for a static THREADS translation
  -h <value>              [Cray] Set the driver mode (default|upc)

VS

{phargrov@pcp-d-5 ~}$ /usr/local/pkg/berkeley_upc-2019.4.0/llvm-upc/bin/clang-upc --help 2>& 1| grep -i upc
  -fno-upc-inline-lib     Disable inlining of UPC runtime calls
  -fno-upc-pre-include    Disable pre-include of UPC specific include files
  -fupc-debug             Generate UPC runtime calls that include debugging information
  -fupc-packed-bits=<value>
                          Specify the UPC packed pointer-to-shared representation (e.g. 20,10,34)
  -fupc-pts-vaddr-order=<value>
                          Specify the UPC pointer-to-shared address field order (first or last)
  -fupc-pts=<value>       Specify the UPC pointer-to-shared representation (packed or struct)
  -fupc-threads-<value>   Specify the number of UPC threads at compile time

Verbose outputs with old and new cce versions:

With clang-based cce/9.0.0:

{hargrove@gert01 ~}$ module swap cce cce/9.0.0
{hargrove@gert01 ~}$ cc -hupc hello.upc -v
Cray clang version 9.0.0 (44f957486d17952efd83575cc9709118c146a616) (based on LLVM 9.0.0svn)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /opt/cray/pe/cce/9.0.0/cce-clang/x86_64/bin
Found candidate GCC installation: /opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0
Selected GCC installation: /opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0
Candidate multilib: .;@m64
Selected multilib: .;@m64
 "/opt/cray/pe/cce/9.0.0/cce-clang/x86_64/bin/clang-9" -cc1 -triple x86_64-unknown-linux-gnu -fcray-gpu -fcray-openmp-rename-outlined-funcs -mllvm -cray-llvm-enable-asm-comments=1 -fenhanced-asm=1 -fomp-local-offload-table -emit-obj -mrelax-all -disable-free -main-file-name hello.upc -static-define -mrelocation-model static -mthread-model posix -mdisable-fp-elim -fmath-errno -masm-verbose -mconstructor-aliases -munwind-tables -fuse-init-array -target-cpu core-avx2 -dwarf-column-info -debugger-tuning=gdb -v -resource-dir /opt/cray/pe/cce/9.0.0/cce-clang/x86_64/lib/clang/9.0.0 -isystem /opt/cray/pe/cce/9.0.0/cce-clang/x86_64/lib/clang/9.0.0/include -isystem /opt/cray/pe/cce/9.0.0/cce/x86_64/include/craylibs -D __CRAYXC -D __CRAY_HASWELL -D __CRAYXT_COMPUTE_LINUX_TARGET -I /opt/cray/rca/2.2.20-7.0.0.1_4.25__g8e3fb5b.ari/include -I /opt/cray/alps/6.6.50-7.0.0.1_3.26__g962f7108.ari/include -I /opt/cray/xpmem/2.2.17-7.0.0.1_3.16__g7acee3a.ari/include -I /opt/cray/gni-headers/5.0.12.0-7.0.0.1_7.23__g3b1768f.ari/include -I /opt/cray/dmapp/7.1.1-7.0.0.1_5.10__g25e5077.ari/include -I /opt/cray/pe/pmi/5.0.14/include -I /opt/cray/ugni/6.0.14.0-7.0.0.1_7.20__ge78e5b0.ari/include -I /opt/cray/udreg/2.3.2-7.0.0.1_4.18__g8175d3d.ari/include -I /opt/cray/wlm_detect/1.3.3-7.0.0.1_3.13__g7109084.ari/include -I /opt/cray/krca/2.2.5-7.0.0.1_4.25__g92086a7.ari/include -I /opt/cray-hss-devel/9.0.0/include -internal-isystem /usr/local/include -internal-isystem /opt/cray/pe/cce/9.0.0/cce-clang/x86_64/lib/clang/9.0.0/include -internal-externc-isystem /include -internal-externc-isystem /usr/include -fdebug-compilation-dir /global/homes/h/hargrove -ferror-limit 19 -fmessage-length 115 -fobjc-runtime=gcc -fdiagnostics-show-option -fcolor-diagnostics -o /tmp/hello-070cb1.o -x upc hello.upc -faddrsig
clang -cc1 version 9.0.0 based upon LLVM 9.0.0svn default target x86_64-unknown-linux-gnu
ignoring nonexistent directory "/include"
ignoring duplicate directory "/opt/cray/pe/cce/9.0.0/cce-clang/x86_64/lib/clang/9.0.0/include"
#include "..." search starts here:
#include <...> search starts here:
 /opt/cray/rca/2.2.20-7.0.0.1_4.25__g8e3fb5b.ari/include
 /opt/cray/alps/6.6.50-7.0.0.1_3.26__g962f7108.ari/include
 /opt/cray/xpmem/2.2.17-7.0.0.1_3.16__g7acee3a.ari/include
 /opt/cray/gni-headers/5.0.12.0-7.0.0.1_7.23__g3b1768f.ari/include
 /opt/cray/dmapp/7.1.1-7.0.0.1_5.10__g25e5077.ari/include
 /opt/cray/pe/pmi/5.0.14/include
 /opt/cray/ugni/6.0.14.0-7.0.0.1_7.20__ge78e5b0.ari/include
 /opt/cray/udreg/2.3.2-7.0.0.1_4.18__g8175d3d.ari/include
 /opt/cray/wlm_detect/1.3.3-7.0.0.1_3.13__g7109084.ari/include
 /opt/cray/krca/2.2.5-7.0.0.1_4.25__g92086a7.ari/include
 /opt/cray-hss-devel/9.0.0/include
 /opt/cray/pe/cce/9.0.0/cce-clang/x86_64/lib/clang/9.0.0/include
 /opt/cray/pe/cce/9.0.0/cce/x86_64/include/craylibs
 /usr/local/include
 /usr/include
End of search list.
 "/opt/cray/pe/cce/9.0.0/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld.gold" --eh-frame-hdr -m elf_x86_64 -static -o a.out /usr/lib/../lib64/crt1.o /usr/lib/../lib64/crti.o /opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0/crtbeginT.o -L/opt/cray/rca/2.2.20-7.0.0.1_4.25__g8e3fb5b.ari/lib64 -L/opt/cray/alps/6.6.50-7.0.0.1_3.26__g962f7108.ari/lib64 -L/opt/cray/xpmem/2.2.17-7.0.0.1_3.16__g7acee3a.ari/lib64 -L/opt/cray/dmapp/7.1.1-7.0.0.1_5.10__g25e5077.ari/lib64 -L/opt/cray/pe/pmi/5.0.14/lib64 -L/opt/cray/ugni/6.0.14.0-7.0.0.1_7.20__ge78e5b0.ari/lib64 -L/opt/cray/udreg/2.3.2-7.0.0.1_4.18__g8175d3d.ari/lib64 -L/opt/cray/pe/atp/2.1.3/libApp -L/opt/cray/pe/cce/9.0.0/cce/x86_64/lib/pkgconfig/../ -L/opt/cray/wlm_detect/1.3.3-7.0.0.1_3.13__g7109084.ari/lib64 -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0/../../../../lib64 -L/lib/../lib64 -L/usr/lib/../lib64 -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0/../../.. -L/opt/cray/pe/cce/9.0.0/cce-clang/x86_64/bin/../lib -L/lib -L/usr/lib -rpath=/opt/cray/pe/cce/9.0.0/cce/x86_64/lib -rpath=/opt/cray/pe/gcc-libs /tmp/hello-070cb1.o --no-as-needed -lAtpSigHandler -lAtpSigHCommData --undefined=_ATP_Data_Globals --undefined=__atpHandlerInstall -lpthread -Ttext-segment=0x20000000 -zmax-page-size=0x20000000 --whole-archive -lhugetlbfs --no-whole-archive -lpgas-dmapp -lfi -lu -lrt --undefined=dmapp_get_flag_nbi -ldmapp -lugni -ludreg -lpthread -lm -lcray-c++-rts -lstdc++ -lxpmem -ldmapp -lpthread -lpmi -lpthread -lalpslli -lpthread -lwlm_detect -lugni -lpthread -lalpsutil -lpthread -lrca -ludreg -lquadmath -lm -lmodules -lm -lrt -lfi -lmodules -lm -lquadmath -lrt -lcraymath -lm -lgfortran -lquadmath -lrt -lf -lm -lpthread -lrt -lu -lrt -ldl -lpthread -lcray-c++-rts -lstdc++ -lm -lcsup -lrt /opt/cray/pe/cce/9.0.0/cce/x86_64/lib/no_mmap.o --start-group -lgcc -lgcc_eh -lc --end-group /opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0/crtend.o /usr/lib/../lib64/crtn.o

With cce/8.7.9:

{hargrove@gert01 ~}$ module swap cce cce/8.7.9
{hargrove@gert01 ~}$ cc -hupc hello.upc -v
rm /tmp/pe_5534/hello_1.s

/opt/cray/pe/cce/8.7.9/cce/x86_64/bin/ccfe -hc -hcpu=haswell -D __CRAYXC -D __CRAY_HASWELL -D __CRAYXT_COMPUTE_LINUX_TARGET -hnetwork=aries -hupc -hupc -nostdinc -isystem /opt/cray/pe/cce/8.7.9/cce/x86_64/include/craylibs -isystem /opt/cray/pe/cce/8.7.9/cce/x86_64/include/basic -isystem /opt/gcc/6.1.0/snos/lib/gcc/x86_64-suse-linux/6.1.0/include -isystem /opt/gcc/6.1.0/snos/lib/gcc/x86_64-suse-linux/6.1.0/include-fixed -isystem /opt/gcc/6.1.0/snos/include -isystem /usr/include -I /opt/cray/rca/2.2.20-7.0.0.1_4.25__g8e3fb5b.ari/include -I /opt/cray/alps/6.6.50-7.0.0.1_3.26__g962f7108.ari/include -I /opt/cray/xpmem/2.2.17-7.0.0.1_3.16__g7acee3a.ari/include -I /opt/cray/gni-headers/5.0.12.0-7.0.0.1_7.23__g3b1768f.ari/include -I /opt/cray/dmapp/7.1.1-7.0.0.1_5.10__g25e5077.ari/include -I /opt/cray/pe/pmi/5.0.14/include -I /opt/cray/ugni/6.0.14.0-7.0.0.1_7.20__ge78e5b0.ari/include -I /opt/cray/udreg/2.3.2-7.0.0.1_4.18__g8175d3d.ari/include -I /opt/cray/wlm_detect/1.3.3-7.0.0.1_3.13__g7109084.ari/include -I /opt/cray/krca/2.2.5-7.0.0.1_4.25__g92086a7.ari/include -I /opt/cray-hss-devel/9.0.0/include -D__PEDRVR_GCC_BASE_ACTIVE__ -hpl=/tmp/pe_5534//pldir -rx8503 -ru87000 -CZ /tmp/pe_5534/hello.T -Cx -CX /tmp/pe_5534/hello.xml -hdecompile=/global/u1/h/hargrove/hello. -S /tmp/pe_5534/hello_1.s -hipa3 -ufile_id=1 hello.upc

/opt/cray/pe/cce/8.7.9/cce/x86_64/bin/optcg /tmp/pe_5534//pldir 1

/opt/cray/pe/cce/8.7.9/binutils/x86_64/x86_64-pc-linux-gnu/bin/as -o /tmp/pe_5534/hello_1.o --64 /tmp/pe_5534/hello_1.s
rm /tmp/pe_5534/hello_1.s
rm a.out

/opt/cray/pe/cce/8.7.9/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld /usr/lib64//crt1.o /usr/lib64//crti.o /opt/gcc/6.1.0/snos/lib/gcc/x86_64-suse-linux/6.1.0//crtbeginT.o /opt/gcc/6.1.0/snos/lib/gcc/x86_64-suse-linux/6.1.0//crtfastmath.o /opt/cray/pe/cce/8.7.9/cce/x86_64/lib/no_mmap.o /tmp/pe_5534/hello_1.o -Bstatic -rpath=/opt/cray/pe/cce/8.7.9/cce/x86_64/lib -L /opt/gcc/6.1.0/snos/lib64 -rpath=/opt/cray/pe/gcc-libs -L /opt/cray/rca/2.2.20-7.0.0.1_4.25__g8e3fb5b.ari/lib64 -L /opt/cray/alps/6.6.50-7.0.0.1_3.26__g962f7108.ari/lib64 -L /opt/cray/xpmem/2.2.17-7.0.0.1_3.16__g7acee3a.ari/lib64 -L /opt/cray/dmapp/7.1.1-7.0.0.1_5.10__g25e5077.ari/lib64 -L /opt/cray/pe/pmi/5.0.14/lib64 -L /opt/cray/ugni/6.0.14.0-7.0.0.1_7.20__ge78e5b0.ari/lib64 -L /opt/cray/udreg/2.3.2-7.0.0.1_4.18__g8175d3d.ari/lib64 -L /opt/cray/pe/atp/2.1.3/libApp -L /opt/cray/pe/cce/8.7.9/cce/x86_64/lib/pkgconfig/../ -L /opt/cray/wlm_detect/1.3.3-7.0.0.1_3.13__g7109084.ari/lib64 --no-as-needed -lAtpSigHandler -lAtpSigHCommData --undefined=_ATP_Data_Globals --undefined=__atpHandlerInstall -L /usr/lib64 -L /lib64 -rpath=/usr/lib64 -rpath=/lib64 -lpthread -Ttext-segment=0x20000000 -zmax-page-size=0x20000000 --whole-archive -lhugetlbfs --no-whole-archive -lpgas-dmapp -lfi -lu -lrt --undefined=dmapp_get_flag_nbi -ldmapp -lugni -ludreg -lpthread -lm -lcray-c++-rts -lstdc++ -lxpmem -ldmapp -lpthread -lpmi -lpthread -lalpslli -lpthread -lwlm_detect -lugni -lpthread -lalpsutil -lpthread -lrca -ludreg -lquadmath -lm -lomp -lrt -lcraymp -lpthread -lrt -ldl -lcray-c++-rts -lstdc++ -lm -lmodules -lm -lrt -lfi -lm -lquadmath -lrt -lcraymath -lm -lgfortran -lquadmath -lrt -lf -lm -lpthread -lrt -lu -lrt -ldl -lcray-c++-rts -lstdc++ -lm -lcsup -lrt --as-needed -latomic --no-as-needed -lstdc++ -lpthread --start-group -lc -lcsup -lgcc_eh -lm -lgcc --end-group -T/opt/cray/pe/cce/8.7.9/cce/x86_64/lib/2.23.1.cce.ld -L /opt/gcc/6.1.0/snos/lib/gcc/x86_64-suse-linux/6.1.0 -L /opt/cray/pe/cce/8.7.9/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib -EL --undefined=__pthread_initialize_minimal /opt/gcc/6.1.0/snos/lib/gcc/x86_64-suse-linux/6.1.0//crtend.o /usr/lib64//crtn.o
rm /tmp/pe_5534/hello_1.o

/opt/cray/pe/cce/8.7.9/binutils/x86_64/bin/objcopy --remove-section=.note.ftn_module_data a.out
rm /tmp/pe_5534//hello_1.xml.Ttmp
rm /tmp/pe_5534//hello_1.cif.Ttmp
rm /tmp/pe_5534//hello.Tfe
rm /tmp/pe_5534//hello.xml
rm /tmp/pe_5534//hello.T
rm /tmp/pe_5534//pldir/iline_1
rm /tmp/pe_5534//pldir/gline_1
rm /tmp/pe_5534//pldir/hello_1.inl
rm /tmp/pe_5534//pldir/PL_path
rm /tmp/pe_5534//pldir/PL_module_list
rm /tmp/pe_5534//pldir/PL_global_data
rmdir /tmp/pe_5534//pldir
rmdir /tmp/pe_5534/

[Paul Hargrove]

Update regarding UPC support:

Applying nm to both Cray's and Intrepid's compilers make it appear highly likely that Cray has derived their UPC support from Intrepid's code. They contain, respectively, 92 and 83 public functions with UPC in the name. Of those, 77 are common to both.

[Paul Hargrove]

It is now possible to at least build on Cori w/ this compiler using non-default module versions.

$ module swap PrgEnv-intel PrgEnv-cray
$ module swap cce cce/9.0.0
$ module swap cray-libsci cray-libsci/19.06.1
$ module swap cray-mpich cray-mpich/7.7.8

Use of -Wno-unused-command-line-argument is highly recommended because Cray's wrappers are not sensitive to the pass (e.g. adding linker options to preprocess-only steps) and the clang front-end is notoriously verbose about options not used for the selected pass(es).

I am using the PR96 branch for testing at the moment.
The following works with just minor tweaks in utils/system-checks.sh! $ env CROSS=cray-aries-slurm CC='cc -Wno-unused-command-line-argument' CXX='CC -Wno-unused-command-line-argument' ./install ...

I hope to get some full CI results over the weekend.

[Paul Hargrove]

With the switch to newer modules listed above, I am able to get a SUCCESSFUL full CI run on Cori with this compiler! So, I am updating the description to say "Support" rather than "Validate".

FWIW, I suspect one can unload, rather than upgrade, the cray-libsci module, but have not yet attempted to confirm this.

It should be simple to update utils/system-checks.sh to allow one to build/install UPC++ with PrgEnv-cray and cce/9.0. What is less simple is making sure that (at least) the cray-mpich module is sufficiently new, and documenting that requirement clearly. I will, of course, plan to PR such work when I can get around to it.

NOTE: I am working to get this combination into CI at least weekly. In the event that such testing shows failures that I did not observe in last night's runs, we may hold off on claiming support for this PrgEnv/compiler.

[Paul Hargrove]

Changes to docs and system-checks now in PR#107

[Paul Hargrove]

Add PrgEnv-cray w/ cce/9.0 as supported

Resolves issue #233

→ <<cset 337e0ac196e8>>