The Cray compiler cannot compile intrinsics code, so we have to use the Intel compiler. As $MKLROOT will be set by the Intel module, MKL will be automatically detected and used.

#!bash
module swap PrgEnv-cray PrgEnv-intel
module load cudatoolkit
module load hwloc

cmake .. -DCMAKE_INSTALL_PREFIX=<ghost_instdir> -DCUDA_HOST_COMPILER=`which gcc` -DCUDA_NVCC_FLAGS="-I${MPICH_DIR}/include -arch=sm_60" -DCMAKE_EXE_LINKER_FLAGS="-dynamic"

The flag -DCMAKE_EXE_LINKER_FLAGS="-dynamic" is also required for the link step of all applications.

To add support for Zoltan, you can use

module load cray_trilinos

and add the cmake flag -DZOLTAN_INCLUDE_DIR=${CRAY_TRILINOS_PREFIX_DIR}/include when configuring GHOST.

Enabling GPUdirect can greatly enhance the communication performance in multi-node runs. This is currently experimental and may not work in all scenarios, especially for non-trival densemats (like scattered views and such). GHOST has to be compiled with -DGHOST_USE_GPUDIRECT and the executable has to be invoked having the MPICH_RDMA_ENABLED_CUDA=1 environment variable set. Note that this does not work for heterogeneous runs, i.e., all ranks must be of the same GHOST type.