matrixMulCUBLAS

[jg piccinali]

Get the src

git clone https://github.com/eth-cscs/SciComp.git

Cuda/6.5

Setup

ssh santis01
cd /users/piccinal/SciComp.git/Training/cuda/mpicublas
salloc
module swap PrgEnv-cray PrgEnv-gnu
module load craype-accel-nvidia35
        # cudatoolkit/6.5.14-1.0502.9613.6.1
        # cray-libsci_acc/3.1.1

Compile

CC -I.  -c matrixMulCUBLAS.cpp -o GNU_matrixMulCUBLAS.o
CC -I.  GNU_matrixMulCUBLAS.o  -o GNU.SANTIS

• ldd GNU.SANTIS |grep cuda

        libcudart.so.6.5 => /opt/nvidia/cudatoolkit6.5/default/lib64/libcudart.so.6.5
        libcuda.so.1 => /opt/cray/nvidia/default/lib64/libcuda.so.1
        libcublas.so.6.5 => /opt/nvidia/cudatoolkit6.5/default/lib64/libcublas.so.6.5 
        libcudart.so.5.5 => not found

Fix

• export LD_LIBRARY_PATH=$CRAY_LD_LIBRARY_PATH:$LD_LIBRARY_PATH
• ldd GNU.SANTIS |grep cuda

        libcudart.so.6.5 => /opt/nvidia/cudatoolkit6.5/6.5.14-1.0502.9613.6.1/lib64/libcudart.so.6.5 
        libcuda.so.1 => /opt/cray/nvidia/default/lib64/libcuda.so.1 
        libcublas.so.6.5 => /opt/nvidia/cudatoolkit6.5/6.5.14-1.0502.9613.6.1/lib64/libcublas.so.6.5 

• ldd GNU.SANTIS |grep libsci_acc

        libsci_acc_gnu_48_nv35.so.3 => /opt/cray/libsci_acc/3.1.1/GNU/48/x86_64/lib/libsci_acc_gnu_48_nv35.so.3

Run

• aprun -n1 ./GNU.SANTIS 128

        CUDA Driver Version / Runtime Version     6.5 / 6.5
        Computing result using CUBLAS...cublasGetVersion=6050

Cuda/5.5

Setup

module swap cudatoolkit/6.5.14-1.0502.9613.6.1 cudatoolkit/5.5.22-1.0502.7944.3.1
module swap cray-libsci_acc/3.1.1 cray-libsci_acc/3.0.2

Compile+Fix

export LD_LIBRARY_PATH=$CRAY_LD_LIBRARY_PATH:$LD_LIBRARY_PATH
CC -I.  -c matrixMulCUBLAS.cpp -o GNU_matrixMulCUBLAS.o
CC -I.  GNU_matrixMulCUBLAS.o  -o GNU.SANTIS

• ldd GNU.SANTIS |grep cuda

        libcudart.so.5.5 => /opt/nvidia/cudatoolkit/5.5.22-1.0502.7944.3.1/lib64/libcudart.so.5.5 
        libcuda.so.1 => /opt/cray/nvidia/default/lib64/libcuda.so.1
        libcublas.so.5.5 => /opt/nvidia/cudatoolkit/5.5.22-1.0502.7944.3.1/lib64/libcublas.so.5.5 

• ldd GNU.SANTIS |grep libsci_acc

libsci_acc_gnu_48_nv35.so.3 => /opt/cray/libsci_acc/3.0.2/GNU/48/x86_64/lib/libsci_acc_gnu_48_nv35.so.3

Run

• aprun -n1 ./GNU.SANTIS 128

        CUDA Driver Version / Runtime Version     6.5 / 5.5
        Computing result using CUBLAS...cublasGetVersion=5050

[jg piccinali]

Daint

Compile (GNU)

• cudatoolkit/6.5.14-1.0502.9613.6.1
• cray-libsci_acc/3.1.1

ldd GNU.DAINT |grep cuda
    libcudart.so.6.5 => /opt/nvidia/cudatoolkit6.5/default/lib64/libcudart.so.6.5 (0x00002ace82fc6000)
    libcuda.so.1 => /opt/cray/nvidia/default/lib64/libcuda.so.1 (0x00002ace83216000)
    libcublas.so.6.5 => /opt/nvidia/cudatoolkit6.5/default/lib64/libcublas.so.6.5 (0x00002ace852b1000)

Run

• OK

Computing result using CUBLAS...cublasGetVersion=6050
NID nid00019 Performance= 1220.73 GFlop/s, Time= 0.107 msec, Size= 131072000 Ops
Computing result using host CPU...done.
NID nid00019 Comparing CUBLAS Matrix Multiply with CPU results: PASS

[jg piccinali]

SCOREP/142

GNU/482

[jg piccinali]