summer school - mpi (maqao/lprofmpi)

[jg piccinali]

Setup

• ssh -Y daint01
• git clone https://github.com/bcumming/summer-school.git
• cd summer-school.git/mpi/fortran/
• module swap PrgEnv-cray PrgEnv-intel

Compile

• make clean
• make FFLAGS="-dynamic -openmp"

ftn -dynamic -openmp  -c stats.f90 -o stats.o
ftn -dynamic -openmp  -c data.f90 -o data.o
ftn -dynamic -openmp  -c operators.f90 -o operators.o
ftn -dynamic -openmp  -c linalg.f90 -o linalg.o
ftn -dynamic -openmp  -c io.f90 -o io.o
ftn -dynamic -openmp *.o main.f90  -o main

Run/Profile

• export G=/apps/ela/sandbox/jgp/graphviz/git/gnu_434
• export PATH=$G/bin:$PATH
• export M=/apps/daint/5.2.UP02/maqao/2.1.1
• export PATH=$M/bin:$PATH
  • run without "/usr/bin/time -p"
• sbatch.sh brisi 5 ./main 24 24 1 "1024 1024 50 0.001" \ "LD_PRELOAD=$M/lib/MPI/INTEL/libopmpi_intel_cscs.so"

No tool

===================================================
                       Welcome to mini-stencil!
 MPI : pid           24
 mesh ::         1024 *        1024     dx =  9.775171056389809E-004
 time ::           50 time steps from 0 ..   1.000000000000000E-003
===================================================
--------------------------------------------------------------------------
 simulation took    3.47360515594482       seconds
                  5088  conjugate gradient iterations   1464.76060795000
  per second
                   231  nonlinear newton iterations
--------------------------------------------------------------------------

lprof/mpi

INTEL

• LD_PRELOAD=$M/lib/MPI/GNU/libopmpi_gnu_cscs.so \ aprun -n 8 -N 8 -d 1 -j 1 INTEL.SANTIS 1024 1024 50 0.001

forrtl: severe (174): SIGSEGV, segmentation fault occurred
Process stopped in PMPI​_File​_close from 
/dsl/opt/cray/mpt/7.2.2/gni/mpich2​-intel/14.0/lib/libmpich​_intel.so.3.0.1 
with signal SIGSEGV ​(Segmentation fault)​.

• DDT

#3 diffusion_serial () 
at fortran/main.f90:181 
#2 io::write_parallel (filename=..., u=..., .tmp.FILENAME.len_V$fe=10) 
at fortran/io.f90:83 
#1 mpi_file_close_ (file=0x7fffffff6498, ierr=0x7fffffff649c) 
at /users/jblerest/ONLINE_PROF_FROGGY/MPI/MPI_wrapp.c:8972 
#0 PMPI_File_close () from 
/dsl/opt/cray/mpt/7.2.2/gni/mpich2-intel/14.0/lib/libmpich_intel.so.3.0.1

comment out main.f90:

!cscs call write_header('output.bov')
!cscs call write_parallel('output.bin', x_new)

================== MPI PROFILE (Rank 0) ====================
MPI_Allreduce    hits: 10436     time: 0.290454      size : 83488 
MPI_Barrier      hits: 8     time: 0.000025      size : 0 
MPI_Cart_coords      hits: 1     time: 0.000003      size : 0 
MPI_Cart_shift       hits: 2     time: 0.000003      size : 0 
MPI_Comm_rank    hits: 1     time: 0.000000      size : 0 
MPI_Comm_size    hits: 1     time: 0.000000      size : 0 
MPI_Dims_create      hits: 1     time: 0.000005      size : 0 
MPI_Finalize     hits: 1     time: 0.000008      size : 0 
MPI_Init     hits: 1     time: 0.001497      size : 0 
MPI_Irecv    hits: 11186     time: 0.024760      size : 34363392 
MPI_Isend    hits: 11186     time: 0.040749      size : 34363392 
MPI_Waitall      hits: 5593      time: 0.089110      size : 0