1. jgpiccinali Avatar jgpiccinali
  2. HPC
  3. pug
  4. Issues

CloverLeaf_MPI

Issue #15 new
jg piccinali repo owner created an issue

CloverLeaf_MPI is a pure MPI/F90 test (with no OpenMP).

Get the src

scorep/1.3

GNU

module swap PrgEnv-cray PrgEnv-gnu
module swap craype/2.05 craype/2.1.3
module swap cray-mpich/6.2.2 cray-mpich/7.0.1
module swap cray-libsci/12.1.3 cray-libsci/13.0.0
module swap pmi/5.0.2-1.0000.9906.117.2.ari pmi/5.0.4-1.0000.10161.132.4.ari
module load  cudatoolkit
module load scorep/1.3
  • make clean
make \
MPI_COMPILER="scorep --mpp=mpi  ftn" \
C_MPI_COMPILER="scorep --mpp=mpi  cc" \
COMPILER=GNU \
CFLAGS_GNU= \
FLAGS_GNU=
  • mv GNU.DAINT.MPI GNU.DAINT.MPI.scorep13

8 mpi tasks

  • grep end_time clover.in

PROFILING

export SCOREP_ENABLE_PROFILING=true
export SCOREP_ENABLE_TRACING=false
export SCOREP_CUDA_ENABLE=no
aprun -n8 -N4 -d1  GNU.DAINT.MPI.scorep13 clover.in

  • Ignore: ERROR:150: Module 'cray-mpich/6.2.2' conflicts with the currently loaded module(s) 'cray-mpich/7.0.1'

  • scorep-score scorep-20140921_1633_608628491056605/profile.cubex

Estimated aggregate size of event trace:                   1501MB
Estimated requirements for largest trace buffer (max_buf): 207MB
Estimated memory requirements (SCOREP_TOTAL_MEMORY):       209MB
(hint: When tracing set SCOREP_TOTAL_MEMORY=209MB to avoid intermediate flushes
 or reduce requirements using USR regions filters.)
flt     type  max_buf[B]     visits time[s] time[%] time/visit[us]  region
         ALL 216,914,561 40,163,008   53.69   100.0           1.34  ALL
         MPI 140,012,255 16,450,744   40.24    74.9           2.45  MPI
         USR  64,356,682 19,451,944    5.17     9.6           0.27  USR
         COM  13,846,040  4,260,320    8.28    15.4           1.94  COM
  • cube scorep-20140921_1633_608628491056605/profile.cubex Screen Shot 2014-09-21 at 17.31.46.png

Comments (6)

  3. jg piccinali reporter
    • edited description

    TRACING

    export SCOREP_ENABLE_PROFILING=false
export SCOREP_ENABLE_TRACING=true
export SCOREP_CUDA_ENABLE=no
aprun -n8 -N4 -d1  GNU.DAINT.MPI.scorep13 clover.in

    • Ignore: ERROR:150: Module 'cray-mpich/6.2.2' conflicts with the currently loaded module(s) 'cray-mpich/7.0.1'

    • vampir83 scorep-20140921_1736_612445138776044/traces.otf2 Screen Shot 2014-09-21 at 17.41.41.png

  4. jg piccinali reporter

    perftools/6.2.1

    GNU

    module swap PrgEnv-cray PrgEnv-gnu
module swap craype/2.05 craype/2.1.3
module swap cray-mpich/6.2.2 cray-mpich/7.0.1
module swap cray-libsci/12.1.3 cray-libsci/13.0.0
module swap pmi/5.0.2-1.0000.9906.117.2.ari pmi/5.0.4-1.0000.10161.132.4.ari
module load perftools/6.2.1         # perftools/621jg
    • make clean
    make \
MPI_COMPILER=ftn \
C_MPI_COMPILER=cc \
COMPILER=GNU \
CFLAGS_GNU= \
FLAGS_GNU=
    • mv GNU.DAINT.MPI GNU.DAINT.MPI.perft621
    pat_build -g mpi,io GNU.DAINT.MPI.perft621

    8 MPI TASKS

    • grep end_time clover.in
    aprun -n8 -N4 -d1  GNU.DAINT.MPI.perft621+pat          clover.in
    • Ignore: ERROR:150: Module 'cray-mpich/6.2.2' conflicts with the currently loaded module(s) 'cray-mpich/7.0.1'
    • pat_report GNU.DAINT.MPI.perft621+pat+32267-3678t > xf.txt
    Table 1:  Profile by Function Group and Function

  Time% |     Time |     Imb. |   Imb. |     Calls |Group
        |          |     Time |  Time% |           | Function
        |          |          |        |           |  PE=HIDE

 100.0% | 3.519134 |       -- |     -- | 2112690.6 |Total
|-----------------------------------------------------------------------
|  56.1% | 1.974115 |       -- |     -- | 2056343.0 |MPI
||----------------------------------------------------------------------
||  33.0% | 1.161910 | 0.256009 |  20.6% |  575020.0 |mpi_waitall_
||  12.8% | 0.451565 | 0.079247 |  17.1% |  718775.0 |mpi_isend_
||   7.1% | 0.251233 | 0.014611 |   6.3% |   37500.0 |mpi_allreduce_
||   2.8% | 0.098882 | 0.027397 |  24.8% |  718775.0 |mpi_irecv_
||======================================================================
|  23.4% | 0.822777 | 0.484992 |  42.4% |       1.0 |USER
||----------------------------------------------------------------------
|  23.4% | 0.822777 | 0.484992 |  42.4% |       1.0 | main
||======================================================================
|  19.3% | 0.679128 |       -- |     -- |   43771.0 |MPI_SYNC
||----------------------------------------------------------------------
||  17.4% | 0.611054 | 0.366420 |  60.0% |   37500.0 |mpi_allreduce_(sync)
||   1.3% | 0.045135 | 0.019517 |  43.2% |    6260.0 |mpi_reduce_(sync)
||======================================================================
|   1.2% | 0.043114 |       -- |     -- |   12575.6 |ETC
||----------------------------------------------------------------------
|   1.0% | 0.036490 | 0.255428 | 100.0% |    6305.6 | raw_write
|=======================================================================
    • app2 *.ap2Screen Shot 2014-09-21 at 17.54.29.png

    Use energy hardware counters

    export PAT_RT_SUMMARY=0
export PAT_RT_PERFCTR=default_samp
    • More infos about the counters (SandyBridge):
      • $CRAYPAT_ROOT/share/counters/Counters.papi_intel_fam6mod45
      • $CRAYPAT_ROOT/share/counters/CounterGroups.intel_fam6mod4
      • man pmpc
      • aprun -n1 papi_avail

    Screen Shot 2014-09-21 at 18.10.47.png

    Screen Shot 2014-09-21 at 18.15.55.png

    Screen Shot 2014-09-21 at 18.13.08.png

  6. jg piccinali reporter

    scorep/1.3

    export SCOREP_ENABLE_PROFILING=false
export SCOREP_ENABLE_TRACING=true
export SCOREP_CUDA_ENABLE=no
export SCOREP_METRIC_PAPI_SEP=@
export SCOREP_METRIC_PAPI=PAPI_FP_OPS@PAPI_NATIVE_craypm:::PM_POWER_CAP:NODE

    Screen Shot 2014-09-21 at 19.13.58.png

    • Empty :-( => Score-P metric plugin to monitor the application external energy and power information on Cray platforms
  7. Log in to comment
Assignee
Type
bug
Priority
major
Status
new
Votes
0
Watchers
1