1. HPC center at NTUU "KPI"
  2. Untitled project
  3. openvampir

Wiki

Clone wiki

openvampir / Home

Developer information

Developer guide

Discussion

Set up MPI environment to generate OTF traces

Install PAPI

Download PAPI sources. At the time of writing, latest version is papi-4.2.0.tar.gz

$ ./configure --prefix=$HOME/papi
$ make
$ make install

Don't forget to add $HOME/papi/bin to your PATH and $HOME/papi/lib to LD_LIBRARY_PATH. Make sure that both PATH and LD_LIBRARY_PATH are set correctly on every compute node (if unsure, set them in your batch script):

export PATH=$HOME/papi/bin:$PATH
export LD_LIBRARY_PATH=$HOME/papi/lib:$LD_LIBRARY_PATH

Some issues possible while compiling PAPI on 32-bit architectures. Please, consult PAPI documentation in this case and this discussion in case of compilation errors.

Install VampirTrace

VampirTrace instruments your MPI program, adding calls to write function enter/leave events to OTF trace files.

Unfortunately, VampirTrace is not packaged as DEB or RPM, so we need to compile it from source. Download VampirTrace sources. At the time of writing, latest version is VampirTrace-5.12.1.tar.gz

A few important points:

  • To keep our system clean, we will VampirTrace install into $HOME/vt.
  • Debian has OpenMPI headers and libraries in /usr/lib/openmpi, so we need to pass that to configure.
  • For some reason vtunify fails to run (at least, from VampirTrace 5.11.1 compiled on my system) if it was compiled as a library. To disable shared library creation, pass --disable-mpiunify-lib to configure.
$ ./configure --with-papi-dir=$HOME/papi --with-mpi-dir=/usr/lib/openmpi --disable-mpiunify-lib --prefix=$HOME/vt
$ make
$ make install

This installs:

  • vtcc, vtc++, vtf70 etc compiler wrappers
  • required support libraries Compiler wrappers try to determine from command line arguments if the program uses MPI, OpenMP or Pthread. Then wrapper adds appropriate instrumentation arguments and calls gcc.

Don't forget to add $HOME/vt/bin to your PATH and $HOME/vt/lib to LD_LIBRARY_PATH. Make sure that both PATH and LD_LIBRARY_PATH are set correctly on every compute node (if unsure, set them in your batch script):

export PATH=$HOME/vt/bin:$PATH
export LD_LIBRARY_PATH=$HOME/vt/lib:$LD_LIBRARY_PATH

OpenMPI installs mpicc, mpic++ etc compiler wrappers that add appropriate OpenMPI include directories and libraries to the command line and call gcc. But we need them to call vt* compilers. Fortunately, OpenMPI is already integrated with VampirTrace, but VampirTrace compiler wrappers are not installed in Debian.

$ mkdir $HOME/bin
$ cd $HOME/bin
$ ln -s /usr/bin/opal_wrapper mpicc-vt
$ ln -s /usr/bin/opal_wrapper mpic++-vt
$ ln -s /usr/bin/opal_wrapper mpicxx-vt
$ ln -s /usr/bin/opal_wrapper mpiCC-vt
$ ln -s /usr/bin/opal_wrapper mpif77-vt
$ ln -s /usr/bin/opal_wrapper mpif90-vt

Don't forget to add $HOME/bin to your PATH.

Test your setup

We will show how to compile and run a simple MPI program with VampirTrace instrumentation.

Save the following C source as mpi_send_recv.c:

#include <stdio.h>
#include <mpi.h>

const int TAG_SOME_DATA = 10;

int main(int argc, char *argv[])
{
  MPI_Init(&argc, &argv);

  int rank;
  MPI_Comm_rank(MPI_COMM_WORLD, &rank);

  int np;
  MPI_Comm_size(MPI_COMM_WORLD, &np);

  if(rank == 0)
  {
    int x = 42;
    for(int i = 1; i < np; i++)
    {
      MPI_Send(&x, 1, MPI_INT, i, TAG_SOME_DATA, MPI_COMM_WORLD);
    }
  }
  else
  {
    int x = 0;
    MPI_Recv(&x, 1, MPI_INT, 0, TAG_SOME_DATA, MPI_COMM_WORLD,
        MPI_STATUS_IGNORE);
    printf("Task %d.  Got data = %d\n", rank, x);
  }

  MPI_Finalize();
  return 0;
}

Compile:

$ mpicc-vt -vt:mpi -std=c99 -L$HOME/otf1.10/lib/ mpi_send_recv.c -o mpi_send_recv

Run:

$ mpirun -np 8 ./mpi_send_recv
Task 1.  Got data = 42
Task 7.  Got data = 42
Task 6.  Got data = 42
Task 3.  Got data = 42
Task 5.  Got data = 42
Task 4.  Got data = 42
Task 2.  Got data = 42

Check that trace files have been generated:

$ ls mpi_send_recv.*
mpi_send_recv.0.def.z
mpi_send_recv.1.events
mpi_send_recv.1.events.z
mpi_send_recv.2.events
mpi_send_recv.2.events.z
[...]

VampirTrace environment variables

  • VT_MEMTRACE=yes -- trace memory allocation (define a counter).
  • VT_IOTRACE=yes -- trace I/O (write I/O events).
  • VT_METRICS=colon:separated:list -- record specified hardware performance counters (for example, VT_METRICS=PAPI_FP_OPS:PAPI_L2_TCM).

Compiling OpenVampir

Compile libotf

Download libotf. At the time of writing, latest version is OTF-1.10.1coelacanth.tar.gz.

Please note that libotf from Debian and Ubuntu repositories is another, unrelated, library, that just happens to have the same name.

Install dependencies of libotf:

$ apt-get install zlib1g-dev

Skipping this step will not lead to a compile failure, but to a weird failure at runtime.

Compile and install libotf into openvampir/third_party/otf:

$ ./configure --prefix=${HOME}/openvampir/third_party/otf
$ make
$ make install

Compile OpenVampir

Compile OpenVampir (qmake should find libotf in $HOME/otf1.10):

$ cd openvampir/src
$ qmake openvampir.pro
$ make

Add libotf to $LD_LIBRARY_PATH if needed (i.e. if openvampir binary complains about its absence):

$ export LD_LIBRARY_PATH=$HOME/otf1.10/lib:$LD_LIBRARY_PATH

Run OpenVampir:

$ ./release/openvampir

or

$ ./debug/openvampir

Parallel make (make -j 4) also works.

Tests

Compile tests:

$ cd openvampir/src
$ qmake tests.pro
$ make

Run tests:

$ ./tests_release/tests

or

$ ./tests_debug/tests

Updated