Wiki

Clone wiki

process / Tutorial

Section Guide

The tutorial is currently broken into three sections. Before you get started using the tools, you need to install and set up your process environment. If you are interested in running parallel jobs without directly interacting with the queuing system, head on to the kslrun section. If you are interested in reproducibly installing and managing software with versions and variants simultaneously made available through the modules environment, head on to the kslinstall section.

Installing process and Required Packages

Shaheen

If you are running on Shaheen, process is already available. You can make it available by loading the system module 'kslrun'

module load kslrun

You should also take a moment to set up your preferences for kslrun.

kslrun -c

This will launch an interactive session prompting you for the configuration file you'd like to modify and asking you which variables you'd like to change. Accepting the default options will create a file in the current working directory with the default configuration options. You can modify this with a text editor if you wish.

Other Environments

process is a pure Python source package. You should be able to install it with a download, unzip, and install:

curl -O http://bitbucket.org/ahmadia/process/downloads/process-0.2.1.tar.gz
tar -zxvf process-0.2.1.tar.gz
cd process-0.2.0
python setup.py install

Unfortunately, I am currently hard-coding some Shaheen-specific stuff into the kslrun and kslinstall 'header' scripts. Please modify these before launching the install.

kslrun

Overview

kslrun is a tool for that abstracts the interface to the batch management system, granting simplicity in executing jobs both in batch and interactive mode. This is useful in a variety of circumstances, particularly if you are working in a tight loop between submitting jobs and working with their output or trying to interact with another tool like configure or a scripted job execution process.

Basic Use

kslrun with no options emulates a simple mpirun script. You can run it with no arguments except for the executable in the current path and expect output in stdout.

Example #1: Hello World!

cp /opt/share/ksl/test/system/hello_world_mpi.c .
module load GNU
mpicc -o hello hello_world_mpi.c
kslrun hello

Example #1 Expected Output

1.0: hello world

Here, kslrun is executing a single process job and redirecting the output to stdout. On Shaheen, the smallest partition available on contains 128 nodes (512 cores), so be aware that your job will be billed as if 512 cores were in use.

Of course, you can use kslrun to take advantage of 512 processes if you wish! On Shaheen, such a command would look like:

Example #2 Hello World, I brought friends!

kslrun -n 512 -p 128 -m VN hello

Here, we use the -n flag to detail the number of MPI processes we wish to use, and -p to request a partition of 128 nodes. We also use the Shaheen-specific -m VN mode flag to request virtual node mode, mapping 4 processes to each node, i.e. 1 per core.

Example #2 Expected Output

512.0: hello world

Advanced Use

kslrun really shines when used to submit multiple jobs, such as a parameter search over a space. Imagine we want to verify that our hello run code works for every power of 2 between 1 and 512. If we had a lot of time on our hands, we could write a for loop using the previous kslrun command, but that wouldn't be very *parallel*, would it?

We introduce two more flags: -j, which specifies the job name, and -r, which disables redirection of job output to stdout and instead uses two files for stdout and stderr.

The -j flag is followed by an argument jobname, which is used both as an argument to the batch system and to specify job output when stdout redirection is disabled. kslrun creates two files, jobname_id.out, and jobname_id.err, for storing stdout and stderr output from your job, then returns immediately, ready to submit another job into the system.

Example #3 Hello World, In powers of 2

for n in 1 2 4 8 16 32 64 128 256 512
do
  echo "kslrun -r -j hello_mpi_$n -n $n -m VN -p 128 ./hello"
  kslrun -r -j hello_mpi_$n -n $n -m VN -p 128 ./hello
done

# wait until jobs have completed on the system
for n in 1 2 4 8 16 32 64 128 256 512
do
cat hello_mpi_${n}_*.out
done

Example #2 Expected Output

kslrun -r -j hello_mpi_1 -n 1 -m VN -p 128 ./hello
kslrun -r -j hello_mpi_2 -n 2 -m VN -p 128 ./hello
kslrun -r -j hello_mpi_4 -n 4 -m VN -p 128 ./hello
kslrun -r -j hello_mpi_8 -n 8 -m VN -p 128 ./hello
kslrun -r -j hello_mpi_16 -n 16 -m VN -p 128 ./hello
kslrun -r -j hello_mpi_32 -n 32 -m VN -p 128 ./hello
kslrun -r -j hello_mpi_64 -n 64 -m VN -p 128 ./hello
kslrun -r -j hello_mpi_128 -n 128 -m VN -p 128 ./hello
kslrun -r -j hello_mpi_256 -n 256 -m VN -p 128 ./hello
kslrun -r -j hello_mpi_512 -n 512 -m VN -p 128 ./hello

1.0: hello world
2.0: hello world
4.0: hello world
8.0: hello world
16.0: hello world
32.0: hello world
64.0: hello world
128.0: hello world
256.0: hello world
512.0: hello world

To explore the other options available in kslrun, use the -h flag for a complete listing of option flags and their descriptions.

kslinstall

Overview

kslinstall is a tool for automating reproducible research, it is responsible for installing packages and their corresponding module files while maintaining records of the installation procedure which can be used to automate a rebuild.

kslinstall reads .ksl files, which specify information about a package's contents, including:

  • how to build it (sources, patches, configuration, and make)
  • how to use it (module files)

More environment setup

Unfortunately, kslinstall is built on top of an experimental module system on Shaheen and Neser, so you will need to modify your environment aggressively to work with them.

module purge
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:\
/opt/share/ksl/tcl/8.5.8/ppc64/lib
export MODULEPATH=/opt/share/ksl/system/modules/shared/tools:\
/opt/share/ksl/system/modules/`uname -p`/compilers:\
/opt/share/ksl/system/modules/`uname -p`/targets:\
/opt/share/ksl/system/modules/`uname -p`/tools:\
/opt/share/ksl/modules/`uname -p`
if [[ `uname -p` == 'ppc64' ]] then
export MODULEPATH=${MODULEPATH}:/opt/share/ksl/modules/ppc450d
fi
module load init

Creating an Install file

Creating an install file for the first time requires some effort. It is recommended that you first review some example install files, then try and adapt one to create a custom build. Here, we take you through the steps to build a variant of the GNU Scientific Library.

There are a number of examples of kslinstall files contained in /opt/share/ksl/examples/

Some of the sample install files rely on patches to the sources they are installing. In this case, the patch files are stored in /opt/share/ksl/examples/patches

First, let's take a look at the install file.

# -*- python -*-
# GSL installation file, KSL supercomputing team
from ksl.process.install.installer import *
from copy import copy

variants = []

base = installer()
base.name='gsl'
base.version='1.14'
base.release='1'
base.license='GPL'
base.vendor='Free Software Foundation'
base.url='http://www.gnu.org/software/gsl/'
base.summary='GNU Scientific Library'
base.packager='aron'
base.source='gsl-1.14.tar.gz'
base.category='scientific'
base.description=\
"""The GNU Scientific Library (GSL) is a numerical library for C and
C++ programmers. It is free software under the GNU General Public
License.

The library provides a wide range of mathematical routines such as
random number generators, special functions and least-squares
fitting. There are over 1000 functions in total with an extensive test
suite."""

base.paths    = []
base.binaries = []
base.includes=['./include']
base.libdirs=['./lib']
base.libs=['libgsl.a']
base.compile_modules=[]
base.required_modules=['blas', 'cblas']
base.notes=""

s0 = configure, '--disable-shared'
s1 = make
s2 = make_install

standard_install = (s0, s1, s2)

base.install_steps = standard_install
base.tag = ''

ppc450d = copy(base)
ppc450d.build_host = 'ppc64'
ppc450d.compile_modules = ['ibm']
ppc450d.target_arch = 'ppc450d'
variants.append(ppc450d)

You will notice that the file syntax is Python. Your responsibility is to create and return a list of variants. Each variant is a specific build of the package that will be carried out by kslinstall. There is a corresponding modulefile built for every variant, but the modulefiles will be all be stored in a common subdirectory.

There are two main elements of information in setting up an install file: package description and configuration/building.

First, you need to provide some basic information about the package, from the name to the version number, as well as a description and specification of a source package. The source package needs to reside in one of the source directories known to kslinstall. The default source directory is /opt/share/ksl/system/sources, but you may specify others at the command line. You also need to provide some information about what becomes available after the package is installed, including libraries, binaries, and include paths. Finally, you should specify what modules you need to load to use the package (required_modules). If you only need a module to compile the package, specify it instead in the list of compile_modules.

Next, you need to specify the steps to configuring and building the package. In this case, a configure, make, and make install suffices. Some magic is happening in the background, as the blas, cblas, and ibm modules are setting environment variables such as CC, CFLAGS, and LIBS that are used by configure to determine the parameters of the build.

You can build this package in place with the following command:

kslinstall -v --build_dir ./build --root_install_dir $PWD/install --root_module_dir $PWD/module gsl-1.14-1.ksl

After the build is finished (it will take a while), you can inspect the results by looking at the directories specified by the three flags:

  • build_dir -- Temporary output of the build process, where the source file gets unpacked and in-place patches, overlays, and configures are done.
  • root_install_dir -- Location to install packages to. Here is where libraries, binaries, include files, and other output from a 'make install' go. The subdirectories are hierarchically managed by kslinstall.
  • root_module_dir -- Location to install modulefiles to. The subdirectories are also hierarchically managed by kslinstall.

We can add a variant with a GNU build for Shaheen by changing compilers in the install file and re-running kslinstall.

...

ppc450d = copy(base)
ppc450d.build_host = 'ppc64'
ppc450d.compile_modules = ['ibm']
ppc450d.target_arch = 'ppc450d'

variants.append(ppc450d)

ppc450d_gnu = copy(ppc450d)
ppc450d_gnu.tag = 'gnu'
ppc450d_gnu.compile_modules = ['gnu']
variants.append(ppc450d_gnu)

The variant build fails to compile because the math libraries are not brought in automatically by the GNU compiler, an error that would ideally have been caught by configure.

Updated