Wiki

In yesterday's ExTASY session, we used "ready-to-use" scripts built using components of the Ensemble Toolkit. Specifically, ExTASY uses the SimulationAnalysisLoop pattern of Ensemble Toolkit (EnTK).

In the previous session, we were introduced to other patterns supported by EnTK:

Pipeline of Ensembles pattern
Ensemble of Pipelines pattern
Ensemble Exchange pattern

In this tutorial session, we will be building "simple workflows" using these patterns but for molecular simulations using these patterns.

Link to documentation: http://radicalensemblemd.readthedocs.io/en/latest/

Preparation

We will be using workflow.iu.xsede.org as the access point to Stampede/ARCHER. You can log into the machine using your XSEDE credentials. ssh <username>@workflow.iu.xsede.org
Once logged in, you can now source the required virtual environment: source $HOME/ve/bin/activate.
The following exercise scripts are configured to be readily run on Stampede/Archer. Exercise 3 alone is configured for Stampede only.

Exercise 1 - Pipeline of Ensembles (PoE) Pattern

The PoE pattern consists of multiple tasks running concurrently and independent of each other; these tasks might be heterogeneous. Executing tasks are associated with a "stage" and only those tasks that belong to the same stage execute concurrently. Each stage waits for all the tasks in the previous stage to complete. See the figure of visual representation of this pattern. The following is a representation of N tasks in each of the M stages.

Pipeline of Ensembles pattern with gromacs workload

In this exercise, we will run a set of gromacs simulations on a remote machine. There are two stages in the pipeline: The first contains tasks which perform the pre-processing (using grompp to merge and convert files); the second stage contains tasks which perform the MD simulations (mdrun). All tasks in each stage complete before the next stage is invoked. Let us see how we can compose this using the Pipeline of Ensembles pattern.

In your home directory, change into the directory building_new_workflows/PoE_pattern. Open the pipeline_of_ensemble.py file.
In the presentation, you came across three primary components: Execution Pattern, Resource Handle and Kernel Plugin. We will identify those components in this script.
From line 31-76, you see a class called Gromacs_simulation of the type of BagofTasks. This class is the execution pattern of type BagofTasks with 2 stages. Once defined, you create an instance of this class at line 120.
Within the execution pattern, in stage_1 we use the custom.grompp kernel plugin and custom.mdrun kernel plugin in stage 2. The plugins are defined in ./kernel_defs/. The kernel definition might not look too intuitive in the first glance, please look at the inline components to understand each section. Once you get the handle of it, you will see the advantage of using the kernel plugins.
Next, in lines 96-107, we define a Resource Handle of SingleClusterEnvironment type targeting a particular resource (Stampede/Archer) and provide the resource requirements for our workflows.

A breakdown of the script:

Locally split a large coordinate file (line 111).
Create our workflow using the BagofTasks execution pattern (lines 31-76 and line 120).
Acquire resources on Stampede/Archer (lines 96-107 and line 123).

Run the workflow on the acquired resources (line 126).
Deallocate resources once workflow is completed (line 129).

Execution command: RADICAL_ENTK_VERBOSE=REPORT python pipeline_of_ensembles.py <resource>

Note

If you want to run on Archer, you have to set the username in line 100 of the script.

Exercise 2 - Ensemble of Pipelines (EoP) Pattern

The pipeline of tasks pattern consists of a bag of independent pipelines. Each pipeline in turn consist of multiple stages of execution, where each stage can contain heterogeneous workloads. Although each stage of a pipeline depends on its predecessor, the pipelines themselves execute independent of each other. The following is a representation of N pipelines with M stages.

Ensemble of Pipelines with gromacs workload

This exercise performs the same set of operations but in a different order. As before, we split a file containing multiple copies of coordinates for a decalanine system into smaller files. There are two stages in the pipeline: The first contains tasks which perform the preprocessing (grompp), second stage contains tasks performing MD simulations (mdrun). Each pipeline executes independent of other pipelines.

Let us see how we can compose this using the Pipeline pattern.

In your home directory, change into the directory building_new_workflows/EoP_pattern. Open the ensemble_of_pipelines.py file.
You will notice that there aren't many changes in this script. The only difference that you will see is that now you import and use the Pipeline pattern as opposed to the BagofTasks pattern. The remaining script is exactly the same as the one used in exercise 1.

Execution command: RADICAL_ENTK_VERBOSE=REPORT python ensemble_of_pipelines.py <resource>

Note

If you want to run on Archer, you have to set the username in line 100 of the script.

Tip

The REPORT level verbose output would have given you a glimpse of the difference in execution in the first two exercises. Try running them again with RADICAL_ENTK_VERBOSE=INFO to see a more detailed output.

Exercise 3 - Ensemble Exchange Pattern

Ensemble Exchange

In your home directory, change into the directory building_new_workflows/EE_pattern. Open the ensemble_exchange.py file.
This script is an example of how Replica Exchange can be performed. Although, a larger script it has the same components as in the previous exercises. Can you identify them ?
You can run this example using RADICAL_ENTK_VERBOSE=REPORT python ensemble_exchange.py <resource>. This example is configured only for xsede.stampede.

These patterns along with the SimulationAnalysisLoop pattern that you worked with, we believe we can support a major class of distributive applications on HPC systems.

Some more exercises

Exercise 4 - "Scaling" the examples

Take up any of the examples that you have worked on today and scale up the number of cores and tasks that you are executing.
Example: Open the Pipeline of Ensembles example (`pipeline_of_ensembles.py`). Use the SingleClusterEnvironment to request 16 cores for a duration of 15 minutes on Stampede or ARCHER and run 64 instances. (Approximately, how long did it take ? What do you expect if you had acquired 64 cores instead ?)
The ExTASY examples from yesterday are also available in the same folder (building_new_workflows/gromacs-lsdmap and building_new_workflows/amber-coco). Modify the the SingleClusterEnvironment to request 32 cores for a duration of 20 minutes on Stampede. Run 2 iterations of the pattern with 32 simulation instances and 1 analysis instances.

Exercise 5 - Creating a custom kernel plugins

In all of the given examples, we provided you with ready-to-use kernel plugins. You can take a crack at creating your own kernel plugin to support your own usecases. Please take a look at http://radicalensemblemd.readthedocs.io/en/latest/dev/dev_writing_kernels.html which describes the process with a step-by-step example.