Running MPI jobs on C4

There are two methods to run MPI jobs on C4: either use the mpisub script, or write your own submission script directly.

Using mpisub

The mpisub script prepares an HTCondor submission script to run an MPI job and submits that script to the scheduler. For example,

mpisub -n 2 ./ping.x

will run the program ping.x on two processors. By default, mpisub will only consider nodes allocated to the class; if you want to allow any old node, supply mpisub the -a flag.

In general, the mpisub script uses the default preference ranking to choose the nodes on which things should be run. However, you may certainly change this. For example, if we wanted to run on different nodes (rather than different slots on the same node), we might try adding the requirement that each process should run on slot 1 of some node. This can be done with the -f flag. For example, running

mpisub -n 2 -f req1.txt ./ping.x

where the file req1.txt includes the line

requirements = SlotID == 1

will force this requirement. In general, the -f flag can be used to fold arbitrary HTCondor submission script directives into the file normally generated by mpisub.

Writing your own submission script

The mpisub script generates a standard HTCondor submission script, but you can certainly produce your own parallel job submission. If you choose to do this, you may use the openmpiscript script in /share/cs-insructional/cs5220/script as the executable in your submission script. But I would recommend running mpisub, possibly with the -f option, for most purposes.