Using IPython with HTCondor

Without MPI

You can use IPython.parallel on C4 with HTCondor as follows:

0. Load the anaconda module.
1. Run the setup_ipython_htc.sh script (under the class script directory, which is in your path). This will set up an htc profile under IPython for you.
2. From the head node, run ipcluster start --n=4 --profile=htc. This will launch four engines on the cluster using HTCondor. You may, of course, change the number of engines to suit yourself.

Now, in a different window, start ipython and run

from IPython.parallel import Client
c = Client(profile="htc")
c.ids

If everything worked correctly, you should see

[0, 1, 2, 3]

As a simple example of how the IPython.parallel interface can be used, see the Monte-Carlo pi example.

With MPI

You can also use MPI with a similar setup to the one described above:

0. Load the anaconda module.
1. Run the setup_ipython_mpi.sh script (under the class script directory, which is in your path). This will set up an mpi profile under IPython for you.
2. From the head node, run ipcluster start --n=4 --profile=mpi. This will launch four engines on the cluster using HTCondor.

With this configuration, you can do anything that you could do with the configuration without MPI; you can also make MPI calls.

Note that you can also use MPI with ordinary Python without setting up an IPython profile or running the ipcluster program. Just create a Python program that uses mpi4py and launch it like any other MPI program.