Wiki
Clone wikiCS5220-S14 / julia-htc
Using Julia with HTCondor
There are two ways to use Julia parallel processing on C4.
- Run a few workers on a local machine (head node or compute node).
The command
julia -p n
providesn
workers on the local machine. If you are just playing around, I would recommend this. - Run Julia interactively using the HTCondor ClusterManager interface.
We have extended the Julia ClusterManager
interface to support HTCondor
on the C4 cluster, so that you can interactively use Julia on the head node
and still run worker processes on the compute nodes. To do this, you will
need the ClusterManager
package; get it by typing
Pkg.add("ClusterManagers")
Once you have installed the package, you can add workers as follows
using ClusterManagers
addprocs_htc(np) # Where np is the number of processes
The scripts and submission files generated by the addprocs_htc
file
will be stored in a hidden subdirectory of your home directory called
.julia-htc
. If there are problems with starting up the workers, you
should look there (and in particular look at the standard error
outputs from the workers, which are stored in the files labeled
julia-XXXX-X.e
).
In principle, it should also be possible to do something non-interactive,
possibly using the -machinefile
argument to Julia inside a script
similar to openmpiscript
. I have not tried this, though.
Updated