Wiki

# Using MoFEM on ARCHIE-WeSt #

##SSH Connection

To login to ARCHIE-WeSt, execute in the terminal:

$ ssh -l username archie-login.hpc.strath.ac.uk

where username should be replaced by your bitbucket username.

Then enter the password when it is prompted.

##Setting up MoFEM

Create directory for MoFEM:

$ mkdir build

Create a directory for the version you wish to use in build:

$ mkdir build/release

Now run the following commands:

$ wget https://bytebucket.org/likask/mofem-joseph/wiki/downloads/.bashrc
$ wget https://bytebucket.org/likask/mofem-joseph/wiki/downloads/modload.sh

The .bashrc file just dowloaded will have been named .bashrc.1, so we need to rename it using:

$ mv .bashrc.1 .bashrc

cd in build/release and execute the follow to obtain MoFEM source

$ git clone https://username@bitbucket.org/likask/mofem-joseph.git

where username is your bitbucket username, and bitbucket password might be prompted.

Ensure you have pulled from CDashTesting in your branch before continuing

cmake MoFEM in build/release using:

$ cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_FLAGS="-I/users/kkb14112/opt/local/include" -DPETSC_DIR=/users/kkb14112/opt/petsc-3.4.4 -DPETSC_ARCH=arch-linux2-c-opt -DMOAB_DIR=/users/kkb14112/opt/local-moab-4.6/ -DCMAKE_INSTALL_PREFIX=/lustre/gla/eng/$username /users/$username/build/release/mofem-joseph/mofem_v0.1/

where username is your ARCHIE-WeSt username. PETSc, MOAB, etc. are installed from Lukasz directory (username kkb14112).

Compile MoFEM using:

$ make -j 6

where -j 6 is multi-processor compiling, using 6 processors

Test compilation using ctest:

$ ctest -D Experimental

and results are sent to CDash Dashboard, where warnings and errors are shown.

##Working with MoFEM on ARCHIE-WeSt

There are two hard drives on ARCHIE-WeSt; lustre is where you run your code, and users is where you move your data to after your job has finished. Therefore, to create the installation directory in lustre, navigate to:

$ cd /users/username/build/release

were username is your ARCHIE-WeSt username, then:

$ make install

To run a benchmark test, for example, brick slice, we need to copy job scripts from Lukasz directory. First go to the brick slice benchmark directory in lustre:

$ cd /lustre/gla/eng/username/mofem/benchmarks/brick_slice

where username is your ARCHIE-WeSt username. Now copy job scripts:

$ cp /lustre/gla/eng/kkb14112/mofem/benchmarks/brick_slice/*job.sh .

Now open the readme file using:

$ less README

and copy the run command for convergence study. Exit and open convergence_study_job.sh in Vim:

$ vim convergence_study_job.sh

Delete everything after /opt/gridware (select multiple lines using V and delete using d) and place the cursor where this text had been. Then paste the above command into the Vim command line adding r ! at the beginning, changing the order and refinement level as required, change the directory for results to the current directory, change the number of processors to "\$NCORES" as we do not yet know the number of processors we will use, and add --showme, for example:

$ r !./convergence_study.sh -f brick_slice.cub -l -1e-5 -o 1,2,3,4,5 -r 1,2 -d . -p "\$NCORES" --showme

This command will add all the operations that require to be undertaken, approximately 70 lines in this case. The end result should look like this:

convergence_study_job.sh

The number of processors is set on line 10 of convergence_study_job.sh:

#$ -pe mpi-verbose 2

This means we are using 2 nodes and there are 12 processors on each node of ARCHIE-WeSt when running in parallel, so here we are using a total of 24 processors. Time is charged per node so it makes sense to use all 12 processors on a node. If you wish to use less than 12 processors, for example 9 processors, change this line to:

#$ -pe smp-verbose 9

However, bear in mind that you will still be charged for 12 processors as you are using one node and no other users can access the remaining processors you are not using. Estimate the run time of your analysis on line 19:

#$ -ac runtime="1h"

Here we are assuming one hour; it is better to over estimate than under estimate. Note, lines starting with #$ are SGE (SUN Grid Engine) directives and lives starting with # are simply comments. Save and exit the file. Run convergence study using:

$ qsub convergence_study_job.sh

You can check the status of the job using:

$ qstat

which will give something similar to the following output:

job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID 
-----------------------------------------------------------------------------------------------------------------
 493574 0.51026 convergenc username     r     05/08/2014 11:50:57 parallel.q@node260.archie.clus     2

Waiting jobs are denoted qw, and running jobs are denoted r. To delete a job:

$ qdel 493574

The usual output files will be created alongside a log file with the job number in the title, for example out.493574. You can view this file in real time using:

$ tail -f out.493574

Once the job has finished, we can now run face splitting arc length job using the same process as above. To perform an anlysis using face splitting for the 2nd order with 1 level of automatic mesh refinement, copt the run command from the README, open the face_split_arc_length_job.sh in Vim, delete as before and paste the command into the command line making the same changes as before. It will look something like this:

r !./face_split_arc_length.sh -f out_spatial_2_1.h5m -a 1 -n 100 -g 2.3e-8 -d . -t 1e-8 -p "\$NCORES" --showme

Or to run an analysis without face splitting, use:

r !./arc_length.sh -f out_spatial_2_1.h5m -a 1 -n 100 -g 2.3e-8 -d . -t 1e-8 -p "\$NCORES" --showme

Select the number of nodes you wish to use and estimate the time, then run the job as before. Once completed your analysis, ensure you move any data you wish to keep to users/username/$DIR_NAME where username is your ARCHIE_WeSt username and delete it from lustre.

Note: if you want others in our group to be able to view files in your lustre area, run the following command:

$ chmod -R ug+rX /lustre/gla/eng/username/*

where username is your ACRHIE-WeSt username.

To pull from the release branch after updates, navigate to:

cd /users/username/build

where username is your ACRHIE-WeSt username, and run the following commands:

$ /lustre/gla/eng/username/bin/mofem_update.sh
$ /lustre/gla/eng/username/bin/mofem_build.sh
$ make
$ make install

where username is your ACRHIE-WeSt username.

References

http://www.archie-west.ac.uk/