These instructions explain how to setup a coupled WRF-LIS-CABLE simulation. For non-coupled simulations, you should simply ignore the instructions for files/options that are irrelevant to your case. There are also some special notes as required for uncoupled simulations.

It is recommended that before running coupled WRF-LIS-CABLE simulations for the first time, you run through the WRF online tutorial as this will explain the sequence of steps for running WRF and the various WRF preprocessor tools, referred to as WPS. WPS is still part of the WRF-LIS-CABLE workflow for configuring the domain (using geogrid.exe) and preparing the WRF boundary conditions (using ungrib.exe, metgrid.exe and real.exe).

There is also a WRF user support forum and it is recommended that for any WRF issues you consult the FAQ.

This guideline also includes information on getting the code.

To run WRF-LIS-CABLE the basic workflow consists of the following steps:

Step 1. Installation and pre-requisite

You need access to a Fortran compiler (only tested with Intel compiler) and Python v3.x.

At NCI, you need to request access to the following projects by MyNCI:

• hh5 (conda environments)
• sx70 (WRF inputs)
• w97 (LIS/LDT inputs)
• ub4 (if you want access to ERA-Interim dataset for meteorological forcing)

Step 2. Get the WRF-LIS-CABLE model

The model is available on: https://bitbucket.org/ccarouge/nuwrf/. You will need to request access the first time by sending an email to cws_help@nci.org.au.

On your preferred HPC system in your preferred location, run:

git clone git@bitbucket.org:ccarouge/nuwrf.git

This will create a nuwrf directory with the model. In your nuwrf directory, the run_build script will compile each component of the code. Before running this script, please update NUWRF_INSTALL_DIR to whatever directory name you want to save the compiled code into. The directory will be created for you. You will need to specify this later when setting the run scripts for WRF-LIS-CABLE.

Note:

• You need to compile each component separately (ldt, lis, wrf, wps)
• If you want to debug then include -o debug <prog name> as follows:

  ./build.sh -p $NUWRF_INSTALL_DIR -c $NUWRFDIR/scripts/python/build/gadi.cfg -o debug lis
  

• If you want to recompile its advised to run 'allclean' first on the command line:

  ./build.sh allclean
  

Once set, compile the model with qsub run_build. When successful the executables are in the NUWRF_INSTALL_DIR that you specified

Step 3. Get the Setup Software

There are two repositories required for setting up each stage of the WRF-LIS-CABLE workflow:

• codes
• templates

Clone these to your preferred HPC system. Here we clone them to the same directory we'll call nuwrf-setup hereafter but you are free to clone those repositories wherever you want:

git clone git@bitbucket.org:ahirsch/codes.git
git clone git@bitbucket.org:ahirsch/templates.git

Several branches are provided in the templates repository. Each branch contains some standard setups for starting out. All of those are setup to the optimal resource requests and timesteps, the preferred WRF physics and the latest CABLE switches: - erai_AUS_50km - Recommended setup to test the workflow. It covers the CORDEX AustralAsia domain at 50 km resolution. - erai_AUS_25km - It is identical to the 50 km setup but at 25 km resolution. - erai_AUS_12km and a nested example erai_AUS_50km_10km_nest - Those are quite expensive to run because of the resolution. Do not use those for testing the setup. - Both use ERA-Interim for the offline met forcing and as boundary conditions for the coupled.

For example, to switch to the erai_AUS_50km branch. Go to your templates directory and run:

git checkout erai_AUS_50km

Step 4. Customise the Setup

Within the nuwrf-setup/ directory, cd to the sub-directory called templates

input_scripts.txt

The first template to modify is input_scripts.txt - items to check/update include:

• EMAIL
• PROJECT (e.g. w35)
• mode: NUWRF (for LIS offline spinup to WRF-LIS coupled - you may as well set to NUWRF so that any long offline runs are ready to switch to coupled)
• start_month, start_year, end_month, end_year - At the moment this will set up the simulation to run the simulations in blocks of months rather than sub-monthly. Generally, we run LIS-CABLE offline as 12 months per job and WRF-LIS-CABLE as 1 month maybe 2.
• len_spinup - longer is preferable for the groundwater parameterisation (e.g. 30 years)
• CODEDIR_ROOT - This is where you have your clone of the WRF-LIS-CABLE code
• EXE_DIR - This is the relative code path for the executables
• RUNDIR_ROOT - Where you plan to run the model (e.g. /scratch/$PROJECT/$USER/)
• SCRIPTSDIR_ROOT - Where you want to archive the run scripts, boundary conditions and outputs
• compile: False if already compiled, True if not compiled
• One thing to keep in mind is that the XPROC and YPROC need to be factors for the number of processors NCPUs. If using the above branches these are already set to an appropriate value

models namelist and configuration files

Then, you may want to customise the models namelist and configuration files to choose your own domain and science options. Depending on your case, you may only need to modify a subset of the files listed below.

namelist.wps

The namelist.wps template defines the domain grid. This information is also needed in namelist.input and ldt.config although, for coupled simulations, these will be set automatically by the scripts as explained later. At the moment, it is configured to run the CORDEX Austral-Asia domain at 25km or 50km depending on which branch you are on.

There is a utility in the CODEDIR_ROOT/WPS/util sub-directory called plotgrids_new.ncl that can plot your domain from the namelist.wps file. This is useful for seeing if the extents are what you wanted before running geogrid.exe

Below is the domain for CORDEX AustralAsia:

template_ldt.config.prelis

Contains the namelist switches for the parameter input datasets. We run WRF-LIS-CABLE on a Lambert domain by default (necessary for the CORDEX AUS44 domain), if running on a different domain you will need to make sure the domain specification is consistent (e.g. lat-lon, mercator etc). The main thing to watch for is the resolution of these datasets. We already have some of these at 1, 10, 50 and 100km. It is recommended to use the ones closest to your domain resolution. You can use the 1km ones anytime, even if for a 50km domain, LDT will just take longer to run and hence waste resources. If you do so, don't forget to update the memory and walltime requirements.

template_ldt.config.postlis

This config file is to run LDT on LIS outputs to prepare for real.exe. It is worth checking but you may not need to modify.

template_lis.config

This contains the namelist switches that are used for both LIS-CABLE and WRF-LIS-CABLE. The switches within ${} notation are auto-populated with information from input_scripts.txt. You will see there are lots of options but you will be most interested in the Land surface models section where you can specify the CABLE switches you want to use. If you are running nested simulations check template_lis.config_2nests to understand what you need to do in the template_lis.config file

template_run_LDTprelis.sh

To run LDT before LIS offline, check the PBS resources.

template_run_LDTpostlis.sh

To run LDT after LIS-offline, check the PBS resources.

template_run_LIS.csh

To run LIS offlince, check the PBS resources. Scaling tests have been done to determine the optimal CPUs for the examples provided.

template_runwrf_raijin.deck

To run WRF. This contains all the namelist options for the WRF part of the code. See here for the definitions or the WRF documentation. Currently set up to use the best performing physics for WRF-LIS-CABLE as determined by Hirsch et al. [2019]

This file will create 3 PBS jobs:

1. to transfer necessary input files to the run directory
2. to run WRF-LIS-CABLE
3. to transfer the outputs to g/data

Check the PBS resources, scaling tests have been done to determine the optimal CPUs for the examples provided

getbdy_grib.deck

To create the WRF boundary condition (BC) files. Currently set up to include the lake initialisation step which is necessary for high resolution runs to be stable

Step 5. Create the setup

Now that you have checked these files and amended as per your requirements, its time to create the decks!

Warning: The setup software must always be launched from the templates/ directory.

You need to run:

cd templates/
module use /g/data/hh5/public/modules
module load conda/analysis3
python <path_to_codes>/prepare_simulation.py

This step will first run the domain generator tool for WRF, geogrid.exe if there are no geo_em.d??.nc files under your SCRIPTSDIR_ROOT directory (set in inputs_script.txt).

Once geogrid.exe has run successfully, it is recommended to check the MAPFAC variables in the geo_em.d??.nc files. Those should be close to 1 for a well-defined domain.

If you need to modify your domain, simply update namelist.wps and remove the geo_em.d??.nc files then simply run prepare_simulation.py again from your templates/ directory.

Once you are satisfied with your domain, you simply run prepare_simulation.py again to setup the rest of the decks:

python <path_to_codes>/prepare_simulation.py

Once completed successfully you can check that things are correct:

• Scripts and outputs are all in SCRIPTSDIR_ROOT as defined in the input_scripts.txt file. In this directory you will find 4 sub-directories:

  1. decks - This contains all the scripts for running each stage of your simulation
  2. bdy_data - This is where all input files will be written to for running your simulation: lis_input.DOMAIN.nc, lis4real_input.DOMAIN.nc, wrfbdy_DOMAIN_YEAR_MONTH, wrffdda_DOMAIN_YEAR_MONTH, wrfinput_DOMAIN_YEAR_MONTH and wrflowinp_DOMAIN_YEAR_MONTH
  3. LIS_output - Where the LIS outputs and restarts are written for both the offline and coupled parts of the simulation!
  4. WRF_output - WRF outputs from the coupled WRF-LIS-CABLE runs so these are predominantly atmospheric variables

The run directory will be whatever you specified for RUNDIR_ROOT in the input_scripts.txt file

IMPORTANT! - IF YOU ARE RUNNING A LONG OFFLINE LIS-CABLE SIMULATION. For climate simulations the CABLE fixed CO2 concentration in the lis.config file needs to be set at an appropriate value representative of the time period of interest. These codes will do this for you automatically, however it is advised that for simulations >5 years you run LIS-CABLE as one model year per job so that the CO2 concentration varies each year. For short spin-up periods you may choose to run this as one job, in which case these codes will take the average CO2 concentration across the spin-up years.

Step 6. Run the simulation

Go to your RUNDIR_ROOT directory. In this directory, you will find 4 sub-directories:

• LDTprelis
• LIS_offline
• LDTpostlis
• coupled_run

For LIS-CABLE you will only care about: LDTprelis and LIS_offline and for WRF-LIS-CABLE you will care about all of them! For WRF standalone, you will only care about coupled_run

Run LDT pre-LIS

Create the input file of the land surface parameters with LDT prelis:

cd LDTprelis 
qsub run_LDTprelis.sh

Run LIS-CABLE offline

cd ../LIS_offline
./run_LIS_YEAR_MONTH

If you are just running LIS-CABLE offline you can stop here! If you want to then run coupled WRF-LIS-CABLE keep reading!

Run LDT post-LIS

Create the information from LIS required for the WRF initial conditions with running LDT post-LIS. This creates the intermediate file SCRIPTSDIR_ROOT/bdy_data/lis4real_input.d01.nc.

cd ../LDTpostlis
qsub run_LDTpostlis.sh 

Run WPS and real.exe

Creates the boundary condition files required for WRF which will be saved in SCRIPTSDIR_ROOT/bdy_data:

cd ../coupled_run
qsub getbdy_grid_YEAR_MONTH.deck

• bdy_data/wrfbdy_DOMAIN_YYYY_mm - Contains the atmospheric BCs at each 6 hourly interval of the coupled simulation
• bdy_data/wrfinput_DOMAIN_YYYY_mm - Contains the initial conditions for the atmospheric fields and land surface fields
• bdy_data/wrflowinp_DOMAIN_YYYY_mm - Contains the surface inputs for the ocean for the duration of the simulation

If you have nested domains there will be a separate set of files for each domain: d01, d02, ... dNN. Note the scripts only handle up to 10 domains currently.

Run WRF-LIS-CABLE

cd ../coupled_run
./runwrf_YYYY_mm_01.deck

When completed successfully the outputs for the atmospheric fields will be transferred to SCRIPTSDIR_ROOT/WRF_output and the CABLE variables in SCRIPTSDIR_ROOT/LIS_output. Since LIS saves its own outputs for the land surface model, it is easy to save the atmospheric and land outputs at different intervals. You can define the output intervals independently in the template_lis.config and template_runwrf_raijin.deck files.