Wiki

Clone wiki

AQMEII-NA_N2O / Building_CMAQ-5.0.1

table of contents

First, a note on usage. CMAQ is a big model with many parts, one of which is CCTM, its Eulerian chemistry and transport component. When one refers (and specifically when I refer) to "building CMAQ," one typically means building CCTM.

introduction

run constraints

US EPA OAQPS has previously run CMAQ-4.7.1 with spatial domain=AQMEII-NA and temporal domain=2008 for the CDC PHASE program. I need to augment the OAQPS run (over the same spatiotemporal domain) with N2O emissions. So why build CMAQ? Why not just extend the version that was run by OAQPS (i.e., the ancestral CMAQ used for our ancestor run) to do what I need?

  1. Adding species to CMAQ is (I'm told) difficult to do to CMAQ versions >= 5.0. CMAQ-5.0 introduced a namelist mechanism for adding new species. CMAQ-5.0.1 is (at this writing) the latest CMAQ version, and fixes bugs in CMAQ-5.0, so I seek to use CMAQ-5.0.1. Note that CMAQ-5.0.1 and CMAQ-5.0 share the same Technical Documentation (documenting their science and engineering) and Operational Guidance (OGD) (documenting how to compile and run them), since CMAQ-5.0.1 is basically just a bugfix version of CMAQ-5.0.
  2. The ancestor run was done on the EPA EMVL terra cluster (better known as terrae). Unfortunately (at this writing) terrae runs are restricted to 72 hours, so I will need to either restructure the run to fit that constraint, or run on the EPA AMAD HPCC cluster (better known as infinity). Ellen Cooter et al in my workgroup recommend the latter.

building CMAQ

the old way to build CMAQ

The "usual" way to build CMAQ components uses a separate builder, bldmake. Specifically, the usual/supported way to build CCTM is to

  1. Compile bldmake for the platform.
  2. Write build high-level configuration information (notably compiler information) to a config.cmaq to be sourced by a bldit.
  3. Write and run a bldit. This is typically a csh script, but one can use any executable that can
    • use the config.cmaq file
    • read and set the appropriate build-time environment variables
    • run bldmake

In addition to building a CCTM instance (and toward that end) the bldit script will generate

These artifacts can presumably be used to more finely characterize the options with which a CCTM instance is built.

a new way to build CMAQ

The currently-public CMAQ-5.0.1 build process is

  • based on CVS, not git. However, Jeff Young at EPA AMAD has made great strides toward rebasing the post-CMAQ-5 build process on git repositories (on EMVL).
  • not fully reproducible, in that it is (IMHO) insufficiently script-driven and repository-backed.

Accordingly I have begun CCTM-build, which is

  • more fully git-based. Particularly, it uses submodules for BLDMAKE, CCTM, ICL, and MECHS.
  • more reproducible. Except for M3LIB (i.e., the libraries required for a CMAQ-5.x build (which is a huge exception!) it is entirely script-driven and repository-backed.

strategy

  1. For comparison purposes, I hope to obtain results that are similar to the ancestor's outputs. I want to build a CMAQ-5.0.1 that is as generic as possible but which satisfy my constraints: i.e., I want to bring in only those configuration options that are
    1. needed to run N2O with reasonable performance and skill. (Note I use performance in the informatic sense--efficiency of computation--and skill to denote scientific effectiveness.)
    2. reflect ancestral configuration options.
  2. I need to run on infinity, so try to find a build already working there that satisfies the previous constraints. Copy/modifying an "known-good" build will also allow me to utilize previously-built CMAQ dependencies, such as bldmake and required libraries.
  3. I will test my infinity-building skill by building and running the CMAQ-5.0.1 benchmark and comparing my output to its reference outputs.

tactics

investigate an infinity build candidate

Donna Schwede has a build on infinity with bldmake tag=master in git repo=i:schwede/cctm/CCTM-git/ that seems pretty close to stock CMAQ-5.0.1. (TODO: verify by checking out CMAQ-5.0.1 from appropriate tag on appropriate repo: work with JOY or SJR.) Its bldmake configuration details are (hopefully) in

roche@imaster:~ $ lsalh $(find ~schwede/cctm/CCTM-git/ -name 'bldit.master')
> -rwxr--r-- 1 schwede mod3 19K May 30 15:07 /home/schwede/cctm/CCTM-git/bldit.master
roche@imaster:~ $ fgrep -e 'config.cmaq' $(find ~schwede/cctm/CCTM-git/ -name 'bldit.master')
> #   setenv M3HOME <source code repository base> set in config.cmaq       #
> #   setenv M3LIB <code libraries> is set it config.cmaq                  #
> #> Source the config.cmaq file to set the build environment
>  source ./config.cmaq
roche@imaster:~ $ lsalh $(find ~schwede/cctm/CCTM-git/ -name 'config.cmaq')
> -rwxr--r-- 1 schwede mod3 3.9K May 30 15:19 /home/schwede/cctm/CCTM-git/config.cmaq
roche@imaster:~ $ fgrep -e 'APPL' $(find ~schwede/cctm/CCTM-git/ -name 'bldit.master')
 > set APPL  = master
 > set MODEL = CCTM_${APPL}
 > set Bld = $Base/BLD_${APPL}
roche@imaster:~ $ lsalh $(find ~schwede/cctm/BLD_master/ -name 'cfg*')
> -rw-r--r-- 1 schwede mod3 3.9K May 30 15:07 /home/schwede/cctm/BLD_master/cfg.CCTM_master
roche@imaster:~ $ lsalh $(find ~schwede/cctm/BLD_master/ -name 'Makefile*')
> lrwxrwxrwx 1 schwede mod3  44 May 30 15:07 /home/schwede/cctm/BLD_master/Makefile -> /home/schwede/cctm/BLD_master/Makefile.intel
> -rwxr--r-- 1 schwede mod3 19K Jul 29 14:11 /home/schwede/cctm/BLD_master/Makefile.intel

saved to

compare variables in candidate, benchmark, and OAQPS builds

Note that the configuration files used by DBS (bldit.master, config.cmaq, cfg.CCTM_master) differ from my benchmark build (see CCTM-build) and from the OAQPS build (bldit.cctm.N5ao, cfg.CCTM_master, and the unfortunately unversioned executable /home/wdx/lib/scom -d -o (saved file dated 12 Feb 13)).

DBS build
var name
DBS valuenote if/how
different from
benchmark
note if/how
different
from OAQPS
APPLmasterCCTM_CMAQv5_0_1
Base/home/schwede/cctm${M3HOME}
Blder${M3HOME}/BLDMAKE-git/bldmake${M3HOME}/BLDMAKE/bldmake -verboseno match
Bld/home/schwede/cctm/BLD_master${M3HOME}/BLD_CCTM_CMAQv5_0_1
bld_osLinux2
CC/share/linux86_64/intel/cc/11.1.059/bin/intel64/iccno match
CFGcfg.${MODEL}
compilerintel
compiler_extifort
cpp_flags-Dparallel
-DSUBST_BARRIER=SE_BARRIER
-DSUBST_COMM=SE_COMM
-DSUBST_DATA_COPY=SE_DATA_COPY
-DSUBST_GATHER=SE_GATHER
-DSUBST_GLOBAL_LOGICAL=SE_GLOBAL_LOGICAL
-DSUBST_GLOBAL_MAX=SE_GLOBAL_MAX
-DSUBST_GLOBAL_MIN=SE_GLOBAL_MIN
-DSUBST_GLOBAL_MIN_DATA=SE_GLOBAL_MIN_DATA
-DSUBST_GLOBAL_SUM=SE_GLOBAL_SUM
-DSUBST_GLOBAL_TO_LOCAL_COORD=
SE_GLOBAL_TO_LOCAL_COORD
-DSUBST_HI_LO_BND_PE=SE_HI_LO_BND_PE
-DSUBST_INIT_ARRAY=SE_INIT_ARRAY
-DSUBST_IN_SYN=SE_IN_SYN
-DSUBST_LOOP_INDEX=SE_LOOP_INDEX
-DSUBST_MODULES=SE_MODULES
-DSUBST_MY_REGION=SE_MY_REGION
-DSUBST_SLICE=SE_SLICE
-DSUBST_SUBGRID_INDEX=SE_SUBGRID_INDEX
-DSUBST_SUM_CHK=SE_SUM_CHK
subset,
see note [1]
CPP_FLAGS""no match
c_compiler/share/linux86_64/intel/cc/11.1.059/bin/intel64/iccno match
c_flags${myCFLAGS} -DFLDMN -I ${MPI_INC}
C_FLAGS${myCFLAGS} -DFLDMN -I ${MPI_INC}no match
EXEC_IDLinux2_x86_64intel
extra_lib-lmpiif-lrdmacm -libumad -lrtsee note [2]
f90_flags${myFRFLAGS} -I.
-I ${IOAPIMOD}
-I ${PARIO}
-I ${STENEX}
-I ${MPI_INC}
no match
F90_FLAGS${myFRFLAGS}
-I ${IOAPIMOD} -I ${PARIO}
-I ${STENEX} -I ${MPI_INC} -I.
no match
FC/share/linux86_64/intel/fc/11.1.059/bin/intel64/ifortno match
FPP/share/linux86_64/intel/fc/11.1.059/bin/intel64/ifortsame but for
intel version
FP/share/linux86_64/intel/fc/11.1.059/bin/intel64/ifortno match
f_compiler/share/linux86_64/intel/fc/11.1.059/bin/intel64/ifortsame but for
intel version
f_flags${myFFLAGS} -I.
-I ${IOAPIMOD}
-I ${MPI_INC}
-I ${PARIO}
-I ${STENEX}
different ordersuperset
of OAQPS
F_FLAGS${myFFLAGS}
-I ${IOAPIMOD} -I ${PARIO}
-I ${STENEX} -I ${MPI_INC} -I.
no match
globalverbose
GlobInc${M3HOME}/ICL-git${M3HOME}/ICLdifferent
repo
IOAPIMOD${M3LIB}/ioapi_3.1/Linux2_x86_64ifortno match
lib_basedir/home/wdx/lib
libraries-L${IOAPIMOD} -lioapi
-L${M3LIB}/mpich/lib -lmpich
${extra_lib}
-L${M3LIB}/netcdf/lib -lnetcdf
-L${PARIO} -lpario
-L${STENEX} -lse_snl
link_flags${myLINK_FLAG}
LINK_FLAGS${myLINK_FLAG}-i-staticno match
LIOAPI${IOAPIMOD} -lioapi
M3HOME/home/yoj/src/repo/project/inf35w/roche/CMAQ-5.0.1/tempno match
M3LIB/home/wdx/lib/x86_64i/intelsame
M3MODEL$cwd${M3HOME}/CCTMpoints to repo
see note [3]
MakeOptset/nullsame
Mechanismcb05tucl_ae6_aqcb05cl_ae5_aq
Mechs${M3HOME}/MECHS-git${M3HOME}/MECHSno match
modelCCTM_masterCCTM_CMAQv5_0_1_Linux2_x86_64intel
MODELCCTM_masterCCTM_CMAQv5_0_1_Linux2_x86_64intel
ModAeroaero/aero6aero5
ModBiogbiog/beis3no match
ModCgrdsspcs/cgrid_spcs_nmlno match
ModCloudcloud/cloud_acm_ae6...ae5
ModCplcouple/gencoor_wrfgencoor
ModDepvdepv/m3dryno match
ModDriverdriver/wrfno match
ModEmisemis/emisno match
ModGasgas/ebi_cb05tuclebi_cb05cl_ae5
ModGridgrid/cartesianno match
ModHadvhadv/yamohyamo
ModHdiffhdiff/multiscalemultiscale
ModInitinit/yamosame!
ModParpar/mpipar_nodistr
ModPaprocan/papa
ModPhotphot/phot_inlinephot_table
ModPlmrsplrise/smokeno match
ModUtilutil/utilutil
ModVadvvadv/wrfvyamo
ModVdiffvdiff/acm2acm2_inline
MPI_INC${M3LIB}/mpich/include
MPI-lmpich/share/linux86_64/mvapich2/mvapich2-1.4/install_x86_64_intel111059/bin
see note [4]
myCC/share/linux86_64/intel/cc/11.1.059/bin/intel64/icc
myCFLAGS-O2
myFC/share/linux86_64/intel/fc/11.1.059/bin/intel64/ifort
myFFLAGS-fixed -132 -O3 -override-limits -fno-alias -mp1
myFRFLAGS-free -O3 -fno-alias -mp1
myLINK_FLAG-i-static
NETCDF${M3LIB}/netcdf/lib -lnetcdf
Optverboseno match
PABase${GlobInc}
PAOptpa_nooppa_noop2
PARIO${M3LIB}/pario${M3LIB}/pario_3.1see note [5]
ParOptset""set
Revisionunsetexplicit
STENEX${M3LIB}/se_snlno match
SUBST_CONST${GlobInc}/fixed/const/CONST.EXTsame name,
diff path
SUBST_EMISPRM${GlobInc}/fixed/emctrl/EMISPRM.EXTresembles
OAQPS
SUBST_EMPR_CH
SUBST_EMPR_VD
SUBST_FILES_ID${GlobInc}/fixed/filenames/FILES_CTM.EXTsame name,
diff path
SUBST_MPI${MPI_INC}/mpif.hsame name,
diff path
SUBST_PACMN_ID${GlobInc}/procan/pa_noop/PA_CMN.EXTsame name,
diff path
SUBST_PACTL_ID${GlobInc}/procan/pa_noop/PA_CTL.EXTsame name,
diff path
SUBST_PADAT_ID${GlobInc}/procan/pa_noop/PA_DAT.EXTsame name,
diff path
SUBST_PE_COMM${GlobInc}/fixed/mpi/PE_COMM.EXTsame name,
diff path
SUBST_RXCMMN${Mechs}/cb05tucl_ae6_aq/RXCM.EXTsame name,
diff path
SUBST_RXDATA${Mechs}/cb05tucl_ae6_aq/RXDT.EXTsame name,
diff path
systemx86_64
Tracertrac0same

Notes:

[1]: DBS' cpp_flags are a proper subset of OAQPS', which also has {-DINTERPB=PINTERPB, -DM3ERR=PM3ERR, -DM3EXIT=PM3EXIT, -DM3WARN=PM3WARN, -DSHUT3=PSHUT3, -DWRITE3=PWRITE3}

[2]: extra_lib is set in a *config.cmaq* in DBS and benchmark (which are CMAQ-5.x builds), so their values are deterministic. Unfortunately in OAQPS, extra_lib was set by the unversioned executable /home/wdx/lib/scom -d -o (saved file dated 12 Feb 13)), and I don't have its build logs to verify which value was actually used.

[3]: M3MODEL is set to /home/schwede/cctm/CCTM-git in config.cmaq, but after that is sourced in bldit.master, it gets set to $cwd.

[4]: benchmark var=mpi == DBS var=MPI == -lmpich

[5]: I did not find PARIO in OAQPS. The differing version in DBS and benchmark is illusory, since

roche@imaster:~ $ ls -ald /home/wdx/lib/x86_64i/intel/pario*
lrwxrwxrwx 1 wdx mod3   11 Apr  3  2012 /home/wdx/lib/x86_64i/intel/pario -> ./pario_3.1
drwxr-xr-x 2 wdx mod3 4096 Jan  6  2012 /home/wdx/lib/x86_64i/intel/pario_3
drwxr-xr-x 2 wdx mod3 4096 Jun 18  2012 /home/wdx/lib/x86_64i/intel/pario_3.1
drwxr-xr-x 2 wdx mod3 4096 Mar 12  2012 /home/wdx/lib/x86_64i/intel/pario_3.1.debug

comparing candidate to benchmark build

summary: the DBS 5.0.1 build seems quite similar to the CMAQ-5.0.1 benchmark build. Main differences seem to be as follows, in decreasing order of perceived importance:

  1. extra_lib, which == -lmpiif in DBS but == -lrdmacm -libumad -lrt in the benchmark. Latter were explicitly overridden for infinity in version of config.cmaq on which I drew, so I'm inclined to use those.
  2. build variables present in benchmark but not DBS:
benchmark var namebenchmark value
Bldmake${M3HOME}/BLDMAKE
CPP/share/linux86_64/intel/fc/11.1.059/bin/intel64/ifort
Cfilecfg.CCTM_CMAQv5_0_1_Linux2_x86_64intel.bld
ICL_CONST${GlobInc}/fixed/const
ICL_EMCTL${GlobInc}/fixed/emctrl
ICL_FILES${GlobInc}/fixed/filenames
ICL_MECH${Mechs}/cb05tucl_ae6_AQ
IOAPI_DIR${IOAPIMOD}
IOAPI_VERSION3.1
IOAPI-L${IOAPIMOD} -lioapi
LIB1-L${STENEX} -lse_snl
LIB2-L${PARIO} -lpario
LIB3""
LIB4-L${M3LIB}/mpich/lib -lmpich ${extra_lib}
LIB5-L${IOAPIMOD} -lioapi
LIB6-L${M3LIB}/netcdf/lib -lnetcdf
LIBS-L${STENEX} -lse_snl -L${PARIO} -lpario -L${M3LIB}/mpich/lib -lmpich ${extra_lib} -L${IOAPIMOD} -lioapi -L${M3LIB}/netcdf/lib -lnetcdf
LINKER/share/linux86_64/intel/fc/11.1.059/bin/intel64/ifort
LNETCDF-L${M3LIB}/netcdf/lib -lnetcdf
mpi-lmpich
MPIRUN/share/linux86_64/mvapich2/mvapich2-1.4/install_x86_64_intel111059/bin/mpiexec
NETCDF_DIR${M3LIB}/netcdf/lib
PAR( -Dparallel )
PoptSE
REPOROOT${M3MODEL}
seLse_snl
Str1( Parallel / Include message passing definitions )
Str2( include SUBST_MPI ${MPI_INC}/mpif.h ; )
STX1( -DSUBST_MODULES SE_MODULES -DSUBST_BARRIER SE_BARRIER )
STX2( -DSUBST_GLOBAL_MAX=SE_GLOBAL_MAX -DSUBST_GLOBAL_MIN=SE_GLOBAL_MIN -DSUBST_GLOBAL_MIN_DATA=SE_GLOBAL_MIN_DATA -DSUBST_GLOBAL_TO_LOCAL_COORD=SE_GLOBAL_TO_LOCAL_COORD -DSUBST_GLOBAL_SUM=SE_GLOBAL_SUM -DSUBST_GLOBAL_LOGICAL=SE_GLOBAL_LOGICAL -DSUBST_LOOP_INDEX=SE_LOOP_INDEX -DSUBST_SUBGRID_INDEX=SE_SUBGRID_INDEX )
STX3( -DSUBST_HI_LO_BND_PE=SE_HI_LO_BND_PE -DSUBST_SUM_CHK=SE_SUM_CHK -DSUBST_INIT_ARRAY=SE_INIT_ARRAY -DSUBST_COMM=SE_COMM -DSUBST_MY_REGION=SE_MY_REGION -DSUBST_SLICE=SE_SLICE -DSUBST_GATHER=SE_GATHER -DSUBST_DATA_COPY=SE_DATA_COPY -DSUBST_IN_SYN=SE_IN_SYN )

comparing candidate to OAQPS build

summary: the candidate build was of CMAQ-5.0.1 on infinity, while the OAQPS build was of CMAQ-4.7.1 on terrae. Other than that :-) the most noticeable differences are

  1. DBS' cpp_flagss are a proper subset of OAQPS': see note [2] above.
  2. DBS' includes are a small proper subset of OAQPS'. OAQPS has many more includes, mostly of the form SUBST_{AE,GC,NR,}_{ADV,DDEP,DEPV,DIFF,EMIS,IC/BC,SCAV,SPC,WDEP}, but also SUBST_{AE_A2AQ,AE_CONCENTRATION,GC_CONCENTRATION,GC_G2AE,GC_G2AQ,IODECL,IOFDESC,IOPARMS,NR_CONCENTRATION,NR_N2AE,NR_N2AQ,TR_T2AQ}
  3. DBS' Modules appears mostly a superset of OAQPS' modules:
    • DBS has no apparent analog of OAQPS' {aero_depv2, ctm_yamo, vyamo}
    • OAQPS has no apparent analog of DBS' {biog/beis3, depv/m3dry, driver/WRF, emis/emis, grid/cartesian, plrise/smoke, spcs/cgrid_spcs_nml}
  4. The corresponding config files from the OAQPS build are
    • bldit.cctm.N5ao, which runs bldmake after doing ...
    • source /home/wdx/lib/scom -d -o, which apparently served a similar purpose as config.cmaq in CMAQ version < 5 builds. Unfortunately, while the latter is currently (and appropriately) versioned, the former was not--it's just another file on the filesystem. (Version dated 12 Feb 13 saved here)
    • cfg.CCTM_master is generated by bldmake from bldit.cctm.N5ao
    • Makefile is generated by bldmake from cfg.CCTM_master

procedure

From the analysis above, I conclude I can use my benchmark CMAQ-5.0.1 build for the N2O run. For the procedure I used to build CMAQ-5.0.1, see CCTM-build.

Updated