(part of the AQMEII-NA_N2O family of projects)

table of contents

open-source notice

Copyright 2013, 2014 Tom Roche Tom_Roche@pobox.com

This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details.

distributed under the GNU Affero General Public License


Suppose you have CMAQ-ready boundary (BC) and initial (IC) condition IOAPI/netCDF files (collectively known as conditions files) containing a number of species (typically those required for air-quality (AQ) applications), but not some particular species of interest. This project takes as input

and outputs a set of well-formed CMAQ BCs or ICs (corresponding one-to-one to the input) containing all of the AQ species, plus N2O.


Uses bash to drive NCL code to

  1. Read appropriate N2O atmospheric concentrations, in ppmv as required by CMAQ BC and IC files. Sources may include
    • hard-coded input from an ungridded source of spatially- and temporally-coarse mean concentrations (e.g., NOAA ESRL HATS N2O concentrations)
    • (planned) spatially- and temporally-matching concentration grids (someday to derive from a 3D-regrid project)
  2. Read {"well-formed," AQ-only, N2O-free, possibly vertically inconsistent} CMAQ BCs and ICs.
  3. Write {well-formed, "reverticalized," AQ-only, N2O-free} CMAQ BCs and ICs, due to unexpected vertical-layer-boundary mismatch between received IC/BCs and meteorology. (Confirming well-formedness of output WRT IOAPI may require use of VERDI.)
  4. Write {well-formed, vertically consistent, N2O-augmented} output files suitable for use in subsequent modeling.


To run this code,

  1. git clone this repo. (See commandlines on the project homepage, where you probably are now.)
  2. cd to its working directory (where you cloned it to).
  3. Setup your applications and paths.
    1. Download a copy of these bash utilities to the working directory.
    2. Open the file in in an editor! You will probably need to edit its functions setup_paths and setup_apps to make it work on your platform. Notably you will want to point it to your PDF viewer and NCL and R executables.
    3. Open the resource strings (in this repo/directory/folder) in an editor. Make the paths point to your inputs, outputs, etc.
    4. You may also want to open the driver (bash) script in an editor and take a look. Having edited resources (in previous steps), the driver should run Out Of The Box, but you might need to tweak something. In the worst case, you could hardcode your paths and apps in the driver.
    5. By default, the driver
      1. will process all of the input conditions files in BC_AQ_RAW_DIR (for BCs) and IC_AQ_RAW_DIR (for ICs). If you don't want this default behavior, override it by creating files to specify the BCs or ICs which you want to process:
        1. Create a text file (at the path specified by BCs_TO_PROCESS_FP) containing the FQ paths to each BC you want to process (one path per line). Those BCs, and only those, will be processed, in the order specified.
        2. The procedure for ICs is analogous: e.g., the relevant path is specified by ICs_TO_PROCESS_FP. Note that the driver will create these text files if run in the default manner, so the user can edit them for subsequent runs if desired (or just delete them).
      2. will not process any raw input conditions files for which matching (by date) output files are found in the appropriate dir/folder for verticalized output (specified by BC_AQ_VERT_DIR or IC_AQ_VERT_DIR). Matching output is presumed to have been previously verticalized (though this is unfortunately not currently checked.)
    6. Once you've got this project working, you may want to fork it. If so, you can automate running your changes with uber_driver.sh (changing that as needed, too).
  4. Run the driver: $ ./BC_IC_N2O_driver.sh (Remember to prefix with GIT_CLONE_PREFIX='GIT_SSL_NO_VERIFY=true' if running on a platform without the necessary SSL certificates, such as infinity or terrae.) This will download inputs, then run ...
    1. write_N2O.ncl, which will write an N2O datasource file (currently this, used for both BCs and ICs) if not available.
    2. verticalize_conditions.ncl, which will loop over the set of paths to CMAQ conditions files passed to it by the driver, changing their vertical layers to match the input meteorology
    3. IC_plus_N2O.ncl, which will loop over the set of paths to CMAQ IC files passed to it by the driver,
      • (planned) verifying consistency of their vertical layers with sample meteorology
      • augmenting those files with species=N2O.
    4. BC_plus_N2O.ncl, which will perform similar functions WRT passed BC files.


  1. Move all these TODOs to issue tracker.
  2. uber_driver.sh or resource_strings.sh: add provision to {copy, not create} N2O datasource file from uber_driver if desired: it's small now ...
  3. verticalize_conditions.ncl: on inconsistency: pretty-print values side-by-side for easier comparison.
  4. A meta-project TODO mostly for regrid_utils: create a home for generic filepath utilities currently in BC_IC_helpers.ncl:
    1. Create home for generic filepath utilities in regrid_utils with (e.g.) filename=filepaths.ncl.
    2. Make it load existing regrid_utils/get_filepath_from_template.ncl
    3. Move generic filepath utilities currently in ./BC_IC_helpers.ncl and regrid_utils/get_filepath_from_template.ncl to regrid_utils/filepaths.ncl. Particularly, move BC_IC_helpers.ncl::get_filepaths_from_file, which has an exact copy @ IOAPI_spatiotemporality_checker/IOAPIST_helpers.ncl.
  5. Some meta-project TODOs (i.e., that belong in every project's TODOs):

    1. *.sh: use "real bash booleans" à la uber_driver.sh.
    2. *.ncl: put all body code (outside of functions) inside a begin/end block per NCL recommendation
    3. *.ncl, *.sh:
      1. *.ncl: exit() -> status_exit(n) with n sequenced.
      2. *.sh: show {errorcodes, return values} to user! for use in debugging.
    4. uber_driver.sh: should be runnable from anywhere in filesystem, e.g.

      START="$(date)" ; echo -e "START=${START}" ; rm -fr <path to workspace/> ; <path to local repo/>/uber_driver.sh ; END="$(date)" ; echo -e " END=${END}" ; echo -e "START=${START}"

    rather than, as current, requiring running from ./, e.g.,

    pushd <path to local repo/> ; START="$(date)" ; echo -e "START=${START}" ; rm -fr <path to workspace/> ; ./uber_driver.sh ; END="$(date)" ; echo -e "  END=${END}" ; echo -e "START=${START}"