Commits

Tom Roche committed 2986b57

adopts current *_driver framework preparatory to writing fully-IOAPI-zed output

uber_driver.sh and GFED_driver.sh resemble peers from other subprojects, including
* uber_driver failfasts
* use of AQMEII_ag_soil output for template (not the CMAQ kludge previously used only for extents)

Also:
* tweaks to vis_regrid_vis.r, check_coord_vars.ncl (to accommodate above)
* added strings to GFED_driver.sh preparatory to fully-IOAPI-zing output

Tested on terrae/EMVL from
* 'env GIT_SSL_NO_VERIFY=true ./GFED_driver.sh': ran to completion, including check_conservation.ncl in just 18 min!
* 'uber_driver.sh' in fresh terminal: ... 22 min!
* (everything else in 10 min)

TODO:
* make GFED generate IOAPI, view in VERDI to ensure proper grid alignment
** ... and handle {name, path to} VERDI in regrid_utils/bash_utilities.sh
* move all these TODOs to https://bitbucket.org/tlroche/gfed-3.1_global_to_aqmeii-na/issues
*** investigate 8% loss of mass from monthly regrids to hourly regrids
* gate running of check_conservation.ncl::aggregation of hourlies: takes too long!
* post EMVL (or ncl-talk first?) regarding stat_dispersion hang
* make_hourlies.ncl::get_output_file_attributes(...): parameterize explicit constants
* check_raw_fractions.ncl: check that
ftp://gfed3:dailyandhourly@zea.ess.uci.edu/GFEDv3.1/Readme.pdf
> Over a day at each grid cell location, the sum of the eight 3-hourly fire fractions should be equal to 1.0.
* check_coord_vars.ncl: should also check monotonicity of coordvars? or is that guaranteed by NetCDF?
* regrid_utils: make commandline netCDF.stats.to.stdout.r take files beginning with numerals, e.g.
- $ Rscript ./netCDF.stats.to.stdout.r netcdf.fp=5yravg_20111219_pure_emis.nc data.var.name=N2O # fails
+ $ Rscript ./netCDF.stats.to.stdout.r netcdf.fp=yravg_20111219_pure_emis.nc data.var.name=N2O # works
* port AQMEII_driver-style failfast to
** other *_driver: CLMCN_driver, EDGAR_driver, GEIA_driver, GFED_driver
** other uber_driver: GFED::uber_driver
* create common project for regrid_resources à la regrid_utils, so I don't hafta hunt down which resource is in which project
* (this project only) use the EPIC input as template, rather than the TEMPLATE_INPUT_FP I shlep to all the projects.
* all regrids: how to nudge off/onshore as required? e.g., soil or burning emissions should never be offshore, marine emissions should never be onshore.
* all regrid maps: add Caribbean islands (esp Bahamas! for offshore burning), Canadian provinces, Mexican states
* complain to ncl-talk about NCL "unsupported extensions," e.g., .ncf and <null/> (e.g., MCIP output)
* determine why '<-' assignment is occasionally required in calls to visualize.*(...)
* fully document platform versions (e.g., linux, compilers, bash, NCL, R)
* test on
** tlrPanP5 (which now has R package=ncdf4, but readAsciiTable of input .txt's is very slow compared to terrae)
** HPCC (once problem with ncdf4 on amad1 is debugged: in process with JOB and KMF)

Comments (0)

Files changed (4)

 
 ### downloading
 WGET_TO_FILE='wget --no-check-certificate -c -O'
+WGET_TO_STREAM="${WGET_TO_FILE} -"
 CURL_TO_FILE='curl -C - -o'
 CURL_TO_STREAM='curl'
 
 ### empirical constants
 export MOLAR_MASS_N2O='44.0128' # grams per mole of N2O, per wolframalpha.com
 
+### AQMEII-NA constants
+
+## datavar attributes
+# TODO: get from template file or from EPIC
+# "real" data
+AQMEIINA_DV_NAME='N2O'
+# IOAPI pads varattr=long_name to length=16 with trailing spaces
+AQMEIINA_DV_LONG_NAME="$(printf '%-16s' ${AQMEIINA_DV_NAME})"
+# IOAPI pads varattr=units to length=16 with trailing spaces
+AQMEIINA_DV_UNITS="$(printf '%-16s' 'moles/s')"
+# IOAPI pads varattr=var_desc to length=80 with trailing spaces
+# Don't single-quote the payload: double-quote it (OK inside parens inside double-quotes)
+AQMEIINA_DV_VAR_DESC="$(printf '%-80s' "Model species ${AQMEIINA_DV_NAME}")"
+
+## "fake" datavar=TFLAG, required by IOAPI (and more to the point (IIUC), VERDI)
+AQMEIINA_TFLAG_NAME='TFLAG'
+# IOAPI pads varattr=long_name to length=16 with trailing spaces
+AQMEIINA_TFLAG_LONG_NAME="$(printf '%-16s' ${AQMEIINA_TFLAG_NAME})"
+# IOAPI pads varattr=units to length=16 with trailing spaces
+AQMEIINA_TFLAG_UNITS="$(printf '%-16s' '<YYYYDDD,HHMMSS>')"
+# IOAPI pads varattr=var_desc to length=80 with trailing spaces
+AQMEIINA_TFLAG_VAR_DESC="$(printf '%-80s' 'Timestep-valid flags:  (1) YYYYDDD or (2) HHMMSS')"
+
+## dimensions and their attributes
+AQMEIINA_DIM_LAYER_NAME='LAY'
+AQMEIINA_DIM_LAYER_LONG_NAME='index of layers above surface'
+AQMEIINA_DIM_LAYER_UNITS='unitless'
+
+AQMEIINA_DIM_TSTEP_NAME='TSTEP'
+AQMEIINA_DIM_TSTEP_UNITS='' # they vary!
+AQMEIINA_DIM_TSTEP_LONG_NAME='timestep'
+
+# dimensions we don't really need, but VERDI/IOAPI/TFLAG does
+AQMEIINA_DIM_DATETIME_NAME='DATE-TIME'
+AQMEIINA_DIM_VAR_NAME='VAR'
+
+AQMEIINA_DIM_X_N='459'
+AQMEIINA_DIM_X_NAME='COL'
+# AQMEIINA_DIM_X_UNITS='unitless' # my invention
+AQMEIINA_DIM_X_UNITS='m'
+# AQMEIINA_DIM_X_LONG_NAME='Fortran-style index to grid columns from lower-left origin' # my invention (TODO: CHECK it's not from top)
+AQMEIINA_DIM_X_LONG_NAME='grid-center offset from center of projection'
+
+AQMEIINA_DIM_Y_N='299'
+AQMEIINA_DIM_Y_NAME='ROW'
+# AQMEIINA_DIM_Y_UNITS='unitless' # my invention
+AQMEIINA_DIM_Y_UNITS='m'
+# AQMEIINA_DIM_Y_LONG_NAME='Fortran-style index to grid columns from lower-left origin' # my invention (TODO: CHECK it's not from top)
+AQMEIINA_DIM_Y_LONG_NAME="${AQMEIINA_DIM_X_LONG_NAME}"
+
 ### artifact visualization
 
 ## for visualization (generally)
 # no, allow user to override repo code: see `get_bash_utils`
 export BASH_UTILS_FP="${WORK_DIR}/${BASH_UTILS_FN}"
 
-export GET_DAILY_EMIS_FN='get_daily_emis_fp.ncl'
+export GET_DAILY_EMIS_FN='get_daily_emis_fp.ncl' # TODO: port to get_filepath_from_template.ncl and remove
 export GET_DAILY_EMIS_FP="${WORK_DIR}/${GET_DAILY_EMIS_FN}"
 
 export GET_INPUT_AREAS_FN='get_input_areas.ncl'
 GFED_GLOBAL_3HOURLY_FRAC_JAN_FN="${GFED_GLOBAL_3HOURLY_FRAC_FN_TEMPLATE/@@/01}"
 export GFED_GLOBAL_3HOURLY_FRAC_JAN_FP="${WORK_DIR}/${GFED_GLOBAL_3HOURLY_FRAC_JAN_FN}"
 
-### CMAQ downloads
+### Template for `raster` regridding and IOAPI-writing.
 
-## CMAQ/AQMEII-NA template: copy of some meteorological data (MCIP? raw WRF?),
-## with 52 "real" datavars, in the IOAPI format, which here is basically a wrapper around netCDF.
-# (IOAPI can be used with other data, and similar wrappers exist for other models.)
-# I removed all but one of the datavars (with NCO 'ncks'). TODO: script that!
-TEMPLATE_INPUT_GZ_URI='https://bitbucket.org/tlroche/geia_regrid/downloads/emis_mole_all_20080101_12US1_cmaq_cb05_soa_2008ab_08c.EXTENTS_INPUT.nc.gz'
+# This file should have all required IOAPI metadata (including the "fake datavar"=TFLAG) for a single datavar=N2O,
+# but also the correct extent, grid, and other spatial parameters for our domain, to use in `raster` regridding.
+TEMPLATE_INPUT_GZ_URI='https://bitbucket.org/tlroche/aqmeii_ag_soil/downloads/emis_mole_N2O_2008_12US1_cmaq_cb05_soa_2008ab_08c.ncf.gz' # final yearly output from AQMEII_ag_soil, VERDI-viewable
+# formerly also used
+# TEMPLATE_EXTENT_GZ_URI='https://bitbucket.org/tlroche/geia_regrid/downloads/emis_mole_all_20080101_12US1_cmaq_cb05_soa_2008ab_08c.EXTENTS_INPUT.nc.gz'
+# et al
 TEMPLATE_INPUT_GZ_FN="$(basename ${TEMPLATE_INPUT_GZ_URI})"
 TEMPLATE_INPUT_GZ_FP="${WORK_DIR}/${TEMPLATE_INPUT_GZ_FN}"
-TEMPLATE_INPUT_ROOT="${TEMPLATE_INPUT_GZ_FN%.*}" # everything left of the LAST dot
-TEMPLATE_INPUT_FN="${TEMPLATE_INPUT_ROOT}"
+# TEMPLATE_INPUT_ROOT="${TEMPLATE_INPUT_GZ_FN%.*}" # everything left of the LAST dot
+# TEMPLATE_INPUT_FN="${TEMPLATE_INPUT_ROOT}"
+# that will collide with this project's final output! so instead
+TEMPLATE_INPUT_FN='aqmeii_ag_soil_yearly.nc'
 export TEMPLATE_INPUT_FP="${WORK_DIR}/${TEMPLATE_INPUT_FN}"
-export TEMPLATE_DATAVAR_NAME='emi_n2o'
+export TEMPLATE_INPUT_DATAVAR_NAME="${AQMEIINA_DV_NAME}"
+export TEMPLATE_INPUT_TFLAG_NAME="${AQMEIINA_TFLAG_NAME}"
+export TEMPLATE_INPUT_BAND='1' # since position of TSTEP in N2O(TSTEP, LAY, ROW, COL)? seems to work
 
 # ----------------------------------------------------------------------
 # intermediate products
     echo -e "${THIS_FN}::${FUNCNAME[0]}: ERROR: REGRID_UTILS_DIR not defined"
     exit 3
   fi
+
+  # Noticed a problem where code committed to regrid_utils was not replacing code in existing ${REGRID_UTILS_DIR}, so ...
+  if [[ -r "${REGRID_UTILS_DIR}" ]] ; then
+    echo -e "${THIS_FN}::${FUNCNAME[0]}: removing existing regrid_utils folder='${REGRID_UTILS_DIR}'"
+    for CMD in \
+      "rm -fr ${REGRID_UTILS_DIR}/" \
+     ; do
+      echo -e "\n$ ${THIS_FN}::${FUNCNAME[0]}::${CMD}\n"
+      eval "${CMD}" # comment this out for NOPing, e.g., to `source`
+      if [[ $? -ne 0 ]] ; then
+        echo -e "${THIS_FN}::${FUNCNAME[0]}::${CMD}: ERROR: failed or not found\n"
+        exit 4
+      fi
+    done
+  fi
+
   if [[ ! -r "${REGRID_UTILS_DIR}" ]] ; then
-#      "rm -fr ${REGRID_UTILS_DIR}/.git" \
+  # assumes GIT_CLONE_PREFIX set in environment by, e.g., uber_driver
     for CMD in \
-      "git clone ${REGRID_UTILS_URI}.git" \
+      "${GIT_CLONE_PREFIX} git clone ${REGRID_UTILS_URI}.git" \
     ; do
       echo -e "\n$ ${THIS_FN}::${FUNCNAME[0]}::${CMD}\n"
       eval "${CMD}" # comment this out for NOPing, e.g., to `source`
       if [[ $? -ne 0 ]] ; then
-        echo -e "$ ${THIS_FN}::${FUNCNAME[0]}::${CMD}: ERROR: failed or not found\n"
-        exit 1
+        echo # newline
+        echo -e "${THIS_FN}::${FUNCNAME[0]}::${CMD}: ERROR: failed or not found"
+        echo -e "\t(suggestion: check that GIT_CLONE_PREFIX is set appropriately in calling environment)"
+        echo # newline
+        exit 5
       fi
     done
   fi
+
   if [[ ! -r "${REGRID_UTILS_DIR}" ]] ; then
-    echo -e "${THIS_FN}::${FUNCNAME[0]}: ERROR: cannot download REGRID_UTILS_DIR=='${REGRID_UTILS_DIR}'"
-    exit 4
+    echo -e "${THIS_FN}::${FUNCNAME[0]}: ERROR: cannot download regrid_utils to '${REGRID_UTILS_DIR}'"
+    exit 6
   fi  
 } # end function get_regrid_utils
 
 
 function setup_resources {
   for CMD in \
-    'get_template' \
+    'get_template_input' \
     'get_output_areas' \
     'download_GFED' \
   ; do
   done
 } # end function setup_resources
 
-### get the "template" file, used for extent-setting and emissions-file creation
-function get_template {
+### get the "template" file used for regridding and IOAPI-writing
+function get_template_input {
   if [[ -z "${TEMPLATE_INPUT_FP}" ]] ; then
     echo -e "${THIS_FN}::${FUNCNAME[0]}: ERROR: TEMPLATE_INPUT_FP not defined"
-    exit 1
+    exit 8
   fi
   if [[ ! -r "${TEMPLATE_INPUT_FP}" ]] ; then
     if [[ ! -r "${TEMPLATE_INPUT_GZ_FP}" ]] ; then
       for CMD in \
         "${WGET_TO_FILE} ${TEMPLATE_INPUT_GZ_FP} ${TEMPLATE_INPUT_GZ_URI}" \
       ; do
-        echo -e "\n$ ${THIS_FN}::${FUNCNAME[0]}::${CMD}\n"
-        eval "${CMD}" # comment this out for NOPing, e.g., to `source`
-        if [[ $? -ne 0 ]] ; then
-          echo -e "$ ${THIS_FN}::${FUNCNAME[0]}::${CMD}: ERROR: failed or not found\n"
-          exit 1
-        fi
+        echo -e "$ ${CMD}"
+        eval "${CMD}"
       done
     fi
     if [[ -r "${TEMPLATE_INPUT_GZ_FP}" ]] ; then
+      # JIC ${TEMPLATE_INPUT_FP} != ${TEMPLATE_INPUT_GZ_FP} - .gz
       for CMD in \
-        "gunzip ${TEMPLATE_INPUT_GZ_FP}" \
+        "gunzip -c ${TEMPLATE_INPUT_GZ_FP} > ${TEMPLATE_INPUT_FP}" \
       ; do
-        echo -e "\n$ ${THIS_FN}::${FUNCNAME[0]}::${CMD}\n"
-        eval "${CMD}" # comment this out for NOPing, e.g., to `source`
-        if [[ $? -ne 0 ]] ; then
-          echo -e "$ ${THIS_FN}::${FUNCNAME[0]}::${CMD}: ERROR: failed or not found\n"
-          exit 1
-        fi
+        echo -e "$ ${CMD}"
+        eval "${CMD}"
       done
     fi
   fi
   if [[ ! -r "${TEMPLATE_INPUT_FP}" ]] ; then
-    echo -e "${THIS_FN}::${FUNCNAME[0]}: ERROR: TEMPLATE_INPUT_FP=='${TEMPLATE_INPUT_FP}' not readable"
-    exit 1
+    echo -e "${THIS_FN}::${FUNCNAME[0]}: ERROR: cannot read TEMPLATE_INPUT_FP=='${TEMPLATE_INPUT_FP}'"
+    exit 9
   fi
-} # end function get_template
+} # end function get_template_input
 
 ### Call NCL helper to get areas of output/AQMEII-NA gridcells.
 ### Depends on functions={get_get_output_areas, get_MSFs} above.
   fi
 } # end function download_GFED_3hour_fractions
 
-### Call NCL helper to check the sum and distribution of the {raw, as downloaded from GFED} daily and 3-hour fraction values.
-### Depends on function get_check_raw_fracs above.
-### Assumes `ncl` is in PATH: see bash_utilities.sh
-### TODO: pass commandline args to NCL
-### for now: just use envvars :-(
-function check_raw_fracs {
-# ncl # bail to NCL and copy script lines, or ...
-  for CMD in \
-    "ncl -n ${CHECK_RAW_FRACS_FP}" \
-  ; do
-  cat <<EOM
-
-About to run command='${CMD}'. WARNING: may seem to hang while processing!
-
-EOM
-    eval "${CMD}"
-    if [[ $? -ne 0 ]] ; then
-      echo -e "$ ${THIS_FN}::${FUNCNAME[0]}::${CMD}: ERROR: failed or not found\n"
-      exit 1
-    fi
-  done
-} # end function check_raw_fracs
-
 ### Call NCL helper to get areas of input/global gridcells.
 ### Depends on function get_get_input_areas above.
 ### Assumes `ncl` is in PATH: see bash_utilities.sh
 # action functions
 # ----------------------------------------------------------------------
 
+### Call NCL helper to check the sum and distribution of the {raw, as downloaded from GFED} daily and 3-hour fraction values.
+### Depends on function get_check_raw_fracs above.
+### Assumes `ncl` is in PATH: see bash_utilities.sh
+### TODO: pass commandline args to NCL
+### for now: just use envvars :-(
+function check_raw_fracs {
+# ncl # bail to NCL and copy script lines, or ...
+  for CMD in \
+    "ncl -n ${CHECK_RAW_FRACS_FP}" \
+  ; do
+  cat <<EOM
+
+About to run command='${CMD}'. WARNING: may seem to hang while processing!
+
+EOM
+    eval "${CMD}"
+    if [[ $? -ne 0 ]] ; then
+      echo -e "$ ${THIS_FN}::${FUNCNAME[0]}::${CMD}: ERROR: failed or not found\n"
+      exit 1
+    fi
+  done
+} # end function check_raw_fracs
+
 ### Call NCL helper to convert the 12 {raw, as downloaded from GFED} monthly emissions native.txt files
 
 ### * from native .txt format to netCDF

check_coord_vars.ncl

   input_fp_n = input_fp_n + 1
 
   template_fp = getenv("TEMPLATE_INPUT_FP")
-  template_dv_name = getenv("TEMPLATE_DATAVAR_NAME")
+  template_dv_name = getenv("TEMPLATE_INPUT_DATAVAR_NAME")
   input_fp_n = input_fp_n + 1
 
 ;----------------------------------------------------------------------
 # * remember to enable netCDF-4 on your system, e.g.
 # > module add netcdf-4.1.2 # on terrae (a cluster on which I work)
 
+# and run this like
+# START="$(date)" ; echo -e "START=${START}" ; rm -fr /tmp/gfed-3.1_global_to_aqmeii-na/ ; ./uber_driver.sh ; END="$(date)" ; echo -e "  END=${END}" ; echo -e "START=${START}"
+
+# ----------------------------------------------------------------------
+# constants
+# ----------------------------------------------------------------------
+
 THIS=$0
 THIS_FN="$(basename ${THIS})"
 
-# TODO: get from commandline
-ACCESS='file'       # use uncommitted code in current directory
+REPO_ROOT='/tmp' # or location of your choice: repo/workspace created in subdir
+PROJECT_NAME='gfed-3.1_global_to_aqmeii-na' # TODO: get from commandline
+REPO_DIR="${REPO_ROOT}/${PROJECT_NAME}"
+# what actually does the work
+DRIVER_FN='GFED_driver.sh'                  # TODO: get from commandline
+
+### following are "booleans": comment out or change value if !TRUE
+# # Does our platform allow git/ssh? (e.g., terrae does not)
+# CAN_GIT_SSH='TRUE'
+# # Does our platform have HTTP certificates? (e.g., terrae does not)
+# HAVE_HTTP_CERTS='TRUE'
+# if latter is false, git must not check for certificates
+if [[ -z "${HAVE_HTTP_CERTS}" || "${HAVE_HTTP_CERTS}" != 'TRUE' ]] ; then
+  # set for ${DRIVER_FN}
+  export GIT_CLONE_PREFIX='env GIT_SSL_NO_VERIFY=true'
+else
+  export GIT_CLONE_PREFIX=''
+fi
+
+### modes of access to sources (see setup_sources): either copy files or get from repo
+ACCESS='file'       # copy code in current directory
+# ACCESS=='file' uses no repository
 # ACCESS='git/ssh'    # outgoing ssh works (preferred if available)
+# REPO_URI_GIT="git@bitbucket.org:tlroche/${PROJECT_NAME}.git"
 # ACCESS='http+cert'  # outgoing ssh blocked, but you have certificates
+# REPO_URI_HTTP="https://tlroche@bitbucket.org/tlroche/${PROJECT_NAME}.git"
 # ACCESS='http-cert'  # (terrae) outgoing ssh blocked, no certs installed
+# also uses REPO_URI_HTTP
 
 # Workaround for when GFED, aka zea.ess.uci.edu, is down (i.e., not available for downloading)
 # Comment out as needed. TODO: get from commandline
 GFED_GLOBAL_DAILY_FRAC_GZ_FAKE_FN='daily.2008.tar.gz'
 GFED_GLOBAL_3HOURLY_FRAC_GZ_FAKE_FN='3hourly.2008.tar.gz'
 
-REPO_ROOT='/tmp' # or location of your choice: repo/workspace created in subdir
-PROJECT_NAME='gfed-3.1_global_to_aqmeii-na' # TODO: get from commandline
-REPO_DIR="${REPO_ROOT}/${PROJECT_NAME}"
-# what actually does the work
-DRIVER_FN='GFED_driver.sh'                  # TODO: get from commandline
-
 # if [[ "${ACCESS}" == 'file' ]], you'll need
 FILE_LIST="./${DRIVER_FN} \
            ./check_conservation.ncl \
              ./${GFED_GLOBAL_3HOURLY_FRAC_GZ_FAKE_FN}"
 fi
 
-if [[ -z "${REPO_DIR}" ]] ; then
-  echo -e "${THIS_FN}: ERROR: REPO_DIR not defined"
-  exit 1
-fi
-if [[ -d "${REPO_DIR}" ]] ; then
-  echo -e "${THIS_FN}: ERROR? repo dir='${REPO_DIR}' exists: move or delete it before running this script"
-  exit 2
-fi
+# ----------------------------------------------------------------------
+# functions
+# ----------------------------------------------------------------------
+
+function setup_sources {
+  if [[ "${ACCESS}" == 'file' ]] ; then
+    if [[ ! -x "./${DRIVER_FN}" ]] ; then
+      echo -e "${THIS_FN}::${FUNCNAME[0]}: ERROR: driver=='${DRIVER_FN}' not executable (with ACCESS==${ACCESS})"
+      exit 1
+    else
+      for CMD in \
+        "mkdir -p ${REPO_DIR}" \
+        "cp ${FILE_LIST} ${REPO_DIR}/ " \
+      ; do
+        echo -e "$ ${CMD}"
+        eval "${CMD}"
+        if [[ $? -ne 0 ]] ; then
+            echo -e "${THIS_FN}::${FUNCNAME[0]}::${CMD}: ERROR: failed (with ACCESS==${ACCESS})\n"
+            exit 2
+        fi
+      done
+    fi
 
-### USER BEWARE: following `if`s should probably be a `case`
+  else # ACCESS != 'file', i.e., from repo
+
+    if   [[ "${ACCESS}" == 'git/ssh' ]] ; then
+      if [[ -n "${CAN_GIT_SSH}" && "${CAN_GIT_SSH}" == 'TRUE' ]] ; then
+        REPO_URI="${REPO_URI_GIT}"
+      else
+        echo -e "${THIS_FN}::${FUNCNAME[0]}: ERROR: cannot use access==git/ssh on this platform\n"
+        exit 3
+      fi
+    elif [[ "${ACCESS}" == 'http+cert' ]] ; then
+      if [[ -n "${HAVE_HTTP_CERTS}" && "${HAVE_HTTP_CERTS}" == 'TRUE' ]] ; then
+        REPO_URI="${REPO_URI_HTTP}"
+      else
+        echo -e "${THIS_FN}::${FUNCNAME[0]}: ERROR: cannot use access==http+cert on this platform\n"
+        exit 4
+      fi
+    elif [[ "${ACCESS}" == 'http-cert' ]] ; then
+      REPO_URI="${REPO_URI_HTTP}"
+    fi
 
-if [[ "${ACCESS}" == 'file' ]] ; then
-  if [[ ! -x "./${DRIVER_FN}" ]] ; then
-    echo -e "${THIS_FN}: ERROR: ACCESS==file, but driver=='${DRIVER_FN}' not executable"
-    exit 3
-  else
-    # TODO: failfast: test retval/errorlevel from `cp` for failure, e.g.,
-    # > cp: cannot stat `./distill_EPIC_N2O.ncl': No such file or directory
     for CMD in \
-      "mkdir -p ${REPO_DIR}" \
-      "cp ${FILE_LIST} ${REPO_DIR}/ " \
+      "pushd ${REPO_ROOT}" \
+      "${GIT_CLONE_PREFIX} git clone ${REPO_URI}" \
+      "popd" \
     ; do
       echo -e "$ ${CMD}"
       eval "${CMD}"
       if [[ $? -ne 0 ]] ; then
-	  echo -e "$ ${THIS_FN}::access=file::${CMD}: ERROR: failed\n"
-	  exit 4
+          echo -e "${THIS_FN}::${FUNCNAME[0]}::${CMD}: ERROR: failed (with ACCESS==${ACCESS})\n"
+          exit 5
       fi
     done
-  fi
+
+  fi # end testing ACCESS
+} # end function setup_sources
+
+# ----------------------------------------------------------------------
+# payload
+# ----------------------------------------------------------------------
+
+if [[ -z "${REPO_DIR}" ]] ; then
+  echo -e "${THIS_FN}:ERROR: REPO_DIR not defined"
+  exit 6
 fi
 
-if   [[ "${ACCESS}" == 'git/ssh' ]] ; then
-  REPO_URI="git@bitbucket.org:tlroche/${PROJECT_NAME}.git"
-elif [[ "${ACCESS}" == 'http+cert' ]] ; then
-  REPO_URI="https://tlroche@bitbucket.org/tlroche/${PROJECT_NAME}.git"
-elif [[ "${ACCESS}" == 'http-cert' ]] ; then
-  REPO_URI="https://tlroche@bitbucket.org/tlroche/${PROJECT_NAME}.git"
+if [[ -d "${REPO_DIR}" ]] ; then
+  echo -e "${THIS_FN}:ERROR? repo dir='${REPO_DIR}' exists: move or delete it before running this script"
+  exit 7
 fi
-# http-cert !require `env GIT_SSL_NO_VERIFY=true` for `git clone`
 
 if [[ -z "${ACCESS}" ]] ; then
-    echo -e "${THIS_FN}: ERROR: ACCESS not defined: did you uncomment one of the choices above?"
-    exit 5
-elif [[ "${ACCESS}" != 'file' ]] ; then
-  for CMD in \
-    "pushd ${REPO_ROOT}" \
-    "git clone ${REPO_URI}" \
-    "popd" \
-  ; do
-    echo -e "$ ${CMD}"
-    eval "${CMD}"
-    if [[ $? -ne 0 ]] ; then
-	echo -e "$ ${THIS_FN}::access != file::${CMD}: ERROR: failed\n"
-	exit 6
-    fi
-  done
+  echo -e "${THIS_FN}: ERROR: ACCESS not defined: did you uncomment one of the choices above?"
+  exit 8
 fi
 
-pushd ${REPO_DIR}
-./${DRIVER_FN}
+#   'setup_resources' \
+for CMD in \
+  'setup_sources' \
+  "pushd ${REPO_DIR}" \
+  "./${DRIVER_FN}" \
+; do
+#   "ls -alhR ${REPO_DIR}" \
+  echo -e "\n$ ${THIS_FN}::main loop::${CMD}\n"
+  eval "${CMD}" # comment this out for NOPing, e.g., to `source`
+  if [[ $? -ne 0 ]] ; then
+    echo -e "${THIS_FN}::main loop::${CMD}: ERROR: failed or not found\n"
+    exit 9
+  fi
+done
 ## AQMEII-NA template
 
 template_input_fp <- Sys.getenv('TEMPLATE_INPUT_FP')
-template_datavar_name <- Sys.getenv('TEMPLATE_DATAVAR_NAME')
+template_datavar_name <- Sys.getenv('TEMPLATE_INPUT_DATAVAR_NAME')
 
 ## monthly input
 
Tip: Filter by directory path e.g. /media app.js to search for public/media/app.js.
Tip: Use camelCasing e.g. ProjME to search for ProjectModifiedEvent.java.
Tip: Filter by extension type e.g. /repo .js to search for all .js files in the /repo directory.
Tip: Separate your search with spaces e.g. /ssh pom.xml to search for src/ssh/pom.xml.
Tip: Use ↑ and ↓ arrow keys to navigate and return to view the file.
Tip: You can also navigate files with Ctrl+j (next) and Ctrl+k (previous) and view the file with Ctrl+o.
Tip: You can also navigate files with Alt+j (next) and Alt+k (previous) and view the file with Alt+o.