QuNex 'turnkey' workflow and 'batch' engine

The QuNex Suite enables 'Turnkey' processing across the supported functions. This feature enables full automation across selected QuNex steps. This is accomplished via the run_turnkey QuNex command that enables one step processing of sessions.

It is less flexible than running individual qunex commands; however, it offers the ability to run the entire QuNex Suite workflow in a single command, either locally or via XNAT.

The QuNex turnkey workflow can be invoked in two ways, which can be specified via the following flag:

--turnkeytype=<turnkey_run_type>

The two turnkey workflow options are local or xnat, which will differ in their parameter specification.

• Note: The command examples below do not provide the exhaustive list of all run_turnkey parameters. To get a full parameter listing invoke help via qunex run_turnkey.

QuNex run_turnkey on a cluster or local workstation

Data assumptions

The raw data folder should be:

${RAW_DATA}
        ├── HCPA001.zip
        ├── hcp_mapping.txt
        └── parameters.txt

The folder needs to contain the individual session data (dicom, bids, compressed format). If the data is compressed, it is expected to hold a single session and be named using the following specification:

<subject id>_<session id>.zip.

For example, in the case below this would be HCPA001.zip.

In addition, the folder needs to contain the two QuNex settings files:

• parameters.txt
• hcp_mapping.txt

The run_turnkey command specification

The processing from data ingestion to creation of processing results is then completed by running the following example command, which specifies how to run the turnkey workflow for a local workstation:

# - Folder name to generate
STUDY_NAME="QUNEX_UseCase"

# - Master Working Directory
WORK_DIR="/gpfs/project/fas/n3/Studies"

# - Location for Raw Data
RAW_DATA="/gpfs/project/fas/n3/software/qunexdata/QUNEXTurnkeyUseCase"

# - Name of session to process
SESSIONS_LIST="HCPA001"

# - Input files
INPUT_BATCH_FILE="${RAW_DATA}/parameters.txt"
INPUT_MAPPING_FILE="${RAW_DATA}/hcp_mapping.txt"

# - Type of QuNex Turnkey run
RUNTURNKEY_TYPE="local"

# - QuNex Turnkey steps
RUNTURNKEY_STEPS=\
"create_study,map_raw_data,create_session_info,setup_hcp,create_batch,hcp_pre_freesurfer,hcp_freesurfer,hcp_post_freesurfer,run_qc_t1w,hcp_fmri_volume,hcp_fmri_surface,run_qc_bold,map_hcp_data,create_bold_brain_masks,compute_bold_stats,create_stats_report,extract_nuisance_signal,preprocess_bold,compute_bold_fc_seed,compute_bold_fc_gbc"

# - Command call
qunex run_turnkey \
  --rawdatainput="${RAW_DATA}" \
  --dataformat="DICOM" \
  --batchfile="${INPUT_BATCH_FILE}" \
  --mappingfile="${INPUT_MAPPING_FILE}" \
  --workingdir="${WORK_DIR}" \
  --projectname="${STUDY_NAME}" \
  --path="${WORK_DIR}/${STUDY_NAME}" \
  --sessions="${SESSIONS_LIST}" \
  --sessionids=${SESSIONS_LIST} \
  --turnkeytype="${RUNTURNKEY_TYPE}" \
  --bolds="all" \
  --runparcellations=all \
  --turnkeysteps="${RUNTURNKEY_STEPS}" \
  --overwritestep="no"

The breakdown of the parameters is as follows (for full in-line help run qunex run_turnkey at the terminal):

• turnkeysteps – specifies QuNex commands that are supported as a turnkey step, see the following section for a complete list of turnkey steps

• rawdatainput – specifies where the input data is to be found

• dataformat – specifies the type of raw data [DICOM, BIDS, HCPLS]

• batchfile – specifies the name of the batch parameter file to be used in this study

• mappingfile – specifies the name of the file that describes the mapping of images to HCP compliant names

• workingdir – specifies the working directory that will hold the study

• projectname – the name of the study

• path – the path to the study (it should be <workingdir>/<projectname>)

• local_batchfile – the name of the generated batch file with session details (it should be <workingdir>/<projectname>/processing/batch.txt)

• sessions – the session names to be included in processing

• sessionids – the session names to be included in processing

• turnkeytype – 'local' when run on a local machine (i.e. not XNAT)

• bolds – which bold files to use when generating BOLD QC and functional connectivity results

• runparcellations – which parcellation files to run

The command will generate QUNEX_UseCase study folder where all the results should be present at the end.

Progress can be tracked by keeping track of the standard out and by log files generated in:

${WORK_DIR}/${STUDY_NAME}/processing/logs/runlogs/

${WORK_DIR}/${STUDY_NAME}/processing/logs/comlogs/

List of Turnkey Steps

Most turnkey steps have exact matching qunex commands with several exceptions that fall into two categories:

• map_raw_data step is only relevant to run_turnkey, which maps files on a local filesystem or in XNAT to the study folder.
• run_qc* and compute_bold_fc* are two groups of turnkey steps that have qunex commands as their prefixes. The suffixes of these commands are options of the corresponding qunex command.

A complete list of turnkey commands can be found in List of commands.