Preprocessing and analysis commands are typically run on batches of sessions that most often represent all the sessions in a study. Additionally, a number of parameters that are to be used in preprocessing and analysis commands are stable and do not change between command invocations. To facilitate processing of batches of sessions from small sets to thousands, QuNex utilizes batch files. The basic name for a batch file is something like
batch.txt; however, any name can be used, which allows setting up separate versions of batch files for different purposes or preprocessing and analysis variants.
A batch file (e.g.
batch.txt) consists of two core elements:
- the parameter header
- the session-level information for all sessions to be processed with the parameters in the header
The information for the parameter header is typically taken from the parameter file (commonly stored in:
/<path_to_sessions_folder>/spec/parameters.txt) to allow flexible deployment and re-generation of the batch file for a given set of sessions.
The final batch files are by default stored in
/<study_folder>/processing/ to allow easy access to all the study-specific batch iterations. In principle the final batch file can be stored anywhere as long as the relevant QuNex command specifies its absolute path.
Batch file group-level header information
The header element of the batch file provides group-level information that will be used across all specified sessions in a form of
_parameter : value pairs, one per line. Here is a concrete example of a group level section (for full details of all the parameter explanation see
# --- basic settings _image_source: hcp _image_target: cifti _log: keep _glm_matrix: both _glm_residuals: forget _bold_preprocess: WM|flanker|rest _base_folder: /data/studies/WM3/sessions _TR: 2.5 _omit: 0 _pignore: hipass=linear|regress=ignore|lopass=linear # --- nuisance signal _shrinknsroi: true # --- smoothing parameters _surface_smooth: 4 _volume_smooth: 4 _voxel_smooth: 3 _smooth_mask: nonzero
Note that lines starting with
# are ignored and can be used for comments or to remove specific parameters.
Please note that if a parameter is specified in the batch file, QuNex will set it to the value specified. This is also the case for parameters that are left empty (e.g.
_event_file :). In this case, the
_event_file parameter is actually being set to an empty string and is being read as such by QuNex. The situation where this would be a misspecification would be, for instance:
_nprocess :. In this case the
_nprocess parameter checks how many concurrent jobs to run. If it is set to an empty string then the QuNex engine cannot set the process number. This behavior is important so that the QuNex engine can interpret all specified parameters based on actual values provided (empty or some value) as opposed to making an assumption that the parameter is absent when in fact it is present with an empty string.
Batch file session-level information following the header
Information for each session has the same format as the session information files. To separate information from individual sessions, each new session section has to start with three dashes (
---) in a line. Continuing the example from above, the session level information could start with:
--- id: NK3 subject: NK3 dicom: /Volumes/pooh/NK/fMRI/PD-fcMRI-TMS/sessions/NK03/dicom raw_data: /Volumes/pooh/NK/fMRI/PD-fcMRI-TMS/sessions/NK03/nii hpc: /Volumes/pooh/NK/fMRI/PD-fcMRI-TMS/sessions/NK03/hpc age: 21 handedness: right gender: male group: control 01: Survey 02: T1w: T1w 0.7mm N2 03: T2w: T2w 0.7mm N2 04: Survey 05: SE-FM-AP: C-BOLD 3mm 48 2.5s FS-P 06: SE-FM-PA: C-BOLD 3mm 48 2.5s FS-A 07: bold1:rest: BOLD 3mm 48 2.5s 08: bold2:task: BOLD 3mm 48 2.5s 09: bold2:task: BOLD 3mm 48 2.5s --- id: NK4 subject: NK4 dicom: /Volumes/pooh/NK/fMRI/PD-fcMRI-TMS/sessions/NK04/dicom raw_data: /Volumes/pooh/NK/fMRI/PD-fcMRI-TMS/sessions/NK04/nii hpc: /Volumes/pooh/NK/fMRI/PD-fcMRI-TMS/sessions/NK04/hpc age: 24 handedness: left gender: female group: intervention 01: Survey 02: T1w: T1w 0.7mm N2 03: T2w: T2w 0.7mm N2 04: Survey 05: SE-FM-AP: C-BOLD 3mm 48 2.5s FS-P 06: SE-FM-PA: C-BOLD 3mm 48 2.5s FS-A 07: bold1:rest: BOLD 3mm 48 2.5s 08: bold2:task: BOLD 3mm 48 2.5s 09: bold2:task: BOLD 3mm 48 2.5s
Note that the information above is also referenced inside the session-specific
Generating batch files for preprocessing
The QuNex batch files described above are the 'workhorse' of the preprocessing workflow as they can flexibly specify all relevant parameters in a header component of the file, followed by a listing of all relevant sessions to work on. Once all the data is gathered, the batch file needs to be compiled. These files (commonly named
batch.txt) list information for all the sessions, as well as study parameters, so that they don't have to be specified with each invocation of a given QuNex command, which flexibly reads the relevant header parameter subset. The key advantage here is that for rapid re-processing or specification of new parameters, even within a single study, use of batch files accelerates this workflow.
To help with generation of batch files, QuNex provides a convenient
create_batch command. The command scours all the subfolders in the study sessions directory for the specified session information files and joins them in the specified target file. If desired, the sessions to add can be specified explicitly or taken from another batch or list file. Run
qunex ?create_batch for more detailed information on the command use.
As noted, the command also prepends the specific batch header parameters. They need to be provided if they are saved in a specific header batch parameters file. The default location of the batch header files is in
/<path_to_study_folder>/sessions/specs/. If the batch file does not exist, an example will be created automatically, placing all the possible parameters into the header, their default values and explanations to allow easy editing. The command also supports appending new sessions to an existing batch file. The final batch file with the appended session information is saved in
/<path_to_study_folder>/processing/<name_of_batch_file>.txt. This file should be used in subsequent processing.
Note that if a batch file already exists then the parameter file will not be added to the header of the batch unless
overwrite parameter is set to
overwrite is set to
append, then the parameters will not be changed; however, any sessions that are not yet present in the batch file will be appended at the end of the batch file.
Please see additional help regarding compiling a batch file on Running HCP minimal preprocessing pipeline page.