Wiki

Clone wiki

qunex / Overview / file_batch_txt

Batch files

Preprocessing and analysis commands are typically run on batches of sessions that most often represent all the sessions in a study. Additionally, a number of parameters that are to be used in preprocessing and analysis commands are stable and do not change between command invocations. To facilitate processing of batches of sessions from small sets to thousands, QuNex utilizes batch files. The basic name for a batch file is something like batch.txt; however, any name can be used, which allows setting up separate versions of batch files for different purposes or preprocessing and analysis variants.

A batch file (e.g. batch.txt) consists of two core elements:

  • the parameter header
  • the session-level information for all sessions to be processed with the parameters in the header

The information for the parameter header is typically taken from the parameter file (commonly stored in: /<path_to_sessions_folder>/spec/parameters.txt) to allow flexible deployment and re-generation of the batch file for a given set of sessions. The final batch files are by default stored in /<study_folder>/processing/ to allow easy access to all the study-specific batch iterations. In principle the final batch file can be stored anywhere as long as the relevant QuNex command specifies its absolute path.

Batch file group-level header information

The header element of the batch file provides group-level information that will be used across all specified sessions in a form of _parameter : value pairs, one per line. Here is a concrete example of a group level section (for full details of all the parameter explanation see ~/qunex/python/qx_utilities/templates/parameters_example.txt):

# --- basic settings

_image_source:      hcp
_image_target:      cifti
_log:               keep
_glm_matrix:        both
_glm_residuals:     forget

_bold_preprocess:   WM|flanker|rest
_base_folder:       /data/studies/WM3/sessions
_TR:                2.5
_omit:              0

_pignore:           hipass=linear|regress=ignore|lopass=linear

# --- nuisance signal

_shrinknsroi:       true

# --- smoothing parameters

_surface_smooth:    4
_volume_smooth:     4
_voxel_smooth:      3
_smooth_mask:       nonzero

Note that lines starting with # are ignored and can be used for comments or to remove specific parameters.

Please note that if a parameter is specified in the batch file, QuNex will set it to the value specified. This is also the case for parameters that are left empty (e.g. _event_file :). In this case, the _event_file parameter is actually being set to an empty string and is being read as such by QuNex. The situation where this would be a misspecification would be, for instance: _nprocess :. In this case the _nprocess parameter checks how many concurrent jobs to run. If it is set to an empty string then the QuNex engine cannot set the process number. This behavior is important so that the QuNex engine can interpret all specified parameters based on actual values provided (empty or some value) as opposed to making an assumption that the parameter is absent when in fact it is present with an empty string.

Batch file session-level information following the header

Information for each session has the same format as the session information files. To separate information from individual sessions, each new session section has to start with three dashes (---) in a line. Continuing the example from above, the session level information could start with:

---
id: NK3
subject: NK3
dicom: /Volumes/pooh/NK/fMRI/PD-fcMRI-TMS/sessions/NK03/dicom
raw_data: /Volumes/pooh/NK/fMRI/PD-fcMRI-TMS/sessions/NK03/nii
hpc: /Volumes/pooh/NK/fMRI/PD-fcMRI-TMS/sessions/NK03/hpc

age: 21
handedness: right
gender: male
group: control

01: Survey
02: T1w:             T1w 0.7mm N2
03: T2w:             T2w 0.7mm N2
04: Survey
05: SE-FM-AP:        C-BOLD 3mm 48 2.5s FS-P
06: SE-FM-PA:        C-BOLD 3mm 48 2.5s FS-A
07: bold1:rest:      BOLD 3mm 48 2.5s
08: bold2:task:      BOLD 3mm 48 2.5s
09: bold2:task:      BOLD 3mm 48 2.5s

---
id: NK4
subject: NK4
dicom: /Volumes/pooh/NK/fMRI/PD-fcMRI-TMS/sessions/NK04/dicom
raw_data: /Volumes/pooh/NK/fMRI/PD-fcMRI-TMS/sessions/NK04/nii
hpc: /Volumes/pooh/NK/fMRI/PD-fcMRI-TMS/sessions/NK04/hpc

age: 24
handedness: left
gender: female
group: intervention

01: Survey
02: T1w:             T1w 0.7mm N2
03: T2w:             T2w 0.7mm N2
04: Survey
05: SE-FM-AP:        C-BOLD 3mm 48 2.5s FS-P
06: SE-FM-PA:        C-BOLD 3mm 48 2.5s FS-A
07: bold1:rest:      BOLD 3mm 48 2.5s
08: bold2:task:      BOLD 3mm 48 2.5s
09: bold2:task:      BOLD 3mm 48 2.5s

Note that the information above is also referenced inside the session-specific session_<pipeline>.txt files.

Generating batch files for preprocessing

The QuNex batch files described above are the 'workhorse' of the preprocessing workflow as they can flexibly specify all relevant parameters in a header component of the file, followed by a listing of all relevant sessions to work on. Once all the data is gathered, the batch file needs to be compiled. These files (commonly named batch.txt) list information for all the sessions, as well as study parameters, so that they don't have to be specified with each invocation of a given QuNex command, which flexibly reads the relevant header parameter subset. The key advantage here is that for rapid re-processing or specification of new parameters, even within a single study, use of batch files accelerates this workflow.

To help with generation of batch files, QuNex provides a convenient create_batch command. The command scours all the subfolders in the study sessions directory for the specified session information files and joins them in the specified target file. If desired, the sessions to add can be specified explicitly or taken from another batch or list file. Run qunex ?create_batch for more detailed information on the command use.

As noted, the command also prepends the specific batch header parameters. They need to be provided if they are saved in a specific header batch parameters file. The default location of the batch header files is in /<path_to_study_folder>/sessions/specs/. If the batch file does not exist, an example will be created automatically, placing all the possible parameters into the header, their default values and explanations to allow easy editing. The command also supports appending new sessions to an existing batch file. The final batch file with the appended session information is saved in /<path_to_study_folder>/processing/<name_of_batch_file>.txt. This file should be used in subsequent processing.

Note that if a batch file already exists then the parameter file will not be added to the header of the batch unless overwrite parameter is set to yes. If overwrite is set to append, then the parameters will not be changed; however, any sessions that are not yet present in the batch file will be appended at the end of the batch file.

Please see additional help regarding compiling a batch file on Running HCP minimal preprocessing pipeline page.

Updated