Wiki

Clone wiki

qunex / Overview / Logging

Logging and log files

The QuNex suite generates a standard set of output log files that are stored in <path_to_study_folder>/processing/logs/ when a given command is run. The different kinds of logs are stored into three subfolders called runlogs, comlogs, and batchlogs. Each of them contains distinct information on the processing that was run and the outputs that were generated during the execution.

Runlog files

Runlog files are the more general log files. They are created when a QuNex command is run independently across a number of sessions. An example of this would be running steps of the HCP minimal preprocessing workflow or BOLD preprocessing. This log will list the exact command that was run and the parameters it was run with. In addition, it will list for each session the relevant information for the files and settings and report the success or failure of processing that session. Last, it will print a short summary of the success in processing each session, one session per line.

By default runlogs are stored in <study_folder_name>/processing/logs/runlogs; however, see logfolder parameter for details.

Runlogs are given unique names compiled using the following specification:

Log-<command name>-<date>_<hour>.<minute>.<microsecond>.log

Comlog files

Comlog files provide a detailed output of the processing generated by a specific command. They are generated each time a process is started. By default, comlog files are saved in the <study_folder_name>/processing/logs/comlogs folder. However, the default location of the logs folder can be changed using the logfolder parameter (see below or by running qunex -o).

A comlog file is generated when the command has started and is named using the following specification:

tmp_<command_name>[_BOLD<N>]_<session code>_<date>_<hour>.<minute>.<microsecond>.log

(Note, [_BOLD<N>] is added when the command is ran for an individual BOLD run. In this case <N> stands for the BOLD number the command was run on.)

This tmp_* file can be inspected (e.g. using tail -f <path to the comlog file>) while the command is running to track its execution. Once the command ends, there are three possible outcomes for the final comlog file:

  1. If the command fails, the comlog file will be renamed to start with error instead of tmp and left in the folder with name:

    error_<command_name>[_BOLD<N>]_<session code>_<date>_<hour>.<minute>.<microsecond>.log

  2. If the command finishes successfully and the log parameter is set to keep (default behavior), the comlog file will be renamed to start with done and left in the comlog folder as:

    done_<command_name>[_BOLD<N>]_<session code>_<date>_<hour>.<minute>.<microsecond>.log

  3. If the command finishes successfully and the log parameter is set to remove, the tmp_* comlog file will be deleted.

  4. If the command finishes and the last file to be generated by the command is present, but the full file check has identified some missing files, the comlog file will be renamed to start with incomplete and left in the comlog folder as:

    incomplete_<command_name>[_BOLD<N>]_<session code>_<date>_<hour>.<minute>.<microsecond>.log

Batchlog files

If a command was run using a scheduler to be submitted to a cluster queue, the standard and error outputs of each of the submitted jobs will be captured and saved into the batchlog folder. The files will be named using the following specification:

<batch system>_<command name>_job<job number>.<date>_<hour>.<minute>.<microsecond>.log

Depending on the specific command that was run, the content of the batchlog file might be unique or match a runlog or a comlog file.

Log specific parameters

There are two parameters that affect logging.

logfolder

logfolder parameter can be used to specify the location where the runlog and comlog files are to be saved if the desired location is other than the default. If left unset (or set to an empty string), the default location for the runlogs and comlogs folder will be <study_folder_name>/processing/logs/.

log

log parameter is used to specify what happens to comlog files after successful completion of a command (see Comlog files above). Possible values are:

  • remove

Comlog files are deleted after a successful completion of the command. If the command fails, the error log will be preserved.

  • keep or study

Comlog files will be saved to the default location in <study>/processing/logs/comlogs that then holds all the comlog files from all the sessions. This is the default used if the --log parameter is not specified.

  • session

Comlog files will be saved within each session's own folder, specifically in <sessions folder>/<session id>/logs/comlogs. Using this option, the logs are saved along with each session, which makes it easy to find logs specific to a session and possibly archive them with the session.

  • hcp

Comlog files will be stored in each session's own hcp folder, specifically in <sessions folder>/<session id>/hcp/<session id>[<hcp suffix>]/logs/comlogs. Using this option, the log files are stored along with the results and other files related to processing using HCPpipelines.

Do note that except remove it is possible to specify multiple options, so that the comlog files are saved in all locations that a user would find them useful. The options can be specified using a comma or pipe (|) separated string (e.g. --log="study|session").

Example

If the following command was used:

qunex preprocess_bold \
   --sessions="<studyfolder>/processing/batch.txt" \
   --sessionsfolder="<studyfolder>/sessions" \
   --bold_actions="shrc" \
   --bold_nuisance="m,V,WM,WB,1d" \
   --glm_matrix="both" \
   --pignore="hipass=linear|regress=mark|lopass=linear" \
   --parsessions=10 \
   --bold_preprocess="rest" \
   --log="study|session" \
   --scheduler="SLURM,time=00-08:00:00,cpus-per-task=1,mem=40000,partition=<queue name>"

For example, if the batch.txt includes 25 sessions, each with 2 BOLD runs marked as rest, the following logs would be generated in 3 separate folders:

1.<studyfolder>/processing/logs/runlogs would hold 4 sets of log files (one for each BOLD):

Log-preprocess_bold-<date>_<hour>.<minute>.<second>.<microsecond>.log

  • The first runlog file would list the scheduled jobs that qunex scheduled to run on the cluster SLURM queuing system. One job for the first 10 sessions, the next for the next 10 sessions, and the last for the remaining 5 sessions.

  • This workflow breakdown is controlled by the --parsessions=10 parameter. If this parameter was set to 5, then there would be 5 logs (5 sessions per job).

  • Each of the jobs scheduled would generate corresponding runlog files. Each of them would list the settings, the detailed report on the command that was executed and a summary report for each of the sessions and bolds that were part of the job.

2.<studyfolder>/processing/logs/comlogs would hold 50 log files named:

done_Preprocess_B<N>_<session code>_<date>_<hour>.<minute>.<microsecond>.log

  • Here each comlog file would correspond to each of the BOLD files for each session (i.e. 2 x 25 = 50) that were part of the job. While the job is ongoing, the comlog file would start with tmp instead. If any of the commands fail, the related comlog file would start with error instead.

  • As both study and session were specified, the same logs would also be saved in each session's folder, specifically in <sessionsfolder>/<sessionid>/logs/comlogs.

3.<studyfolder>/processing/logs/batchlogs would hold 3 log files named:

SLURM_preprocessBOLD_job<job number>.<date>_<hour>.<minute>.<microsecond>.log

  • Each of them would store the combined standard and error outputs of the 3 jobs submitted to the queuing system (in this case SLURM).
  • Here the content of these files would match the respective runlog files, as the qunex command generates the same standard and error output as it stores in the runlog files.

Updated