Articulatory strategy biomarker =============================== This folder contains the files needed to repeat the analysis reported in the following paper. T. Sorensen, A. Toutios, L. Goldstein, and S. Narayanan, “Test-retest repeatability of articulatory strategies using real-time magnetic resonance imaging.” Proceedings of Interspeech 2017, submitted. Contact information ------------------- Tanner Sorensen Signal Analysis and Interpretation Laboratory <> Directories ----------- - `data` - copy MRI data-sets to this folder - `graphics` - graphical output of MATLAB programs - `manual_annotations` - manually annotations which serve as inputs to MATLAB programs - `mat` - segmentation results, articulatory strategy biomarkers output by MATLAB programs - `scripts` - MATLAB programs - `constrictions` - manually annotate constriction locations, automatically measure constriction degrees - `data_scripts` - convert segmentation file format - `dMRI_GUI` - manually annotate onset and offset of constrictions - `factor_analysis` - perform guided factor analysis on segmentation results - `segmentation` - files used to segment the articulators in rt-MRI videos - `stats` - perform statistical analysis to link articulatory strategies to morphology - `strategies` - extract articulatory strategies - `util` - configuration file, plotting function - `main.m` - extracts articulatory strategies from track files - `text` - paper - `figure` - vector graphics - `main.tex` - run pdflatex, bibtex, pdflatex, pdflatex on this file to compile paper - `mybib.bib` - BibTeX file for the bibliography Instructions ------------ In order to reproduce the analysis reported in the paper, do the following steps in order. Manual annotation steps and segmentation steps can be omitted, as the outputs are included in the repository. 1. Clone the repository Clone this repository to the home directory (i.e., the directory denoted by the shortcut `~` in the terminal). 2. Download the imaging data The imaging data can be obtained at Copy the downloaded folder to the address `~/data`. The data-set is documented in the following paper. J. Töger, T. Sorensen, K. Somandepalli, A. Toutios, S. G. Lingala, S. Narayanan, and K. S. Nayak, “Test-retest repeatability of human speech biomarkers from static and real-time dynamic magnetic resonance imaging,” Journal of the Acoustical Society of America, submitted. 3. Manual annotation Manually annotate the time-stamps for the formation and release of constrictions and save spreadsheets `~/manual_annotations/timestamps_rep1.xlsx` and `~/manual_annotations/timestamps_rep2.xlsx`. The timestamps were manually annotated by clicking through the frames of the AVI files in the directories under `~/data/segmentation_results` using the MATLAB GUI `~/scripts/dMRI_GUI/inspect_dmri.m` This is opened by adding `~/scripts/dMRI_GUI` and subdirectories to the MATLAB search path, cd'ing to the folder whose children are `avi` and `wav`, and then typing the following command into the MATLAB Command Window: `inspect_dmri`. The annotator can then click around using the arrows or mouse to find the timestamps as frame numbers. The rows of the spreadsheets correspond to a file-speaker pair. The columns contain the following information. - file - (string) AVI file name - speaker - (string) speaker name - Apa-clo-start - (integer) onset frame number for bilabial closure - Apa-clo-end - (integer) offset frame number for bilabial closure - Apa-rel-start - (integer) onset frame number for bilabial release - Apa-rel-end - (integer) offset frame number for bilabial release - Ata-clo-start - (integer) onset frame number for coronal closure - Ata-clo-end - (integer) offset frame number for coronal closure - Ata-rel-start - (integer) onset frame number for coronal release - Ata-rel-end - (integer) offset frame number for coronal release - Aka-clo-start - (integer) onset frame number for velar closure - Aka-clo-end - (integer) offset frame number for velar closure - Aka-rel-start - (integer) onset frame number for velar release - Aka-rel-end - (integer) offset frame number for velar release - Aia-pal-clo-start - (integer) onset frame number for palatal approximation - Aia-pal-clo-end - (integer) offset frame number for palatal approximation - Aia-phar-clo-start - (integer) onset frame number for pharyngeal approximation - Aia-phar-clo-end - (integer) offset frame number for pharyngeal approximation 4. Articulator contour segmentation - Run `~/analysis/scripts/segmentation/wrap_make_hpc_csv.m` to generate csv files containing the start and stop times for segmentation. - Within each folder `~/analysis/scripts/segmentation/wrappers_<subj>`, where `<subj>` stands for subject ID, make a template using the script `wrap_make_template.m` and generate the files to segment the AVI files using `wrap_make_batch.m`. Then move the files generated in the folder `~/analysis/scripts/segmentation/wrappers_<subj>/cluster` to the HPC cluster and run them by executing the shell script in that folder. - After the segmentation is done, copy the results back to the local hard disk. 5. Manual annotation Create the files `~/manual_annotations/tvlocs_rep1.mat` and `~/manual_annotations/tvlocs_rep2.mat`, which contain the coronal, palatal, velar, nasopharyngeal, and hypopharyngeal phonetic places of articulation. The bilabial place covers the whole lip region and is not manually specified. The files were generated using the script `~/scripts/constrictions/tvlocs.m`. This is opened by adding '~/scripts/constrictions' to the MATLAB search path and then typing the following command into the MATLAB Command Window: `tvlocs(dataset)`, where `dataset` is either `rep1` or `rep2`. The annotator can then click around, using the menu to select the participant, the buttons to select the place of articulation, and the pointer to click on the place of articulation (once at the anterior/superior end and once at the posterior/inferior end of the place of articulation). 6. Extract articulatory strategy biomarkers by running `~/scripts/main.m` in MATLAB. 7. Perform statistical analysis by running `~/scripts/stats/test-retest.R` in RStudio. 8. Compile PDF from the terminal with the following commands: ``` cd ~/text pdflatex main.tex bibtex main.aux pdflatex main.tex pdflatex main.tex ``` Software versions ----------------- Linux distribution release: Distributor ID: Ubuntu Description: Ubuntu 16.04.2 LTS Release: 16.04 Codename: xenial MATLAB MATLAB Version: (R2016b) MATLAB License Number: 623588 Operating System: Linux 4.4.0-72-generic #93-Ubuntu SMP Fri Mar 31 14:07:41 UTC 2017 x86_64 Java Version: Java 1.7.0_60-b19 with Oracle Corporation Java HotSpot(TM) 64-Bit Server VM mixed mode MATLAB Version 9.1 (R2016b) Simulink Version 8.8 (R2016b) Bioinformatics Toolbox Version 4.7 (R2016b) Control System Toolbox Version 10.1 (R2016b) Curve Fitting Toolbox Version 3.5.4 (R2016b) DSP System Toolbox Version 9.3 (R2016b) Financial Toolbox Version 5.8 (R2016b) Image Processing Toolbox Version 9.5 (R2016b) Instrument Control Toolbox Version 3.10 (R2016b) Optimization Toolbox Version 7.5 (R2016b) Parallel Computing Toolbox Version 6.9 (R2016b) Signal Processing Toolbox Version 7.3 (R2016b) Simscape Version 4.1 (R2016b) Simscape Multibody Version 4.9 (R2016b) Simulink Control Design Version 4.4 (R2016b) Stateflow Version 8.8 (R2016b) Statistics and Machine Learning Toolbox Version 11.0 (R2016b) Symbolic Math Toolbox Version 7.1 (R2016b) R _ platform x86_64-pc-linux-gnu arch x86_64 os linux-gnu system x86_64, linux-gnu status major 3 minor 3.3 year 2017 month 03 day 06 svn rev 72310 language R version.string R version 3.3.3 (2017-03-06) nickname Another Canoe RStudio Version 1.0.136 – © 2009-2016 RStudio, Inc. Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/538.1 (KHTML, like Gecko) RStudio Safari/538.1 Qt/5.4.0