HTTPS SSH

Articulatory strategy biomarker

This folder contains the files needed to repeat the analysis reported in the following paper.

T. Sorensen, A. Toutios, L. Goldstein, and S. Narayanan, “Test-retest repeatability of articulatory strategies using real-time magnetic resonance imaging.” Proceedings of Interspeech 2017, submitted.

Contact information

Tanner Sorensen
Signal Analysis and Interpretation Laboratory
tsorense@usc.edu

Directories

  • data - copy MRI data-sets to this folder
  • graphics - graphical output of MATLAB programs
  • manual_annotations - manually annotations which serve as inputs to MATLAB programs
  • mat - segmentation results, articulatory strategy biomarkers output by MATLAB programs
  • scripts - MATLAB programs
    • constrictions - manually annotate constriction locations, automatically measure constriction degrees
    • data_scripts - convert segmentation file format
    • dMRI_GUI - manually annotate onset and offset of constrictions
    • factor_analysis - perform guided factor analysis on segmentation results
    • segmentation - files used to segment the articulators in rt-MRI videos
    • stats - perform statistical analysis to link articulatory strategies to morphology
    • strategies - extract articulatory strategies
    • util - configuration file, plotting function
    • main.m - extracts articulatory strategies from track files
  • text - paper
    • figure - vector graphics
    • main.tex - run pdflatex, bibtex, pdflatex, pdflatex on this file to compile paper
    • mybib.bib - BibTeX file for the bibliography

Instructions

In order to reproduce the analysis reported in the paper, do the following steps in order. Manual annotation steps and segmentation steps can be omitted, as the outputs are included in the repository.

  1. Clone the repository
    Clone this repository to the home directory (i.e., the directory denoted by the shortcut ~ in the terminal).
  2. Download the imaging data
    The imaging data can be obtained at http://sail.usc.edu/span/test-retest. Copy the downloaded folder to the address ~/data. The data-set is documented in the following paper.
    J. Töger, T. Sorensen, K. Somandepalli, A. Toutios, S. G. Lingala, S. Narayanan, and K. S. Nayak, “Test-retest repeatability of human speech biomarkers from static and real-time dynamic magnetic resonance imaging,” Journal of the Acoustical Society of America, submitted.
  3. Manual annotation
    Manually annotate the time-stamps for the formation and release of constrictions and save spreadsheets ~/manual_annotations/timestamps_rep1.xlsx and ~/manual_annotations/timestamps_rep2.xlsx. The timestamps were manually annotated by clicking through the frames of the AVI files in the directories under ~/data/segmentation_results using the MATLAB GUI ~/scripts/dMRI_GUI/inspect_dmri.m This is opened by adding ~/scripts/dMRI_GUI and subdirectories to the MATLAB search path, cd'ing to the folder whose children are avi and wav, and then typing the following command into the MATLAB Command Window: inspect_dmri. The annotator can then click around using the arrows or mouse to find the timestamps as frame numbers. The rows of the spreadsheets correspond to a file-speaker pair. The columns contain the following information.
    • file - (string) AVI file name
    • speaker - (string) speaker name
    • Apa-clo-start - (integer) onset frame number for bilabial closure
    • Apa-clo-end - (integer) offset frame number for bilabial closure
    • Apa-rel-start - (integer) onset frame number for bilabial release
    • Apa-rel-end - (integer) offset frame number for bilabial release
    • Ata-clo-start - (integer) onset frame number for coronal closure
    • Ata-clo-end - (integer) offset frame number for coronal closure
    • Ata-rel-start - (integer) onset frame number for coronal release
    • Ata-rel-end - (integer) offset frame number for coronal release
    • Aka-clo-start - (integer) onset frame number for velar closure
    • Aka-clo-end - (integer) offset frame number for velar closure
    • Aka-rel-start - (integer) onset frame number for velar release
    • Aka-rel-end - (integer) offset frame number for velar release
    • Aia-pal-clo-start - (integer) onset frame number for palatal approximation
    • Aia-pal-clo-end - (integer) offset frame number for palatal approximation
    • Aia-phar-clo-start - (integer) onset frame number for pharyngeal approximation
    • Aia-phar-clo-end - (integer) offset frame number for pharyngeal approximation
  4. Articulator contour segmentation
    • Run ~/analysis/scripts/segmentation/wrap_make_hpc_csv.m to generate csv files containing the start and stop times for segmentation.
    • Within each folder ~/analysis/scripts/segmentation/wrappers_<subj>, where <subj> stands for subject ID, make a template using the script wrap_make_template.m and generate the files to segment the AVI files using wrap_make_batch.m. Then move the files generated in the folder ~/analysis/scripts/segmentation/wrappers_<subj>/cluster to the HPC cluster and run them by executing the shell script in that folder.
    • After the segmentation is done, copy the results back to the local hard disk.
  5. Manual annotation
    Create the files ~/manual_annotations/tvlocs_rep1.mat and ~/manual_annotations/tvlocs_rep2.mat, which contain the coronal, palatal, velar, nasopharyngeal, and hypopharyngeal phonetic places of articulation. The bilabial place covers the whole lip region and is not manually specified. The files were generated using the script ~/scripts/constrictions/tvlocs.m. This is opened by adding '~/scripts/constrictions' to the MATLAB search path and then typing the following command into the MATLAB Command Window: tvlocs(dataset), where dataset is either rep1 or rep2. The annotator can then click around, using the menu to select the participant, the buttons to select the place of articulation, and the pointer to click on the place of articulation (once at the anterior/superior end and once at the posterior/inferior end of the place of articulation).
  6. Extract articulatory strategy biomarkers by running ~/scripts/main.m in MATLAB.
  7. Perform statistical analysis by running ~/scripts/stats/test-retest.R in RStudio.
  8. Compile PDF from the terminal with the following commands:
cd ~/text  
pdflatex main.tex  
bibtex main.aux  
pdflatex main.tex  
pdflatex main.tex

Software versions

Linux distribution release:

Distributor ID: Ubuntu
Description:    Ubuntu 16.04.2 LTS
Release:    16.04
Codename:   xenial

MATLAB

MATLAB Version: 9.1.0.441655 (R2016b)
MATLAB License Number: 623588
Operating System: Linux 4.4.0-72-generic #93-Ubuntu SMP Fri Mar 31 14:07:41 UTC 2017 x86_64
Java Version: Java 1.7.0_60-b19 with Oracle Corporation Java HotSpot(TM) 64-Bit Server VM mixed mode
MATLAB                                                Version 9.1         (R2016b)
Simulink                                              Version 8.8         (R2016b)
Bioinformatics Toolbox                                Version 4.7         (R2016b)
Control System Toolbox                                Version 10.1        (R2016b)
Curve Fitting Toolbox                                 Version 3.5.4       (R2016b)
DSP System Toolbox                                    Version 9.3         (R2016b)
Financial Toolbox                                     Version 5.8         (R2016b)
Image Processing Toolbox                              Version 9.5         (R2016b)
Instrument Control Toolbox                            Version 3.10        (R2016b)
Optimization Toolbox                                  Version 7.5         (R2016b)
Parallel Computing Toolbox                            Version 6.9         (R2016b)
Signal Processing Toolbox                             Version 7.3         (R2016b)
Simscape                                              Version 4.1         (R2016b)
Simscape Multibody                                    Version 4.9         (R2016b)
Simulink Control Design                               Version 4.4         (R2016b)
Stateflow                                             Version 8.8         (R2016b)
Statistics and Machine Learning Toolbox               Version 11.0        (R2016b)
Symbolic Math Toolbox                                 Version 7.1         (R2016b)

R

               _                           
platform       x86_64-pc-linux-gnu         
arch           x86_64                      
os             linux-gnu                   
system         x86_64, linux-gnu           
status                                     
major          3                           
minor          3.3                         
year           2017                        
month          03                          
day            06                          
svn rev        72310                       
language       R                           
version.string R version 3.3.3 (2017-03-06)
nickname       Another Canoe

RStudio

Version 1.0.136 – © 2009-2016 RStudio, Inc.
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/538.1 (KHTML, like Gecko) RStudio Safari/538.1 Qt/5.4.0