Wiki
Clone wikitestvox / TestVox_Configuration_Recipes
TestVox Configuration Recipes
This page lists recipes to configure different types of listening tests with TestVox. You can copy this content into the config.yaml file in your experiment directory and edit it appropriately.
For more details on what the different configuration options are, see TestVox_Configuration.
Listening Tests
Transcription Task
In the transcription task, participants listen to a bunch of audio files and type in their transcriptions. This can be used to either transcribe audio for further processing, or for evaluating the intelligibility of synthesized speech.
Experiment Directory Setup
my_experiment_transcription/
config.yaml
mp3/
data01.mp3
data02.mp3
data03.mp3
Configuration File
# copy this to config.yaml and edit it.
# Experiment to evaluate intelligibility of synthesized MT Output
testvox_config:
base_media_directory: mp3
pagetitle: CMU Transcription Task
testvox_steps:
- name: listening_task
task_type: transcriptiontask
instruction: >-
Listen to the short audio clip below, and
type in what was said in it.
data:
- filename: data01.mp3
- filename: data02.mp3
- filename: data03.mp3
data_randomize: Yes # Present the different data files in random order
audio_autoplay: No
A/B Preference Task
In the A/B task, participants listen to two audio clips at a time. They then choose from a list of options and mark their preference. This task is often used to compare two different TTS models. The same utterance is synthesized with both models, and participants will pick the model they think is better.
Experiment Directory Setup
my_experiment_abtask/
config.yaml
mp3/
condition_1/
data01.mp3
data02.mp3
data03.mp3
condition_2/
data01.mp3
data02.mp3
data03.mp3
Configuration File
# copy this to config.yaml and edit it.
# Experiment to evaluate which synthesis model sounds more natural
testvox_config:
base_media_directory: mp3
pagetitle: Carnegie Mellon University
testvox_steps:
- name: listening_task
task_type: abtask
instruction: >-
Listen to the two audio clips below, and
tell us which one you think sounds more natural.
directory_a: condition_1
directory_b: condition_2
ab_randomize: Yes # Randomize order of A-clip and B-clip presented to participants
data:
- filename: data01.mp3 # Same filename must exist in both condition directories
- filename: data02.mp3
- filename: data03.mp3
data_randomize: Yes # Present the different data files in random order
audio_autoplay: No
MCQ / Likert Tasks
In these tasks, participants listen to an audio clip, and select from a set of options what their opinion is about the audio. They could rate it on a scale of 1 to 5, or mark one or more options (natural, intelligible) for the clip.
These tasks are called "radiotask" or "checktask" based on whether participant can pick only one option, or more.
Experiment Directory Setup
my_experiment_rate/
config.yaml
mp3/
data01.mp3
data02.mp3
data03.mp3
Configuration File
# copy this to config.yaml and edit it.
# Experiment to rate audio clips
testvox_config:
base_media_directory: mp3
pagetitle: TestVox -- Rate Audio
testvox_steps:
- name: listening_task
task_type: radiotask # or checktask, if more than one options can be selected
instruction: >-
Listen to the short audio clip below, and
rate the naturalness of speech.
task_options:
- It sounds very natural.
- It sounds somewhat natural.
- It sounds neither natural nor robotic.
- It sounds somewhat robotic.
- It sounds very robotic.
data:
- filename: data01.mp3
- filename: data02.mp3
- filename: data03.mp3
data_randomize: Yes # Present the different data files in random order
audio_autoplay: No
Word Choice Task
This task is suited for participants to listen to an audio clip and mark certain words from its transcript. For example, participants can select words that were emphasized in the audio.
Experiment Directory Setup
my_experiment_wordchoice/
config.yaml
mp3/
data01.mp3
data02.mp3
data03.mp3
Configuration File
# copy this to config.yaml and edit it.
# Experiment to evaluate intelligibility of synthesized MT Output
testvox_config:
base_media_directory: mp3
pagetitle: Emphasis Detection Task
testvox_steps:
- name: listening_task
task_type: wordchoicetask
instruction: >-
Listen to the short audio clip below, and
mark the words that you think were emphasized.
select_type: radio # only one word can be marked. use 'check' otherwise.
data:
- filename: data01.mp3
text: Hi There.
enabled_flags: "10" # One bit per word in text. 1 if word can be marked. 0 otherwise
- filename: data02.mp3
text: This is an example.
enabled_flags: "1001"
- filename: data03.mp3
text: All that glitters is not gold.
enabled_flags: "101101"
data_randomize: Yes # Present the different data files in random order
audio_autoplay: No
Entry/Exit Surveys
You can add entry/exit surveys by creating additional entries in 'testvox_steps' section of the configuration file. Place entry_survey before your listening task, and exit_survey after your listening task.
Example Exit Survey
testvox_config:
# Your settings
testvox_steps:
- name: listening_task
# Your listening task here (see listening task recipes)
- name: exit_survey
survey_scope: experiment
task_type: surveyform
instruction: >-
Please answer the following questions. These responses will help us
classify your response appropriately.
questions:
- name: native_language
type: select
text: What is your first language?
options:
- US English
- UK English
- Marathi
- Other
- name: other_languages
type: text
text: What other languages are you fluent in? (e.g. french, german, hindi)
- name: age_range
type: select
text: Please select your age group
options:
- Under 18
- 18 to 35
- 36 to 50
- Above 50
- name: tts_expertise
type: select
text: How often do you use text-to-speech software?
options:
- I never use TTS
- I sometimes use TTS
- I often use TTS
- I am a TTS developer
Updated