Clone wiki

testvox / TestVox_Configuration_Recipes

TestVox Configuration Recipes

This page lists recipes to configure different types of listening tests with TestVox. You can copy this content into the config.yaml file in your experiment directory and edit it appropriately.

For more details on what the different configuration options are, see TestVox_Configuration.

Listening Tests

Transcription Task

In the transcription task, participants listen to a bunch of audio files and type in their transcriptions. This can be used to either transcribe audio for further processing, or for evaluating the intelligibility of synthesized speech.

Experiment Directory Setup

my_experiment_transcription/
  config.yaml
  mp3/
    data01.mp3
    data02.mp3 
    data03.mp3

Configuration File

# copy this to config.yaml and edit it.

# Experiment to evaluate intelligibility of synthesized MT Output

testvox_config:
  base_media_directory: mp3
  pagetitle: CMU Transcription Task

testvox_steps:
  - name: listening_task
    task_type: transcriptiontask

    instruction: >-
      Listen to the short audio clip below, and 
      type in what was said in it.

    data:
      - filename: data01.mp3
      - filename: data02.mp3
      - filename: data03.mp3

    data_randomize: Yes  # Present the different data files in random order

    audio_autoplay: No

A/B Preference Task

In the A/B task, participants listen to two audio clips at a time. They then choose from a list of options and mark their preference. This task is often used to compare two different TTS models. The same utterance is synthesized with both models, and participants will pick the model they think is better.

Experiment Directory Setup

my_experiment_abtask/
  config.yaml
  mp3/
    condition_1/
      data01.mp3
      data02.mp3 
      data03.mp3
    condition_2/
      data01.mp3
      data02.mp3
      data03.mp3

Configuration File

# copy this to config.yaml and edit it.

# Experiment to evaluate which synthesis model sounds more natural

testvox_config:
  base_media_directory: mp3
  pagetitle: Carnegie Mellon University

testvox_steps:
  - name: listening_task
    task_type: abtask

    instruction: >-
      Listen to the two audio clips below, and 
      tell us which one you think sounds more natural.

    directory_a: condition_1
    directory_b: condition_2
    ab_randomize: Yes  # Randomize order of A-clip and B-clip presented to participants

    data:
      - filename: data01.mp3  # Same filename must exist in both condition directories
      - filename: data02.mp3
      - filename: data03.mp3

    data_randomize: Yes  # Present the different data files in random order

    audio_autoplay: No

MCQ / Likert Tasks

In these tasks, participants listen to an audio clip, and select from a set of options what their opinion is about the audio. They could rate it on a scale of 1 to 5, or mark one or more options (natural, intelligible) for the clip.

These tasks are called "radiotask" or "checktask" based on whether participant can pick only one option, or more.

Experiment Directory Setup

my_experiment_rate/
  config.yaml
  mp3/
    data01.mp3
    data02.mp3 
    data03.mp3

Configuration File

# copy this to config.yaml and edit it.

# Experiment to rate audio clips

testvox_config:
  base_media_directory: mp3
  pagetitle: TestVox -- Rate Audio

testvox_steps:
  - name: listening_task
    task_type: radiotask  # or checktask, if more than one options can be selected

    instruction: >-
      Listen to the short audio clip below, and 
      rate the naturalness of speech.

    task_options:
      - It sounds very natural.
      - It sounds somewhat natural.
      - It sounds neither natural nor robotic.
      - It sounds somewhat robotic.
      - It sounds very robotic.
    data:
      - filename: data01.mp3
      - filename: data02.mp3
      - filename: data03.mp3

    data_randomize: Yes  # Present the different data files in random order

    audio_autoplay: No

Word Choice Task

This task is suited for participants to listen to an audio clip and mark certain words from its transcript. For example, participants can select words that were emphasized in the audio.

Experiment Directory Setup

my_experiment_wordchoice/
  config.yaml
  mp3/
    data01.mp3
    data02.mp3 
    data03.mp3

Configuration File

# copy this to config.yaml and edit it.

# Experiment to evaluate intelligibility of synthesized MT Output

testvox_config:
  base_media_directory: mp3
  pagetitle: Emphasis Detection Task

testvox_steps:
  - name: listening_task
    task_type: wordchoicetask

    instruction: >-
      Listen to the short audio clip below, and 
      mark the words that you think were emphasized.

    select_type: radio  # only one word can be marked. use 'check' otherwise.

    data:
      - filename: data01.mp3
        text: Hi There.
        enabled_flags: "10"  # One bit per word in text. 1 if word can be marked. 0 otherwise

      - filename: data02.mp3
        text: This is an example.
        enabled_flags: "1001"

      - filename: data03.mp3
        text: All that glitters is not gold.
        enabled_flags: "101101"

    data_randomize: Yes  # Present the different data files in random order

    audio_autoplay: No

Entry/Exit Surveys

You can add entry/exit surveys by creating additional entries in 'testvox_steps' section of the configuration file. Place entry_survey before your listening task, and exit_survey after your listening task.

Example Exit Survey

testvox_config:
  # Your settings

testvox_steps:
  - name: listening_task
    # Your listening task here (see listening task recipes)

  - name: exit_survey
    survey_scope: experiment
    task_type: surveyform

    instruction: >-
      Please answer the following questions. These responses will help us
      classify your response appropriately.

    questions:
      - name: native_language
        type: select
        text: What is your first language?
        options:
          - US English
          - UK English
          - Marathi
          - Other

      - name: other_languages
        type: text
        text: What other languages are you fluent in? (e.g. french, german, hindi)

      - name: age_range
        type: select
        text: Please select your age group
        options:
          - Under 18
          - 18 to 35
          - 36 to 50
          - Above 50

      - name: tts_expertise
        type: select
        text: How often do you use text-to-speech software?
        options:
          - I never use TTS
          - I sometimes use TTS
          - I often use TTS
          - I am a TTS developer

Updated