TestVox Configuration

TestVox Configuration has to be specified as a config.yaml file. As the extension suggests, we use the YAML syntax for the configuration.

Selected Notes on YAML

TestVox configuration essentially consists of dicts (maps) and lists. Here is how you create these structures in YAML:

For a dict, we use colons. Example: For the dict

{'name': 'Foo', 
 'institute': 'CMU'}

The YAML representation is:

name: Alok
institute: CMU

For lists, we use hyphens. Example: The list

['A', 'B', 'C']

can be represented in YAML as:

- A
- B
- C

Nested structures can be specified in YAML using indentation. Remember: use only blank space for indentation. YAML does not accept TAB characters.

Example nested structure:

{'person': 
     {'name': 'Foo',
      'institute': 'CMU',
      'aliases': ['bar', 'baz']
      }
}

in YAML, becomes:

person:
  name: Foo
  institute: CMU
  aliases:
    - bar
    - baz

TestVox Configuration Structure

The TestVox configuration is essentially a dictionary of three keys:

testvox_config: Global settings
testvox_amturk_config: Amazon M-Turk related settings
testvox_steps: Steps (listening task/survey) that are parts of experiments.

Thus, any config.yaml file starts with this form:

testvox_config:
  # some settings

testvox_amturk_config:
  # some settings. This section is optional

testvox_steps: - # step 1. At least one step required. - # step 2

The following subsections explain these three configuration sections.

Global TestVox Configuration (testvox_config)

The global configuration consists of these possibilities:

testvox_config:
  base_media_directory: relpath
  pagetitle: Your Title
  logo_url: relative/url or http://absolute.url

The base_media_directory is a mandatory option. It tells, within your zipfile, the location of where the audio files are.

Page title and Logo are shown on the listening task page and you can configure these to suit your organization. By default, the TestVox logo will be shown, along with a title that is 'TestVox'.

Page title and logo can be configured per experiment. They can not be configured per step. That is, you can't have a pagetitle that is different for survey versus listening task.

TestVox M-Turk settings (testvox_amturk_config)

Let's say you are looking for a transcription task, and you have 20 utterances that you want transcribed. It is not a good idea to post all 20 at once for participants on Amazon M-Turk. Participants who look for quick tasks may get bored a few utterances down in your task and not do more of your HITs. Thus, showing around 5 utterances in one session may be a good idea.

The testvox_amturk_config looks like this:

testvox_amturk_config:
  num_hits: 10
  num_clips_per_hit: 5

Suppose your listening task specified 50 clips, TestVox will partition them into num_hits sets, and create a unique URL for each set. These sets can then be uploaded over to Mechanical Turk.

The testvox_amturk_config section is optional. If it is not specified, the partitions/sets are not created, and all clips have to be done in one session. This is ideal for locally recruited participants.

TestVox Steps (testvox_steps)

Typically, an experiment will involve one or two steps: a listening task, and an exit survey. The set of steps is a list under the 'testvox_steps' section of the configuration.

Each testvox_step requires these mandatory settings:

testvox_steps:
  - name:
    task_type:
    instruction:
  - name:
    task_type:
    instruction:

Name can be anything. Typical names are 'listening_task' and 'entry_survey' or 'exit_survey'. Instructions are displayed to users while they do the task. Task type specifies what task will be displayed to the user. This can be one of:

surveyform (Display a survey)
transcriptiontask (Transcribe audio clips)
abtask (Preference task)
radiotask (Listen to a clip and choose an option.. Likert/MOS are radio tasks)
checktask (Listen to a clip and choose one or more options)
wordchoicetask (Listen to a clip and choose words from transcripts)

Surveys

For each step of type 'surveyform', you need to specify a list of 'questions'. Each question must have a "name", a "text" (the question that will be displayed), and a "type". The "type" can be "select", "radio", "check" or "text", depending on which type of HTML question needs to be asked. The "select", "radio" and "check" types require a "option" key that's a list of options to present to user.

Example:

testvox_steps:
 - name: example_survey
   task_type: surveyform
   instruction: >-
     Something with
     multiple lines
   questions:
     - name: x
       type: select  # (or radio, or check)
       text: "What is ...?"
       options:
         - option1
         - option2
     - name: y
       type: text
       text: "Type this..."

Listening Tasks

Each listening task must have the following: "name", "data", "data_randomize".

Data contains a list of items, typically filenames, that get shown to users. data_randomize is a flag (Yes, No) that specifies whether utterances from "data" should be presented to the users in the listed order, or whether they should be randomized.

testvox_steps:
  - name: listening_task
    data: 
      - #data items
    data_randomize: Yes # (or No)

The "data" section itself is used differently by different listening task types. See the TestVox Configuration Recipes to learn about the variety of configuration here.

However, every item in the "data" section must have a "filename" which corresponds to the file that should be played to the user.

Wiki

testvox / TestVox_Configuration