TestVox HOWTO

There are two main things to do when setting up and using TestVox.

1. Set up and start the webserver
2. Set up listening tasks

Setting up TestVox Webserver

TestVox has a built in web server powered by CherryPy that you can use for serving your listening tasks. This allows for easy deployment:

• Download TestVox prebuilt-package and unzip it on a publicly accessible computer.

• Set up a username and password for administration

  cd TestVox-prebuilt
  python scripts/change_admin_password.py

• Start the webserver:

  python testvox_server.py --ip 0.0.0.0 --port 8080

• The default port is 8080. You can change it to the standard webserver port (80) by changing the commandline argument. However, remember that you need root access to do that. Also, make sure any other webserver installed on your machine (Apache/IIS/Lighttpd/NginX) is stopped. Otherwise TestVox will not be able to use port 80.

• You should now be able to access TestVox by visiting http://your.computer:your_port/admin. Login to this administration portal using credentials you created earlier.

Remember: Your web server should be running the entire time you want to host any experiments. It is recommended to start the webserver in a screen session, or using nohup so it doesn't accidentally terminate.

Now you are all set to run listening tests. For advanced webserver setup, and installing TestVox behind a standard Webserver (Apache/NginX/etc) or for deploying TestVox over Google App Engine, please see TestVox_Advanced_Deployment.

Setting up Listening Tests

A typical use case for running listening tests with TestVox consists of the following steps:

1. Set up an Experiment Directory
2. Prepare Audio Data
3. Write the Configuration File
4. Upload Experiment to TestVox
5. Conduct Listening Tests
6. Download Results
7. Delete Uploaded Experiments

Set up an Experiment Directory

TestVox requires you to configure your experiment and upload it to the server as a zip file. This zip file includes the configuration file for your listening task, audio files, and any other relevant set of files.

To create a new experiment, start by creating a new directory as follows:

Create a directory for your experiment

mkdir my_experiment1
mkdir my_experiment1/mp3           # store audio files
touch my_experiment1/config.yaml   # configuration file

Note that you can use any name instead of 'my_experiment1', and 'mp3'. For purposes of this documentation, we will use these names. The configuration file must be called 'config.yaml'.

Prepare Audio Data

Speech synthesis output will typically be wav files. You should convert them to mp3 so that they can be played with the flash player that TestVox uses. You can use "lame" to convert wav to mp3.

lame -V1 file.wav file.mp3

Move all the mp3 files in your experiment's mp3 directory. You can have your own directory structure within the directory for your audio files.

Here is an example directory structure:

my_experiment/
  config.yaml
  mp3/
    tts_model_A/
      1.mp3
      2.mp3
    tts_model_B/
      1.mp3
      2.mp3  

Write the Configuration File

TestVox configuration file is always called config.yaml and it should be present in the main directory of your experiment. As the name suggests, we use YAML as the configuration language. YAML is a superset of JSON. You don't need to know all of YAML to set up experiments. But remember: Indentation is important :-)

For the types of listening tests that FestVox Supports, we have created sample configuration files that you can copy and modify to set up your experiment. See TestVox_Configuration_Recipes

To learn more about the configuration file, and help with customizing it to your needs, visit TestVox_Configuration.

Upload Experiment to TestVox

Once you have created the experiment directory, zip it up. TestVox includes a utility script for this task.

python /path/to/TestVox-prebuilt/scripts/create_experiment_zipfile.py my_experiment1 my_experiment1.zip

Now visit the TestVox administration interface at http://your.server:your_port/admin. Log in with the appropriate credentials and upload your zipfile. The webpage will ask for a description of the experiment. Write something that will help you quickly identify the experiment. (E.g. tts_model_A vs tts_model_B for project Z).

Conduct Listening Tests

Once you have uploaded an experiment, the admin page (http://your.server:your_port/admin) will show you a link called "Local URL". This is the URL that you can send out to participants. The URL has a format like:

http://your.server:your_port/begin?exp_id=xxxxx

If you want, you can create links for individual people by adding the username you would want to save. To do that, just add at the end of the URL another variable: username.

Thus, you can create http://your.server:your_port/begin?exp_id=xxxxxx&username=aup

If you do not specify any username, then TestVox will prompt for a name to be entered before the test starts. On the other hand, if the username is 'testvox_test', then TestVox will enter PREVIEW mode, and you can look at whether tests are working properly. But responses from this special user will not be saved.

If you had configured your experiment for Amazon Mechanical Turk, you can create HITs on MTurk. See Listening Tests on Mechanical Turk for the details.

Download Results

The TestVox administration interface will allow you to download results of experiments as a csv file. One row of the CSV file contains answers submitted by one participant.

Delete Experiment

It is a good idea to delete uploaded experiments once they are completed. The TestVox administration interface allows you to delete experiments. When you delete an experiment, all data associated with it, including results, are deleted. Make sure you download the results before deleting.