1. Alok Parlikar Avatar Alok Parlikar
  2. Untitled project
  3. testvox

Wiki

Clone wiki

testvox / Listening Tests on Mechanical Turk

View History

Listening Tests on Mechanical Turk

TestVox supports conducting listening tests on Amazon Mechanical Turk. The tests are hosted on the TestVox server, and are put out to Mechanical Turk as ExternalQuestion. A helper script has been provided to interact with Amazon Services to create and manage hits.

Results of tests on AMTurk will be saved along with any locally conducted tests in the same database, and can be downloaded from the admin page.

Configuring Experiments to use AMTurk

You have to explicitly configure an experiment for Mechanical Turk. The way to do that is to add the 'testvox_amturk_config' section to your config.yaml file.

testvox_amturk_config:
  num_hits: 10  # Your data will be partitioned into 10 hits
  num_clips_per_hit: 5  # Each session on M-Turk (Each HIT) will consist of 5 clips

Creating HITs on Mechanical Turk

If an experiment has the testvox_amturk_config section appropriately filled, the administration page will show a link called "AMTurk Config" next to the experiment. Click that. This will download a file of the name amturk_config_xxxxxxx.yaml.

Edit the downloaded configuration file to set the title, keywords, etc. appropriately.

Now you can use the scripts/amturk.py program to interact with Amazon Mechanical Turk.

# Check AMTurk Balance
python scripts/amturk.py bal

# Use -l or --live-turk to use actual turk, instead of sandbox. Default is to use sandbox
# Check Balance on actual turk
python scripts/amturk.py --live-turk bal

# Create HITs for experiment
# The --limit-to-country option here is not required. Default is to allow everybody

python scripts/amturk.py --live-turk posttask -f amturk_config_xxxxx.yaml \
    --workers-per-hit 10 \
    --approval-percentage 80 \
    --price-per-assignment 0.02
    --limit-to-country US


# DO NOT DELETE the amturk_config_xxxxx.yaml now. 
# The HIT information is saved in it and required later.


# Getting list of completed assignments
python scripts/amturk.py --live-turk listassignments -f amturk_config_xxxxx.yaml

# Rejecting hits that were completed too soon
python scripts/amturk.py --live-turk reject -f amturk_config_xxxxx.yaml \
    --min-time 30

# Approve all un-rejected assignments
python scripts/amturk.py --live-turk approveall -f amturk_config_xxxxx.yaml

# Delete all created HITs for an experiment
# Note: This will auto-approve all unreviewed work.
python scripts/amturk.py --live-turk deletetask -f amturk_config_xxxxx.yaml

Updated