Audio pre-screening

The audio pre-screening experiment is designed to check whether raters can reliably hear small differences between audio samples before they take part in a larger listening study.

Raters are asked to use headphones and complete the task in a quiet environment. Each slate shows a reference clip together with one or two test clips, and the rater must listen before making a decision. After the session, the outro page shows a per-question correct or incorrect summary.

How it works

Jobs created in the app use a lexicographic sampling strategy. They also try to include the dataset reference and anchors whenever those are available.

Depending on the dataset and job setup, a slate can contain either one or two test clips. In both cases, the task is intended as a screening step rather than a full preference study.

Dataset format

The canonical audio pre-screening dataset uses one folder per question. Each folder contains a config.json file and either one audio file or two audio files.

Single-audio question

Some screening tasks ask the rater to listen to a single clip and answer a simple prompt:

group_name/
  config.json
  clip.mp3

Example from the canonical dataset:

{
    "title": "Hearing Test",
    "question": "Left or right?",
    "description": ["Does the sound come from the left or right side of your headphones?"]
}

Two-audio discrimination question

Other tasks present two clips and ask the rater to choose the correct one:

group_name/
  config.json
  clip_a.mp3
  clip_b.mp3

Example:

{
    "title": "Hearing Test",
    "question": "Which audio file is the original?",
    "description": [
        "One of these audio files has been processed with an algorithm that degrades the audio quality. Select the original, unprocessed audio file."
    ]
}

The canonical dataset also includes language-focused pairs with multi-line instructions, for example:

{
    "title": "Preposition Test",
    "question": "Which version sounds correct?",
    "description": [
        "You will hear two very similar sentences.",
        "Only one of them sounds natural to a native speaker.",
        "Pay attention to the preposition in the sentence."
    ]
}

and:

{
    "title": "Word Order Test",
    "question": "Which version sounds correct?",
    "description": [
        "You will hear a subordinate clause with a small difference.",
        "Only one version has a natural word order.",
        "Pay attention to the position of the verb at the end of the sentence."
    ]
}

In practice, title, question, and description are the main dataset-level fields used by the canonical pre-screening audio set.

Options

This experiment currently has no dedicated experiment-level options in the app form. Most of the behavior comes from the dataset contents and the job configuration.

Job setup

For pre-screening jobs, the app form defaults to a lexicographic strategy so raters see a stable, predictable sequence of screening questions.

The public create-job API supports selecting the top-level strategy type, but it does not expose the more detailed pre-screening strategy fields used by the app form to always include references and anchors when available.

Configuration via API

Below is a minimal experiment configuration accepted by the public API when creating an audio pre-screening experiment.

config = {
    'preScreenAudio': {},
}

When creating a draft job through the public API, you can use a payload such as:

{
    "rater_pool": "b1ou403",
    "num_sessions": 1,
    "num_comparisons": 2,
    "num_training": 0,
    "strategy": "lexicographic"
}