ACR audio

The Absolute Category Rating (ACR) audio experiment presents raters with individual audio stimuli and asks them to rate each one on a categorical quality scale. ACR is a widely used methodology standardized in ITU-T P.910 for evaluating speech and audio quality.

Unlike pairwise comparisons where stimuli are evaluated relative to each other, ACR asks raters to judge each stimulus independently. This makes it well suited for large-scale evaluations where the number of conditions would make pairwise comparisons impractical.

Categories

By default, the rating scale uses the standard five-point ACR scale:

Score	Label
5	Excellent
4	Good
3	Fair
2	Poor
1	Bad

Categories are fully customizable. You can change the number of categories (between 2 and 10), their labels, and their scores.

Options

Hidden reference

When enabled, a hidden reference is included among the stimuli that raters evaluate. Since the rater does not know which stimulus is the reference, this can be used to verify that raters are paying attention (the reference should receive high scores).

Show reference

When enabled, an explicitly labeled reference audio sample is shown alongside the stimulus being rated. This gives raters a baseline against which to judge the quality of the stimulus.

Waveform

This option controls how the waveform is rendered. Setting it to stimulus renders a waveform based on each audio sample. Setting it to reference uses the waveform of the reference for all audio samples, preventing raters from using the waveform to identify the reference or draw conclusions about the quality of the audio.

Minimum play duration

This option controls the minimum duration (in milliseconds) for which the audio must be played before raters can submit their response.

Configuration via API

Below is an example configuration to get you started when using our API to create an ACR audio experiment.

config = {
    'acrAudio': {
        'categories': [
            {'score': 1, 'label': 'Bad'},
            {'score': 2, 'label': 'Poor'},
            {'score': 3, 'label': 'Fair'},
            {'score': 4, 'label': 'Good'},
            {'score': 5, 'label': 'Excellent'},
        ],
        'hiddenReference': True,
        'showReference': False,
        'showWaveform': 'stimulus',
        'minPlayDuration': 1000,
    },
}