The Absolute Category Rating (ACR) audio experiment presents raters with individual audio stimuli and asks them to rate each one on a categorical quality scale. ACR is a widely used methodology standardized in ITU-T P.910 for evaluating speech and audio quality.
Unlike pairwise comparisons where stimuli are evaluated relative to each other, ACR asks raters to judge each stimulus independently. This makes it well suited for large-scale evaluations where the number of conditions would make pairwise comparisons impractical.
By default, the rating scale uses the standard five-point ACR scale:
| Score | Label |
|---|---|
| 5 | Excellent |
| 4 | Good |
| 3 | Fair |
| 2 | Poor |
| 1 | Bad |
Categories are fully customizable. You can change the number of categories (between 2 and 10), their labels, and their scores.
When enabled, a hidden reference is included among the stimuli that raters evaluate. Since the rater does not know which stimulus is the reference, this can be used to verify that raters are paying attention (the reference should receive high scores).
When enabled, an explicitly labeled reference audio sample is shown alongside the stimulus being rated. This gives raters a baseline against which to judge the quality of the stimulus.
This option controls how the waveform is rendered. Setting it to stimulus renders a waveform based on each audio sample. Setting it to reference uses the waveform of the reference for all audio samples, preventing raters from using the waveform to identify the reference or draw conclusions about the quality of the audio.
This option controls the minimum duration (in milliseconds) for which the audio must be played before raters can submit their response.
Below is an example configuration to get you started when using our API to create an ACR audio experiment.
config = {
"acrAudio": {
"categories": [
{"score": 1, "label": "Bad"},
{"score": 2, "label": "Poor"},
{"score": 3, "label": "Fair"},
{"score": 4, "label": "Good"},
{"score": 5, "label": "Excellent"},
],
"hiddenReference": True,
"showReference": False,
"showWaveform": "stimulus",
"minPlayDuration": 1000,
},
}