The Absolute Category Rating (ACR) video experiment presents raters with individual video stimuli and asks them to rate each one on a categorical quality scale. This is a standard methodology for evaluating perceived video quality, commonly used for assessing video codecs, enhancement algorithms, and generative video models.
Unlike pairwise comparisons where stimuli are evaluated relative to each other, ACR asks raters to judge each stimulus independently. This makes it well suited for large-scale evaluations where the number of conditions would make pairwise comparisons impractical.
By default, the rating scale uses the standard five-point ACR scale:
| Score | Label |
|---|---|
| 5 | Excellent |
| 4 | Good |
| 3 | Fair |
| 2 | Poor |
| 1 | Bad |
Categories are fully customizable. You can change the number of categories (between 2 and 10), their labels, and their scores.
By default, videos are played without audio. If your videos contain audio and you wish to play it, you can enable it here.
When enabled, raters are required to view the video in fullscreen mode. This can be useful for evaluating video quality at the intended display size.
This option controls the minimum duration (in milliseconds) for which the video must be played before raters can submit their response.
Our ACR video experiment requires fragmented mp4s. Videos uploaded to our platform are automatically fragmented but if you are self-hosting data, you can use the ffmpeg command below to generate fragmented mp4s.
We do not transcode videos uploaded to our platform. For high quality and wide browser compatibility, we recommend encoding videos using ffmpeg with the following settings:
ffmpeg -i INPUT \
-preset veryslow \
-keyint_min 2 -g 24 -sc_threshold 0 \
-c:v libx264 -pix_fmt yuv420p -crf 17 \
-c:a aac -b:a 256k -ac 2 -ar 48000 \
-movflags +frag_keyframe+empty_moov+default_base_moof \
OUTPUT.mp4CRF values below 18 are generally considered visually lossless. Similarly for audio, compression at 256 kbps or above is considered unnoticeable [1, 2].
Below is an example configuration to get you started when using our API to create an ACR video experiment.
config = {
"acrVideo": {
"categories": [
{"score": 1, "label": "Bad"},
{"score": 2, "label": "Poor"},
{"score": 3, "label": "Fair"},
{"score": 4, "label": "Good"},
{"score": 5, "label": "Excellent"},
],
"hasAudio": False,
"fullscreen": False,
"minPlayDuration": 1000,
},
}[1] Pras et al. (2009). Subjective Evaluation of MP3 Compression for Different Musical Genres.
[2] Hines et al. (2014). Perceived Audio Quality for Streaming Stereo Music.