The survey video experiment presents raters with individual video stimuli and asks them to answer custom survey questions about each one. Unlike ACR video where raters select a score from a fixed category scale, the survey experiment supports open-ended text responses, single-choice (radio), and multiple-choice (checkbox) questions.
This is useful when you need qualitative feedback or structured responses that go beyond a simple quality rating, such as asking raters to describe what they see, identify specific artifacts, or classify content attributes.
You can configure up to 10 survey questions per experiment. Each question requires a kind (the question type), a question (the prompt), and a dimension (a short label used to identify the question in the results).
Three question types are supported:
| Kind | Description |
|---|---|
textarea | An open-ended text response field. |
radio | A single-choice question where the rater selects one option. |
checkbox | A multiple-choice question where the rater selects one or more options. |
For radio and checkbox questions, you must also provide a list of choices. For checkbox questions, you can optionally set minSelect and maxSelect to constrain the number of selections.
By default, videos are played without audio. If your videos contain audio and you wish to play it, you can enable it here.
When enabled, raters are required to view the video in fullscreen mode.
This option controls the minimum duration (in milliseconds) for which the video must be played before raters can submit their response.
Our survey video experiment requires fragmented mp4s. Videos uploaded to our platform are automatically fragmented but if you are self-hosting data, you can use the ffmpeg command below to generate fragmented mp4s.
We do not transcode videos uploaded to our platform. For high quality and wide browser compatibility, we recommend encoding videos using ffmpeg with the following settings:
ffmpeg -i INPUT \
-preset veryslow \
-keyint_min 2 -g 24 -sc_threshold 0 \
-c:v libx264 -pix_fmt yuv420p -crf 17 \
-c:a aac -b:a 256k -ac 2 -ar 48000 \
-movflags +frag_keyframe+empty_moov+default_base_moof \
OUTPUT.mp4CRF values below 18 are generally considered visually lossless. Similarly for audio, compression at 256 kbps or above is considered unnoticeable [1, 2].
Below is an example configuration to get you started when using our API to create a survey video experiment.
config = {
"surveyVideo": {
"survey": [
{
"kind": "textarea",
"dimension": "Overall",
"question": "How do you feel about the video quality?",
},
{
"kind": "radio",
"dimension": "Realism",
"question": "How realistic does this video look?",
"choices": [
{"label": "Very realistic"},
{"label": "Somewhat realistic"},
{"label": "Not realistic"},
],
},
],
"hasAudio": False,
"fullscreen": False,
"minPlayDuration": 1000,
},
}[1] Pras et al. (2009). Subjective Evaluation of MP3 Compression for Different Musical Genres.
[2] Hines et al. (2014). Perceived Audio Quality for Streaming Stereo Music.