Skip to main content

Real or not?

The binary video experiment presents raters with a single video and asks them to decide which of two categories it belongs to. By default, these categories are Real and Generated, but the labels can be customized.

This experiment is useful for discrimination tasks where you want to measure whether raters can tell apart reference videos from synthetic or otherwise manipulated videos. In each trial, raters watch one video and choose between the two configured labels.

Binary video interface

Categories​

The two response labels are configurable. By default, the interface uses:

KeyLabel
realReal
fakeGenerated

You can replace these with labels that better match your task, as long as there are still exactly two categories.

Options​

Minimum play duration​

This option controls the minimum duration (in milliseconds) for which the video must be played before raters can submit their response.

Reference probability​

This option controls how often a reference video is shown instead of a non-reference video. A value of 0 means reference videos are never used, a value of 1 means only reference videos are used, and 0.5 means reference and non-reference videos are shown equally often.

This is useful when the dataset contains reference videos and you want to measure how reliably raters can distinguish them from generated or manipulated examples.

Audio​

By default, videos are played without audio. If your videos contain audio and you want raters to hear it, enable this option.

Give feedback​

If enabled, raters hear immediate audio feedback after each response indicating whether their decision was correct.

Allow seeking​

If enabled, raters can seek backward and forward while reviewing the video. If disabled, the video must be watched without manual seeking.

Allow replay​

If enabled, raters can replay the video. If disabled, the video can only be played once before a response is submitted.

Configuration via API​

Below is an example configuration to get you started when using our API to create a binary video experiment.

config = {
    'binaryVideo': {
        'labels': {
            'real': 'Real',
            'fake': 'Generated',
        },
        'referenceProbability': 0.5,
        'minPlayDuration': 1000,
        'hasAudio': False,
        'giveFeedback': True,
        'allowSeeking': True,
        'allowReplay': True,
    },
}