Survey image

The survey image experiment presents raters with individual images and asks them to answer custom survey questions about each one. Unlike ACR image where raters select a score from a fixed category scale, the survey experiment supports open-ended text responses, single-choice (radio) and multiple-choice (checkbox) questions, and slider ratings.

This is useful when you need qualitative feedback or structured responses that go beyond a simple quality rating, such as asking raters to describe what they see, identify specific artifacts, or classify image attributes.

Survey questions

You can configure up to 10 survey questions per experiment. Each question requires a kind (the question type), a question (the prompt), and a dimension (a short label used to identify the question in the results).

The following question types are supported:

Kind	Description
textarea	An open-ended text response field.
radio	A single-choice question where the rater selects one option.
checkbox	A multiple-choice question where the rater selects one or more options.
discrete_slider	A slider that snaps to one of the labelled categories.
continuous_slider	A slider that can be placed anywhere between the labelled categories.

For radio and checkbox questions, you must also provide a list of choices. For checkbox questions, you can optionally set minSelect and maxSelect to constrain the number of selections.

Slider questions instead require a list of categories (2 to 10 {label, score} objects giving the labels shown along the slider and the score recorded at each one) and a responseType, which must be discrete for discrete_slider and continuous for continuous_slider. A discrete slider records the score of the category it snaps to; a continuous slider records a score interpolated between the two neighbouring categories. Sliders do not take choices, minSelect or maxSelect.

Options

Cropping

Images may be cropped for practical reasons if the full-resolution images do not fit on the screen and you do not wish to scale them down. The maximum crop size controls the size of the crop. The initial crop is chosen uniformly at random from all possible crops.

If you enable allow crop refresh, raters can request a different random crop if they wish to see a different part of the image.

Minimum view duration

This option controls the minimum duration (in milliseconds) for which the image must be displayed before raters can submit their response.

Configuration via config.json

Instead of (or in addition to) setting a single survey for the whole experiment, you can define survey questions per group directly in the dataset using a config.json file. This is useful when different images need different questions: each group carries its own survey, which overrides the experiment-level configuration set in the app or via the API.

Place a config.json next to the image in each group folder:

├── source_1/
│   ├── config.json
│   └── sample.png
├── source_2/
│   ├── config.json
│   └── sample.png
├── ...

The survey array uses the same question format described in Survey questions:

{
    "survey": [
        {
            "kind": "textarea",
            "dimension": "Overall",
            "question": "Describe the quality of this image."
        },
        {
            "kind": "radio",
            "dimension": "Content type",
            "question": "What does this image depict?",
            "choices": [
                { "label": "Photograph" },
                { "label": "Illustration" },
                { "label": "AI-generated" },
                { "label": "Other" }
            ]
        }
    ]
}

A group's config.json survey takes precedence over the experiment-level survey. You can put the survey array at the root of the config as shown above, or nest it under the experiment key (surveyImage.survey), which takes precedence over a root-level survey.

Configuration via API

Below is an example configuration to get you started when using our API to create a survey image experiment.

config = {
    'surveyImage': {
        'survey': [
            {
                'kind': 'textarea',
                'dimension': 'Overall',
                'question': 'Describe the quality of this image.',
            },
            {
                'kind': 'radio',
                'dimension': 'Content type',
                'question': 'What does this image depict?',
                'choices': [
                    {'label': 'Photograph'},
                    {'label': 'Illustration'},
                    {'label': 'AI-generated'},
                    {'label': 'Other'},
                ],
            },
            {
                'kind': 'checkbox',
                'dimension': 'Artifacts',
                'question': 'Which artifacts do you notice? (Select all that apply)',
                'choices': [
                    {'label': 'Blurriness'},
                    {'label': 'Color banding'},
                    {'label': 'Noise'},
                    {'label': 'Compression artifacts'},
                    {'label': 'None'},
                ],
                'minSelect': 1,
            },
        ],
        'allowCropRefresh': True,
        'maxCropSize': 512,
        'minViewDuration': 1000,
    },
}