Skip to main content

Surveys

Surveys ask raters questions and record their answers. They appear in three places:

  • Survey experiments ask questions about each stimulus: survey audio, survey image, survey video and survey embedded.
  • Pre- and post-surveys ask questions once per session, in any experiment type. A pre-survey is shown at the beginning of a session and a post-survey after it completes. Training sessions skip both.
  • Saved surveys are reusable question sets kept in the app's Surveys section. Load one into a survey experiment's questions with Load survey, or choose one as an experiment's Pre-survey or Post-survey. Its questions and rules are copied into the experiment, so editing the saved survey later does not change existing experiments. To manage saved surveys through the API, see the surveys tutorial.

Survey builder

Survey format​

A survey holds its questions and, optionally, its visibility rules:

{
"questions": [
{
"kind": "textarea",
"dimension": "Comments",
"question": "Anything else?"
}
]
}

rules is optional: leave it out when every question is always shown.

In an experiment's configuration, a survey experiment's survey goes under its type's key (surveyAudio.survey, surveyImage.survey, surveyVideo.survey or surveyEmbed.survey), and pre- and post-surveys go under preSurvey and postSurvey. A plain array of questions, the format used before visibility rules, is still accepted everywhere and is read as a survey without rules.

Questions​

A survey has up to 10 questions. Each question requires a kind (the question type), a question (the prompt), and a dimension (a short label used to identify the question in the results). Give every question its own dimension: visibility rules find questions by dimension.

The following question types are supported:

KindDescription
textareaAn open-ended text response field.
radioA single-choice question where the rater selects one option.
checkboxA multiple-choice question where the rater selects one or more options.
highlightA transcript in which the rater highlights words by clicking or dragging.
discrete_sliderA slider that snaps to one of the labelled categories.
continuous_sliderA slider that can be placed anywhere between the labelled categories.

For radio and checkbox questions, you must also provide a list of choices. For checkbox questions, you can optionally set minSelect and maxSelect to constrain the number of selections.

Slider questions instead require a list of categories (2 to 10 {label, score} objects giving the labels shown along the slider and the score recorded at each one) and a responseType, which must be discrete for discrete_slider and continuous for continuous_slider. A discrete slider records the score of the category it snaps to; a continuous slider records a score interpolated between the two neighbouring categories. Sliders do not take choices, minSelect or maxSelect.

Highlight questions​

highlight questions are only available in survey experiments, not in pre- and post-surveys or saved surveys. They take neither choices nor categories. The transcript the rater marks up is not part of the survey itself: it comes from a transcript string in the group's config.json. Groups whose transcript is missing or empty hide the highlight question.

Raters highlight the transcript one word at a time, and words are separated by whitespace. Text in Thai, Lao, Khmer, Burmese, Chinese or Japanese, which is written without spaces, can instead be highlighted one character at a time. To choose the boundaries yourself, separate words with |. The | is not shown to raters, and once a transcript contains one, no text is split into characters:

{ "transcript": "今日は|いい|天気です。" }

Visibility rules​

Rules show some questions only after certain answers, for example asking what kind of issue a rater noticed only when they report one. Questions named in a rule's show list start hidden; other questions are always visible.

{
"questions": [
{
"kind": "radio",
"dimension": "Audio issue",
"question": "Is there any issue with the audio?",
"choices": [{ "label": "Yes" }, { "label": "No" }]
},
{
"kind": "checkbox",
"dimension": "Issue type",
"question": "What kind of issue?",
"choices": [
{ "label": "Noise" },
{ "label": "Clipping" },
{ "label": "Distortion" },
{ "label": "Other" }
],
"minSelect": 1
},
{ "kind": "textarea", "dimension": "Noise detail", "question": "Describe the noise." },
{
"kind": "textarea",
"dimension": "Other detail",
"question": "What other issue did you notice?"
}
],
"rules": [
{
"when": [{ "dimension": "Audio issue", "choices": ["Yes"] }],
"show": ["Issue type"]
},
{
"when": [{ "dimension": "Issue type", "choices": ["Noise"] }],
"show": ["Noise detail"]
},
{
"when": [{ "dimension": "Issue type", "choices": ["Other"] }],
"show": ["Other detail"]
}
]
}

A follow-up question appearing after a "Yes" answer

Each rule applies when all its when conditions match. A condition matches any selected choice label in its choices list. If several rules show the same question, any matching rule shows it. A hidden question's answers do not trigger other rules.

Conditions refer to radio or checkbox questions earlier in the configured question order. Use unique dimensions, existing question dimensions, and exact choice labels. Shuffling affects the display order only.

In the app, edit the rules as JSON in Visibility rules, below the question editor. Saved surveys keep their rules, and loading or choosing one copies its rules along with its questions. Through the API, send them as the survey's rules, which you can leave out when there are none.

Visibility rules below the survey builder

Hidden questions keep their position in submitted answers with only kind and dimension; omit their answer and response.

Shuffling questions​

Survey experiments can show their questions in a random order. Shuffling per session keeps the same order across every slate of a session, while shuffling per slate draws a new order for each slate. When supplying a configuration via the API, set shuffleQuestions to none, session, or slate.

Per-group surveys​

Instead of (or in addition to) setting a single survey for the whole experiment, a survey experiment can ask different questions per group. Place a config.json next to the stimuli in each group folder:

├── source_1/
│ ├── config.json
│ └── sample.wav
├── source_2/
│ ├── config.json
│ └── sample.wav
├── ...

Each config.json holds the group's survey, in the same format:

{
"survey": {
"questions": [
{
"kind": "radio",
"dimension": "Content type",
"question": "What type of audio is this?",
"choices": [{ "label": "Speech" }, { "label": "Music" }, { "label": "Other" }]
}
]
}
}

A group's survey takes precedence over the experiment's, and its rules come with it, so the experiment's rules never apply to a group's own questions. You can put the survey at the root of the config as shown above, or nest it under the experiment key (for example surveyAudio.survey), which takes precedence over a root-level survey.