Skip to content
API
/
API reference
/
/
Create a job

Mabyduck API (0.8.3)

API for managing subjective studies on Mabyduck.

Download OpenAPI description
openapi.json
openapi.yaml
Languages
Servers
https://api.mabyduck.com

Datasets

Operations related to datasets.

Operations

Experiments

Operations related to experiments.

Operations

List all experiments

Request

Returns a list of experiments belonging to the given project.

Security
ApiKeyAuth
Path
project_idstring^[a-zA-Z0-9]+$required
curl -i -X GET \
  'https://api.mabyduck.com/projects/{project_id}/experiments/' \
  -H 'Authorization: Api-Key <YOUR_TOKEN_HERE>'

Responses

Bodyapplication/jsonArray [
idstringrequired
namestring<= 128 charactersrequired

A description identifying the experiment.

typestringrequired
languagestring
  • en - English
  • fr - French
  • de - German
  • zh-hans - Chinese (simplified)
  • es - Spanish
  • hi - Hindi
  • pl - Polish
  • vi - Vietnamese
Enum"en""fr""de""zh-hans""es""hi""pl""vi"
dataset_idsArray of stringsread-onlyrequired
training_dataset_idsArray of stringsread-onlyrequired
job_idsArray of stringsread-onlyrequired
date_createdstring(date-time)read-onlyrequired
date_updatedstring(date-time)(Last updated)read-onlyrequired
configany
]
Response
application/json
[
  {
    "id": "string",
    "name": "string",
    "type": "string",
    "language": "en",
    "dataset_ids": [ … ],
    "training_dataset_ids": [ … ],
    "job_ids": [ … ],
    "date_created": "2019-08-24T14:15:22Z",
    "date_updated": "2019-08-24T14:15:22Z",
    "config": null
  }
]

Create an experiment

Request

Creates a new draft experiment with the specified configuration.

Security
ApiKeyAuth
Path
project_idstring^[a-zA-Z0-9]+$required
Body
required
namestring<= 128 charactersrequired

A description identifying the experiment.

typestringrequired

Type of the experiment (e.g., "mushra", "pairwise_image")

Enum"acr_audio""acr_video""acr_image""multi_acr_embed""multi_acr_video""binary_audio""binary_image""binary_video""inpainting_image""mushra"
languagestring
  • en - English
  • fr - French
  • de - German
  • zh-hans - Chinese (simplified)
  • es - Spanish
  • hi - Hindi
  • pl - Polish
  • vi - Vietnamese
Enum"en""fr""de""zh-hans""es""hi""pl""vi"
datasetsArray of integersrequired

List of dataset IDs to use in this experiment

training_datasetsArray of integers

List of training dataset IDs (optional)

titlestring or null<= 128 characters

A short title displayed at the top of the experiment screen.

questionstring or null<= 512 characters

A question or instructions displayed at the top of the experiment screen.

descriptionstring or null

An optional longer explanation of how to evaluate the data.

introductionstring or null<= 1024 characters

A brief introduction explaining how to participate in the experiment. Leave empty to use default introduction.

configany
curl -i -X POST \
  'https://api.mabyduck.com/projects/{project_id}/experiments/' \
  -H 'Authorization: Api-Key <YOUR_TOKEN_HERE>' \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "string",
    "type": "acr_audio",
    "language": "en",
    "datasets": [
      0
    ],
    "training_datasets": [
      0
    ],
    "title": "string",
    "question": "string",
    "description": "string",
    "introduction": "string",
    "config": null
  }'

Responses

Bodyapplication/json
idstringrequired
namestring<= 128 charactersrequired

A description identifying the experiment.

typestringrequired
languagestring
  • en - English
  • fr - French
  • de - German
  • zh-hans - Chinese (simplified)
  • es - Spanish
  • hi - Hindi
  • pl - Polish
  • vi - Vietnamese
Enum"en""fr""de""zh-hans""es""hi""pl""vi"
dataset_idsArray of stringsread-onlyrequired
training_dataset_idsArray of stringsread-onlyrequired
job_idsArray of stringsread-onlyrequired
date_createdstring(date-time)read-onlyrequired
date_updatedstring(date-time)(Last updated)read-onlyrequired
configany
Response
application/json
{
  "id": "string",
  "name": "string",
  "type": "string",
  "language": "en",
  "dataset_ids": [
    "string"
  ],
  "training_dataset_ids": [
    "string"
  ],
  "job_ids": [
    "string"
  ],
  "date_created": "2019-08-24T14:15:22Z",
  "date_updated": "2019-08-24T14:15:22Z",
  "config": null
}

Retrieve an experiment

Request

Returns detailed information for a single experiment identified by experiment_id.

Security
ApiKeyAuth
Path
experiment_idstring^[a-zA-Z0-9]+$required
project_idstring^[a-zA-Z0-9]+$required
curl -i -X GET \
  'https://api.mabyduck.com/projects/{project_id}/experiments/{experiment_id}/' \
  -H 'Authorization: Api-Key <YOUR_TOKEN_HERE>'

Responses

Bodyapplication/json
idstringrequired
namestring<= 128 charactersrequired

A description identifying the experiment.

typestringrequired
languagestring
  • en - English
  • fr - French
  • de - German
  • zh-hans - Chinese (simplified)
  • es - Spanish
  • hi - Hindi
  • pl - Polish
  • vi - Vietnamese
Enum"en""fr""de""zh-hans""es""hi""pl""vi"
datasetsArray of objects(Dataset)read-onlyrequired
datasets[].​idstringrequired
datasets[].​namestring<= 64 charactersrequired

A short name identifying the dataset.

datasets[].​date_createdstring(date-time)read-onlyrequired
datasets[].​date_last_usedstring(date-time)(Last used)read-onlyrequired
datasets[].​is_deletedboolean

The files in this dataset have been deleted.

datasets[].​statusstring
  • draft - Draft
  • processing - Processing
  • ready - Ready
  • error - Error
Enum"draft""processing""ready""error"
training_datasetsArray of objects(Dataset)read-onlyrequired
training_datasets[].​idstringrequired
training_datasets[].​namestring<= 64 charactersrequired

A short name identifying the dataset.

training_datasets[].​date_createdstring(date-time)read-onlyrequired
training_datasets[].​date_last_usedstring(date-time)(Last used)read-onlyrequired
training_datasets[].​is_deletedboolean

The files in this dataset have been deleted.

training_datasets[].​statusstring
  • draft - Draft
  • processing - Processing
  • ready - Ready
  • error - Error
Enum"draft""processing""ready""error"
jobsArray of objects(Job)read-onlyrequired
jobs[].​idstringrequired
jobs[].​experiment_idstringread-onlyrequired
jobs[].​num_sessionsinteger(Number of sessions)[ 0 .. 10000 ]required

How often the task is performed.

jobs[].​num_comparisonsinteger(Number of comparisons per rater)[ 1 .. 1000 ]required

The maximum number of groups of stimuli evaluated by each rater.

jobs[].​num_traininginteger(Number of training comparisons)[ 0 .. 2147483647 ]

Set to zero to disable training.

jobs[].​date_createdstring(date-time)read-onlyrequired
jobs[].​date_updatedstring(date-time)(Last updated)read-onlyrequired
jobs[].​date_launchedstring or null(date-time)
jobs[].​rater_poolobjectread-onlyrequired
jobs[].​rater_pool.​idstringrequired
jobs[].​rater_pool.​kindinteger[ -2147483648 .. 2147483647 ]
  • 0 - Anyone
  • 100 - Project members
  • 400 - Crowd-sourced raters
  • 500 - AI raters
  • 600 - Rater pool
Enum0100400500600
jobs[].​rater_pool.​namestringrequired
jobs[].​rater_pool.​labelstring<= 32 characters

Optional name for the rater pool displayed to our users.

jobs[].​statusstringread-onlyrequired
jobs[].​configany

Configures stimulus selection strategy.

titlestring or null<= 128 characters

A short title displayed at the top of the experiment screen.

questionstring or null<= 512 characters

A question or instructions displayed at the top of the experiment screen.

descriptionstring or null

An optional longer explanation of how to evaluate the data.

introductionstring or null<= 1024 characters

A brief introduction explaining how to participate in the experiment. Leave empty to use default introduction.

date_createdstring(date-time)read-onlyrequired
date_updatedstring(date-time)(Last updated)read-onlyrequired
configany
Response
application/json
{
  "id": "string",
  "name": "string",
  "type": "string",
  "language": "en",
  "datasets": [
    { … }
  ],
  "training_datasets": [
    { … }
  ],
  "jobs": [
    { … }
  ],
  "title": "string",
  "question": "string",
  "description": "string",
  "introduction": "string",
  "date_created": "2019-08-24T14:15:22Z",
  "date_updated": "2019-08-24T14:15:22Z",
  "config": null
}

List all jobs

Request

Returns a list of jobs belonging to the given experiment.

Security
ApiKeyAuth
Path
experiment_idstring^[a-zA-Z0-9]+$required
project_idstring^[a-zA-Z0-9]+$required
curl -i -X GET \
  'https://api.mabyduck.com/projects/{project_id}/experiments/{experiment_id}/jobs/' \
  -H 'Authorization: Api-Key <YOUR_TOKEN_HERE>'

Responses

Bodyapplication/jsonArray [
idstringrequired
experiment_idstringread-onlyrequired
num_sessionsinteger(Number of sessions)[ 0 .. 10000 ]required

How often the task is performed.

num_comparisonsinteger(Number of comparisons per rater)[ 1 .. 1000 ]required

The maximum number of groups of stimuli evaluated by each rater.

num_traininginteger(Number of training comparisons)[ 0 .. 2147483647 ]

Set to zero to disable training.

date_createdstring(date-time)read-onlyrequired
date_updatedstring(date-time)(Last updated)read-onlyrequired
date_launchedstring or null(date-time)
rater_poolobjectread-onlyrequired
rater_pool.​idstringrequired
rater_pool.​kindinteger[ -2147483648 .. 2147483647 ]
  • 0 - Anyone
  • 100 - Project members
  • 400 - Crowd-sourced raters
  • 500 - AI raters
  • 600 - Rater pool
Enum0100400500600
rater_pool.​namestringrequired
rater_pool.​labelstring<= 32 characters

Optional name for the rater pool displayed to our users.

statusstringread-onlyrequired
configany

Configures stimulus selection strategy.

]
Response
application/json
[
  {
    "id": "string",
    "experiment_id": "string",
    "num_sessions": 10000,
    "num_comparisons": 1,
    "num_training": 2147483647,
    "date_created": "2019-08-24T14:15:22Z",
    "date_updated": "2019-08-24T14:15:22Z",
    "date_launched": "2019-08-24T14:15:22Z",
    "rater_pool": { … },
    "status": "string",
    "config": null
  }
]

Create a job

Request

Creates a new draft job with the specified configuration.

Security
ApiKeyAuth
Path
experiment_idstring^[a-zA-Z0-9]+$required
project_idstring^[a-zA-Z0-9]+$required
Body
required
rater_poolintegerrequired

ID of the rater pool to use for this job

num_sessionsinteger>= 1required

Number of sessions (times the task is performed)

num_comparisonsinteger>= 1required

Number of comparisons per session

num_traininginteger>= 0

Number of training comparisons

Default 2
max_repetitionsinteger(Maximum number of repetitions)[ 0 .. 2147483647 ]

How often an individual rater is allowed to repeat the task.

min_rest_timeinteger(Minimum rest time)[ 0 .. 2147483647 ]

How many minutes should pass before a rater can repeat the task.

strategystring
  • randomized - randomized
  • lexicographic - lexicographic
  • active - active
  • neighbor - neighbor
Default "randomized"
Enum"randomized""lexicographic""active""neighbor"
notestring or null

An optional note for this job. Not visible to raters.

curl -i -X POST \
  'https://api.mabyduck.com/projects/{project_id}/experiments/{experiment_id}/jobs/' \
  -H 'Authorization: Api-Key <YOUR_TOKEN_HERE>' \
  -H 'Content-Type: application/json' \
  -d '{
    "rater_pool": 0,
    "num_sessions": 1,
    "num_comparisons": 1,
    "num_training": 2,
    "max_repetitions": 2147483647,
    "min_rest_time": 2147483647,
    "strategy": "randomized",
    "note": "string"
  }'

Responses

Bodyapplication/json
idstringrequired
experiment_idstringread-onlyrequired
num_sessionsinteger(Number of sessions)[ 0 .. 10000 ]required

How often the task is performed.

num_comparisonsinteger(Number of comparisons per rater)[ 1 .. 1000 ]required

The maximum number of groups of stimuli evaluated by each rater.

num_traininginteger(Number of training comparisons)[ 0 .. 2147483647 ]

Set to zero to disable training.

date_createdstring(date-time)read-onlyrequired
date_updatedstring(date-time)(Last updated)read-onlyrequired
date_launchedstring or null(date-time)
rater_poolobjectread-onlyrequired
rater_pool.​idstringrequired
rater_pool.​kindinteger[ -2147483648 .. 2147483647 ]
  • 0 - Anyone
  • 100 - Project members
  • 400 - Crowd-sourced raters
  • 500 - AI raters
  • 600 - Rater pool
Enum0100400500600
rater_pool.​namestringrequired
rater_pool.​labelstring<= 32 characters

Optional name for the rater pool displayed to our users.

statusstringread-onlyrequired
configany

Configures stimulus selection strategy.

Response
application/json
{
  "id": "string",
  "experiment_id": "string",
  "num_sessions": 10000,
  "num_comparisons": 1,
  "num_training": 2147483647,
  "date_created": "2019-08-24T14:15:22Z",
  "date_updated": "2019-08-24T14:15:22Z",
  "date_launched": "2019-08-24T14:15:22Z",
  "rater_pool": {
    "id": "string",
    "kind": 0,
    "name": "string",
    "label": "string"
  },
  "status": "string",
  "config": null
}

Retrieve a job

Request

Returns detailed information for a single job identified by job_id.

Security
ApiKeyAuth
Path
experiment_idstring^[a-zA-Z0-9]+$required
job_idstring^[a-zA-Z0-9]+$required
project_idstring^[a-zA-Z0-9]+$required
curl -i -X GET \
  'https://api.mabyduck.com/projects/{project_id}/experiments/{experiment_id}/jobs/{job_id}/' \
  -H 'Authorization: Api-Key <YOUR_TOKEN_HERE>'

Responses

Bodyapplication/json
idstringrequired
experiment_idstringread-onlyrequired
num_sessionsinteger(Number of sessions)[ 0 .. 10000 ]required

How often the task is performed.

num_comparisonsinteger(Number of comparisons per rater)[ 1 .. 1000 ]required

The maximum number of groups of stimuli evaluated by each rater.

num_traininginteger(Number of training comparisons)[ 0 .. 2147483647 ]

Set to zero to disable training.

date_createdstring(date-time)read-onlyrequired
date_updatedstring(date-time)(Last updated)read-onlyrequired
date_launchedstring or null(date-time)
rater_poolobjectread-onlyrequired
rater_pool.​idstringrequired
rater_pool.​kindinteger[ -2147483648 .. 2147483647 ]
  • 0 - Anyone
  • 100 - Project members
  • 400 - Crowd-sourced raters
  • 500 - AI raters
  • 600 - Rater pool
Enum0100400500600
rater_pool.​namestringrequired
rater_pool.​labelstring<= 32 characters

Optional name for the rater pool displayed to our users.

statusstringread-onlyrequired
configany

Configures stimulus selection strategy.

max_repetitionsinteger(Maximum number of repetitions)[ 0 .. 2147483647 ]

How often an individual rater is allowed to repeat the task.

min_rest_timeinteger(Minimum rest time)[ 0 .. 2147483647 ]

How many minutes should pass before a rater can repeat the task.

Response
application/json
{
  "id": "string",
  "experiment_id": "string",
  "num_sessions": 10000,
  "num_comparisons": 1,
  "num_training": 2147483647,
  "date_created": "2019-08-24T14:15:22Z",
  "date_updated": "2019-08-24T14:15:22Z",
  "date_launched": "2019-08-24T14:15:22Z",
  "rater_pool": {
    "id": "string",
    "kind": 0,
    "name": "string",
    "label": "string"
  },
  "status": "string",
  "config": null,
  "max_repetitions": 2147483647,
  "min_rest_time": 2147483647
}

Get job cost breakdown

Request

Returns cost breakdown information for a specific job.

Security
ApiKeyAuth
Path
experiment_idstring^[a-zA-Z0-9]+$required
job_idstring^[a-zA-Z0-9]+$required
project_idstring^[a-zA-Z0-9]+$required
curl -i -X GET \
  'https://api.mabyduck.com/projects/{project_id}/experiments/{experiment_id}/jobs/{job_id}/costs/' \
  -H 'Authorization: Api-Key <YOUR_TOKEN_HERE>'

Responses

Bodyapplication/json
num_sessionsintegerrequired
costnumber(double)required
cost_per_additional_sessionnumber(double)required
currencystringrequired
tokenstringrequired

Time-limited token to confirm cost calculation.

token_expires_inintegerrequired

Token expiry time in seconds.

token_expires_atstring(date-time)required

Token expiry timestamp (ISO 8601).

Response
application/json
{
  "num_sessions": 0,
  "cost": 0.1,
  "cost_per_additional_session": 0.1,
  "currency": "string",
  "token": "string",
  "token_expires_in": 0,
  "token_expires_at": "2019-08-24T14:15:22Z"
}

Launch a job

Request

Launches the specified job. Requires a valid job costs token.

Security
ApiKeyAuth
Path
experiment_idstring^[a-zA-Z0-9]+$required
job_idstring^[a-zA-Z0-9]+$required
project_idstring^[a-zA-Z0-9]+$required
Body
required
tokenstringrequired

Time-limited token to confirm job launch.

curl -i -X POST \
  'https://api.mabyduck.com/projects/{project_id}/experiments/{experiment_id}/jobs/{job_id}/launch/' \
  -H 'Authorization: Api-Key <YOUR_TOKEN_HERE>' \
  -H 'Content-Type: application/json' \
  -d '{
    "token": "string"
  }'

Responses

Bodyapplication/json
tokenstringrequired

Time-limited token to confirm job launch.

Response
application/json
{
  "token": "string"
}

List rater pools

Request

Returns a list of rater pools available for a given experiment.

Security
ApiKeyAuth
Path
experiment_idstring^[a-zA-Z0-9]+$required
project_idstring^[a-zA-Z0-9]+$required
Query
kindinteger

Filter by rater pool kind.

Enum0100400500600
curl -i -X GET \
  'https://api.mabyduck.com/projects/{project_id}/experiments/{experiment_id}/rater_pools/?kind=0' \
  -H 'Authorization: Api-Key <YOUR_TOKEN_HERE>'

Responses

Bodyapplication/jsonArray [
idstringrequired
kindinteger[ -2147483648 .. 2147483647 ]
  • 0 - Anyone
  • 100 - Project members
  • 400 - Crowd-sourced raters
  • 500 - AI raters
  • 600 - Rater pool
Enum0100400500600
namestringrequired
labelstring<= 32 characters

Optional name for the rater pool displayed to our users.

]
Response
application/json
[
  {
    "id": "string",
    "kind": 0,
    "name": "string",
    "label": "string"
  }
]

List all raters

Request

Returns a list of raters of a given experiment.

Security
ApiKeyAuth
Path
experiment_idstring^[a-zA-Z0-9]+$required
project_idstring^[a-zA-Z0-9]+$required
curl -i -X GET \
  'https://api.mabyduck.com/projects/{project_id}/experiments/{experiment_id}/raters/' \
  -H 'Authorization: Api-Key <YOUR_TOKEN_HERE>'

Responses

Bodyapplication/jsonArray [
idstringrequired
userobject(User)required
user.​idstringrequired
user.​namestring(Full name)<= 256 charactersrequired
metaany
]
Response
application/json
[
  {
    "id": "string",
    "user": { … },
    "meta": null
  }
]

List all ratings

Request

Returns a list of ratings belonging to the given experiment.

Security
ApiKeyAuth
Path
experiment_idstring^[a-zA-Z0-9]+$required
project_idstring^[a-zA-Z0-9]+$required
Query
trainingstring

Whether to include training data.

Enum"false""true"
curl -i -X GET \
  'https://api.mabyduck.com/projects/{project_id}/experiments/{experiment_id}/ratings/?training=false' \
  -H 'Authorization: Api-Key <YOUR_TOKEN_HERE>'

Responses

Bodyapplication/jsonArray [
scorenumber or null(double)
rater_idintegerrequired
slate_idintegerread-onlyrequired
session_idstringrequired
job_idstringrequired
stimulus_idintegerread-onlyrequired
stimulusstringrequired
dimensionstring or null<= 32 characters
is_trainingbooleanrequired
metaany

For example, expected score range.

is_correctboolean or nullread-onlyrequired

Checks if score is within expectation.

First checks for explicit correctness in meta['isCorrect']. Falls back to comparing score within bounds given in meta['expectedScores'] (inclusive).

Returns: True if score is correct, False if incorrect, or None if neither.

]
Response
application/json
[
  {
    "score": 0,
    "rater_id": 0,
    "slate_id": 0,
    "session_id": "string",
    "job_id": "string",
    "stimulus_id": 0,
    "stimulus": "string",
    "dimension": "string",
    "is_training": true,
    "meta": null,
    "is_correct": true
  }
]

List all results

Request

Returns a list of metrics derived from the results of an experiment.

Security
ApiKeyAuth
Path
experiment_idstring^[a-zA-Z0-9]+$required
project_idstring^[a-zA-Z0-9]+$required
Query
metricstring

Filter by this type of metric.

Enum"bayesian_elo""elo""mean""stats""win_loss"
sourcestring

An optional comma-separated list of job or experiment IDs.

curl -i -X GET \
  'https://api.mabyduck.com/projects/{project_id}/experiments/{experiment_id}/results/?metric=bayesian_elo&source=string' \
  -H 'Authorization: Api-Key <YOUR_TOKEN_HERE>'

Responses

Bodyapplication/jsonArray [
kindstring(Kind805Enum)required
  • bayesian_elo - Elo
  • elo - Elo (vanilla)
  • mean - MOS
  • win_loss - Win/loss
  • stats - Statistics
Enum"bayesian_elo""elo""mean""win_loss""stats"
dimensionstring or null<= 32 characters

Used by experiments evaluating multiple criteria simultaneously.

sourcestring or nullread-onlyrequired

Return the source of the metric (job or experiment).

scoresArray of anyread-onlyrequired

Return the scores for each method.

optionsobjectread-onlyrequired

Front-end options.

options.​property name*anyadditional property
]
Response
application/json
[
  {
    "kind": "bayesian_elo",
    "dimension": "string",
    "source": "string",
    "scores": [ … ],
    "options": { … }
  }
]

List all sessions

Request

Returns a list of sessions belonging to the given experiment.

Security
ApiKeyAuth
Path
experiment_idstring^[a-zA-Z0-9]+$required
project_idstring^[a-zA-Z0-9]+$required
Query
trainingstring

Whether to include training data.

Enum"false""true"
curl -i -X GET \
  'https://api.mabyduck.com/projects/{project_id}/experiments/{experiment_id}/sessions/?training=false' \
  -H 'Authorization: Api-Key <YOUR_TOKEN_HERE>'

Responses

Bodyapplication/jsonArray [
idstringrequired
job_idstringread-onlyrequired
experiment_idstringread-onlyrequired
date_createdstring(date-time)read-onlyrequired
date_updatedstring(date-time)(Last updated)read-onlyrequired
date_completedstring or null(date-time)(Completion date)
metaany
raterobject(Rater)required
rater.​idstringrequired
rater.​userobject(User)required
rater.​user.​idstringrequired
rater.​user.​namestring(Full name)<= 256 charactersrequired
rater.​metaany
]
Response
application/json
[
  {
    "id": "string",
    "job_id": "string",
    "experiment_id": "string",
    "date_created": "2019-08-24T14:15:22Z",
    "date_updated": "2019-08-24T14:15:22Z",
    "date_completed": "2019-08-24T14:15:22Z",
    "meta": null,
    "rater": { … }
  }
]

List all slates

Request

Returns a list of slates and ratings belonging to the given experiment.

Security
ApiKeyAuth
Path
experiment_idstring^[a-zA-Z0-9]+$required
project_idstring^[a-zA-Z0-9]+$required
Query
trainingstring

Whether to include training data.

Enum"false""true"
curl -i -X GET \
  'https://api.mabyduck.com/projects/{project_id}/experiments/{experiment_id}/slates/?training=false' \
  -H 'Authorization: Api-Key <YOUR_TOKEN_HERE>'

Responses

Bodyapplication/jsonArray [
idintegerread-onlyrequired
rater_idstringread-onlyrequired
session_idstringread-onlyrequired
job_idstringread-onlyrequired
ratingsArray of objects(Rating)read-onlyrequired
ratings[].​scorenumber or null(double)
ratings[].​stimulus_idintegerread-onlyrequired
ratings[].​stimulusstringrequired
ratings[].​conditionstringrequired
ratings[].​dimensionstring or null<= 32 characters
ratings[].​metaany

For example, expected score range.

ratings[].​is_correctboolean or nullread-onlyrequired

Checks if score is within expectation.

First checks for explicit correctness in meta['isCorrect']. Falls back to comparing score within bounds given in meta['expectedScores'] (inclusive).

Returns: True if score is correct, False if incorrect, or None if neither.

date_updatedstring(date-time)(Last updated)read-onlyrequired
metaany or null

For example, the time it took to complete the slate.

One of:

For example, the time it took to complete the slate.

any
is_trainingbooleanrequired
]
Response
application/json
[
  {
    "id": 0,
    "rater_id": "string",
    "session_id": "string",
    "job_id": "string",
    "ratings": [ … ],
    "date_updated": "2019-08-24T14:15:22Z",
    "meta": {},
    "is_training": true
  }
]

Rubrics

Operations related to rubrics and leaderboards.

Operations

Metrics

Operations