Skip to content
API
/
/
List all raters

Mabyduck API (0.7.5)

API for managing subjective studies on Mabyduck.

Download OpenAPI description
openapi.json
openapi.yaml
Languages
Servers
https://api.mabyduck.com

Datasets

Operations related to datasets.

Operations

Experiments

Operations related to experiments.

Operations

List all experiments

Request

Returns a list of experiments belonging to the given project.

Security
ApiKeyAuth
Path
project_idstring^[a-zA-Z0-9]+$required
curl -i -X GET \
  'https://api.mabyduck.com/projects/{project_id}/experiments/' \
  -H 'Authorization: Api-Key <YOUR_TOKEN_HERE>'

Responses

Bodyapplication/jsonArray [
idstringrequired
namestring<= 128 charactersrequired

A description identifying the experiment.

typestringrequired
languagestring
  • en - English
  • fr - French
  • de - German
  • zh-hans - Chinese (simplified)
  • es - Spanish
  • hi - Hindi
  • pl - Polish
  • vi - Vietnamese
Enum"en""fr""de""zh-hans""es""hi""pl""vi"
dataset_idsArray of stringsread-onlyrequired
job_idsArray of stringsread-onlyrequired
training_dataset_idsArray of stringsread-onlyrequired
date_createdstring(date-time)read-onlyrequired
date_updatedstring(date-time)(Last updated)read-onlyrequired
configany
]
Response
application/json
[
  {
    "id": "string",
    "name": "string",
    "type": "string",
    "language": "en",
    "dataset_ids": [ … ],
    "job_ids": [ … ],
    "training_dataset_ids": [ … ],
    "date_created": "2019-08-24T14:15:22Z",
    "date_updated": "2019-08-24T14:15:22Z",
    "config": null
  }
]

Create an experiment

Request

Creates a new draft experiment with the specified configuration.

Security
ApiKeyAuth
Path
project_idstring^[a-zA-Z0-9]+$required
Body
required
namestring<= 128 charactersrequired

A description identifying the experiment.

versionintegerrequired

ID of the experiment version/type to use

datasetsArray of integersrequired

List of dataset IDs to use in this experiment

training_datasetsArray of integers

List of training dataset IDs (optional)

configany
languagestring
  • en - English
  • fr - French
  • de - German
  • zh-hans - Chinese (simplified)
  • es - Spanish
  • hi - Hindi
  • pl - Polish
  • vi - Vietnamese
Enum"en""fr""de""zh-hans""es""hi""pl""vi"
curl -i -X POST \
  'https://api.mabyduck.com/projects/{project_id}/experiments/' \
  -H 'Authorization: Api-Key <YOUR_TOKEN_HERE>' \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "string",
    "version": 0,
    "datasets": [
      0
    ],
    "training_datasets": [
      0
    ],
    "config": null,
    "language": "en"
  }'

Responses

Bodyapplication/json
idstringrequired
namestring<= 128 charactersrequired

A description identifying the experiment.

typestringrequired
languagestring
  • en - English
  • fr - French
  • de - German
  • zh-hans - Chinese (simplified)
  • es - Spanish
  • hi - Hindi
  • pl - Polish
  • vi - Vietnamese
Enum"en""fr""de""zh-hans""es""hi""pl""vi"
dataset_idsArray of stringsread-onlyrequired
job_idsArray of stringsread-onlyrequired
training_dataset_idsArray of stringsread-onlyrequired
date_createdstring(date-time)read-onlyrequired
date_updatedstring(date-time)(Last updated)read-onlyrequired
configany
Response
application/json
{
  "id": "string",
  "name": "string",
  "type": "string",
  "language": "en",
  "dataset_ids": [
    "string"
  ],
  "job_ids": [
    "string"
  ],
  "training_dataset_ids": [
    "string"
  ],
  "date_created": "2019-08-24T14:15:22Z",
  "date_updated": "2019-08-24T14:15:22Z",
  "config": null
}

Retrieve an experiment

Request

Returns detailed information for a single experiment identified by experiment_id.

Security
ApiKeyAuth
Path
experiment_idstring^[a-zA-Z0-9]+$required
project_idstring^[a-zA-Z0-9]+$required
curl -i -X GET \
  'https://api.mabyduck.com/projects/{project_id}/experiments/{experiment_id}/' \
  -H 'Authorization: Api-Key <YOUR_TOKEN_HERE>'

Responses

Bodyapplication/json
idstringrequired
namestring<= 128 charactersrequired

A description identifying the experiment.

typestringrequired
languagestring
  • en - English
  • fr - French
  • de - German
  • zh-hans - Chinese (simplified)
  • es - Spanish
  • hi - Hindi
  • pl - Polish
  • vi - Vietnamese
Enum"en""fr""de""zh-hans""es""hi""pl""vi"
datasetsArray of objects(Dataset)read-onlyrequired
datasets[].​idstringrequired
datasets[].​namestring<= 64 charactersrequired

A short name identifying the dataset.

datasets[].​date_createdstring(date-time)read-onlyrequired
datasets[].​date_last_usedstring(date-time)(Last used)read-onlyrequired
datasets[].​is_deletedboolean

The files in this dataset have been deleted.

datasets[].​statusstring
  • draft - Draft
  • processing - Processing
  • ready - Ready
  • error - Error
Enum"draft""processing""ready""error"
jobsArray of objects(Job)read-onlyrequired
jobs[].​idstringrequired
jobs[].​experiment_idstringread-onlyrequired
jobs[].​num_sessionsinteger(Number of sessions)[ 0 .. 10000 ]required

How often the task is performed.

jobs[].​date_createdstring(date-time)read-onlyrequired
jobs[].​date_updatedstring(date-time)(Last updated)read-onlyrequired
jobs[].​date_launchedstring or null(date-time)
jobs[].​poolintegerread-onlyrequired
jobs[].​statusstringread-onlyrequired
jobs[].​configany

Configures stimulus selection strategy.

training_datasetsArray of objects(Dataset)read-onlyrequired
training_datasets[].​idstringrequired
training_datasets[].​namestring<= 64 charactersrequired

A short name identifying the dataset.

training_datasets[].​date_createdstring(date-time)read-onlyrequired
training_datasets[].​date_last_usedstring(date-time)(Last used)read-onlyrequired
training_datasets[].​is_deletedboolean

The files in this dataset have been deleted.

training_datasets[].​statusstring
  • draft - Draft
  • processing - Processing
  • ready - Ready
  • error - Error
Enum"draft""processing""ready""error"
date_createdstring(date-time)read-onlyrequired
date_updatedstring(date-time)(Last updated)read-onlyrequired
configany
Response
application/json
{
  "id": "string",
  "name": "string",
  "type": "string",
  "language": "en",
  "datasets": [
    { … }
  ],
  "jobs": [
    { … }
  ],
  "training_datasets": [
    { … }
  ],
  "date_created": "2019-08-24T14:15:22Z",
  "date_updated": "2019-08-24T14:15:22Z",
  "config": null
}

List all jobs

Request

Returns a list of jobs belonging to the given experiment.

Security
ApiKeyAuth
Path
experiment_idstring^[a-zA-Z0-9]+$required
project_idstring^[a-zA-Z0-9]+$required
curl -i -X GET \
  'https://api.mabyduck.com/projects/{project_id}/experiments/{experiment_id}/jobs/' \
  -H 'Authorization: Api-Key <YOUR_TOKEN_HERE>'

Responses

Bodyapplication/jsonArray [
idstringrequired
experiment_idstringread-onlyrequired
num_sessionsinteger(Number of sessions)[ 0 .. 10000 ]required

How often the task is performed.

date_createdstring(date-time)read-onlyrequired
date_updatedstring(date-time)(Last updated)read-onlyrequired
date_launchedstring or null(date-time)
poolintegerread-onlyrequired
statusstringread-onlyrequired
configany

Configures stimulus selection strategy.

]
Response
application/json
[
  {
    "id": "string",
    "experiment_id": "string",
    "num_sessions": 10000,
    "date_created": "2019-08-24T14:15:22Z",
    "date_updated": "2019-08-24T14:15:22Z",
    "date_launched": "2019-08-24T14:15:22Z",
    "pool": 0,
    "status": "string",
    "config": null
  }
]

Create a job

Request

Creates a new draft job with the specified configuration.

Security
ApiKeyAuth
Path
experiment_idstring^[a-zA-Z0-9]+$required
project_idstring^[a-zA-Z0-9]+$required
Body
required
rater_poolintegerrequired

ID of the rater pool to use for this job

num_sessionsinteger>= 1required

Number of sessions (times the task is performed)

num_comparisonsinteger>= 1required

Number of comparisons per session

num_traininginteger>= 0

Number of training comparisons

Default 2
max_repetitionsinteger(Maximum number of repetitions)[ 0 .. 2147483647 ]

How often an individual rater is allowed to repeat the task.

min_rest_timeinteger(Minimum rest time)[ 0 .. 2147483647 ]

How many minutes should pass before a rater can repeat the task.

strategystring
  • randomized - randomized
  • active - active
  • neighbour - neighbour
Default "randomized"
Enum"randomized""active""neighbour"
notestring or null

An optional note for this job. Not visible to raters.

curl -i -X POST \
  'https://api.mabyduck.com/projects/{project_id}/experiments/{experiment_id}/jobs/' \
  -H 'Authorization: Api-Key <YOUR_TOKEN_HERE>' \
  -H 'Content-Type: application/json' \
  -d '{
    "rater_pool": 0,
    "num_sessions": 1,
    "num_comparisons": 1,
    "num_training": 2,
    "max_repetitions": 2147483647,
    "min_rest_time": 2147483647,
    "strategy": "randomized",
    "note": "string"
  }'

Responses

Bodyapplication/json
idstringrequired
experiment_idstringread-onlyrequired
num_sessionsinteger(Number of sessions)[ 0 .. 10000 ]required

How often the task is performed.

date_createdstring(date-time)read-onlyrequired
date_updatedstring(date-time)(Last updated)read-onlyrequired
date_launchedstring or null(date-time)
poolintegerread-onlyrequired
statusstringread-onlyrequired
configany

Configures stimulus selection strategy.

Response
application/json
{
  "id": "string",
  "experiment_id": "string",
  "num_sessions": 10000,
  "date_created": "2019-08-24T14:15:22Z",
  "date_updated": "2019-08-24T14:15:22Z",
  "date_launched": "2019-08-24T14:15:22Z",
  "pool": 0,
  "status": "string",
  "config": null
}

List all raters

Request

Returns a list of raters of a given experiment.

Security
ApiKeyAuth
Path
experiment_idstring^[a-zA-Z0-9]+$required
project_idstring^[a-zA-Z0-9]+$required
curl -i -X GET \
  'https://api.mabyduck.com/projects/{project_id}/experiments/{experiment_id}/raters/' \
  -H 'Authorization: Api-Key <YOUR_TOKEN_HERE>'

Responses

Bodyapplication/jsonArray [
idintegerread-onlyrequired
userobject(User)required
user.​idstringrequired
user.​namestring(Full name)<= 256 charactersrequired
metaany
]
Response
application/json
[
  {
    "id": 0,
    "user": { … },
    "meta": null
  }
]

List all ratings

Request

Returns a list of ratings belonging to the given experiment.

Security
ApiKeyAuth
Path
experiment_idstring^[a-zA-Z0-9]+$required
project_idstring^[a-zA-Z0-9]+$required
Query
trainingstring

Whether to include training data.

Enum"false""true"
curl -i -X GET \
  'https://api.mabyduck.com/projects/{project_id}/experiments/{experiment_id}/ratings/?training=false' \
  -H 'Authorization: Api-Key <YOUR_TOKEN_HERE>'

Responses

Bodyapplication/jsonArray [
scorenumber or null(double)
rater_idintegerrequired
slate_idintegerread-onlyrequired
session_idstringrequired
job_idstringrequired
stimulus_idintegerread-onlyrequired
stimulusstringrequired
dimensionstring or null<= 32 characters
is_trainingbooleanrequired
metaany

For example, expected score range.

is_correctboolean or nullread-onlyrequired

Checks if score is within expectation.

First checks for explicit correctness in meta['isCorrect']. Falls back to comparing score within bounds given in meta['expectedScores'] (inclusive).

Returns: True if score is correct, False if incorrect, or None if neither.

]
Response
application/json
[
  {
    "score": 0,
    "rater_id": 0,
    "slate_id": 0,
    "session_id": "string",
    "job_id": "string",
    "stimulus_id": 0,
    "stimulus": "string",
    "dimension": "string",
    "is_training": true,
    "meta": null,
    "is_correct": true
  }
]

List all results

Request

Returns a list of metrics derived from the results of an experiment.

Security
ApiKeyAuth
Path
experiment_idstring^[a-zA-Z0-9]+$required
project_idstring^[a-zA-Z0-9]+$required
Query
metricstring

Filter by this type of metric.

Enum"bayesian_elo""elo""mean""stats""win_loss"
sourcestring

An optional comma-separated list of job or experiment IDs.

curl -i -X GET \
  'https://api.mabyduck.com/projects/{project_id}/experiments/{experiment_id}/results/?metric=bayesian_elo&source=string' \
  -H 'Authorization: Api-Key <YOUR_TOKEN_HERE>'

Responses

Bodyapplication/jsonArray [
kindstring(KindEnum)required
  • bayesian_elo - Elo
  • elo - Elo (vanilla)
  • mean - MOS
  • win_loss - Win/loss
  • stats - Statistics
Enum"bayesian_elo""elo""mean""win_loss""stats"
dimensionstring or null<= 32 characters

Used by experiments evaluating multiple criteria simultaneously.

sourcestring or nullread-onlyrequired

Return the source of the metric (job or experiment).

scoresArray of objectsread-onlyrequired

Return the scores for each method.

optionsobjectread-onlyrequired

Front-end options.

options.​property name*anyadditional property
]
Response
application/json
[
  {
    "kind": "bayesian_elo",
    "dimension": "string",
    "source": "string",
    "scores": [ … ],
    "options": { … }
  }
]

List all sessions

Request

Returns a list of sessions belonging to the given experiment.

Security
ApiKeyAuth
Path
experiment_idstring^[a-zA-Z0-9]+$required
project_idstring^[a-zA-Z0-9]+$required
Query
trainingstring

Whether to include training data.

Enum"false""true"
curl -i -X GET \
  'https://api.mabyduck.com/projects/{project_id}/experiments/{experiment_id}/sessions/?training=false' \
  -H 'Authorization: Api-Key <YOUR_TOKEN_HERE>'

Responses

Bodyapplication/jsonArray [
idstringrequired
job_idstringread-onlyrequired
experiment_idstringread-onlyrequired
date_createdstring(date-time)read-onlyrequired
date_updatedstring(date-time)(Last updated)read-onlyrequired
date_completedstring or null(date-time)(Completion date)
metaany
raterobject(Rater)required
rater.​idintegerread-onlyrequired
rater.​userobject(User)required
rater.​user.​idstringrequired
rater.​user.​namestring(Full name)<= 256 charactersrequired
rater.​metaany
]
Response
application/json
[
  {
    "id": "string",
    "job_id": "string",
    "experiment_id": "string",
    "date_created": "2019-08-24T14:15:22Z",
    "date_updated": "2019-08-24T14:15:22Z",
    "date_completed": "2019-08-24T14:15:22Z",
    "meta": null,
    "rater": { … }
  }
]

List all slates

Request

Returns a list of slates and ratings belonging to the given experiment.

Security
ApiKeyAuth
Path
experiment_idstring^[a-zA-Z0-9]+$required
project_idstring^[a-zA-Z0-9]+$required
Query
trainingstring

Whether to include training data.

Enum"false""true"
curl -i -X GET \
  'https://api.mabyduck.com/projects/{project_id}/experiments/{experiment_id}/slates/?training=false' \
  -H 'Authorization: Api-Key <YOUR_TOKEN_HERE>'

Responses

Bodyapplication/jsonArray [
idintegerread-onlyrequired
rater_idstringread-onlyrequired
session_idstringread-onlyrequired
job_idstringread-onlyrequired
ratingsArray of objects(Rating)read-onlyrequired
ratings[].​scorenumber or null(double)
ratings[].​stimulus_idintegerread-onlyrequired
ratings[].​stimulusstringrequired
ratings[].​conditionstringrequired
ratings[].​dimensionstring or null<= 32 characters
ratings[].​metaany

For example, expected score range.

ratings[].​is_correctboolean or nullread-onlyrequired

Checks if score is within expectation.

First checks for explicit correctness in meta['isCorrect']. Falls back to comparing score within bounds given in meta['expectedScores'] (inclusive).

Returns: True if score is correct, False if incorrect, or None if neither.

date_updatedstring(date-time)(Last updated)read-onlyrequired
metaany or null

For example, the time it took to complete the slate.

One of:

For example, the time it took to complete the slate.

any
is_trainingbooleanrequired
]
Response
application/json
[
  {
    "id": 0,
    "rater_id": "string",
    "session_id": "string",
    "job_id": "string",
    "ratings": [ … ],
    "date_updated": "2019-08-24T14:15:22Z",
    "meta": {},
    "is_training": true
  }
]

Rubrics

Operations related to rubrics and leaderboards.

Operations

Metrics

Operations