Mabyduck supports direct evaluation of APIs through API request stimuli. These are .req.json files that stand in for media files in your datasets. API request datasets are organized like other datasets. For instance:
├──providers.json (optional, but recommended)
├──source_1/
│ ├──elevenlabs.req.json
│ └──cartesia.req.json
├──source_2/
│ ├──elevenlabs.req.json
│ └──cartesia.req.json
├──source_3/
│ ├──...API request stimuli allow your experiments to fetch content dynamically from external APIs instead of using uploaded media files. Each stimulus is defined as a request configuration that the platform executes on-demand.
When a rater views an API request stimulus:
- The platform reads your request configuration (
.req.jsonfile). - Dataset secrets are securely injected into the request.
- The platform makes the API call on your behalf.
- The response is processed and streamed to the rater.
Your API credentials never leave our servers and are never exposed to raters.
A request file defines everything needed to call an external API:
{
"url": "https://api.openai.com/v1/images/generations",
"method": "POST",
"headers": {
"Authorization": "Bearer #{SECRETS.OPENAI_KEY}",
"Content-Type": "application/json"
},
"body": {
"model": "dall-e-3",
"prompt": "A futuristic city with neon lights",
"n": 1,
"size": "1024x1024"
},
"response_format": "json_url",
"response_extraction_path": "data[0].url"
}| Field | Description |
|---|---|
url | The API endpoint URL |
response_format | How to process the API response (see Response formats) |
| Field | Description |
|---|---|
method | HTTP method (GET, POST, etc.). Defaults to GET |
headers | Request headers as key-value pairs |
body | JSON body to send with the request |
response_extraction_path | JMESPath expression to extract data from JSON responses |
When you have multiple stimuli calling the same API, you can define shared configurations in a providers.json file at the root of your dataset. This avoids repetition and makes your dataset easier to maintain.
{
"dalle_3": {
"url": "https://api.openai.com/v1/images/generations",
"method": "POST",
"headers": {
"Authorization": "Bearer #{SECRETS.OPENAI_KEY}",
"Content-Type": "application/json"
},
"body": {
"model": "dall-e-3",
"n": 1,
"size": "1024x1024"
},
"response_format": "json_url",
"response_extraction_path": "data[0].url"
},
"stable_diffusion": {
"url": "https://api.stability.ai/v1/generation/stable-diffusion-xl/text-to-image",
"method": "POST",
"headers": {
"Authorization": "Bearer #{SECRETS.STABILITY_KEY}",
"Content-Type": "application/json"
},
"response_format": "json_base64",
"response_extraction_path": "artifacts[0].base64"
}
}Once you have defined providers, your .req.json files become much simpler:
{
"provider": "dalle_3",
"body": {
"prompt": "A futuristic city with neon lights",
"quality": "hd"
}
}The provider field tells the platform to start with the dalle_3 configuration and merge in any overrides you specify. When a request file references a provider:
- Objects are merged recursively.
- Lists and primitives are replaced.
- Setting a value to
nullremoves it.
For example, if your provider sets "size": "1024x1024" but you want a different size:
{
"provider": "dalle_3",
"body": {
"prompt": "A mountain landscape",
"size": "512x512"
}
}The response_format field tells the platform how to process the API response:
| Format | Description | Use when |
|---|---|---|
binary | Stream response bytes directly | API returns raw image/audio/video data |
json_url | Extract a URL from JSON, then fetch that URL | API returns a JSON with a hosted file URL |
json_base64 | Extract a Base64 string from JSON, decode it | API returns Base64-encoded file data in JSON |
redirect_url | Follow redirects and stream the final response | API redirects to the file location |
multipart | Handle multipart responses | API returns mixed content types |
For json_url and json_base64 formats, use response_extraction_path to specify where the data is located in the JSON response. This uses JMESPath syntax.
Example responses and extraction paths:
// Response: {"data": [{"url": "https://..."}]}
// Path: "data[0].url"
// Response: {"artifacts": [{"base64": "iVBORw0..."}]}
// Path: "artifacts[0].base64"
// Response: {"result": {"image": {"url": "https://..."}}}
// Path: "result.image.url"API keys and other credentials should be stored as dataset secrets, not in your configuration files. Reference them using the placeholder syntax:
#{SECRETS.SECRET_NAME}Example:
{
"headers": {
"Authorization": "Bearer #{SECRETS.OPENAI_KEY}",
"X-Api-Key": "#{SECRETS.CUSTOM_API_KEY}"
}
}The placeholders are replaced with your actual secrets when the request is made. Your credentials are never exposed to raters or visible in browser developer tools.
Here's a complete example dataset for comparing two image generation models:
providers.json
{
"dalle": {
"url": "https://api.openai.com/v1/images/generations",
"method": "POST",
"headers": {
"Authorization": "Bearer #{SECRETS.OPENAI_KEY}",
"Content-Type": "application/json"
},
"body": {
"model": "dall-e-3",
"n": 1,
"size": "1024x1024"
},
"response_format": "json_url",
"response_extraction_path": "data[0].url"
},
"midjourney": {
"url": "https://api.example.com/midjourney/generate",
"method": "POST",
"headers": {
"Authorization": "#{SECRETS.MJ_API_KEY}",
"Content-Type": "application/json"
},
"response_format": "json_url",
"response_extraction_path": "imageUrl"
}
}sunset/dalle.req.json
{
"provider": "dalle",
"body": {
"prompt": "A beautiful sunset over the ocean, photorealistic"
}
}sunset/midjourney.req.json
{
"provider": "midjourney",
"body": {
"prompt": "A beautiful sunset over the ocean, photorealistic"
}
}mountain/dalle.req.json
{
"provider": "dalle",
"body": {
"prompt": "Snow-capped mountains at dawn, cinematic lighting"
}
}mountain/midjourney.req.json
{
"provider": "midjourney",
"body": {
"prompt": "Snow-capped mountains at dawn, cinematic lighting"
}
}