Embedded experiments allow to evaluate custom content generated by javascript code. Instead of uploading static media files, you upload JavaScript files that render content dynamically during the experiment.
This is ideal for evaluating:
- AI-generated content — Images, audio, or text from your models
- Interactive experiences — Content that responds to user input
- Real-time API outputs — Content fetched from external services
When a rater participates in your experiment, your JavaScript code runs inside a secure sandbox. Your code can:
- Render any HTML content you need
- Make network requests to external APIs (with your secrets securely injected)
- Signal when the content is ready for rating
The rater then evaluates your content using the rating interface depending on the type of the embedded experiment.
Embedded experiments use JavaScript files organized like other datasets:
├──source_1/
│ ├──method_a.js
│ ├──method_b.js
│ └──config.json (optional)
├──source_2/
│ ├──method_a.js
│ ├──method_b.js
├──source_3/
│ ├──...Each .js file represents a different stimulus.
Your JavaScript runs inside a page with a <div id="content"> container where you render your content.
const container = document.getElementById("content");
// Create your content
const message = document.createElement("h1");
message.textContent = "Hello from my experiment!";
message.style.textAlign = "center";
container.appendChild(message);
const container = document.getElementById("content");
const img = document.createElement("img");
img.src = "https://images.unsplash.com/photo-1585533530535-2f4236949d08";
img.style.maxHeight = "100%";
container.appendChild(img);
Your JavaScript has access to three functions to communicate the state of your content to the experiment system.
Currently we ignore notify stimulus experiment lifecycle callback functions as we just evaluate your JavaScript code and the raters are able to evaluate once the code is rendered on the page.
Call this when your content is fully loaded and ready for the rater to evaluate. This is the most important function — it tells the system that your stimulus presentation is complete.
const img = document.createElement("img");
img.src = "https://example.com/generated-image.png";
img.onload = () => {
// Image has loaded, content is ready for rating
notifyStimulusComplete();
};
container.appendChild(img);By default,
notifyStimulusComplete()is called automatically when your script finishes executing. You only need to call it manually if your content loads asynchronously (e.g., fetching from an API or loading images).
Call this when your content begins presenting. This is useful when there's a delay before your main content appears (e.g., loading spinners, API calls).
// Show loading state
container.innerHTML = "<p>Generating content...</p>";
// Fetch content from API
const response = await fetch("https://api.example.com/generate");
const data = await response.json();
// Content is now ready to display
notifyStimulusStart();
// Render the actual content
container.innerHTML = `<img src="${data.image_url}">`;By default,
notifyStimulusStart()is called automatically when your script begins executing. You only need to call it manually if you want to delay the "started" signal until your content is actually visible.
Call this when something goes wrong and your content cannot be displayed. The experiment will handle the error gracefully.
try {
const response = await fetch("https://api.example.com/generate");
if (!response.ok) {
throw new Error(`API returned ${response.status}`);
}
const data = await response.json();
// ... render content ...
} catch (error) {
// Report the error to the experiment system
notifyStimulusFailed(error.message);
}If your JavaScript needs to call external APIs that require authentication, you can store your API keys securely with the dataset and use placeholders as dummy values that we later use to inject the real values. Dataset secrets can be set during dataset upload (see the Secrets section) or after uploading the dataset via the Edit menu. Your secrets are never exposed to raters — they are injected server-side when requests are made.
We use the following syntax to reference dataset secrets inside embedded stimuli on datasets -#{SECRETS.<secret name>}. For example, if you wanted to make a request to an Open AI API inside the embedded JS code, but you wanted to hide away your API key, you could do the following:
Use the placeholder syntax in your URLs or headers:
const response = await fetch("https://api.openai.com/v1/images/generations", {
method: "POST",
headers: {
Authorization: "Bearer #{SECRETS.OPENAI_KEY}",
"Content-Type": "application/json",
},
body: JSON.stringify({
prompt: "A mountain landscape",
n: 1,
}),
});The #{SECRETS.OPENAI_KEY} placeholder is automatically replaced with your actual API key when the request is made.
We currently support any requests made with fetch, XHR and even WebSocket.
It is very important to note that we support dataset secrets placeholders ONLY in the urls of requests and also in request headers, but NOT inside request bodies.
For details on adding secrets to your dataset, see Dataset Secrets.
- Always handle errors — Use
notifyStimulusFailed()to report problems gracefully - Show loading states — Give raters feedback while content is being generated
- Test locally first — Develop and test your JavaScript before uploading
- Test with a demo experiment/job second - After uploading the dataset and configuring an experiment, make sure to test it with a demo job/experiment second.
- Keep secrets secure — Never hardcode API keys; use the
#{SECRETS.*}placeholder syntax