Skip to content
Experiments
/
/
Introduction
Last updated

Introduction

Embedded experiments allow to evaluate custom content generated by JavaScript code. Instead of uploading static media files, you upload JavaScript files that render content dynamically during the experiment. This is ideal for evaluating, for instance, interactive experiences where the content depends on user input.

How it works

When a rater participates in your experiment, your JavaScript code runs inside a secure sandbox. Your code can:

  1. Render arbitrary HTML content inside an iframe.
  2. Make network requests to external APIs (with your secrets securely injected).
  3. Signal when the content is ready for rating.

The rater then evaluates your content using a rating interface which depends on the type of the embedded experiment.

Dataset structure

Embedded experiments use JavaScript files organized like other datasets:

├──source_1/
│   ├──anam.js
│   ├──tavus.js
│   ├──heygen.js
│   └──config.json  (optional)
├──source_2/
│   ├──anam.js
│   ├──tavus.js
│   ├──heygen.js
├──source_3/
│   ├──...

Each .js file represents a different condition. For example, different AI models or different prompts.

Writing your experiment javascript code

Your JavaScript runs inside an iframe with a <div id="content"> container where you render your content.

Basic example

const container = document.getElementById("content");

// Create your content
const message = document.createElement("h1");
message.textContent = "Hello from my experiment!";
message.style.textAlign = "center";

container.appendChild(message);
Simple embedded stimulus

Example with an image

const container = document.getElementById("content");

const img = document.createElement("img");
img.src = "https://images.unsplash.com/photo-1585533530535-2f4236949d08";
img.style.maxHeight = "100%";

container.appendChild(img);
Simple embedded stimulus with an image

Lifecycle callback functions

Your JavaScript has access to three functions to communicate the state of your content to the experiment system.

notifyStimulusComplete()

Call this when your content is fully loaded and ready for the rater to evaluate.

const img = document.createElement("img");

img.onload = () => {
  // Image has loaded, content is ready for rating
  notifyStimulusComplete();
};

img.src = "https://example.com/generated-image.png";

container.appendChild(img);

By default, notifyStimulusComplete() is called automatically when the end of your script is reached. You only need to call it manually if your content loads asynchronously (e.g., fetching from an API or loading images).

notifyStimulusStart()

Call this when your content begins presenting. This is useful when there's a delay before your main content appears (e.g., loading spinners, API calls).

// Show loading state
container.innerHTML = "<p>Generating content...</p>";

// Fetch content from API
const response = await fetch("https://api.example.com/generate");
const data = await response.json();

// Content is now ready to display
notifyStimulusStart();

// Render the actual content
container.innerHTML = `<img src="${data.image_url}">`;

By default, notifyStimulusStart() is called automatically when your script begins executing. You only need to call it manually if you want to delay the "start" signal until your content is actually visible.

notifyStimulusFailed(message)

Call this when something goes wrong and your content cannot be displayed. The experiment will handle the error gracefully.

try {
  const response = await fetch("https://api.example.com/generate");

  if (!response.ok) {
    throw new Error(`API returned ${response.status}`);
  }

  const data = await response.json();
  // ... render content ...
} catch (error) {
  // Report the error to the experiment system
  notifyStimulusFailed(error.message);
}

Using API keys and dataset secrets

If your JavaScript needs to call external APIs that require authentication, you can use dataset secrets to do so securely. Dataset secrets are set during dataset creation and are injected into requests automatically.

For example, if you wanted to make a request to an Open AI API inside the embedded JS code, but you wanted to hide your API key, you could do the following:

const response = await fetch("https://api.openai.com/v1/images/generations", {
  method: "POST",
  headers: {
    Authorization: "Bearer #{SECRETS.OPENAI_KEY}",
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    prompt: "A mountain landscape",
    n: 1,
  }),
});

The #{SECRETS.OPENAI_KEY} placeholder is automatically replaced with your actual API key when the request is made.

Requests support

Secrets are supported in urls and headers of requests made with fetch, XHR and WebSocket.

Previous page
Next page