Strategies

Sampling strategies control which stimuli to evaluate next. Our experiments support randomized, lexicographic, uniform, exhaustive, and active sampling strategies. These can be configured when creating a job, where the options offered depend on the type of experiment:

Active sampling strategies try to select sets of stimuli whose evaluation will be most informative. This can significantly reduce the number of comparisons needed for robust results. Several papers have explored active selection strategies in the context of pairwise comparisons [e.g., 1, 2, 3].

Active strategies are only offered for pairwise audio, image, and video experiments, which compare exactly two stimuli per slate. They are also unavailable when an experiment tracks no Elo metrics (for example, when its datasets contain too many conditions), or when it is configured with named rating dimensions.

Datasets on Mabyduck are a collection of folders corresponding to different sources or contexts, and media files within a folder correspond to different conditions applied to the same source. Our strategies decide how to sample folders as well as how to sample conditions within folders.

Each job maintains its own state for the purpose of active selection. That is, when creating multiple jobs for one experiment, each job will sample stimuli independently.

Randomized

Our randomized sampling strategy randomizes the order in which folders and conditions are selected, but otherwise tries to sample stimuli as evenly as possible. For each session, folders are shuffled once and then sampled deterministically in round-robin fashion. Conditions are sampled at random without replacement from a list of conditions when at least twice as many conditions are available as there are stimuli per slate, and there are at least four conditions (this last requirement is dropped for sessions of fewer than ten comparisons). When not enough conditions remain to fill a slate, a new list is formed and used to fill the slate. This approach ensures that each condition is sampled with equal frequency while not creating a predictable order. Otherwise, conditions are sampled with replacement.

Lexicographic

This strategy samples folders in lexicographic order. This gives you precise control over the order in which stimuli are presented. Within each folder, conditions are sampled randomly following the same strategy as for the randomized strategy.

If you also wish to control the conditions that are presented, you can do so by limiting the number of conditions per folder to the number of stimuli per slate. For example, for a single-stimulus ACR experiment, each folder would have a single media file.

Uniform

This strategy samples each stimulus evenly across all sessions of a job, rather than sampling conditions evenly within each session. The order of the stimuli is shuffled once per job, so that the stimuli a session receives stay the same whenever the order is recomputed.

The uniform strategy only applies to experiments showing a single stimulus per slate, and it is the default strategy for those experiment types, such as ACR, rubric, survey and inpainting experiments. It does not support required conditions, nor references that are included with a probability, which is why it is not offered for the Real or not? experiment types.

Exhaustive

Sometimes you want every possible comparison rather than a sample. The exhaustive strategy rates every pair of conditions within a folder, and does so for every folder. Pairs are spread evenly across the job's sessions in a pseudorandom order that is fixed for the job, so a partially completed job still gives you an unbiased view of the whole dataset. Enough sessions to cover all pairs will rate each pair exactly once; any further sessions start the sequence over, adding repeat measurements just as evenly.

Like the active strategies, this strategy is offered for pairwise audio, image, and video experiments, which compare exactly two stimuli per slate.

Maximize information gain

Let $S_A$ represent the performance of condition $A$, and $S_B$ represent the performance of another condition $B$. $S_A$ and $S_B$ represent the true underlying performance of the methods, which is unknown. That is, they are real-valued random variables.

Typically, the goal of an experiment is to learn the extent to which $S_A > S_B$ is true. There may be many conditions in our experiment. For some pairs, we may already have a good idea about their relative performance, while for others we are still uncertain. We want to focus our data collection efforts on pairs where additional data will be the most impactful.

Let $Y_{AB} \in \{0, 1\}$ denote the outcome of a pairwise comparison between conditions $A$ and $B$. That is, $Y_{AB}$ represents the decision by a rater. Further, let $\mathcal{D}$ be the data already collected. The information maximization strategy tries to select conditions $A$ and $B$ so as to maximize the following mutual information:

$$I[S_A > S_B; Y_{AB} \mid \mathcal{D}] = H[S_A > S_B | \mathcal{D}] - H[S_A > S_B | \mathcal{D}, Y_{AB}]$$

We use mean-field variational inference to approximate the posterior distribution over scores $S_A$ and $S_B$ using Gamma distributions. This makes it fairly straightforward to compute or estimate the entropies on the right-hand side.

As a guardrail and for practical reasons, we do not only consider the single most informative pair. Instead, we construct a list of pairs by considering every condition, and pairing it with the most informative other condition. We then randomly sample from this list of pairs.

Neighbors

This simple strategy orders conditions based on their current Elo scores, then lists all possible pairs of conditions with neighboring scores. Stimulus pairs are then sampled from this list uniformly at random.

Top-K

Often you only care about identifying the best few conditions rather than ranking every condition precisely. The top-K strategy focuses on efficiently identifying and ranking only the top $K$ conditions. You choose $K$ when creating the job; it defaults to $1$, which focuses the experiment on finding the single best condition.

---

References

[1] Chen et al. (2013). Pairwise ranking aggregation in a crowdsourced setting.
[2] Maystre and Grossglauser (2017). Just Sort It! A Simple and Effective Approach to Active Preference Learning.
[3] Mikhailiuk et al. (2020). ASAP: Active sampling for pairwise comparisons via approximate message passing and information gain maximization.