Guide 03
Introduction
A benchmark is not just a list of questions with an answer key. It is an argument about performance: this system, under these conditions, can do this kind of work to this standard. Designing a credible benchmark requires choosing an appropriate task population, constructing valid examples, preventing information leakage, selecting defensible scoring rules, and quantifying uncertainty.
This playbook focuses on benchmarks used to compare language-model systems, evaluation pipelines, and agents. The same principles apply to internal release holdouts as well as public leaderboards. It complements Guide 01, which covers operational evals, and Guide 05, which defines key metrics.
- This guide explains how to produce
- a benchmark specification, curate a defensible test dataset, audit leakage and contamination risks, evaluate differences statistically, and publish a reproducible scorecard.
Chapter one
Scoping what the benchmark claims
Every score implies a claim
Suppose a new assistant scores 84% on “financial reasoning.” What does the number mean? It could reflect arithmetic with clean tables, extracting information from annual reports, answering regulatory questions, or navigating a live transaction workflow. Without a target task population and protocol, “84%” conveys little more than a ranking inside one measurement setup.
Begin with a claim sentence: “For English-language retail customer-support tasks sampled from a defined ticket population in the last quarter, system X completes authorized workflows without policy violations at rate Y, under the specified tool budget and environment.” Even if a public benchmark cannot sample a real production population, explain which behaviors it covers and which it excludes.
A benchmark’s construct is the ability or behavior it intends to measure. Construct underrepresentation occurs when tasks cover only a narrow slice of that ability. Construct-irrelevant variance occurs when scores move because of unrelated factors: answer formatting, a specific prompt template, hidden metadata, a brittle test harness, or model-specific compatibility. Both make a leaderboard misleading.
Define the unit, task and permitted environment
Document the evaluation unit: individual question, conversation, code repository, user journey, or environment state. Specify inputs, permitted tools, time or token budgets, retrieval access, external network access, concurrency, tool errors, and any human intervention. A coding task executed with unrestricted internet access is not comparable to a closed-book task; a support agent’s success can depend on tool state rather than language skill alone.
For agent benchmarks, distinguish trajectory from outcome. A system can take an unexpected but valid route to the desired result. Grade final state when outcome matters; separately grade mandatory process constraints when a route itself is required for safety or policy.
The ladder of defensible claims
Benchmarks support claims of different strength. A closed-set result such as “87 of 100 cases passed” describes exactly what occurred in one run. A carefully sampled, graded and repeated benchmark may support an estimate of performance on a defined task population. A claim about broader competence requires additional evidence that the evaluated tasks represent the relevant settings. A claim about real-world reliability requires evidence from deployed workflows, including tools, users and operational conditions.
The stronger claim is not automatically the more useful one. A narrow benchmark for authorised address changes may provide excellent release evidence for an address-change assistant. A broad academic benchmark may be better for diagnosing general model differences but worse for deciding whether that feature can ship. Before presenting a score, ask which rung of the claim ladder the evidence supports and stop there.
This is the rationale behind Stanford’s HELM framework, which reports performance across scenarios and metrics rather than presenting one context-free measure as definitive. A transparent limitation is a mark of methodological care, not a weakness to hide.
Select primary and secondary outcomes
Pick a primary metric corresponding to the claimed capability, functional pass rate for code, task success for agents, factual-support rate for retrieval QA, or human preference under a well-defined protocol. Report guardrails and trade-offs: harmful actions, latency, cost, abstention, environmental failures, and coverage across subgroups. Avoid giving a single “intelligence score” when the benchmark actually contains heterogeneous, weakly related tasks.
A good scoping table looks like this:
| Item | Required answer |
|---|---|
| Population | Where did tasks come from? What is not represented? |
| Construct | What behavior does passing actually demonstrate? |
| Unit | What counts as one independent example? |
| Conditions | Prompts, tools, retrieval, budget, retries, state |
| Grading | Exact checks, experts, model judges; how validated? |
| Primary claim | What comparison or threshold is predeclared? |
| Limitations | Where does the claim not transfer? |
Decide whether ranking is even the right goal
A benchmark built to diagnose failures may intentionally over-sample hard cases and should not claim population-level accuracy. A public benchmark meant to rank models requires consistent protocols and uncertainty estimates. A safety audit may prioritise finding failure modes rather than estimating prevalence. These are all legitimate uses, but they answer different questions.
The result should be a benchmark charter approved before task construction. The HELM framework is a useful example of treating performance as multidimensional rather than a single-context metric.
Chapter two
Dataset construction
Start with a sampling frame
If your claim concerns actual users, define the source population before sampling. For example: “All resolved support requests in a specified date range that the assistant would have been authorized to handle, excluding cases requiring unavailable private records.” Record inclusion and exclusion rules. Without a sampling frame, you cannot know whether a benchmark resembles real workloads.
For a population-representative test, use probability sampling where feasible. For risk-focused tests, stratify by scenario, difficulty, language or severity, and deliberately include underrepresented but important cases. Keep strata and sampling weights. A convenience collection of “interesting” questions is excellent for stress testing and often poor for estimating everyday success rates.
Construct tasks with authentic context
A benchmark example should include what a legitimate solver needs: inputs, effective policy or reference material, valid tool schemas and initial state, evaluation outcome, and ambiguity notes. Cases that demand knowledge unavailable to the solver measure guessing or benchmark familiarity rather than intended competence.
A compact benchmark record:
{
"item_id": "support-0218",
"source_family": "ticket-family-74",
"split": "private_holdout",
"locale": "en-GB",
"scenario": "authorised_address_change",
"input_fixture": "support-0218-input-v2",
"environment_fixture": "support-0218-state-v2",
"success_check": "address_changed_after_required_verification",
"process_guardrail": "verification_must_precede_write",
"reference_version": "policy-2026-09",
"rubric_version": "support-v2",
"difficulty": "multi_step",
"licence_and_consent": "restricted_internal",
"review_status": "expert_verified"
}This is an illustrative schema, not real case data. Store large or sensitive inputs in a controlled fixture store; the record can reference immutable snapshots by hash.
Sampling weights and challenge sets
Suppose support traffic contains 70% routine requests, 20% policy exceptions and 10% account changes. A benchmark created with equal numbers from the three groups is useful for diagnosing each category, but its simple average is not the expected result for the real traffic mix. To estimate production-mix quality, compute a weighted result using the target proportions.
If a system passes 98%, 85% and 70% of the three categories respectively, the traffic-weighted result is
That is 92.6% under the stipulated mix, not the unweighted average of 84.3%. Both numbers have a role: the former estimates common traffic under the stated assumptions, and the latter gives equal emphasis to the three categories. Neither should conceal the 70% account-change performance if those failures are consequential.
Document the source of population weights, how they change over time, and whether cases within each stratum were actually sampled representatively. A manually curated challenge set cannot become a probability sample simply by assigning plausible weights.
Audit task quality before scoring systems
Ask an expert to attempt each task using only the published inputs and permissions. If experts cannot determine a valid outcome, the benchmark is not ready. Test the grader on intentionally correct, incorrect and edge-case trajectories. Cases that require an exact but arbitrary wording, path or action sequence may penalise valid solutions. Conversely, a loose grader can reward success messages even when no action occurred.
For stochastic environments, identify all sources of randomness. Where feasible, freeze state and seeds, or use repeated trials and report variability. If tool APIs change over time, snapshot the test environment or document the dependence.
Separate cases by family, not only by row
A 1,000-row dataset may contain only 100 distinct scenarios with ten near-identical rewrites each. Split by source incident, document family, user workflow, or problem template, not by random row, so development and test sets do not share nearly identical tasks. Detect exact duplicates using normalised hashes; inspect semantic near-duplicates manually or through audited similarity tooling. Deduplication must not eliminate legitimate cases that differ in an outcome-determining detail.
Make the dataset governable
Record provenance, legal basis for source use, rights and licences, potential personal data, owner, version, review status, and change log. Consider who may access private holdouts and how to avoid re-identification of users. Publish a dataset card describing composition, construction, known biases and prohibited uses even when the items themselves remain private.
Quality gate: a benchmark dataset is ready when each item is answerable, traceable, scoped, licensed appropriately, assigned to a documented stratum, and paired with a tested grader. Quantity cannot compensate for invalid examples.
Chapter three
Leakage
Leakage happens inside the evaluation process
In this playbook, leakage means information that should be unavailable for a given test influences model selection, prompts, grading, or execution. It is a procedural failure that can occur even when the base model never trained on the test set. Contamination (Chapter 04) focuses on the model or system having encountered benchmark-specific content through pretraining, tuning, retrieval, or other exposure. The phenomena overlap, but their remedies differ.
Common leakage paths include:
- Development-to-test overlap: paraphrases or duplicates from the same incident appear in both pools.
- Adaptive holdout use: repeated candidate comparisons expose the held-out set to optimization.
- Answer exposure: the correct label or reference solution appears in a tool result, metadata field, test fixture, or agent-accessible file.
- Grader leakage: a judge sees the answer key while it is supposed to evaluate independently, or a scoring script inadvertently includes ground truth in prompts.
- Temporal leakage: features or source documents postdate the decision being evaluated.
- Human leakage: reviewers recognise benchmark items or know model identity in a supposedly blind comparison.
Concrete failure: the hidden answer in the tool schema
A benchmark asks an agent to update a user’s mailing address. The test environment supplies a tool called update_address_expected_742 where the suffix exposes the reference state. The agent may take a shortcut unrelated to real task competence. A second variant places expected_status=passed in a file visible to the candidate. The benchmark now tests exploitation of test infrastructure as much as compliance with user instructions.
Repair: conduct a candidate-side visibility audit. For each field the model can observe, ask whether a legitimate production system would expose it and whether it leaks the required answer. Keep hidden grader artefacts and expected outcomes isolated from agent sandboxes.
Establish hard boundaries
Use access-controlled private holdouts, separate environments for prompts and graders, and strict data-flow rules. Give developers failing categories and selected development examples; avoid releasing the entire private answer set after every iteration. Maintain an exposure ledger recording which benchmark versions, task IDs and labels were seen by which models, tools and teams.
For time-series or policy-sensitive tasks, freeze the source snapshot available at the task’s nominal time. If a case requires a policy that changed yesterday, specify whether the system is expected to know the old or new rule; otherwise apparent mistakes may be reference inconsistencies.
Leakage routes that are easy to miss
Leakage is not limited to accidentally pasting an answer key into a prompt. A development team may copy repeatedly failed holdout cases into its prompt examples, allowing the next candidate to overfit the supposed final test. A retrieval tool may expose labels in file names or metadata. A stateful benchmark might preserve changes from the previous attempt, enabling a later run to see outcomes that should have been hidden. A grader may inspect the reference answer while generating evidence intended for the model under test.
Controls follow the information flow. Give implementers access to development cases but restrict final holdouts. Separate fixtures intended for the solver from private grading metadata. Reset state between trials. Review all tool schemas and retrieved context for label-bearing fields. Restrict logs and issue reports that reveal full holdout answers. Record who accessed which version and when.
One useful audit is an information-boundary map: for every field in the dataset, specify whether the model may see it, whether the grader may see it, and where it may appear in logs. If the same object is passed wholesale to both solver and scorer, the boundary is already at risk. This is ordinary test-engineering discipline applied to probabilistic systems.
Test for leakage systematically
- Compare exact and near-duplicate families across splits.
- Inspect prompts, retrieval payloads and tool responses for reference labels or hidden solutions.
- Review evaluation code for accidental ground-truth access.
- Ask whether metadata, file names and directory structure reveal the answer.
- Log every use of the locked set and rotate it when repeated tuning erodes independence.
- Test “canary” cases with deliberately uninformative labels to catch impossible performance or answer-channel exposure; interpret any signal carefully.
Chapter four
Contamination
Why public benchmarks become difficult to interpret
Models can encounter benchmark questions, answers, or close variants during pretraining or subsequent tuning. Even without intentional misconduct, widely copied tasks can circulate through web pages, papers, code repositories and evaluation reports. A system may then perform well because it memorised familiar material rather than generalising to unseen tasks. This is contamination, and it challenges the external validity of benchmark claims.
Contamination is not binary. Distinguish verbatim question-and-answer exposure, near-duplicate exposure, exposure to answer explanations, prompt-template familiarity, and benign familiarity with the general domain. Knowledge of arithmetic is not contamination just because a benchmark contains arithmetic. The concern is exposure that gives an unfair or unrepresentative advantage on the particular tested items.
Detection strategies and their limits
Training-corpus matching: if the model provider has corpus access, compare benchmark strings and near-duplicates against pretraining and fine-tuning data. This can uncover exposure but may miss transformed passages. Black-box probes: look for suspicious verbatim continuation, highly specific phrasing recognition, or anomalous performance on known versus newly composed variants. These are circumstantial indicators, not reliable proof of training exposure. Performance comparisons: evaluate fresh private tasks matched in difficulty and format to old public tasks; a gap invites investigation but may reflect distribution differences rather than contamination.
The 2024 contamination survey and 2025 survey describe why detection remains challenging across white-box and black-box settings. Never state “this model memorised the benchmark” based solely on an observed score jump.
Why contamination is difficult to prove absent
A model provider may not disclose the full pretraining corpus, instruction-tuning data, synthetic-data sources or benchmark-derived evaluation feedback. Even an exact-string search can miss paraphrases and translated versions. Conversely, a model’s ability to complete a familiar question does not by itself prove it memorised the exact benchmark answer. Researchers studying pretraining-data detection, including Shi et al., demonstrate that identifying exposure is itself a statistical inference problem.
There is no universal contamination detector with a clean guarantee for all closed models. Stronger design choices include newly created private cases, restricted distribution, independently sourced variants, time-separated data collection and evidence-rich tasks requiring live state. Freshness also has limits: a recent question can still be easy for reasons unrelated to generalisation. Hidden tests improve resistance to memorisation but reduce public reproducibility, so the protocol and aggregate task distribution need especially clear documentation.
When publishing comparisons, distinguish known exposure, plausible exposure, and unassessed exposure. Do not transform the absence of evidence into evidence of absence. The reported score remains useful as a result on the specified benchmark, but the transfer claim should be appropriately narrow.
Mitigation by design
- Freshness: create newly authored or newly collected tasks; document creation dates and model availability windows.
- Private holdouts: restrict full task and answer-key access; accept that secrecy does not itself guarantee representativeness.
- Family separation: prevent overlap with evaluation development and tuning material.
- Dynamic variants: generate or sample new problems with equivalent intended difficulty, then validate them for ambiguity and grader correctness.
- Task diversity: use multiple independent sources, formats, environments and outcome checks.
- Disclosure: report known or suspected exposure, data-access constraints and residual uncertainty.
A common mistake is to paraphrase every public question automatically and declare the benchmark contamination-free. The model may still know the underlying answer or task family; paraphrasing may also change difficulty. Treat rewritten items as new tasks requiring validation, not as a cleansing operation.
A model-provider request
When comparing providers, ask for training and fine-tuning data cut-offs where available, known benchmark exclusions, evaluation-related training uses, and policies governing public benchmark exposure. Providers may lack visibility or disclose only partial information. Record these limitations in the score report rather than implying that contamination risk has been eliminated.
When contamination cannot be ruled out
Narrow the claim. “System X scored 86% on version 2.1 of Benchmark Y under our prompt and tool setup” may be justified. “System X generalises to all unseen tasks better than Y” is not supported without fresh, appropriate evaluation evidence. The strongest benchmarks treat representativeness, leakage and contamination as distinct risks and design controls for each.
Chapter five
Statistical validity
Scores are estimates, not immutable properties
Suppose a system passes 82 of 100 sampled tasks. The observed pass rate is 82%, but it is not necessarily the population pass rate. If tasks were independently and randomly sampled from a defined population, a confidence interval can convey sampling uncertainty. If tasks are a hand-picked challenge set, the same calculation does not turn the result into a representative population estimate.
For an independent binary proportion, the common rough sample-size formula at worst-case prevalence is:
With a 95% normal critical value of 1.96, p = 0.5 and desired half-width e = 0.05, this gives roughly 385 independent cases before finite-population or design adjustments. Use a Wilson or exact interval rather than a naive normal interval for small samples or extreme proportions. The formula is planning guidance, not a substitute for a power analysis or a proper sampling design.
Read the paired results, not just the headline
Consider 200 test tasks. Both systems pass 130, both fail 25, A alone passes 15, and B alone passes 30. A passes 145 tasks (72.5%); B passes 160 (80.0%). The observed gain is 7.5 percentage points. The critical evidence lies in the 45 discordant pairs: B improves 30 cases but regresses on 15.
| Matched outcome | Cases |
|---|---|
| Both pass | 130 |
| A passes, B fails | 15 |
| A fails, B passes | 30 |
| Both fail | 25 |
This table reveals trade-offs that two standalone percentages conceal. Review the 15 regressions for severity and common causes. Where binary paired outcomes and independent task units are appropriate, McNemar’s test can assess evidence against equal marginal success probabilities. A paired bootstrap can also give an interval for the percentage-point difference. If many tasks come from the same case family, resample families rather than rows.
A statistically detectable difference may still be operationally unimportant. Conversely, a small test can fail to establish a difference even when it matters in production. Report effect size, uncertainty and decision consequences together.
Compare systems with pairing
If systems A and B answer the same tasks, compare the within-task differences. This usually provides a more precise view of improvement than treating A and B’s scores as unrelated samples. For binary paired outcomes, record how often only A passes, only B passes, both pass, or neither passes. The number of discordant pairs drives much of the evidence for a change. For paired continuous scores, compute a paired difference and uncertainty interval.
For bootstrapping, resample the independent sampling units, typically tasks or source families, not individual judge calls or paraphrases that share a task. Recompute the full metric on each resample, preserving the A/B pairing. If your dataset is stratified or clustered, respect that design in resampling and weighting. State your confidence level and interval method in the report.
Watch the sources of false certainty
Multiple comparisons: selecting the best of dozens of models after repeatedly evaluating the same holdout tends to inflate the winner’s apparent advantage. Use a confirmation holdout, an explicit comparison plan, or appropriate multiplicity controls. Cluster dependence: 500 prompts based on ten documents are not 500 independent tests of document generalisation. Label error: an exact confidence interval around a wrong grader score is still misleading. Distribution mismatch: a random sample from the wrong population estimates the wrong quantity. Missing outcomes: if tool failures or parsing errors are removed rather than counted, results may be upward-biased.
A small p-value is not equivalent to a meaningful product gain; a non-significant difference is not proof of equivalence. Define the smallest operationally important gain or tolerable regression before looking at results. For a critical guardrail, consider non-inferiority or upper confidence bounds on failure rates rather than optimising a generic mean.
Reporting score changes honestly
Imagine B improves from 80% to 83% on a finite test set. Report the number of cases, matched design, difference in percentage points (+3 pp), uncertainty interval, important slices, judge/label uncertainty, and cost. Avoid saying “3% better” unless you specify whether that means absolute points or a relative change. If the interval includes practically important regressions, the evidence is not yet enough for a confident claim of improvement.
For stochastic agents, specify attempt and retry rules. pass@k (one or more successful attempts) and pass^k (all k attempts successful) address different operational questions. Aggregate these metrics across tasks carefully; raising the pooled single-trial success rate to the k-th power generally does not recover mean per-task consistency when tasks differ in difficulty.
Minimum statistics checklist
Report numerator, denominator, unit of analysis, dataset version, sampled population, task weights, missing data, grader definition, point estimate, uncertainty interval, slicing rules, and any sequential or adaptive test selection. A dashboard number without these details is not ready for scientific or business comparison.
Chapter six
Reporting and maintenance
Publish a scorecard that can be reproduced
A benchmark report should let another qualified team understand what was measured, reproduce the setup where access permits, and identify where the result might mislead. At minimum, include:
| Field | Example of what to disclose |
|---|---|
| Benchmark identity | Name, version, release date, dataset hash |
| Scope | Construct, population, exclusions, languages, modalities |
| Sampling | Source, strata, weights, sample counts, missing outcomes |
| System | Model version, prompt, tools, retrieval, decoding, budget |
| Grading | Rubric, code, judge version, human audit, adjudication |
| Results | Primary metric, uncertainty, slices, guardrails, efficiency |
| Reproducibility | Code commit, run manifest, environment, attempt policy |
| Integrity | Leakage audit, contamination risk, access/exposure history |
| Limits | Claims not supported, known task or grader defects |
Where cases cannot be public for privacy, licensing or benchmark-integrity reasons, publish the protocol, aggregate composition and access process. “Private benchmark” should not be an exemption from explaining methodology.
What a reproducibility package actually contains
A serious benchmark release has more than a CSV of final scores. Its accompanying record should describe the dataset, task-generating process, allowed tools, decoding settings, retry budget, grader rules, annotation protocol and exact system builds. Publish executable scoring code or a sufficiently precise scoring specification. Include any filtering, exclusions and postprocessing rules, particularly those applied after seeing results.
The Datasheets for Datasets proposal provides a useful template for documenting provenance and intended use. Model cards add a complementary discipline for describing evaluated systems and their limitations. Together, these practices encourage reporting that survives staff turnover or tool changes.
For restricted datasets, publish hashes, version identifiers, task-category counts and a controlled replication pathway. Explain why the items are private and what evidence independent auditors may inspect. Privacy constraints and benchmark integrity may justify withholding raw cases; they do not justify omitting the rules by which results were produced.
Use versioning deliberately
Patch version: correct typos or metadata that do not change tasks or scores; verify this is genuinely score-neutral. Minor version: add tasks or optional reporting without redefining the primary comparison; do not silently mix results across different sample sets. Major version: change labels, task populations, scoring rules or environment requirements. This is a recommended local convention, not a universal standard. Whatever system you adopt, never place a new score next to an old one without showing whether the two are comparable.
For every edit, retain the old item, reason, reviewer, effective version, and whether historical runs should be regraded. When a grading bug is discovered, publish a correction with the affected models and revised uncertainty if possible. Avoid changing the answer key quietly to protect a favourite model’s ranking.
Keep the benchmark challenging, but valid
As models improve, a benchmark can saturate: most systems pass most tasks, leaving little power to distinguish them. Add fresh, representative harder tasks with explicit new-version reporting. Avoid solving saturation by simply adding obscure trivia or ambiguously specified puzzles. Difficulty should reflect the intended construct, not arbitrary trickiness.
Reassess category weights and task distributions as usage changes. For persistent evaluation, maintain two components: a stable longitudinal set for tracking known behaviours and a refreshed set for new capabilities and drift. Public leaderboards should announce evaluation protocols and dates, and avoid implying cross-version comparability without re-running systems.
Benchmark governance
Assign separate responsibilities for content construction, expert review, grader validation, result publication and appeals. Establish a process for participants to report ambiguous cases or vulnerabilities without obtaining the whole private answer set. Record competing interests and any benchmark sponsor influence. Invite external replication and publish corrections when warranted.
Final audit
Before sharing a score, ask: Can a competent solver succeed with the provided information? Does the scoring reward the claimed behaviour? Could answers have leaked? Could the model have encountered the items before? Was the sample representative or intentionally adversarial? Is statistical uncertainty reported correctly? Are efficiency, safety and subgroup failures visible? Can someone reconstruct the experiment?
A benchmark earns trust through its protocol and stewardship, not its name or the size of its leaderboard.
Guide 03
References and further reading
- Liang et al. (2022), Holistic Evaluation of Language Models (HELM).
- Gebru et al., Datasheets for Datasets.
- Deng et al. (2024), Unveiling the Spectrum of Data Contamination in Language Models.
- Cheng et al. (2025), A Survey on Data Contamination for Large Language Models.
- Chen et al. (2021), Evaluating Large Language Models Trained on Code.
- Anthropic (2026), Demystifying evals for AI agents.
- EleutherAI Language Model Evaluation Harness.
- Shi et al. (2023), Detecting Pretraining Data from Large Language Models.
Data Gradient