Guide 05Volume 01, Model Evaluation

The Evals Metrics Dictionary

Pass@k, Elo, win rates, confidence intervals, Krippendorff’s alpha, and more.

Series
Guides, Vol. 01
Contents
5 terms
Written by
Head of Training, Data Gradient
Guide 05, openedFrontispiece and index

The Evals Metrics Dictionary

5 terms, in alphabetical order

Written by Head of Training at Data Gradient

Guide 05

Introduction

Metrics are compact descriptions of experiments. They become misleading when their denominators, units, independence assumptions or aggregation conventions are hidden. “Pass@5 is 70%,” “our Elo is 1,250,” and “the judge agrees 92% of the time” may all be true yet answer different questions from the ones a reader assumes.

This dictionary gives working definitions, formulas, interpretation advice and failure modes. Each entry follows the same pattern: what it measures, how to compute it, worked example, when to use it, what to report, and what it cannot prove. It is a reference to use beside Guide 01, Guide 02, Guide 03 and Guide 04.

Confidence intervalsA to ZWin rate

Confidence intervals

05.01

Definition

A confidence interval (CI) quantifies uncertainty from a stated statistical procedure. In the frequentist interpretation, a 95% confidence-interval method would cover the true parameter in approximately 95% of hypothetical repetitions of the sampling process if the model assumptions hold. It does not mean there is a 95% probability that the fixed true parameter lies in this realised interval.

The interval is only as meaningful as the sampling design, labels and estimator. A 95% CI around results from a convenience sample does not establish uncertainty for a different production population. A narrow CI around an invalid benchmark is precisely misleading.

Binary pass rates: prefer Wilson to a naive normal interval

If xx of nn independent sampled tasks pass, let p^=x/n\hat p=x/n, and let zz be the standard normal critical value (approximately 1.96 for 95%). A Wilson interval is:

p^+z2/(2n) ± zp^(1−p^)/n+z2/(4n2)1+z2/n.\frac{\hat p + z^2/(2n) \ \pm\ z\sqrt{\hat p(1-\hat p)/n+z^2/(4n^2)}}{1+z^2/n}.

For 82 successes in 100 independent tasks, the point estimate is 82%; the approximate 95% Wilson interval is 73.3% to 88.3%. This expresses sampling uncertainty under the assumed Bernoulli design. It does not include uncertainty from incorrect reference labels or a biased sample.

Comparing A and B

If both systems run on the same tasks, analyse paired differences. For each task, compute A’s score minus B’s, or keep the paired binary outcomes. Resample complete tasks or independent source families rather than individual model calls. A bootstrap procedure can form an interval from the resulting distribution of aggregate paired differences. With correlated multi-turn tasks or stratified samples, resampling must respect the structure and weights.

If you select the best model out of dozens after repeated testing on the same holdout, ordinary CIs may overstate the evidence for the chosen winner. State the selection protocol and use a fresh confirmation set or appropriate correction. Repeated peeking at results also changes error properties unless handled by a predeclared sequential method.

Interpreting uncertainty from zero failures

A sample with no observed failures is not evidence that the failure probability is zero. Under independent trials with a constant failure probability, a common approximate one-sided 95% upper bound after n zero-failure trials is 3/n. With 100 trials and zero observed failures, the bound is around 3%; with 1,000 such trials, around 0.3%. These bounds rely on the sampling model and do not cover risks the test failed to exercise.

This is especially important for safety metrics. A team may need targeted scenario testing to identify severe failure mechanisms and representative sampling to estimate incidence. The two data sources answer different questions. Do not merge their denominators without a clear design.

Confidence intervals also address only the uncertainty represented by the statistical model. They do not correct a biased task pool, erroneous labels, changed grader, or missing outcomes. A tight interval around an invalid measure is still invalid.

Confidence intervals versus prediction intervals

A confidence interval for an average estimates uncertainty in a population quantity. A prediction interval concerns a future observation, which may vary much more than the mean. A 95% CI for mean judge accuracy does not say any individual judgment has a 95% chance of being correct.

What to report

Always include metric estimate, interval limits, confidence level, method, unit of independence, sample size, sampling frame, weights, missing-data handling and whether multiple comparisons were made. For rare-event safety metrics, zero observed events needs an upper uncertainty bound, not the claim “risk is zero.”

Do not infer: causal improvement or real-world representativeness just because two intervals do not overlap, or because they do. Use a direct interval for the A-B difference and check practical significance.

Research: Guide 03, Statistical validity.

Elo

05.02

Definition

Elo is a sequential rating system originally developed for competitions. In model evaluation it is often used to summarise results of pairwise comparisons. Ratings describe relative performance within a specific comparison pool and protocol. An Elo of 1,200 has no universal meaning across unrelated leaderboards, and a difference of 50 points need not be meaningful without uncertainty and rating-scale details.

The conventional expected score for A against B is:

EA=11+10(RB−RA)/400.E_A = \frac{1}{1 + 10^{(R_B-R_A)/400}}.

After an outcome, a common update rule is:

RA′=RA+K(SA−EA),R'_A = R_A + K(S_A-E_A),

where RAR_A and RBR_B are current ratings, KK is the update factor and SAS_A is 1 for an A win, 0.5 for a tie, or 0 for a loss. Different implementations can use different starting points, K schedules, treatment of ties and matchmaking assumptions.

Worked example

A and B are both rated 1,000. Their expected scores are 0.5 each. If A wins with K=32, A gains 32(1−0.5)=1632(1-0.5)=16 points and B loses 16. New ratings are 1,016 and 984. A single match is a noisy update, not compelling evidence that A is superior.

Elo versus Bradley-Terry

Many current AI leaderboards display Elo-like scores derived from a Bradley-Terry statistical model fitted jointly to pairwise outcomes. These are not necessarily sequential Elo updates. A common form is:

P(A beats B)=eθAeθA+eθB.P(A\text{ beats }B) = \frac{e^{\theta_A}}{e^{\theta_A}+e^{\theta_B}}.

The latent strengths θ\theta are estimated from the match data and mapped to an Elo-style scale for display. State the actual estimator rather than using the word “Elo” indiscriminately. Ratings cannot be compared meaningfully across disconnected match graphs without anchoring assumptions.

Ratings depend on the comparison graph

Elo and Bradley-Terry methods learn from who played whom. Consider three systems: A is compared mostly with C, while B is compared mostly with D, and the two groups have no common opponents. Without bridge matches or additional assumptions, the relative strength of the groups is not identified by the observed comparisons. A leaderboard may still display numbers, but the scale connecting those groups is arbitrary.

This is why a useful rating report includes match coverage, the date range, version identities, uncertainty and sampling rules. Crowd-sourced comparisons may overrepresent prompts people find engaging or models users choose to compare. A change in who participates or which models are paired can move ratings even when the systems themselves do not change.

The Arena FAQ describes its use of a Bradley-Terry model to fit pairwise votes. That is an instructive example of why the rating method should be named precisely. Sequential Elo, batch Bradley-Terry and other models may produce related-looking but nonidentical scales.

What changes a rating?

Opponent selection matters: A beating strong, frequently compared models is different from A beating a narrow group of weaker systems. Prompt difficulty, user demographics, candidate order, draw handling, model version changes and voting interface all influence results. A rating change may reflect a different comparison pool, not a change in the model itself.

Report a rating with its method, scale, comparison window, number of battles, confidence interval, tie handling, and evaluation protocol. When there are few matches or a disconnected pool, uncertainty can be substantial. Model selection from an Elo leaderboard should also consider task-specific guardrails, cost and latency.

Do not infer: absolute intelligence, success rate on unseen production tasks, or statistical superiority solely from a higher point estimate.

Research: Zheng et al., Chatbot Arena; Arena rating methodology.

Krippendorff’s alpha

05.03

Definition

Krippendorff’s alpha (α) is a chance-adjusted measure of inter-rater reliability. It assesses how much annotators disagree relative to the disagreement expected under a chance model derived from observed values. Unlike two-rater nominal kappa, alpha’s framework can work with different numbers of annotators, missing ratings and nominal, ordinal, interval or ratio variables, when the appropriate disagreement measure and design assumptions are used.

Its core definition is:

α=1−DoDe,\alpha = 1 - \frac{D_o}{D_e},

where DoD_o is observed disagreement and DeD_e is expected disagreement. The implementation uses a coincidence matrix and a distance function for the chosen measurement level. Merely substituting a 0/1 disagreement rule for ordinal or continuous ratings can discard information.

Interpretation

  • α = 1: perfect agreement.
  • α = 0: agreement at the chance benchmark defined by the measure.
  • α below 0: more disagreement than the chance benchmark.
  • Undefined: can occur when expected disagreement is zero or the data are insufficient for the coefficient.

An alpha of 0.75 is not “75% of annotations correct.” It also does not establish that the majority labels reflect truth. Raters could agree on an incorrectly designed rubric.

Worked conceptual example

Three reviewers label customer answers as supported, unsupported, or uncertain. Most items receive all three labels, but a few have only two because of scheduling. Alpha can use the available pairwise comparisons without dropping every incompletely rated item. However, if missing reviewers systematically skip the hardest cases, the coefficient may represent a biased subset of the task. Report the missingness pattern and the number of pairable units.

When labels are heavily imbalanced, for instance, nearly everything is supported, raw agreement may be high while chance-adjusted reliability becomes difficult to interpret or unstable. Inspect category prevalence, per-category disagreement, item ambiguity and uncertainty alongside alpha.

Choosing a distance function

For nominal labels, different categories are simply different. For ordinal scores such as poor < adequate < good, distance should reflect the ordered structure, using a defensible ordinal function. For numeric measurements, interval or ratio distances may be relevant. Alpha values computed using different distance functions are not directly interchangeable.

A reliability result needs context

Consider two annotation studies that both report nominal alpha of 0.75. The first has balanced categories, hundreds of independently selected cases and several raters with substantial overlap. The second has an overwhelmingly dominant category and 20 cases chosen for convenience. The shared coefficient does not make the studies equally informative. Inspect category prevalence, overlap structure, sample size and confidence intervals before using the number for a consequential decision.

Reliability also differs from validity. If three reviewers consistently misunderstand the same policy exception, they can agree strongly and still produce incorrect labels. Validate reference labels against independent evidence and qualified adjudication where the task allows it. If judgments are inherently subjective, clearly identify whose preferences are represented rather than claiming a universal ground truth.

Finally, reliability can drift after annotator turnover, a revised handbook or a new model category. Repeat the measurement on fresh overlapping samples and keep original labels when guidelines change. See Guide 04 for the operational design behind these statistics.

Acceptance and reporting

Values around 0.80 are often cited as a reliability convention, with 0.667 sometimes mentioned as a lower bound for tentative conclusions. These are not universal acceptance rules. Evaluate whether the instrument is adequate for the consequence of error. A safety-critical dataset with severe false negatives may require additional per-class checks regardless of alpha.

Report the coefficient, variable level/distance, item count, number of raters, overlap design, missingness, raw agreement, category distribution, uncertainty interval, software/version, and adjudication process. Bootstrap at the independent-item level when appropriate.

Do not infer: validity, human expertise, truth of the reference labels, or suitability for all decisions from alpha alone.

Research: Krippendorff’s alpha: methodology and worked example.

Pass@k

05.04

Definition

Pass@k measures the probability that at least one of k attempts at a task succeeds, when the success check is fixed and the attempts follow the stated sampling policy. It is used in code generation, mathematical problem solving, search and some agent evaluations where multiple candidate solutions can be generated and tested.

For a task with independent, identically distributed trial success probability pp, the theoretical relationship is:

pass@⁡k=1−(1−p)k.\operatorname{pass@}k = 1 - (1-p)^k.

The observed pass@k of a set of k trials is simply 1 if any trial passes, and 0 otherwise; average this indicator across tasks to estimate performance for that attempt policy. Do not assume one candidate has k retries if the product only permits one.

The finite-sample estimator

In coding benchmarks, one often generates nn candidate solutions for a problem, observes cc passing solutions, and wants to estimate pass@k for k≤nk \le n. The standard unbiased combinatorial estimator described by Chen et al. is:

pass@⁡k̂=1−(n−ck)(nk).\widehat{\operatorname{pass@}k} = 1 - \frac{\binom{n-c}{k}}{\binom{n}{k}}.

If n−c<kn-c < k, the numerator is zero: every group of k distinct sampled solutions contains at least one success. Compute this per problem and then average across problems; do not pool all successful attempts into one giant bag if tasks differ in difficulty. The estimator assumes an appropriate sample of outputs under the defined generation procedure.

Worked example

For one programming task, generate 10 outputs; 3 pass the tests. Then:

pass@⁡3̂=1−(73)(103)=1−35120≈70.8%.\widehat{\operatorname{pass@}3} = 1 - \frac{\binom{7}{3}}{\binom{10}{3}} = 1 - \frac{35}{120} \approx 70.8\%.

This does not mean the system succeeds 70.8% of the time on its first output. It estimates the chance that a set of three candidates contains at least one correct answer under the sampling design and adequacy of the tests.

What the estimator assumes

The combinatorial estimator is useful because it estimates the fraction of size-k subsets containing a passing candidate, given n sampled candidates and c observed passes. It is not a guarantee about a real user experience: a system that generates three answers and picks the first without testing them does not obtain the benefit of pass@3. The selection or verification procedure must be part of the product protocol.

Sampling also matters. If all candidates are identical because decoding is deterministic, generating more copies does not create the independent chances implied by 1−(1−p)k1-(1-p)^k. If attempts share context, adapt after seeing failures, or use different budgets, the independent and identically distributed approximation can be inappropriate. Report the actual procedure and measure it empirically.

Finally, distinguish a task-average from an attempt-average. A benchmark with one easy problem and one nearly impossible problem can have a respectable pooled success fraction while showing poor coverage on the hard task. Compute per-task pass@k and aggregate according to the benchmark’s defined task weighting; include task counts and uncertainty.

A different metric: pass^k

Pass^k (read “pass to the k”) asks whether all k attempts succeed, useful for reliability. For an individual task with independent success probability pip_i, passk⁡i=pik\operatorname{pass^k}_i = p_i^k. For a benchmark with differing task difficulties, the relevant aggregate is the mean of pikp_i^k, not generally (mean pi)k(\text{mean }p_i)^k.

If one task has a 75% independent per-trial success chance, the chance of all three succeeding is 0.753≈42.2%0.75^3 \approx 42.2\%; the chance of at least one success is 1−0.253=98.4%1-0.25^3 = 98.4\%. These metrics answer nearly opposite product questions.

Use, report, avoid

Use pass@k if users or systems genuinely generate and select among k candidates. Report task count, number of generated candidates n, k, decoding policy, test validity, retry/selection protocol, aggregation and uncertainty. Use pass^k for consistent success across repeated deployments or agent runs. Do not call pass@k “accuracy” without explaining the attempt budget, and do not confuse a capable search process with dependable first-try performance.

Research: Chen et al., Evaluating Large Language Models Trained on Code; Anthropic, Demystifying evals.

Win rate

05.05

Definition

Win rate reports how often candidate A wins head-to-head comparisons with B under a given set of prompts and judging rules. It is intuitive but has several non-equivalent conventions, especially for ties.

Let W be A wins, L be A losses, T be ties and N=W+L+TN=W+L+T. The tie-split win score is:

win score⁡A=W+0.5TN.\operatorname{win\ score}_A = \frac{W + 0.5T}{N}.

A win rate including ties as non-wins is simply W/NW/N; a decisive-only win rate is W/(W+L)W/(W+L) when W+L>0W+L>0. These values should never be labelled identically without explanation.

Worked example

In 100 comparisons, A wins 46, loses 34 and ties 20. Then:

  • Strict wins / all comparisons = 46/100=46%46/100 = 46\%.
  • Tie-split win score = (46+10)/100=56%(46 + 10)/100 = 56\%.
  • Wins among decisive comparisons = 46/80=57.5%46/80 = 57.5\%.

All three are arithmetically correct. Only the protocol tells the reader which is appropriate. A tie-split score of 56% indicates A’s average match score exceeds 50% in this evaluated set, not that A will be preferred by 56% of all possible users or tasks.

Ties deserve explicit handling

A tie is not necessarily a failure of a judge. Two answers may be equally acceptable, or the difference may be too small to support a preference. A protocol that requires a winner on every case can manufacture distinctions and amplify order effects. Decide in advance whether ties are permitted and whether “both unacceptable” is a distinct outcome from “equally good.”

For example, A may be concise but omit a necessary qualification; B may include the qualification but introduce an unsupported policy claim. A judge should not be pushed to decide which is prettier if both violate a mandatory requirement. A defensible design first checks critical criteria, then asks for a preference among candidates that satisfy them. This separates eligibility from ranking.

With multiple raters, report the distribution of A wins, B wins, ties and invalid comparisons. For a high-stakes benchmark, inspect whether disagreement comes from borderline responses, incomplete evidence or systematic bias. A model judge’s preference is an estimate of the model judge’s verdict under the tested rubric, not automatically a measure of human utility.

Paired design and judge effects

Use the same task for the two candidates; randomise or balance candidate presentation order; blind identities; allow ties where legitimate; and measure order sensitivity. If several judges compare the same response pair, these are correlated judgments. If the same base question appears in many paraphrases, account for clustering by question family.

For a single fixed pair of models, the comparison results can be summarised by wins, losses and ties with an uncertainty interval. When comparing many models, pairwise win rates need not produce a coherent transitive ranking, A can beat B, B can beat C, and C can beat A in different task distributions. Use a clearly specified model such as Bradley-Terry if a global ranking is required, and disclose its assumptions.

Use, report, avoid

Use win rates for subjective preference, clarity comparisons and relative quality where a legitimate human or calibrated judge can compare alternatives. Report prompt population, pair count, evaluator type, tie rule, order-balancing method, uncertainty and slices. Do not confuse a model-judge win rate with direct human preference, and do not compare win rates collected on different prompt populations as if controlled.

Research: Zheng et al., MT-Bench; Wang et al., position bias.

More terms

05.06

The following metrics recur in AI-evaluation reports. They are grouped by the question they answer rather than by a supposed hierarchy of importance.

Classification and detection

TermDefinitionMain caution
Accuracy(TP+TN)/(TP+TN+FP+FN)(TP+TN)/(TP+TN+FP+FN)Misleading for rare classes or unequal error costs
Precision (PPV)TP/(TP+FP)TP/(TP+FP): of flagged positives, how many really are positive?Depends on prevalence in the evaluated population
Recall (sensitivity, TPR)TP/(TP+FN)TP/(TP+FN): of actual positives, how many are found?Can increase while false positives become excessive
Specificity (TNR)TN/(TN+FP)TN/(TN+FP)High specificity can coexist with poor recall
False positive rateFP/(FP+TN)=1−specificityFP/(FP+TN) = 1-\text{specificity}Denominator is actual negatives, not all cases
False negative rateFN/(FN+TP)=1−recallFN/(FN+TP) = 1-\text{recall}Critical for harmful misses
F1 scoreHarmonic mean 2PR/(P+R)2PR/(P+R)Ignores true negatives and makes precision/recall equally weighted by convention
Balanced accuracyMean of sensitivity and specificityStill misses different costs of errors
AUROCProbability-based discrimination ranking across thresholdsMay obscure poor positive precision for rare events
AUPRCArea under precision-recall curve across thresholdsDepends on prevalence and integration convention

Before using the table, define the positive event. In a safety review, is positive “harm detected” or “safe response”? Reversing the label changes which errors matter and how readers interpret recall.

Rating, ranking and agreement

TermDefinitionMain caution
Raw agreementProportion of items for which raters’ labels matchDoes not adjust for chance or class prevalence
Cohen’s kappaChance-adjusted agreement for two raters on categorical outcomesCan be counterintuitive with prevalence imbalance
ICCFamily of intraclass-correlation models for numerical ratingsMust specify model, single vs average and consistency vs absolute agreement
Spearman correlationRank association between two variablesHigh correlation does not imply absolute agreement
Kendall’s tauRank concordance minus discordance under a specified variantTie handling and sample design matter
Mean absolute error (MAE)Mean absolute difference from a reference scoreRequires a meaningful numerical scale
Brier scoreMean squared error of predicted probabilities vs binary outcomesInputs must be interpretable as probabilities
Expected calibration error (ECE)Weighted average gap between predicted confidence and observed accuracy over binsHighly sensitive to binning and sample size; not proof of calibration

A model’s 4/5 judge score is not necessarily a probability, and a strong rank correlation may coexist with systematic score inflation. Choose the statistic that reflects the measurement claim.

Retrieval, grounding and generation

Exact match (EM) is the fraction of predictions identical to the reference after a specified normalisation. It works for strict identifiers or categorical outputs, but unfairly penalises equally valid paraphrases. Token-level F1 measures token overlap with a reference; it can reward surface similarity without establishing truth. ROUGE measures overlap for text generation in various ways; useful for some summarisation comparisons, but not a standalone factuality measure. Faithfulness/groundedness aims to determine whether output claims are supported by specified evidence, usually requiring detailed annotation or a validated judge. Citation precision can measure the share of citations that actually support a claim; citation coverage is the share of material claims with appropriate support. Merely counting citation markers is not either one.

For retrieval systems, distinguish retrieval recall@k (the proportion of known relevant items found among the top k, under a defined relevance set), precision@k (the fraction of the top k results judged relevant), MRR (mean reciprocal rank of the first relevant result) and NDCG (discounted graded relevance compared with an ideal ranking). These assess retrieved lists, not necessarily downstream answer quality.

Operational and safety metrics

Task success rate: proportion of tasks whose defined final outcome is achieved; state whether errors, timeouts and unscorable runs count as failures. Violation rate: harmful or disallowed events per defined unit of exposure, with severity, denominator and surveillance method. Abstention rate: proportion of tasks for which the model declines or requests more information; interpret alongside appropriateness of abstention. Coverage: share of inputs on which an automated grader or policy makes a decision rather than abstains. p95 latency: 95th percentile of request latency in a defined population and interval, not the latency of the slowest 5% averaged together. Cost per successful task: total measured expense divided by successfully completed tasks under a fixed budget and accounting method; may penalise repeated attempts correctly.

Worked metric selection for one AI assistant

Take a system that retrieves product-policy documents and answers customer questions. No single metric captures whether it is reliable. Retrieval recall@k tests whether necessary evidence appears among the returned documents; groundedness assesses whether answer claims follow from the supplied evidence; task success checks whether the user received the correct outcome; and prohibited-action rate verifies that the assistant did not change account state outside its authority.

The measurements should be linked but not conflated. A response can be faithful to the wrong retrieved policy, so high groundedness does not prove policy correctness. The right document can be retrieved while the answer invents a fact, so retrieval recall does not guarantee truth. A well-written explanation can accompany an unauthorised action, so helpfulness cannot substitute for state verification.

Report each metric with its population, unit, grading method and uncertainty. For the first release, prioritise the measures that correspond to real product harms and outcomes. Add finer diagnostics when an aggregate result fails to identify the root cause. A smaller collection of interpretable metrics is more valuable than a dashboard filled with mutually inconsistent percentages.

A complete metric card

Before adding any number to a dashboard, record:

Metric name and version:
Question answered / construct:
Population and inclusion rules:
Unit of observation:
Numerator and denominator or scoring formula:
Treatment of ties, retries, abstentions, errors and missing values:
Sampling and weighting:
Grader or reference standard:
Aggregation and relevant slices:
Uncertainty method:
Known failure modes:
Decision threshold and owner:

Choose metrics by decision, not fashion

If your concern is an agent making prohibited writes, a generic helpfulness rating or leaderboard Elo cannot substitute for verified tool-state checks. If your concern is whether users prefer one explanation, exact match is too narrow. If your concern is whether a judge can be trusted, agreement statistics need class-specific errors, human reference quality and temporal robustness alongside them.

A metric is useful when its assumptions are visible, its errors are understood, and it changes a real decision. Otherwise, it is decoration.

Guide 05

References and further reading

End of guide 05

Guide 05, Volume 01

The Evals Metrics Dictionary

Written by Head of Training at Data Gradient

Published by AI Circle for its community