Confidence intervals
05.01Definition
A confidence interval (CI) quantifies uncertainty from a stated statistical procedure. In the frequentist interpretation, a 95% confidence-interval method would cover the true parameter in approximately 95% of hypothetical repetitions of the sampling process if the model assumptions hold. It does not mean there is a 95% probability that the fixed true parameter lies in this realised interval.
The interval is only as meaningful as the sampling design, labels and estimator. A 95% CI around results from a convenience sample does not establish uncertainty for a different production population. A narrow CI around an invalid benchmark is precisely misleading.
Binary pass rates: prefer Wilson to a naive normal interval
If of independent sampled tasks pass, let , and let be the standard normal critical value (approximately 1.96 for 95%). A Wilson interval is:
For 82 successes in 100 independent tasks, the point estimate is 82%; the approximate 95% Wilson interval is 73.3% to 88.3%. This expresses sampling uncertainty under the assumed Bernoulli design. It does not include uncertainty from incorrect reference labels or a biased sample.
Comparing A and B
If both systems run on the same tasks, analyse paired differences. For each task, compute A’s score minus B’s, or keep the paired binary outcomes. Resample complete tasks or independent source families rather than individual model calls. A bootstrap procedure can form an interval from the resulting distribution of aggregate paired differences. With correlated multi-turn tasks or stratified samples, resampling must respect the structure and weights.
If you select the best model out of dozens after repeated testing on the same holdout, ordinary CIs may overstate the evidence for the chosen winner. State the selection protocol and use a fresh confirmation set or appropriate correction. Repeated peeking at results also changes error properties unless handled by a predeclared sequential method.
Interpreting uncertainty from zero failures
A sample with no observed failures is not evidence that the failure probability is zero. Under independent trials with a constant failure probability, a common approximate one-sided 95% upper bound after n zero-failure trials is 3/n. With 100 trials and zero observed failures, the bound is around 3%; with 1,000 such trials, around 0.3%. These bounds rely on the sampling model and do not cover risks the test failed to exercise.
This is especially important for safety metrics. A team may need targeted scenario testing to identify severe failure mechanisms and representative sampling to estimate incidence. The two data sources answer different questions. Do not merge their denominators without a clear design.
Confidence intervals also address only the uncertainty represented by the statistical model. They do not correct a biased task pool, erroneous labels, changed grader, or missing outcomes. A tight interval around an invalid measure is still invalid.
Confidence intervals versus prediction intervals
A confidence interval for an average estimates uncertainty in a population quantity. A prediction interval concerns a future observation, which may vary much more than the mean. A 95% CI for mean judge accuracy does not say any individual judgment has a 95% chance of being correct.
What to report
Always include metric estimate, interval limits, confidence level, method, unit of independence, sample size, sampling frame, weights, missing-data handling and whether multiple comparisons were made. For rare-event safety metrics, zero observed events needs an upper uncertainty bound, not the claim “risk is zero.”
Do not infer: causal improvement or real-world representativeness just because two intervals do not overlap, or because they do. Use a direct interval for the A-B difference and check practical significance.
Research: Guide 03, Statistical validity.
Data Gradient