Guide 04
Introduction
Human evaluation is often called the gold standard of AI quality assessment. That label is true only when the humans have the right expertise, the instructions are clear, the work is performed under appropriate conditions, and the measurement process is auditable. People can disagree for legitimate reasons; poorly designed annotation studies can systematically reward style over correctness; and high apparent agreement can hide an overly simple or biased task.
This handbook treats human evaluation as a research and operational system. It covers when to use people, how to recruit the right reviewers, how to design rating instruments, how to measure agreement, and how to maintain quality without wasting expertise or mistreating contributors. It connects directly to LLM-as-a-Judge Handbook (Guide 02) when human labels are used to validate an automated grader.
- Who this is for
- research teams, annotation operations leads, QA managers, benchmark designers and domain specialists.
Chapter one
When human evaluation is required
Humans are necessary when the task contains judgment
Some evaluation claims have objectively observable outcomes: code passes a test, a database entry was written, a citation points to a valid source. Others are legitimately about human interpretation: was a response useful to an intended user; did it preserve the nuance of a legal explanation; was a mental-health support response respectful and appropriate; was a creative answer compelling? Humans can provide context and domain reasoning that deterministic checks cannot.
The first decision is not “humans or AI?” but which parts of the evaluation require which kind of evidence. A human panel should not be asked to determine whether a payment transaction completed when a verified transaction log is available. Conversely, a text-similarity score should not replace expert assessment of whether a policy interpretation is legally sound.
Three kinds of human role matter:
- Ground-truth creation: experts determine the correct result or acceptable set for a task.
- Preference or experience judgment: intended users or trained reviewers compare quality where multiple outcomes are valid.
- Audit and adjudication: humans investigate model-judge failures, ambiguous references and high-severity exceptions.
The first and second roles can demand different reviewers. An end user may be the best judge of whether a support answer is understandable, but not of whether it contains a subtle regulatory error. For safety-sensitive domains, qualified professionals need to assess domain correctness within appropriate governance.
Define whose judgment counts
“Human preference” is incomplete without a population. Are the raters customers, general crowdworkers, multilingual experts, physicians, security analysts, or internal product owners? Their perspectives and incentives differ. Write a target-rater statement: “This rubric represents decisions a trained UK insurance claims specialist would make when given the specified policy and case file,” or “This study estimates preference among users of a particular application.” Those claims should not be swapped interchangeably.
When the goal is user preference, a convenience panel of technical staff may not represent users. When the goal is professional correctness, untrained user preference is insufficient. Sometimes a dual panel is appropriate: experts grade factual or safety accuracy while representative users rate clarity and practical usefulness.
Whose judgment is the reference?
“Human preference” is not a universal standard. A physician checking clinical terminology, a user assessing whether a support answer was understandable, and a compliance specialist reviewing an authorised transaction each judge a different construct. The appropriate panel depends on the claim. A domain expert may be best placed to detect factual errors, while a representative user may be best placed to judge ease of understanding or usefulness in context.
For a high-stakes or disputed task, do not turn disagreement into an automatic majority vote without first asking what expertise or perspective is necessary. Define which dimensions require specialist qualification, which need lived user experience, and which can be evaluated by a broad panel. A combined study may report several distinct outcomes instead of forcing them into a single “human score.”
The NIST ARIA Evaluation Planning Manual distinguishes model testing, red teaming and user testing as complementary forms of evidence. This helps explain why expert annotations and real-user outcomes may both be needed: one measures a defined property under inspection, while the other tests how the complete system functions in a human setting.
Determine whether evaluation is formative or summative
Formative review improves a system while it is being built: reviewers can discuss cases, surface missing categories and revise instructions. Summative evaluation estimates performance under a fixed protocol: reviewers should be blinded when feasible, instructions should be frozen, and changes should be logged rather than silently introduced mid-study.
Human evaluation can also be continuous: small random audits estimate typical quality, while incident-triggered reviews investigate severe failures. Separate these sample types. A set of user complaints can teach you about failure patterns but cannot estimate overall failure prevalence without knowing how complaints are generated.
Respect limits and risk
Some requests expose reviewers to disturbing or sensitive material. Screen and minimise such exposure, obtain informed consent where appropriate, provide breaks and support routes, and avoid assigning work outside a person’s competence. Treat human evaluation as an accountable process rather than an unlimited source of inexpensive labels.
A human-evaluation decision should specify the construct, required expertise, target population, judgement type, decision stakes, sampling frame, ethical constraints and how automated checks complement the review.
Chapter two
Expert recruitment
Recruit for the task, not the title
An expert review panel is only as credible as its relevant competence. “AI expert” may be meaningless for evaluating a specialist clinical claim; “native speaker” does not automatically imply skill in assessing culturally sensitive translations. Start with a job analysis: list knowledge required, decisions the reviewer must make, tools available, and mistakes that would matter.
For a financial-report extraction eval, reviewers may need to read financial statements and distinguish accounting measures. For multilingual customer support, reviewers should understand the target language, product context and local usage. For security-sensitive tool actions, reviewers may need workflow and access-control expertise. Identify which qualifications are mandatory and which can be learned in training.
Screening should resemble real work
Ask candidates to complete representative sample tasks using the actual instructions. Include easy cases, boundary cases, and a short opportunity to explain reasoning. Assess the ability to apply evidence and identify uncertainty rather than simply match an undisclosed answer key. A screener should be reviewed for fairness and accessibility and should not require extended unpaid labour.
Record qualification evidence and task scope while minimising unnecessary personal data. If certification or professional status is required, verify it appropriately. Do not assume that educational pedigree predicts quality on the exact evaluation rubric.
Design the panel deliberately
A practical configuration may include primary reviewers, overlap reviewers for reliability measurement, and senior adjudicators for unresolved cases. For contextual or cultural judgments, recruit a genuinely relevant range of perspectives, not a token reviewer whose input is later ignored. Identify potential conflicts of interest: people who built one competing system may bring valuable domain knowledge but should not secretly evaluate its blind comparison as an impartial rater.
Avoid panel drift during long studies. If half of the reviewers are replaced after a rubric update, observed shifts in scores may reflect composition changes. Keep calibration cases, reviewer cohort labels and versioned instructions so you can analyse such effects.
Qualification is evidence, not a job title
Relevant experience matters, but expertise should also be demonstrated on the actual annotation task. A strong screening batch includes straightforward cases, carefully chosen boundary cases and at least one example where insufficient evidence is the right answer. Examine the reviewer’s decision and supporting rationale, not just a single aggregate percentage. This separates gaps in knowledge from confusion caused by poor instructions.
Training should use a calibration set that is different from the locked qualification set. In calibration, reviewers can discuss cases, challenge reference labels and refine the handbook. In qualification, decisions should be independent, scored against an adjudicated standard and documented by criterion. If reviewers repeatedly disagree on the same item, that is evidence about the item’s quality as well as about reviewer performance.
A specialist panel can still introduce systematic blind spots. Recruit for the languages, jurisdictions, document types, accessibility needs and lived contexts that the intended tasks actually involve. Report where the panel is thin instead of claiming it represents every possible user. These practices support both valid measurement and fair treatment of the people doing the work.
Training as a measured process
Give reviewers a clear handbook, examples of correct and incorrect application, a way to ask questions, and feedback on a pilot round. Distinguish instruction ambiguities from individual mistakes. If trained experts consistently disagree, investigate whether the criterion is inherently subjective rather than forcing compliance with an arbitrary “gold” response.
Reviewers should understand when to select uncertain, insufficient_evidence, not_applicable, or escalate. These are not signs of poor performance. They protect data integrity when the task exceeds the available evidence or reviewer’s competence.
Pay and working conditions affect quality
Set transparent compensation for total expected work, including onboarding, calibration, reading context, tool setup and adjudication. Use realistic throughput estimates from pilot studies rather than assuming that every item can be judged at the speed of a one-sentence poll. Provide reasonable breaks and appeal mechanisms. For emotionally difficult content, use exposure limits and specialist support.
A strong review programme includes a reviewer qualification matrix, representative screening task, training materials, conflict-of-interest controls, compensation plan and oversight assignments.
Chapter three
Annotation design
Write instructions for independently reproducible decisions
A useful annotation guideline states the construct, the allowed evidence, what to ignore, label definitions, boundary examples, escalation rules and output format. Consider grading whether an assistant’s policy answer is supported by its cited source. Reviewers need the question, answer, source excerpt and governing policy version. They should not be instructed to infer the merchant’s intent or assume facts not present in the task.
Use the smallest response format that captures the decision. A binary label works for clearly defined compliance; ordinal ratings suit graded coverage; pairwise comparisons suit preference among two valid answers; span annotation pinpoints unsupported claims; free-text comments help diagnose failures. The more complex the instrument, the greater the training and fatigue burden.
Example: a three-category groundedness task
| Label | Exact rule | Distinguishing example |
|---|---|---|
fully_supported | Every material claim follows from the provided evidence | The response repeats the policy condition accurately |
contains_unsupported_claim | One or more material claims exceed or contradict evidence | Adds a non-existent exception |
cannot_determine | Source is missing, contradictory or insufficient | Document excerpt omits the relevant clause |
Ask reviewers to select the label and highlight the evidence or unsupported span. A confidence field is helpful only if reviewers understand how it will be used; self-reported confidence is not necessarily calibrated probability. Use a separate channel for ambiguous-case notes rather than silently folding them into a numeric score.
Control presentation bias
Where feasible, blind which model produced each response. For pairwise evaluation, randomise A/B order and balance position across the sample. Use consistent typography; do not show one candidate in a more polished format. Decide whether response length matters to the target construct. If evaluating factual correctness, reviewers should be instructed not to reward verbosity; if evaluating concision, assess it explicitly.
Keep the prompt and relevant context equal across candidates. Do not show a reference answer if the task is meant to measure independent reasoning. If references are necessary, explain whether they are authoritative, illustrative or one of several acceptable answers.
A rubric excerpt for evidence-based labels
A short instruction fragment is often more useful than a vague five-page description of “good quality.” For grounded answer review, the handbook might say:
Mark supported when every material factual claim follows from the supplied, authoritative sources. Mark unsupported when at least one material claim is contradicted or goes beyond a source set explicitly designated complete. Mark insufficient evidence when necessary sources are absent, conflicting or known to be incomplete. Do not judge grammar or warmth in this field.
The definition should be paired with boundary examples: a harmless paraphrase, a fabricated exception, a cited passage that does not support the claim, and a case where the document itself is contradictory. In the pilot, two reviewers should independently label the same cases and explain the evidence behind difficult choices. Record which handbook sentences they interpreted differently.
The annotation interface matters too. Show the evidence in a stable order, prevent accidental identity cues when comparing systems, make missing context visible and preserve a way to flag a defective task. A polished rubric cannot compensate for a tool that hides half the source passage.
Pilot before scaling
Run a small pilot with several reviewers, preferably including at least two raters on overlapping items. Examine label distributions, completion times, missing evidence, explanations and disagreements. Conduct a debrief: which clauses were confusing? Did people know when to abstain? Were any correct answers impossible under the supplied context? Revise the instructions, then pilot again if the changes are substantial.
Do not combine pre- and post-revision labels as if they were collected under one stable instrument. Tag annotation schema and guideline versions, and re-label affected items where comparability matters.
An annotation record
{
"item_id": "policy-0038",
"reviewer_id": "rater-pseudonym-12",
"guideline_version": "groundedness-v3",
"presentation_order": "candidate_B_first",
"label": "contains_unsupported_claim",
"evidence_span": "refund will arrive tomorrow",
"reason_code": "unsupported_timing_claim",
"uncertainty": "none",
"duration_seconds": 132,
"created_at": "2026-11-12T14:00:00Z"
}The record is fictional. In live systems, avoid storing personally identifying details in openly distributed export files. Preserve underlying candidate and source versions through secure references.
Before collecting labels, prepare a reviewer-ready guideline, worked examples, annotation schema, pilot report and freeze date for the production rubric.
Chapter four
Agreement
Disagreement is information
Agreement measures how consistently reviewers apply a scheme. It does not automatically establish the scheme is correct or that the judgments represent user preference. Two reviewers who always select the majority category may agree on most items while missing the rare, important failures. Conversely, disagreement may reflect legitimate ambiguity in preference or differences across cultural contexts.
Start with a transparent table of raw labels and count how many reviewers assessed each item. Report both raw agreement and an appropriate reliability coefficient when the study warrants it. Analyse disagreements by scenario, severity, reviewer group and rubric clause. If a label is used only twice, interval estimates will be unstable regardless of the headline statistic.
Which coefficient answers which question?
Percent agreement is straightforward but ignores chance agreement. Cohen’s kappa is commonly used for two raters on nominal categories under its assumptions. Krippendorff’s alpha can handle multiple raters, missing assignments and different measurement levels by choosing an appropriate disagreement function. Intraclass correlation (ICC) may be suitable for numeric ratings when its specific model assumptions and the desired form of agreement are understood. Do not report all coefficients indiscriminately; pick the one that matches the annotation design.
Krippendorff’s alpha is conceptually:
Here is observed disagreement and is disagreement expected under chance assignment given the category distribution. For nominal labels, category mismatches contribute disagreement; for ordered or continuous ratings, a justified distance function is used. A value of 1 indicates perfect agreement, 0 indicates chance-level agreement under the measure’s assumptions, and negative values indicate worse-than-chance agreement. Undefined cases can occur when there is no expected variation.
A worked diagnostic example
Suppose two reviewers grade 100 policy answers. They agree on 94. Raw agreement is 94%. But if 95% of all items are straightforward correct answers, even an always-“supported” reviewer can agree with another reviewer frequently. Compute class prevalence, separate agreement on the rare unsupported class, and inspect the disputed items. A 94% headline agreement is not evidence that critical failures are reliably detected.
When measuring alpha on partially overlapping panels, report how many items received at least two labels and how missing assignments were generated. Alpha accommodates missing ratings computationally; it does not make biased reviewer assignment or missing-not-at-random patterns harmless. Use bootstrap uncertainty intervals at the item level when appropriate, with enough independent items to make the interval meaningful.
Separate ambiguity from reviewer error
Imagine two qualified reviewers reading “You may return accessories within 30 days.” One marks an answer about a headset as supported; the other says the document does not establish whether a headset is an accessory. Their disagreement may be reasonable. A forced consensus does not resolve the underlying category definition.
A good adjudicator checks what evidence was actually available, applies the published decision rule and records one of several outcomes: a reviewer misapplied a clear rule, the instructions need revision, the source needs clarification, or the case is genuinely unscorable. Only the first is a straightforward reviewer error. The others require changes to the task or reference material.
Preserve initial labels and rationales even after resolving a case. A final answer key is valuable for consistent scoring, but the original disagreement indicates where the construct was difficult to operationalise. Report both an agreement statistic and the percentage of items requiring adjudication. When a revised guideline changes past labels, version the rubric and avoid comparing scores across versions as though nothing changed.
Use disagreement to improve the process
Tag disagreements into categories: insufficient evidence, rubric ambiguity, inattentive annotation, genuine subjective divergence and domain expertise gaps. Senior adjudicators should apply a documented protocol and preserve original labels. If the benchmark requires one answer key, adjudication determines the operational reference; the initial disagreement remains useful evidence about uncertainty.
Distinguish inter-rater reliability (different reviewers on the same items) from intra-rater stability (one reviewer revisiting items later). Both can matter, but retesting the same person risks recall effects. Balance workload and avoid turning “agreement with the majority” into a compliance incentive that suppresses legitimate specialist dissent.
Thresholds require justification
Sometimes teams cite 0.80 as a desirable reliability level or 0.667 as a tentative-use convention for Krippendorff’s alpha. These are conventions, not universal laws. What level is sufficient depends on the type of decision, severity of error, number of labels, prevalence and consequences of disagreement. Predefine the acceptance criterion and a plan for cases that remain disputed.
A reliability report should include the coefficient and distance function, raw agreement, number of items and raters, missingness, uncertainty, label prevalence, subgroup breakdown and adjudication summary.
Chapter five
Quality control
Quality is built into workflow design
The least reliable QA strategy is to collect thousands of labels and search for mistakes at the end. Quality control begins with answerable tasks, suitable reviewers, accessible evidence, tested instructions and realistic workloads. It continues with overlap assignment, spot audits, calibration rounds and clear escalation.
A strong program typically uses several controls:
- Pilot calibration before the production batch; fix instruction defects early.
- Independent overlap on a planned subset of items; do not let raters coordinate before initial labels.
- Known-reference items where answers are genuinely unambiguous; rotate and review them to avoid memorisation.
- Risk-targeted audits on unusual, consequential or contested items.
- Random audits to estimate ordinary error rates, not merely to find dramatic failures.
- Adjudication with recorded reason codes and a pathway to amend a flawed reference answer.
- Feedback and retraining on specific error patterns, with requalification where necessary.
Estimating error without cherry-picking easy audits
Two audit samples serve different purposes. A random sample of completed annotations can estimate error frequency under the documented selection design. A targeted sample concentrates attention on high-risk, ambiguous or anomalous cases and is valuable for finding problems. Mixing them together without marking their origins creates a misleading overall error rate.
Suppose 100 randomly selected annotations contain 6 verified errors, while 100 risk-targeted annotations contain 24. The combined 15% error figure is not a valid estimate of ordinary project quality unless the selection probabilities justify it. The random sample suggests a 6% error rate for the population it represents, with substantial uncertainty; the targeted sample demonstrates where the review process is vulnerable. Both should be reported, separately.
When quality varies by category or reviewer, stratified auditing may be more efficient than an undifferentiated random draw. Keep sampling probabilities, denominator definitions and adjudication procedures. Audit labels, not just submitted records, and ensure reviewers can appeal a gold label that conflicts with the documented evidence. The objective is to correct measurement, not to manufacture agreement.
Design checks that cannot be gamed trivially
Hidden gold questions can detect obvious quality problems, but repeated exposure turns them into a memorised exam. Do not use trick questions that lack sufficient context and then punish reviewers for failing them. Combine gold performance with evidence quality, disagreement patterns and independent audits. Speed is a weak proxy: very fast work may be careless, but a highly competent reviewer can also be efficient. Flag unusual behaviour for review rather than automatically discarding work.
Use a severity-aware escalation process
A disputed comma in a summary and a disputed policy statement authorising a financial transfer should not receive the same treatment. Define critical, major and minor disagreement classes. A critical disagreement may require an independent specialist, additional source material, temporary exclusion of the item, or suspension of an automated decision. Document how final decisions were reached.
Monitor reviewer and item effects
Track item difficulty, disagreement rate, missing evidence, time spent, guideline version and reviewer workload. Separate item effects (ambiguous cases cause everyone problems) from reviewer effects (one reviewer struggles with one policy type). If agreement collapses after a policy update, investigate the change before assuming the workforce degraded.
Do not expose individual reviewer quality metrics broadly without need. Use pseudonyms, limit access and handle performance conversations fairly. Fair compensation and a channel to dispute inaccurate gold labels improve quality and trust.
Audit the audit
Gold labels can be wrong; senior reviewers can drift; automated quality dashboards can conceal missing submissions. Periodically re-review the gold pool, sample adjudicator decisions and test analytics for denominator and missing-data errors. For any benchmark released externally, describe the review process and known limitations rather than promising infallibility.
A quality-control programme needs a documented policy specifying the overlap percentage, audit sampling method, escalation criteria, gold-set governance, reviewer feedback process, incident definitions and a documented appeal mechanism. Choose percentages based on pilot variance, risk and cost, not habit.
Chapter six
Cost, throughput, and scale
Budget from observed effort
A human evaluation project is a production pipeline: recruiting, training, item preparation, review, overlap, adjudication, QA, reporting and rework all consume time. A common planning error is to calculate only the number of initial ratings and ignore everything else.
An illustrative formula for direct rating hours is:
where is items, is average reviews per item, and is minutes per review. For 2,000 items with two independent reviews averaging four minutes each, direct rating time is hours. At a hypothetical fully loaded rate of £40 per hour, that is roughly £10,667 for rating alone, excluding recruitment, training, QA, adjudication, platform costs, privacy operations and project management. Real compensation must reflect local requirements and the expertise and sensitivity of the work.
Design capacity around complexity
Track throughput separately for easy items, expert cases, long-context materials, adversarial content and disputes. A single average can understate the effort of consequential reviews. Pilot several dozen or hundreds of items, depending on task variability, before locking a budget. Reviewers handling high-context cases may need substantial reading time, tool access and breaks.
A useful pipeline has queues for ready to label, first label complete, overlap required, disputed, adjudicated, QA passed and exported. Each state should have an owner and an explicit exit rule. Measure end-to-end turnaround, not only annotations per hour; idle time while waiting for policy clarifications can dominate elapsed project time.
A complete cost model
Direct rating time is only part of the budget. Add the costs of screening, paid training, double annotation, adjudication, rework, specialist consultation, tooling, secure data handling and project coordination. For a study that needs 4,000 independent judgments, an average of four minutes per judgment implies about 267 direct rating hours. If the 4,000 judgments represent two ratings on each of 2,000 items, and 15% of items require an additional ten-minute expert adjudication, that adds 50 expert hours before other operational overhead.
Model the arrivals and exits of each review queue. A process with enough first-pass reviewers can still stall if one specialist must resolve every ambiguous case. Estimate adjudication demand from the pilot, track its variation, and reserve capacity for model updates or incidents. Use observed timing distributions rather than treating every label as a fixed-duration transaction.
Efficiency improvements should preserve measurement validity. Better interface design and clearer task instructions may reduce review time without increasing error. Replacing domain reviewers with cheaper generalists may do the opposite. Evaluate changes to the annotation operation using held-out qualification cases, agreement analysis and fresh audits, just as you would validate a model judge.
Spend expertise where it changes decisions
Use deterministic checks for objectively verifiable properties, qualified annotators for standardised subjective review, senior experts for complex or critical cases, and calibrated LLM judges for suitable low-risk tasks. This is tiered review, not a licence to replace experts indiscriminately. Choose routing rules using measured grader error, risk and review costs.
Active sampling can help discover errors efficiently: select examples where a model judge is uncertain, where two judges disagree, or where a new scenario appears. But an actively selected pool cannot estimate population-wide accuracy without accounting for its selection. Keep an independent random audit alongside targeted sampling.
A capacity plan
For each queue, estimate expected arrivals per week, average processing time, number of qualified reviewers, proportion requiring second review and adjudication, holiday or absence coverage, and acceptable delay. Reserve extra capacity for incident spikes or new model releases. When capacity becomes constrained, reduce low-value annotation rather than silently shortening review time for critical cases.
Human evaluation as a living reference system
As policies, models and user behaviours change, the annotation handbook and gold set must be maintained. Version every revision, retrain affected reviewers, compare old and new labels on a bridging sample when longitudinal comparability matters, and retire stale cases. Feed recurring disagreements back into product requirements; they often expose ambiguous policy or user needs that no grader can solve alone.
Final checklist
Can we articulate who reviewers represent? Are they qualified for the decisions? Is the rubric answerable and piloted? Are labels blinded where appropriate? Are disagreement and uncertainty preserved? Is the QC process itself audited? Have we budgeted humane conditions and overhead? Can we reproduce and explain every adjudicated label? If so, the human evaluation system is much closer to producing evidence that deserves to be trusted.
Guide 04
References and further reading
- Hämäläinen and Alnajjar (2021), Human Evaluation of Creative NLG Systems: An Interdisciplinary Survey.
- Krippendorff’s Alpha: Calculating Intercoder Reliability.
- Liu et al. (2023), G-Eval.
- Zheng et al. (2023), Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.
- Anthropic (2026), Demystifying evals for AI agents.
- OpenAI, Evaluation best practices.
- NIST (2026), ARIA Evaluation Planning Manual.
Data Gradient