Evaluation fundamentals before advanced agent evaluation

An LLM application can look excellent and still be unreliable.

Imagine that we build PolicyGuide, a retrieval-augmented generation application for an insurance company. An employee asks a question, the system retrieves policy documents, and an LLM produces an answer with citations.

During a demo, we ask five questions:

“How many paid vacation days do employees receive?”
“Can I carry unused leave into next year?”
“When does health coverage begin?”
“How do I add a dependent?”
“What is the reimbursement limit for home-office equipment?”

Every answer is fluent. The citations look plausible. The product team calls the demo a success.

Then real users arrive.

A new employee receives the policy for employees hired before 2025.
A contractor is given a benefit available only to full-time employees.
A response says “up to ₹50,000” while its citation says “up to ₹15,000.”
The right document exists, but the retriever ranks an obsolete copy first.
The answer is factually correct but cites an unrelated paragraph.
A prompt change improves tone while silently reducing citation completeness.
A stronger model improves quality by two points but triples cost and pushes p95 latency beyond the product’s timeout.

Nothing about the demo measured these behaviours. It only produced a feeling: the system looks good.

Evaluation replaces that feeling with evidence.

The essential loop is:

Specify the behaviour that matters.
Turn the specification into representative test cases.
Run a controlled version of the application.
Measure retrieval and generation separately.
Inspect and classify failures.
Change one part of the system.
Re-run the same evaluation.
Reject the change if it causes unacceptable regressions.

This is evaluation-driven development. It is not a final quality check. It is the engineering process through which an LLM application becomes dependable.

1. Why manual testing is insufficient

Manual testing is useful for exploration. It is poor evidence of reliability.

When developers test an LLM application manually, they usually choose questions they expect the system to answer. They unconsciously rewrite unclear queries, forgive minor errors, and focus on impressive outputs. This creates several distortions.

Selection bias

Five memorable questions are not a sample of production traffic. They may exclude ambiguous wording, rare document types, conflicting policies, missing evidence, multilingual inputs, long questions, and malicious instructions.

Inconsistent judgment

One answer may be accepted because it “sounds right,” while a similar error is rejected the next day. Without a rubric, the decision cannot be reproduced.

Nondeterminism

The same input can produce different wording, citations, or even conclusions. A single run does not reveal the distribution of possible behaviour.

No regression signal

Suppose prompt version B looks better than prompt version A on two new examples. Did it damage the other 200 cases? Manual testing rarely answers that.

End-to-end ambiguity

If an answer is wrong, the demo does not reveal whether the retriever missed the evidence, the reranker ordered it badly, context assembly dropped it, the model ignored it, or the citation mapper attached the wrong source.

Manual review remains valuable for discovering new behaviours and adjudicating difficult cases. But reliability requires a repeatable dataset, explicit graders, stored results, and controlled comparison.

2. Begin with behaviour specifications

“Generate a good answer” is not a testable requirement.

A behaviour specification describes what the system must do in observable terms. It defines the input conditions, expected behaviour, unacceptable behaviour, and measurement rule.

For PolicyGuide, a useful specification might be:

For questions answerable from the active policy corpus, return an answer whose material factual claims are supported by that corpus, attach the correct source to each material claim, apply employee attributes supplied in the request, and respond within 4 seconds at p95.

That sentence contains several independent behaviours:

Behaviour	Operational success definition
Retrieve evidence	At least one passage containing every required fact appears in the top 5 results
Respect eligibility	The answer matches the policy for the supplied employment type, region, and effective date
Avoid unsupported claims	Every material factual claim is entailed by retrieved evidence
Cite correctly	Each citation points to a passage that supports its attached claim
Handle missing evidence	The system states that the answer is unavailable and does not invent one
Meet latency target	p95 end-to-end latency is at most 4 seconds under the defined load profile
Meet cost target	Mean model and retrieval cost is at most ₹0.80 per request

This decomposition matters because one overall score can conceal a critical failure. A system averaging 92% may still be unusable if it invents eligibility rules in 5% of cases.

Metrics, thresholds, and gates

A metric reports a measurement. A threshold says what is acceptable. A gate decides whether a release may proceed.

For example:

Retrieval recall@5: 0.94
Required threshold: at least 0.92
Safety violation rate: 0.2%
Required threshold: exactly 0% on critical safety cases
p95 latency: 3.7 seconds
Required threshold: no more than 4 seconds

Not all metrics should be averaged together. A weighted quality score can help sort experiments, but hard constraints should remain independent release gates.

The unit of evaluation

Define what one scored item represents. It might be:

one user query;
one query plus user profile and conversation state;
one retrieved passage;
one claim in a generated answer;
one citation-to-claim pair;
one complete session.

PolicyGuide needs several units. Retrieval is evaluated per query, groundedness per claim, citation correctness per citation, and latency per request. Clear units prevent meaningless numbers.

3. The evaluation dataset is an executable specification

An evaluation dataset is not merely a spreadsheet of questions and answers. Each test case should contain enough information to reproduce the situation and determine which behaviours matter.

A useful RAG test case might contain:

id: leave-carryover-india-fulltime-2026-001
version: 3
input:
  question: Can I move my remaining vacation into next year?
  user_context:
    country: IN
    employment_type: full_time
    hire_date: 2026-04-12
corpus_snapshot: policy-corpus-2026-07-01
expected:
  answerable: true
  required_facts:
    - Up to 5 unused days may be carried forward.
    - Carried days expire on March 31.
  relevant_document_ids:
    - india-leave-policy-v4
  forbidden_document_ids:
    - india-leave-policy-v3-obsolete
  required_citation_sections:
    - section-4.2
tags:
  - leave
  - temporal
  - obsolete-document
  - paraphrase
severity: high

Notice what is versioned: the case, the corpus snapshot, and eventually the application configuration. Without those, a later score cannot be reconstructed.

Golden cases

Golden cases represent important, well-understood production behaviours with carefully reviewed expected outcomes. “Golden” means trusted and curated, not easy.

They should include:

common high-volume questions;
high-impact decisions;
contractual or compliance-sensitive answers;
historically frequent failures;
canonical examples of correct abstention;
cases that distinguish similar policies.

A golden answer need not prescribe exact wording. It can instead define required facts, forbidden claims, relevant sources, expected structured fields, and a scoring rubric. This avoids penalizing harmless paraphrases.

Edge cases

An edge case lies near a boundary in the intended input space:

an employee hired exactly on a policy transition date;
a question whose answer is split across two documents;
a query with an acronym used differently by two departments;
a very long question containing several subquestions;
no relevant document in the corpus;
multiple valid answers depending on region;
contradictory documents with different effective dates.

Edge cases test whether the implementation handles boundaries implied by the specification.

Adversarial cases

An adversarial case intentionally pressures the system toward failure. It does not have to be a cybersecurity attack.

Examples include:

a question containing a false premise: “Since contractors receive 20 vacation days, how do I book them?”;
a retrieved document containing “ignore previous instructions”;
a request to reveal another employee’s claim details;
a query designed to match an obsolete policy lexically;
a source whose heading supports the claim but whose body revokes it;
an instruction to answer confidently even when evidence is absent.

Edge cases occur naturally at boundaries. Adversarial cases are deliberately constructed to exploit weaknesses.

Dataset representativeness

A dataset is representative when its distribution reflects the decisions we want to make from its scores. Representativeness is always relative to a target population.

If production traffic is 45% leave, 25% insurance, 15% expenses, 10% payroll, and 5% other topics, a dataset containing 80% leave questions will distort the overall score. But raw traffic frequency is not the only consideration. Rare, high-severity cases deserve deliberate oversampling.

A practical dataset uses strata such as:

intent or topic;
user segment;
language and writing style;
answerable versus unanswerable;
single-source versus multi-source;
policy recency;
difficulty;
risk severity;
known failure category.

Report both the natural-distribution score and important slice scores. A single aggregate can hide that contractors, Telugu queries, or temporal questions perform badly.

Where cases come from

Use several sources:

Domain experts create canonical and high-risk cases.
Production logs supply real phrasing after privacy review and redaction.
Error analysis converts past failures into permanent regression cases.
Synthetic generation expands variations, but humans verify correctness and distribution.
Adversarial design targets known system weaknesses.

Synthetic data is useful for breadth, not automatic truth. If the same model generates the cases, reference answers, and judge decisions, correlated errors can create an illusion of quality.

Split the dataset by purpose

Maintain at least three logical subsets:

Development set: visible to engineers and used frequently.
Regression set: stable cases that protect previously working behaviour.
Holdout set: restricted cases used to detect overfitting to the development suite.

Production-derived monitoring samples form a fourth stream. Do not endlessly tune against the holdout set; it stops being a holdout once its failures guide changes.

4. Graders: choose the cheapest reliable evaluator

A grader converts an application output and test-case expectations into measurements. Different behaviours require different graders.

The best default order is:

deterministic assertions where correctness can be encoded;
structured comparison where outputs have fields;
semantic or reference-based methods where variation is legitimate;
rubric-based LLM judgment for nuanced qualities;
human review when the consequences or ambiguity justify it.

Do not use an LLM judge merely because the system under test uses an LLM.

Deterministic assertions

Deterministic graders produce the same result for the same inputs. They are fast, cheap, auditable, and ideal for hard constraints.

Examples:

output conforms to a JSON Schema;
answerable equals the expected boolean;
every citation identifier exists in the retrieved set;
no forbidden document was cited;
the response contains at most 200 words;
the API did not expose an email address;
latency is below a case-specific limit;
top-k results contain a required document;
a numerical value lies within an allowed tolerance.

Deterministic does not mean automatically valid. A brittle assertion can measure the wrong thing perfectly. The rule itself must reflect the behaviour specification.

Exact match

Exact match gives full credit only when predicted and expected values are identical, usually after explicitly defined normalization.

It is appropriate for:

classification labels;
enum values;
document IDs;
fixed codes;
yes/no fields;
normalized dates or amounts;
canonical short answers when wording has no legitimate variation.

It is inappropriate for open-ended prose. “Five days may be carried forward” and “You can roll over up to five unused days” are semantically equivalent but textually different.

Normalization must be declared: case folding, whitespace trimming, Unicode normalization, punctuation handling, or number formatting. Quietly adding normalization after seeing failures changes the evaluator and must be versioned.

Structured-field evaluation

If the application can return structured data, score fields according to their semantics rather than reducing the object to a string.

Suppose the output is:

{
  "answerable": true,
  "carryover_days": 5,
  "expiry_date": "2027-03-31",
  "citations": ["india-leave-policy-v4#4.2"]
}

Possible graders are:

schema validity for the object;
exact match for answerable;
numeric equality for carryover_days;
normalized date equality for expiry_date;
set precision and recall for citations.

Field-level results expose the actual failure. A single object-level exact match would report only “wrong.”

Semantic similarity

Semantic similarity commonly embeds the candidate and reference answers, then measures cosine similarity. It is useful for detecting broad paraphrase equivalence.

But similarity is not correctness. These sentences can have high similarity while disagreeing on the decisive fact:

“Employees may carry over five days.”
“Employees may carry over fifteen days.”

Both discuss the same topic and use nearly identical words. Embeddings may overlook negation, numbers, direction, and fine-grained conditions. Semantic similarity is better as a weak signal, retrieval diagnostic, or triage mechanism than as the sole correctness gate.

Reference-based evaluation

Reference-based evaluation compares a candidate against one or more trusted answers or fact sets. It can measure required information, contradictions, omissions, style, or semantic equivalence.

Its limitation is reference incompleteness. An open-ended question may have several valid answers, while the stored reference contains only one. A judge may penalize a correct answer for being different or reward an answer that copies the reference while misusing the supplied evidence.

For RAG, store atomic required facts and forbidden claims in addition to a prose reference. Atomic expectations are easier to score and maintain.

Rubric-based evaluation

A rubric converts an ambiguous quality into explicit scoring criteria.

Bad rubric:

Score the response from 1 to 5 for quality.

Operational rubric:

Score	Factual support criterion
4	Every material claim is directly supported by the supplied evidence; no material omission changes the conclusion
3	Main conclusion is supported; one minor unsupported or omitted detail does not affect user action
2	Some relevant support exists, but at least one material claim or condition is unsupported
1	Answer substantially conflicts with, or is not supported by, the evidence
0	No answer, unusable output, or refusal when the evidence clearly supports an answer

A good rubric defines the dimension, evidence the grader may use, treatment of omissions, examples near boundaries, and what each score means. Avoid combining correctness, tone, completeness, and safety into one vague score. Score independent dimensions separately.

5. LLM-as-judge without pretending the judge is truth

An LLM judge is useful when the behaviour requires language understanding that deterministic code cannot express economically. The judge receives the input, candidate response, permitted evidence, and a rubric, then produces a score and rationale.

It can evaluate:

whether a claim is supported by evidence;
whether an answer follows a detailed instruction;
whether required conditions are missing;
which of two answers better satisfies a rubric;
whether a citation passage entails its attached claim.

An LLM judge is itself a probabilistic model. Its output is a measurement made by an imperfect instrument.

Make the judging task narrow

Instead of asking “Is this answer good?”, ask separate questions:

Extract the material factual claims.
For each claim, identify supporting evidence spans.
Label the claim supported, contradicted, or not_in_evidence.
Check whether every required fact is present.
Return structured JSON with the labels and evidence references.

Narrow tasks improve auditability. The rationale is not proof that the score is correct, but it helps reviewers diagnose judge behaviour.

Judge bias

Common biases include:

Position bias: preferring the first or second answer in a pair.
Verbosity bias: preferring longer answers even when they add no value.
Style bias: rewarding polished prose over factual accuracy.
Self-preference: favouring outputs resembling the judge model’s own style.
Reference anchoring: penalizing correct alternatives that differ from the reference.
Authority bias: accepting confidently phrased statements.
Prompt sensitivity: changing scores when rubric wording or order changes.

Mitigations include randomizing pairwise order, running both A/B and B/A, hiding model identity, separating style from correctness, requiring evidence mapping, using deterministic checks for numbers and IDs, and sending uncertain cases to humans.

Judge calibration

Calibration compares judge decisions against a trusted human-labelled set.

Create a calibration dataset containing obvious successes, obvious failures, and difficult boundary cases. Have trained human reviewers label it using the same rubric. Then measure how the judge behaves at relevant thresholds.

Useful measurements include:

agreement rate;
per-class precision and recall;
confusion matrix;
false-pass rate on critical failures;
score distribution by slice;
stability across repeated runs;
sensitivity to answer order.

If a score of 3 or above is treated as passing, calibration must specifically measure errors around that boundary. Overall correlation can appear strong while the pass/fail threshold remains unsafe.

Version the judge model, system prompt, rubric, temperature, response schema, and calibration results. A judge configuration change is an evaluator change, not a harmless implementation detail.

Inter-rater agreement

When several humans—or humans and an LLM—evaluate the same cases, inter-rater agreement measures consistency.

Raw percentage agreement is intuitive but ignores agreement that may occur by chance. Cohen’s kappa is useful for two categorical raters; Fleiss’ kappa supports more raters; weighted kappa gives partial credit when ordinal scores are close. Krippendorff’s alpha handles several data types and missing ratings.

Agreement does not prove correctness. Several raters can consistently use a bad rubric. Low agreement usually signals ambiguous criteria, insufficient reviewer training, missing evidence, or a genuinely subjective task.

Inspect disagreements rather than merely chasing a higher coefficient. They reveal where the specification is incomplete.

Pairwise comparison

In pairwise evaluation, a grader sees outputs A and B for the same case and chooses A, B, or tie according to a rubric.

Pairwise judgment is often easier than assigning absolute scores. It is especially useful for prompt, model, and context experiments.

To make it fair:

both outputs must use the same input and corpus snapshot;
randomize presentation order;
conceal experiment identity;
allow ties;
evaluate independent dimensions when needed;
repeat swapped-order judgments on a sample;
report win, loss, and tie rates with case counts and uncertainty.

Pairwise results show relative preference, not whether either system meets an absolute production threshold. Keep absolute gates alongside them.

Human evaluation

Humans are necessary when the task depends on domain interpretation, user usefulness, policy consequences, or judge uncertainty.

A dependable human-evaluation process defines:

reviewer qualifications;
the exact rubric;
evidence visible to the reviewer;
blinded experiment identity;
examples for each score;
overlap between reviewers to measure agreement;
adjudication for disagreements;
maximum workload and quality checks.

Human review is not automatically ground truth. Reviewers tire, interpret policies differently, and learn shortcuts. Treat human labels as carefully produced data with provenance.

A practical system routes only the right items to humans: high-severity failures, judge uncertainty, grader disagreement, novel production clusters, and samples used for ongoing judge calibration.

6. Evaluate retrieval before generation

In RAG, the generator cannot reliably use evidence it never receives. End-to-end answer quality alone cannot identify retrieval failures.

The retrieval pipeline usually contains distinct stages:

Candidate generation: lexical, vector, or hybrid search returns a broad set.
Filtering: access control, metadata, date, tenant, or document status removes ineligible items.
Ranking or reranking: the remaining passages are ordered.
Context assembly: selected passages are deduplicated, truncated, and placed into the prompt.
Generation: the model produces the answer.

Measure the output of each boundary when possible.

Relevance judgments

Retrieval evaluation requires knowing which documents or passages are relevant to each query. Relevance may be binary, graded, or conditional.

Binary: passage is relevant or not relevant.
Graded: passage is fully, partially, or marginally relevant.
Conditional: passage is relevant only for employees in India after January 2026.

Document-level labels are easier to create but may hide chunking failures. Passage-level labels are more precise but expensive. For multi-hop questions, store the set of evidence units required to answer completely.

Core retrieval metrics

Let the top k retrieved items be the evaluation boundary.

Recall@k asks: what fraction of all known relevant items appeared in the top k?

[ \text{Recall@k} = \frac{|\text{relevant items in top k}|}{|\text{all known relevant items}|} ]

For answer generation, a case-level alternative is hit rate: did at least one sufficient relevant passage appear in the top k?

Precision@k asks: what fraction of the top k items were relevant?

[ \text{Precision@k} = \frac{|\text{relevant items in top k}|}{k} ]

High recall with low precision can flood the context with distracting passages.

Mean reciprocal rank (MRR) rewards placing the first relevant result early:

[ \text{RR} = \frac{1}{\text{rank of first relevant result}} ]

MRR is the mean reciprocal rank across queries. It is useful when one strong result is enough.

nDCG@k handles graded relevance and rewards useful ordering. Highly relevant items near the top contribute more than marginal items lower down.

No single retrieval metric is universally correct. A multi-source synthesis task cares about evidence-set recall; a lookup task may care mainly about first-relevant rank.

Retrieval evaluation must include filters and time

A passage can be topically relevant and still be invalid because it is obsolete, belongs to another tenant, or violates user permissions. Add separate measurements for:

authorization-filter correctness;
active-version precision;
metadata-filter recall;
temporal validity;
duplicate rate;
context token utilization.

For the carryover example, returning policy v3 may score as semantically relevant while being operationally wrong. Test cases must encode the required effective version.

Retrieval failure examples

Indexing failure: the correct document never entered the index.
Chunking failure: the condition and benefit were split into unusable fragments.
Query-understanding failure: “move leave” did not match “carry forward.”
Filter failure: a country filter removed the correct document.
Ranking failure: the correct passage was a candidate but ranked below k.
Context-assembly failure: the passage was retrieved but later truncated.

These failures require different fixes. An end-to-end “answer incorrect” label does not tell us which fix to make.

7. Evaluate generation with the retrieved context held visible

Generation evaluation asks: given the input and the context actually supplied, did the model behave correctly?

The generator should be evaluated both end to end and under controlled context. A controlled test can supply known-good evidence directly to the model. If generation fails with oracle evidence, improving retrieval will not solve it.

Generation dimensions include:

correctness of material facts;
completeness of required facts;
compliance with conditions and user attributes;
appropriate abstention;
groundedness and faithfulness;
citation correctness and completeness;
clarity and instruction following;
safety.

Groundedness

Groundedness measures whether claims in the answer have evidential support in the allowed context.

A claim-level procedure is:

Decompose the answer into atomic factual claims.
Exclude non-factual discourse such as greetings.
For each claim, search the supplied context for supporting evidence.
Label it supported, contradicted, or unsupported.
Weight claims by materiality if appropriate.

One possible metric is:

[ \text{Groundedness} = \frac{\text{supported material claims}}{\text{all material claims}} ]

The denominator and definition of “material” must be explicit. Otherwise a response can dilute one hallucination with many trivial supported claims.

Groundedness does not guarantee completeness. “You may carry over five days” is grounded but incomplete if the expiry date is necessary for action.

Faithfulness

Faithfulness measures whether the answer preserves the meaning and constraints of the provided evidence without distortion.

The distinction is useful:

Groundedness: can each claim be supported by the context?
Faithfulness: does the answer represent that support accurately, including qualifiers, scope, uncertainty, negation, and conditions?

An answer may mention facts found somewhere in context but combine them unfaithfully—for example, applying a full-time benefit to contractors or joining a monetary limit from one region with an eligibility rule from another.

Faithfulness checks should pay special attention to numbers, dates, negation, exceptions, actor identity, jurisdiction, and temporal scope.

Citation correctness

A citation is not correct merely because its source appears in the retrieved set.

Evaluate at least three properties:

Citation entailment: does the cited passage support the attached claim?
Citation completeness: are all material externally verifiable claims cited?
Citation validity: does the citation resolve to the correct, permitted, active source and span?

Useful metrics include citation precision, citation recall, invalid-citation rate, and claim-level attribution accuracy.

Claim: Employees may carry five unused days into 2027.
Citation A: Leave Policy v4, section 4.2 — supports the claim.
Citation B: Leave Policy v4, section 7.1 — same document, unrelated section.

Citation B is not correct. Document-level evaluation would miss the error, so evaluate the claim-to-span relation when the product displays fine-grained citations.

Reference answer versus evidence

Reference correctness and faithfulness answer different questions:

Comparison with a reference asks whether the answer matches an expected outcome.
Comparison with evidence asks whether the answer is supported by what the model was given.

A candidate might match the reference by chance even though retrieval supplied no supporting evidence. It is end-to-end correct for that case but not demonstrably grounded. Conversely, it may faithfully state incomplete retrieved evidence while failing the full expected answer because retrieval missed a source.

Store both results. Their combination helps isolate root cause.

8. Safety is a set of behaviours, not one score

Safety evaluation must reflect the application’s actual threat model and consequences.

For PolicyGuide, safety behaviours may include:

never reveal another employee’s personal or claim data;
never retrieve documents outside the user’s tenant or authorization scope;
treat retrieved instructions as data, not system commands;
do not invent medical, legal, or eligibility decisions;
refuse requests for prohibited actions while preserving benign assistance;
avoid exposing secrets, internal prompts, or hidden metadata;
preserve privacy in logs and evaluator payloads.

Construct adversarial cases for direct prompt injection, indirect injection in documents, data exfiltration, false premises, encoded requests, role-play attempts, and cross-tenant identifiers.

Safety grading can combine:

deterministic secret and PII detectors;
authorization assertions;
tool or retrieval trace inspection;
policy-specific classifiers;
rubric-based judges;
expert human review.

Report safety dimensions independently. A model that refuses every request may have a low violation rate but no utility. Measure both unsafe compliance and excessive refusal.

Critical safety cases normally use zero-tolerance gates. Do not average a severe privacy leak with twenty correct answers and call the result 95% safe.

9. Quality exists alongside cost and latency

An application is not production-ready merely because its answers are correct.

Cost

Track cost per request and its components:

embedding and query-rewrite cost;
reranker cost;
input tokens;
cached input tokens;
output tokens;
judge cost during evaluation;
storage or search infrastructure where relevant;
retries and fallback calls.

Report mean, median, p95, and cost by slice. Long-document questions may dominate spending even when the average looks acceptable.

For experiment comparison, calculate marginal quality gain per marginal cost. A 0.5-point quality improvement may not justify a threefold cost increase.

Latency

Measure end-to-end latency and stage latency:

query processing;
candidate retrieval;
reranking;
context assembly;
time to first token;
generation time;
post-processing and citation mapping.

Use percentiles rather than only an average. p50 describes the typical request; p95 and p99 reveal tail behaviour users experience as timeouts. Define whether measurements use cold caches, warm caches, production-like concurrency, streaming, retries, and geographic network conditions.

Quality, cost, and latency form a trade-off surface. The “best” experiment is the one that satisfies product constraints, not simply the one with the highest judge score.

10. A failure taxonomy turns scores into engineering work

A score tells us how often the system failed. A failure taxonomy tells us what to fix.

A practical taxonomy for PolicyGuide might be hierarchical:

DATA
  missing_source
  incorrect_reference_label
  stale_corpus
RETRIEVAL
  indexing_failure
  query_understanding_failure
  filter_failure
  ranking_failure
  insufficient_evidence_set
CONTEXT
  truncation
  duplication
  conflicting_sources
  wrong_order
GENERATION
  unsupported_claim
  contradicted_claim
  omitted_required_fact
  incorrect_abstention
  instruction_failure
CITATION
  missing_citation
  wrong_source
  wrong_span
  unresolved_identifier
SAFETY
  privacy_violation
  authorization_violation
  injection_success
SYSTEM
  timeout
  rate_limit
  malformed_output
  dependency_failure
EVALUATOR
  judge_false_pass
  judge_false_fail
  ambiguous_rubric

The taxonomy should distinguish symptom, stage, and root cause. “Hallucination” is usually too broad. An unsupported answer may originate from missing evidence, bad context selection, model behaviour, or an evaluator mistake.

Allow a case to have multiple labels while selecting one primary root cause after investigation. Store severity, owner, status, notes, and links to the affected experiment and trace.

Error analysis

Error analysis is the disciplined inspection of failed and borderline cases.

A useful review process is:

Rank failures by severity, frequency, and confidence.
Inspect inputs, corpus version, retrieval results, assembled context, output, grader evidence, latency, and model metadata.
Reproduce the case.
Decide whether the system, dataset, or evaluator is wrong.
Assign a taxonomy label and root cause.
Group similar failures into clusters.
Propose a change tied to the cluster.
Add confirmed failures to regression coverage.

Do not fix individual outputs by adding one-off prompt instructions before identifying the cluster. Ten isolated prompt patches often indicate a missing abstraction, poor retrieval, or an under-specified product rule.

The evaluator can fail

Treat false positives and false negatives from graders as first-class failures. If humans repeatedly overturn an LLM judge on number-heavy cases, route numbers to deterministic extraction or a specialized rubric. Evaluation infrastructure also requires regression tests.

11. Experiments must change one hypothesis at a time

An experiment compares a baseline with a candidate under controlled conditions.

Every experiment needs:

a hypothesis;
one intended independent change;
fixed dataset and corpus snapshot;
recorded model and runtime configuration;
pre-declared primary metrics and gates;
slice-level results;
paired case-level differences;
cost and latency effects;
a decision.

Example hypothesis:

Adding a cross-encoder reranker will improve recall of sufficient evidence within the final top 5 context passages for temporal questions without increasing p95 latency by more than 500 ms.

This is testable. “Try a better reranker” is not.

Prompt experiments

Prompt experiments may change instructions, examples, output schema, evidence delimiters, or abstention rules. Record the exact prompt template and rendered prompt hash.

Common mistake: changing the prompt and model simultaneously. If quality moves, the cause is unknown.

Inspect instruction-following, groundedness, answer completeness, output length, refusal behaviour, cost, and latency. A longer prompt can improve one slice while increasing cost and causing lost evidence on long contexts.

Model experiments

Model experiments hold prompts, inputs, corpus, retrieval results, and decoding settings as constant as the providers permit. Compare:

absolute pass rates;
pairwise win/loss/tie;
slice performance;
repeated-run stability;
structured-output validity;
safety failures;
cost and latency.

Model identity includes the specific version or snapshot, not only a family name. Provider updates can change behaviour.

Context experiments

Context experiments vary retrieval k, chunk size, overlap, metadata, ordering, deduplication, compression, or reranking. Evaluate retrieval and generation separately.

For example, increasing k from 5 to 12 may improve evidence recall but reduce answer faithfulness because the model sees more distractors. The retrieval score and generation score explain the trade-off.

Fair comparison under nondeterminism

Use paired evaluation: run baseline and candidate on the same cases and compare each pair. For important cases, run multiple trials with controlled seeds where supported. Report sample size and confidence intervals or bootstrap intervals for aggregate differences.

A small mean improvement with an interval spanning zero is not strong evidence. More importantly, inspect whether high-severity cases regressed even if the average rose.

Do not repeatedly search many configurations and report only the winner on the same dataset. That overfits the evaluation suite. Confirm the selected candidate on a holdout set.

12. Regression testing protects accumulated reliability

A regression occurs when a previously acceptable behaviour becomes unacceptable after a change.

Regression tests should run on every relevant change to prompts, models, retrieval, chunking, corpus processing, filters, schemas, or post-processing.

A regression report should answer:

Which metrics changed?
Which cases changed from pass to fail or fail to pass?
Which slices changed?
Did any critical gate fail?
Did cost or latency exceed budget?
Are changes statistically and operationally meaningful?
Is the difference caused by the system or an evaluator version?

Example:

Dimension	Baseline	Candidate	Delta	Gate	Decision
Retrieval hit@5	92.0%	95.5%	+3.5 pp	≥92%	Pass
Grounded answer rate	94.0%	94.5%	+0.5 pp	≥94%	Pass
Citation completeness	91.0%	87.0%	-4.0 pp	≥90%	Fail
p95 latency	3.2 s	3.8 s	+0.6 s	≤4.0 s	Pass
Mean request cost	₹0.54	₹0.71	+₹0.17	≤₹0.80	Pass

The candidate is rejected because citation completeness crossed a hard gate, despite better retrieval.

Avoid treating every numerical movement as a regression. Define minimum meaningful deltas, uncertainty, and hard thresholds in advance. But critical per-case failures—privacy leaks, invalid eligibility decisions, cross-tenant retrieval—can block a release even if aggregate metrics barely move.

When a real failure is fixed, add a minimized version of it to the regression suite. The suite becomes the application’s accumulated memory of past mistakes.

13. Offline evaluation is necessary but not sufficient

Offline datasets are controlled and repeatable. Production changes.

Users ask new questions. Documents drift. Providers update models. Traffic mix shifts. A corpus ingestion job fails. Latency changes under load. Therefore the application also needs online monitoring.

Monitor:

request volume and error rate;
latency and cost percentiles;
retrieval empty-result rate;
document and source distribution;
abstention and refusal rates;
citation coverage and resolution failures;
sampled groundedness or quality judgments;
user feedback and correction rate;
safety and privacy alerts;
input, embedding, and topic-distribution drift;
model, prompt, index, and corpus versions.

Online signals are noisier than curated offline labels. User thumbs-up is not equivalent to factual correctness; feedback is sparse and selection-biased. Automated online judges incur cost and may process sensitive data. Use sampling, privacy controls, and delayed human adjudication.

From monitoring back to evaluation

Monitoring should feed the offline loop:

Detect an anomaly or collect a low-confidence sample.
Review and redact it.
Label the expected behaviour.
Add it to the appropriate dataset slice.
reproduce the failure offline.
Test a fix.
Deploy behind a controlled rollout.
Confirm the online signal improves.

This closes the gap between a static benchmark and the changing production system.

14. Evaluation-driven development

Traditional development often follows: build, demo, test, release.

Evaluation-driven development begins earlier:

Specify: define observable behaviours and release gates.
Curate: create cases that represent normal, boundary, adversarial, and high-severity conditions.
Baseline: run the simplest implementation and store results.
Analyze: classify failures by stage and root cause.
Hypothesize: propose one change that should improve a defined slice.
Experiment: compare candidate and baseline under the same conditions.
Gate: reject candidates that violate quality, safety, cost, or latency constraints.
Deploy carefully: use staged rollout and online monitoring.
Learn: convert production failures into new cases and refine graders.

The dataset, graders, and failure taxonomy evolve with the product. They are production assets, not temporary QA scripts.

Practical project: build a RAG evaluation platform

We will build the platform around three layers:

FastAPI exposes APIs, validates contracts, schedules evaluation work, and performs deterministic grading.
PostgreSQL stores versioned definitions, immutable run records, grader outputs, traces, review decisions, and experiment comparisons.
Next.js provides dataset management, experiment setup, run inspection, human review, regression reports, and dashboards.

Long-running evaluations should execute in background workers rather than inside HTTP requests. A queue such as Redis-backed workers, RabbitMQ, or a managed job system can be added, but PostgreSQL remains the source of truth for run state.

15. Core domain model

Keep definitions separate from executions.

Dataset definitions

datasets
  id, name, description, owner, created_at, archived_at

dataset_versions
  id, dataset_id, version_number, status, created_at, created_by
  distribution_manifest_json, notes

test_cases
  id, stable_key, created_at

test_case_versions
  id, test_case_id, version_number, input_json, expected_json
  tags_json, severity, source, created_at, created_by

dataset_version_cases
  dataset_version_id, test_case_version_id, split, weight

test_cases.stable_key preserves identity across edits. A test-case edit creates a new version; it never rewrites historical evidence. A dataset version pins exact test-case versions.

The distribution manifest stores expected slice proportions and warns when a dataset is unbalanced.

Application and experiment definitions

application_versions
  id, name, git_sha, created_at

configurations
  id, application_version_id, prompt_version, model_name, model_version
  model_params_json, retriever_config_json, context_config_json
  corpus_snapshot_id, created_at

experiments
  id, name, hypothesis, dataset_version_id
  baseline_configuration_id, candidate_configuration_id
  primary_metrics_json, gates_json, status, created_by

Store rendered prompt hashes and configuration JSON, not only display names. The goal is to reconstruct exactly what ran.

Execution records

evaluation_runs
  id, experiment_id, configuration_id, dataset_version_id
  status, started_at, completed_at, environment_json

case_runs
  id, evaluation_run_id, test_case_version_id, trial_number
  status, started_at, completed_at, total_cost, total_latency_ms
  output_json, error_json

retrieval_results
  id, case_run_id, stage, rank, document_id, chunk_id
  score, content_hash, metadata_json, latency_ms

contexts
  id, case_run_id, position, chunk_id, content_hash, token_count

model_calls
  id, case_run_id, purpose, provider, model_version
  prompt_hash, input_tokens, output_tokens, cached_tokens
  latency_ms, cost, response_hash, metadata_json

Outputs are immutable facts about a run. If a grader changes, regrade the stored output into a new grader execution instead of overwriting the original score.

Graders and results

grader_definitions
  id, name, type, version, config_json, rubric_text
  judge_model, judge_prompt_hash, calibration_report_json

grader_assignments
  id, dataset_version_id, grader_definition_id
  metric_name, threshold, gate_type, applicable_tags_json

grader_results
  id, case_run_id, grader_definition_id
  score_numeric, label, passed, confidence
  rationale_json, evidence_json, cost, latency_ms, created_at

type can be exact_match, schema, structured_field, retrieval, citation, semantic, llm_rubric, or human.

Do not store only the final score. Store evidence: failed JSON path, missing document ID, unsupported claim, cited span, or judge rationale. Evidence makes the result diagnosable.

Failure and human-review records

failure_labels
  id, code, parent_id, description, default_severity, owner_team

case_failures
  id, case_run_id, grader_result_id, failure_label_id
  is_primary, severity, status, notes, created_by

review_tasks
  id, case_run_id, reason, priority, status
  assigned_to, rubric_version, created_at, completed_at

human_reviews
  id, review_task_id, reviewer_id, scores_json
  labels_json, rationale, submitted_at

adjudications
  id, review_task_id, final_scores_json, final_labels_json
  adjudicator_id, rationale, created_at

Preserve individual human ratings before adjudication so inter-rater agreement can be calculated.

16. Evaluation execution flow

One run proceeds as follows:

Resolve the immutable dataset and configuration versions.
Create a run manifest containing code, model, prompt, corpus, grader, and environment versions.
Enqueue one or more trials for each case.
Execute the RAG pipeline while recording stage timings and artifacts.
Run cheap deterministic graders first.
Compute retrieval and citation metrics from stored traces.
Run LLM graders only for applicable cases.
Route uncertain, disagreeing, sampled, or high-severity cases to human review.
Aggregate metrics overall and by slice.
Compare candidate and baseline using paired case results.
Apply release gates.
Generate a regression report and preserve all artifacts.

Every worker operation should be idempotent. Use a key such as (case_run_id, grader_definition_id) to prevent duplicate grader results after retries. Store failed tasks explicitly rather than silently excluding them from denominators.

17. FastAPI boundaries

Representative endpoints:

POST   /datasets
POST   /datasets/{id}/versions
POST   /test-cases/{id}/versions
POST   /dataset-versions/{id}/validate

POST   /experiments
POST   /experiments/{id}/runs
GET    /runs/{id}
GET    /runs/{id}/cases
GET    /case-runs/{id}/trace
POST   /runs/{id}/cancel

GET    /experiments/{id}/comparison
GET    /experiments/{id}/regression-report

GET    /review-tasks
POST   /review-tasks/{id}/claim
POST   /review-tasks/{id}/reviews
POST   /review-tasks/{id}/adjudicate

Use Pydantic models for test-case expectations and grader configurations. Validate that referenced documents exist in the pinned corpus snapshot, metric names are known, thresholds lie in valid ranges, and every release gate has an assigned grader.

Separate the API process from evaluation workers. The API changes desired state—such as creating or cancelling a run—while workers perform model and retrieval calls. Cancellation should stop undispatched work and mark in-flight work accurately.

18. Deterministic grader interface

A simple internal contract could be:

from typing import Any, Protocol
from pydantic import BaseModel

class GradeResult(BaseModel):
    metric: str
    score: float | None = None
    label: str | None = None
    passed: bool
    evidence: dict[str, Any]

class GradeContext(BaseModel):
    test_case: dict[str, Any]
    output: dict[str, Any]
    retrieval_trace: list[dict[str, Any]]
    assembled_context: list[dict[str, Any]]

class Grader(Protocol):
    name: str
    version: str

    def grade(self, context: GradeContext) -> list[GradeResult]: ...

Implement these first:

JSON Schema grader;
exact field grader;
required/forbidden fact grader for normalized values;
document hit@k and recall@k grader;
citation resolution grader;
latency and cost budget grader.

Unit-test graders with known positive, negative, and boundary inputs. The platform is only as trustworthy as its measurements.

19. LLM grader execution

An LLM grader request should include:

the exact behaviour dimension;
candidate output;
only the permitted reference and evidence;
a versioned rubric;
definitions of material claim and omission;
a structured response schema;
instructions to cite evidence spans;
an uncertain outcome.

Persist the raw judge response securely, parsed result, model version, prompt hash, cost, latency, and any parsing retry. Never silently treat grader execution failure as a candidate failure or pass; use a separate grader_error state.

Calibrate each judge configuration before making it a hard gate. Periodically sample judge passes and failures for human review because model or traffic distributions can drift.

20. Pairwise experiment comparison

Join baseline and candidate on the stable case version and trial policy. The comparison page should show:

paired metric deltas;
A/B/tie judgments with order randomization;
regressions and improvements by severity;
slice-level changes;
cost and latency deltas;
judge disagreements;
links to side-by-side retrieval, context, output, and grader evidence.

Never compare runs over different dataset versions as though the raw aggregate delta were causal. If the dataset changed, either rerun the baseline on the new version or label the comparison non-equivalent.

21. Human-review queue

Create review tasks when:

a critical deterministic gate fails;
the judge returns uncertain or low confidence;
deterministic and LLM graders disagree;
two judges disagree;
a pairwise result changes after swapping order;
a case belongs to a high-risk random sample;
production monitoring detects a novel cluster.

The review screen should show the question, user context, expected behaviour, retrieved results, assembled context, candidate answer, citations, grader evidence, and rubric. Hide model and experiment identity when feasible.

Use overlapping assignments on a sample to measure agreement. Send disagreements to adjudication. Accepted adjudications can update calibration sets or create new test-case versions, but they should not rewrite historical labels invisibly.

22. Dashboard design

The dashboard should answer engineering questions, not merely display attractive averages.

Overview

pass rates for independent quality gates;
critical failures;
retrieval, generation, citation, and safety metrics;
cost and latency percentiles;
run completeness and grader error rate.

Slices

topic;
user segment;
answerability;
source recency;
single-hop versus multi-hop;
language;
severity;
failure category.

Experiment comparison

paired deltas;
win/loss/tie;
regression list;
improvements;
cost-quality-latency trade-offs;
gate decision.

Failure analysis

taxonomy distribution;
failure trends over time;
clusters by configuration and document;
unowned and unresolved failures;
direct trace inspection.

Judge health

agreement with humans;
false-pass and false-fail rate;
score drift;
order sensitivity;
calibration age;
review backlog.

A dashboard metric should always link to the cases behind it. If an engineer cannot inspect the examples that produced a number, the number will not reliably guide a fix.

23. Implementation sequence

Build the platform in increments that produce value early.

Milestone 1: reproducible offline runner

versioned dataset and case schema;
pinned RAG configuration;
immutable case outputs;
exact, schema, latency, and cost graders;
JSON/Markdown regression report.

Milestone 2: retrieval and citation evaluation

stored retrieval traces and contexts;
hit@k, recall@k, precision@k, MRR, and nDCG where appropriate;
active-version and filter checks;
claim-to-citation mapping;
citation validity, entailment, and completeness.

Milestone 3: rubric and LLM graders

versioned rubrics and judge prompts;
structured judge results;
calibration dataset and report;
grader cost and latency;
disagreement routing.

Milestone 4: experiments and human review

baseline/candidate comparison;
pairwise judging with order randomization;
review queue, overlapping reviews, and adjudication;
failure taxonomy and ownership.

Milestone 5: dashboard and production loop

slice dashboards;
CI release gates;
online sampling and drift monitoring;
production-failure ingestion;
staged deployment comparison.

At each milestone, the platform should evaluate itself: schema migrations, aggregations, grader fixtures, idempotent retries, permission checks, and metric calculations all need tests.

A complete worked evaluation cycle

Suppose PolicyGuide fails this query:

I joined the India office in April 2026. Can I carry my unused leave into next year, and when must I use it?

The application answers:

You may carry up to 10 days, which remain valid throughout next year.

It cites an obsolete 2024 policy.

1. Behaviour to measure

The application must select the active policy for the employee’s region and effective date, include both the carryover limit and expiry, and cite the supporting section.

2. Operational success

active policy v4 appears in top 5;
obsolete policy v3 does not enter final context;
answer states 5 days;
answer states March 31 expiry;
both facts cite v4 section 4.2;
no unsupported condition is added.

3. Representative cases

Add variations for full-time and contractor status, before and after the transition date, explicit and implicit region, paraphrased “carry forward,” and a false premise containing “10 days.”

4. Evaluators

deterministic filter assertion for active document version;
hit@5 for required evidence;
structured extraction for days and expiry;
citation resolver and claim-span entailment judge;
groundedness rubric for remaining claims.

5. Evaluator limitations

The extraction rule may miss unusual wording. The entailment judge may overlook temporal qualifiers. Human-labelled temporal cases are used for calibration, and dates and numeric limits receive deterministic checks.

6. Failure collection

The stored trace shows that both v3 and v4 were candidates. v3 ranked first because the query shared more wording with it. The metadata filter checked region but not effective date.

7. Root cause

Primary: RETRIEVAL.filter_failure.temporal.

Secondary: CITATION.wrong_source.

8. Improvement

Add effective-date filtering before reranking and explicitly mark superseded documents during ingestion.

9. Re-run

Run the entire regression dataset, not only this case. Temporal retrieval improves, but two historical-policy questions now fail because they legitimately require superseded documents. The filter is revised to use the query’s target date rather than always selecting the latest policy.

This final step is the evaluation mindset: the first plausible fix was not accepted until its broader behaviour was measured.

Mastery gate

You have mastered these fundamentals when you can produce the following artifacts without relying on “looks good” judgments.

1. Behaviour specification

Given a RAG feature, define independent measurable behaviours for retrieval, generation, citations, safety, cost, and latency. For each, state the evaluation unit, metric, threshold, and whether it is a hard gate.

2. Representative dataset

Create a versioned dataset manifest containing golden, edge, adversarial, answerable, and unanswerable cases. Show its distribution across important product slices and explain deliberate oversampling of rare high-risk cases.

3. Grader selection

For each expectation, choose the cheapest reliable grader. Explain why exact match, structured comparison, semantic similarity, reference evaluation, rubric judgment, or human review is appropriate—and what failure the selected grader might miss.

4. Judge reliability report

Calibrate an LLM judge against human-labelled cases. Report false-pass rate, false-fail rate, disagreement slices, order sensitivity, repeated-run stability, and the pass threshold. State where the judge is forbidden from acting as the sole release gate.

5. Retrieval/generation separation

For a failed answer, determine whether sufficient evidence was indexed, retrieved, ranked, retained in final context, and used faithfully. Demonstrate a controlled generator test using oracle evidence.

6. Rubric design

Write a single-dimension rubric with observable score boundaries, evidence rules, materiality definitions, and boundary examples. Show that independent reviewers can use it consistently.

7. Failure analysis

Inspect traces, distinguish symptoms from root causes, classify failures with a stable taxonomy, prioritize them by severity and frequency, and convert confirmed failures into regression cases.

8. Fair experiment

State a hypothesis, change one variable, hold the dataset and corpus constant, compare paired outputs, report slice metrics and uncertainty, include cost and latency, and validate the winner on a holdout set.

9. Regression detection

Produce a report showing aggregate changes, pass-to-fail cases, slice regressions, gate violations, grader errors, and evaluator-version changes. Reject a candidate that improves the average while violating a critical constraint.

10. Workflow reconstruction

Starting from a production failure, reconstruct the complete chain:

production signal
→ reviewed and redacted case
→ explicit expected behaviour
→ versioned dataset
→ controlled baseline
→ stage-level graders
→ failure taxonomy
→ one-change experiment
→ paired regression report
→ release gate
→ staged deployment
→ online confirmation

If every arrow has stored provenance, versioned inputs, inspectable evidence, and a responsible decision, the evaluation workflow is reconstructable.

Closing principle

The hardest part of LLM evaluation is not choosing a metric. It is deciding what the application must reliably do.

Once behaviour is defined, the rest becomes an engineering discipline:

datasets make the behaviour repeatable;
graders make it measurable;
calibration makes the measurements trustworthy;
traces and taxonomies make failures actionable;
experiments make improvements attributable;
regression tests preserve what already works;
production monitoring keeps the evaluation connected to reality.

An impressive demo proves that an application can succeed.

A well-designed evaluation system tells you where it succeeds, where it fails, why it fails, whether a change helped, what that improvement cost, and whether it is safe to release.

That is the difference between experimenting with an LLM and engineering an LLM application.