The Most Important LLM Optimization May Be a Better Dataset

How production failures become reliable training and evaluation data

An LLM application has stopped improving.

The team has rewritten the system prompt six times. It has added more examples, changed the wording of tool descriptions, adjusted temperature, and switched between two models. Every change fixes a demo and breaks another. Offline scores move by a point or two, but users still report the same failures: the agent chooses the wrong tool, retrieves an irrelevant policy, invents a parameter, or produces an answer that looks polished but does not complete the task.

This is usually described as a model or prompt problem. Often it is an evidence problem.

The team does not yet know precisely which behaviour fails, how often it fails, for whom it fails, or what a correct trajectory should have looked like. Prompt changes cannot reliably optimize a behaviour that has not been specified, measured, and represented in data.

Data-centric generative AI begins with a different loop:

Observe a failure, specify the missing behaviour, collect representative evidence, turn that evidence into a controlled dataset, run the smallest appropriate experiment, and measure whether the deployed system actually improved.

“Collect more data” is not the strategy. The strategy is to collect the right evidence under a documented contract.

1. Data is part of the system, not fuel poured into a model

In an ordinary software system, behaviour lives mainly in code. In a generative AI system, behaviour is distributed across several components:

model weights;
prompts and examples;
retrieval corpora, indexes, and ranking logic;
tool schemas and permissions;
workflow and stopping rules;
training data;
evaluation data;
the production distribution on which the system is used.

This makes data a system component. A customer-support agent may fail because the model cannot follow a refund policy, but it may also fail because the relevant policy was absent from the index, the retriever selected an obsolete page, the tool result was truncated, or the evaluation set contained only simple refund requests.

The first-principles objective is not “make the model better.” It is:

[ Q = \mathbb{E}{x \sim P{production}}[U(S(x))] ]

Here, (S) is the complete system, (x) is a production case, (P_{production}) is the real usage distribution, and (U) is the utility of the outcome. System quality therefore depends on both behaviour and distribution. A 95% success rate on easy English questions can coexist with a 40% success rate for multilingual users or long documents.

Start with a behaviour specification

A behaviour specification converts a vague complaint into observable conditions.

“The agent is unreliable” is not a specification. A useful specification might be:

When a user asks to cancel an order that has not shipped, the agent must retrieve the currently effective cancellation policy, call get_order before cancel_order, request confirmation before the write action, use the authenticated user's order ID, and report the tool-confirmed result without claiming success if the tool failed.

That specification exposes several measurable behaviours:

retrieval relevance and policy freshness;
tool selection;
tool ordering;
argument correctness;
authorization and confirmation;
grounded final reporting;
recovery after tool failure.

Only now can we derive data requirements. We need positive examples of correct cancellation trajectories, negative examples for invalid states, examples near policy boundaries, tool failures, ambiguous user requests, multilingual phrasing, and cases where cancellation is not allowed. The required unit may be a full trajectory, not an input-output pair.

A practical data-requirements document should state:

Field	Question
Target behaviour	What must become more reliable?
Unit of data	Turn, document, retrieval result, tool call, trajectory, or session?
Population	Which users, languages, products, regions, and time periods matter?
Required evidence	What makes a label defensible?
Expected failure slices	Where do we suspect the behaviour breaks?
Decision enabled	Will the data choose a prompt, retrieval, workflow, or training change?
Acceptance metric	What improvement counts, and what must not regress?

If the data will not support a concrete decision, it probably should not be collected.

2. Production is evidence, but not automatically a dataset

Useful data sources include production traces, user feedback, support tickets, expert-written cases, policy documents, tool logs, incident reports, public benchmarks, curated domain datasets, and synthetic examples. Each source observes a different part of reality and carries a different bias.

Production traces are especially valuable because they show what users actually asked and what the system actually did. A useful production trace should make the run reconstructable:

{
  "trace_id": "tr_...",
  "timestamp": "...",
  "tenant_segment": "...",
  "input_ref": "encrypted-object-ref",
  "prompt_version": "prompt:cancel:v17",
  "model_version": "provider/model/revision",
  "retrieval_snapshot": "kb:2026-08-12",
  "retrieved_document_ids": ["policy:cancel:42"],
  "tool_schema_version": "orders-api:v8",
  "tool_calls": [],
  "latency_ms": 0,
  "token_cost": 0,
  "automated_checks": {},
  "user_feedback": null,
  "outcome": "unknown"
}

Store references to sensitive payloads instead of copying raw content into every analytics system. Record versions of every behaviour-shaping component. Without those versions, a trace says that something failed but cannot tell us which system produced the failure.

Feedback is a signal, not a gold label

User feedback is useful but confounded:

thumbs-down may mean factually wrong, too slow, impolite, or merely unwanted;
no feedback does not mean success;
users who respond are not representative of all users;
a task may complete correctly even when the user dislikes the answer;
a user may reward a confident but incorrect response.

Preserve the raw signal and its context. Do not silently convert thumbs_down = true into label = incorrect. Instead, use feedback to prioritize review or combine it with stronger evidence such as tool outcomes, task completion, follow-up correction, abandonment, or expert adjudication.

Representativeness must be tested

Production data is not necessarily representative of the behaviour we want to improve. Logging may exclude privacy-sensitive workflows. Enterprise tenants may be overrepresented. A recent incident can dominate the sample. Only failed runs omit successful counterexamples. English traffic can hide regional failure rates.

Compare the candidate dataset with production across meaningful slices:

task and intent;
language and locale;
input length and complexity;
customer or product segment;
tool and document source;
time period and policy version;
success, failure, and abstention;
safety risk and business impact.

Coverage is not one number. It is a matrix. A dataset can match the overall language distribution while missing every high-risk multilingual cancellation case.

Keep four stores separate

One raw event can influence several datasets, but the stores must have different contracts:

Monitoring data estimates current production health. It should be recent, time-aware, and sampled to represent live traffic.
Training or development data is allowed to change the system. Prompt examples, retriever tuning cases, and fine-tuning records belong here.
Evaluation data estimates generalization. It must remain hidden from the optimization process and stable enough for comparison.
Investigation data helps diagnose incidents. It may be biased, temporary, and rich in sensitive details; it is not automatically suitable for training or evaluation.

If a case guides a prompt edit, it is no longer a clean test of that edit. Move it to development and replace it with an independently sourced evaluation case.

3. Human annotation is an engineering system

When correctness depends on policy, intent, nuance, or trajectory quality, human annotation supplies evidence unavailable from simple automated checks. But “ask people to label it” merely moves ambiguity from the model to the annotator.

Build a taxonomy around decisions

A label taxonomy is a controlled vocabulary for the distinctions the pipeline needs. For an agent reliability system, a run might have labels on multiple axes:

final outcome: success, partial success, failure, indeterminate;
failure stage: interpretation, retrieval, planning, tool selection, arguments, execution, verification, response;
root cause: missing knowledge, stale knowledge, ambiguous request, model error, tool error, permission denial, orchestration bug;
severity: cosmetic, recoverable, task-blocking, harmful;
correction target: prompt, retrieval corpus, ranking, workflow, tool contract, model adaptation, no action;
confidence: high, medium, low;

These axes should not be collapsed into one enormous class list. “High-severity retrieval failure” is a composition of labels, not necessarily a unique class.

Taxonomies should be:

exhaustive enough to capture decisions;
mutually exclusive where required, with explicit multi-label fields elsewhere;
stable enough for trend analysis;
versioned, because new failure modes will appear;
equipped with other and insufficient_evidence so annotators are not forced to invent certainty.

Annotation guidelines are executable policy for humans

Good annotation guidelines define:

the exact unit being labelled;
the evidence an annotator may use;
each label in operational language;
positive, negative, and boundary examples;
precedence rules when several labels seem valid;
when to abstain;
how to handle missing or contradictory evidence;
how to escalate expert-only cases;
what rationale or corrected answer must be recorded;
which content must not be copied because of privacy or licensing.

For example, distinguish retrieval_failure from reasoning_failure:

Label retrieval_failure when the evidence required for a correct answer was absent from the documents available to the model. Label reasoning_failure when sufficient evidence was present but the output contradicted or failed to use it. If the evidence itself is ambiguous or obsolete, label knowledge_conflict and escalate.

That definition prevents the model from being blamed for a missing document and prevents the retriever from being blamed when the correct passage was already present.

Ambiguity is information

Label ambiguity can come from unclear guidelines, insufficient evidence, genuinely subjective preferences, or cases that sit at a policy boundary. Do not erase this information with majority vote alone.

Store individual annotations, confidence, rationale, and adjudication. A high-disagreement slice may reveal an underspecified product requirement. The correct engineering action may be to clarify policy or let the system ask the user a question—not to train the model harder.

Measure agreement, then diagnose it

Inter-annotator agreement asks whether annotators apply the label system consistently beyond chance. Raw percent agreement is easy to understand but can be misleading when one label dominates. Cohen's kappa for two annotators is:

[ \kappa = \frac{p_o - p_e}{1 - p_e} ]

where (p_o) is observed agreement and (p_e) is agreement expected from the annotators' label frequencies. Use an appropriate statistic for the job—such as Cohen's kappa, Fleiss' kappa, or Krippendorff's alpha—but never treat the coefficient as the diagnosis.

Inspect agreement per label and slice. Review confusion pairs. Discuss disagreements. Revise guidelines. Relabel a calibration batch. Domain experts should adjudicate high-impact cases. A single aggregate agreement score can hide that annotators agree on easy successes and disagree on the exact failure class the experiment needs.

4. Weak supervision: scale judgment without pretending heuristics are truth

Manual labels are expensive, so teams often use weak supervision: noisy signals that cover more data than humans can annotate.

Examples of heuristic labels include:

tool returned an error → probable execution failure;
required citation absent → probable grounding failure;
user immediately rephrased the same request → possible task failure;
response mentions success but no successful write tool call exists → likely false success claim;
policy document retrieved from an expired version → likely stale retrieval;
an evaluator model assigns a low rubric score → candidate for review.

Each heuristic should be a versioned labelling function that can emit a label or abstain. Measure its coverage, estimated precision on a human-labelled calibration set, correlations with other heuristics, and failure slices. Several rules copied from the same underlying detector are not independent votes.

The Snorkel work formalized this pattern: labelling functions can have unknown accuracy and correlated errors, and their outputs can be combined into probabilistic labels rather than treated as ground truth. The important lesson is not a particular library; it is that rule provenance, overlap, conflict, and calibration are part of the dataset. The original Snorkel paper reports this approach in detail.

Weak labels are most useful for triage, pre-labelling, or large training pools. Keep a smaller human-reviewed gold set to estimate their error. Never use the same heuristic both to label the training data and as the only evaluation metric; the system can learn to satisfy the heuristic without improving the intended behaviour.

5. Synthetic data is a controlled experiment, not free truth

Synthetic data is generated rather than directly observed. In generative AI, a teacher model can create instructions, responses, tool trajectories, critiques, counterexamples, or transformations of seed cases.

It is valuable when real examples are rare, privacy prevents reuse, a boundary needs systematic coverage, or a deterministic simulator can verify the answer. It is dangerous when the generator shares the same blind spots as the model being improved.

Derive synthetic examples from a coverage gap

Do not ask a model to “generate 10,000 diverse support questions.” Start with a data requirement:

We have few examples of cancellation requests made after packing but before carrier pickup, especially when the user requests both cancellation and refund, and the policy differs by region.

Then parameterize the missing space:

order state;
region and policy version;
user intent;
ambiguity level;
language and phrasing style;
tool success or failure;
permission state;
expected action.

Sample combinations intentionally, generate cases, and retain those that fill a measured coverage gap.

Teacher-model and self-instruct methods

A teacher-model generation pipeline normally provides a strong model with the behaviour specification, grounded evidence, output schema, and seed examples. The teacher proposes a case and correction; an independent verifier checks what it can.

Self-instruct methods bootstrap instruction data by asking a model to generate new tasks and solutions from seeds, filtering invalid or overly similar generations, and using the retained set for adaptation. The Self-Instruct paper is a useful reference because filtering similar or invalid generations is part of the method, not an optional cleanup step.

For an agent, synthetic generation can vary more than wording. It can create:

alternative user phrasings for the same intent;
hard negatives where a similarly named tool is wrong;
counterfactual tool results;
missing-document and stale-document conditions;
permission denials and transient failures;
adversarial instructions inside retrieved content;
corrected trajectories with required confirmations.

Diversity must be measured

Synthetic-data diversity is not the number of records. A teacher can produce thousands of surface variations that share one reasoning pattern. Measure diversity across semantic clusters, taxonomy cells, languages, lengths, entities, solution strategies, tool sequences, and difficulty.

Useful checks include:

coverage of the parameter grid;
cluster entropy and the share of the largest cluster;
lexical and embedding-distance distributions;
novel combinations relative to seeds;
distinct tool trajectories;
performance differences across slices;
human review of the smallest and largest clusters.

Recursive training on model-generated data can lose rare modes of the original distribution; research on the curse of recursion is a warning against replacing real evidence with self-reproducing model output. Keep real data in the mixture, preserve provenance, and evaluate long-tail behaviour separately.

Filter before the data becomes authoritative

Filtering synthetic data should use several independent gates:

schema and parse validation;
policy or source-grounding checks;
deterministic execution in a simulator where possible;
contradiction and impossible-state checks;
duplication and similarity checks;
teacher-confidence or multi-judge scoring;
difficulty and coverage checks;
spot checks by humans, stratified by generator and failure slice.

For verifiable tasks, execution beats an evaluator model. Run the generated SQL against a sandbox, validate tool arguments against the schema, recompute arithmetic, or replay the trajectory in a simulator. For subjective tasks, use a rubric and periodically calibrate model judges against humans.

Record the teacher model, prompt, temperature, seed IDs, source documents, verifier versions, rejection reasons, and human edits. Synthetic data without generation provenance cannot be debugged.

6. Duplicates, leakage, and contamination can manufacture progress

Exact and near duplicates

Deduplication starts with canonicalization and exact hashes. Normalize only what is irrelevant to the behaviour: perhaps whitespace or generated IDs, but not numbers, negation, policy dates, or tool results that may change the answer.

Exact hashes will miss near duplicates: paraphrases, templated traces with different IDs, or synthetic variations with the same underlying case. Common techniques include:

token shingles with Jaccard similarity;
MinHash and locality-sensitive hashing for scalable candidate retrieval;
SimHash for approximate textual similarity;
embedding nearest neighbours for semantic similarity;
task-specific signatures such as (intent, policy, tool sequence, expected outcome).

Near-duplicate detection is a candidate-generation problem followed by a decision rule. Embedding similarity alone can incorrectly remove useful boundary pairs such as “cancel before shipment” and “cancel after shipment.” Deduplicate within behaviour-aware groups and inspect threshold errors.

Duplicates matter because they distort class balance, overweight common templates, encourage memorization, and leak across splits. Research found that deduplicating language-model data reduced memorized output and train-test overlap while maintaining or improving training efficiency. See Deduplicating Training Data Makes Language Models Better.

Leakage and contamination are lineage failures

Data leakage is any path by which information unavailable at real inference time influences training or model selection. Examples include:

using the eventual tool result as an input feature when the system must predict before calling the tool;
including post-resolution support notes in the prompt;
fitting preprocessing thresholds on the test set;
selecting a prompt after repeatedly inspecting test failures.

Contamination is a particularly important form of leakage in which evaluation content, answers, or close derivatives enter training or development data. Train-evaluation overlap can be exact, semantic, or genealogical. A synthetic paraphrase of a test case is contaminated even when no sentence matches.

Prevent it with split order and lineage:

define immutable evaluation sources first;
assign a lineage_group_id to each real case and every derivative;
split by group, user/session, document, and time as the application requires;
deny evaluation IDs and ancestors to generation jobs;
run exact, fuzzy, semantic, and source-document overlap checks;
quarantine uncertain matches;
record every test exposure to humans and optimization systems;
rotate or replenish evaluation sets when they have influenced development.

Do not generate training variants from evaluation seeds. Do not let an evaluator prompt containing test answers enter the teacher's retrieval index. Benchmark-contamination research exists precisely because overlap can inflate apparent generalization; even developers should report their overlap methodology and statistics. See the analysis in Language model developers should report train-test overlap.

7. Spend annotation effort where it changes the decision

Random sampling is useful for estimating prevalence, but improvement often requires focused sampling.

Hard-example mining and failure clustering

Hard-example mining selects cases the current system gets wrong, handles inconsistently, or solves only with excessive cost. Candidate signals include:

disagreement among models or prompt versions;
low calibrated confidence;
evaluator failure;
repeated retries or unusually long trajectories;
tool-call corrections;
high retrieval-score ambiguity;
user escalation;
success under a stronger system but failure under the deployed one.

Hard cases alone create a distorted training distribution, so keep a representative replay set of ordinary cases. Otherwise, the system can improve on yesterday's failures while regressing on routine traffic.

Failure clustering turns thousands of alerts into recurring families. Embed a structured failure representation—not just the final response—using the user intent, retrieved evidence, tool sequence, verifier outputs, and short redacted summaries. Cluster candidates, then ask humans to name and merge clusters. Algorithms suggest structure; experts decide whether the structure corresponds to an actionable root cause.

A useful cluster is not merely semantically coherent. It should lead to a different intervention. If two clusters both require the same retrieval fix, separating them may not be operationally valuable.

Active learning and uncertainty sampling

Active learning asks which unlabeled examples should be annotated next to improve the system most per unit cost. Uncertainty sampling chooses examples where a classifier or judge is least certain—for example, high entropy or a small margin between the top two labels.

But model confidence is not truth, and uncertainty-only sampling can overselect noise, outliers, or one boundary. Deep active-learning research notes the trade-off between uncertainty and representativeness. See A Survey of Deep Active Learning.

A robust selection score can combine:

[ Priority(x) = aU(x) + bD(x) + cI(x) + dR(x) - eC(x) ]

where (U) is uncertainty, (D) is diversity contribution, (I) is business impact, (R) is underrepresented-slice value, and (C) is annotation cost. Maintain a random sample alongside active selections so prevalence estimates remain unbiased.

Curriculum, balance, coverage, and the long tail

Curriculum design controls which examples appear and in what mixture or order. A sensible curriculum may begin with clean canonical cases, add variations and boundary conditions, then add noisy production and adversarial cases. The purpose is not always “easy to hard”; it is to expose the learning system to a stable representation of the behaviour before emphasizing rare corrections.

Class balancing prevents a dominant class such as success from overwhelming rare but important failures. Balance by sampling, weighting, or targeted collection—but preserve natural prevalence in the evaluation set used to estimate user impact. Training balance and evaluation representativeness are different goals.

Distribution coverage should be tracked across the behaviour matrix. Define target coverage for risk-relevant cells and measure deficits. Raw record count is misleading when 80% of examples differ only in phrasing.

Long-tail cases are individually rare but collectively important: unusual tool combinations, mixed-language requests, accessibility-related phrasing, old account states, or policy conflicts. Prioritize them by severity multiplied by frequency, with an explicit floor for catastrophic cases whose frequency is hard to estimate.

Adversarial examples deliberately probe boundaries: prompt injections in retrieved documents, Unicode confusables, misleading tool output, contradictory instructions, oversized inputs, forged citations, and near-identical intents with opposite required actions. Keep them in a distinct suite so a flood of synthetic attacks does not distort estimates of normal production quality.

8. Lineage makes a dataset reproducible and governable

A dataset is not a folder of JSONL files. It is a versioned view over records with histories.

Lineage, provenance, and versioning

Provenance answers, “Where did this record come from?” It includes source system, collection time, consent or legal basis where applicable, source-document version, producer, and licence or usage restriction.

Data lineage answers, “What happened to it?” It records redaction, normalization, annotation, adjudication, synthesis, filtering, deduplication, splitting, and export. Model and prompt lineage connect the record to the experiment and deployment that consumed it.

Each record should have a stable ID and immutable parent links:

{
  "record_id": "rec_9f...",
  "source_trace_id": "tr_31...",
  "parent_ids": ["rec_seed_7..."],
  "lineage_group_id": "family_52...",
  "source_type": "production_trace",
  "transformations": [
    {"name": "pii_redaction", "version": "3.2", "at": "..."},
    {"name": "human_adjudication", "guideline_version": "failures:1.4", "at": "..."}
  ],
  "split": "train",
  "dataset_version": "agent-failures:2026.08.3"
}

Dataset versioning should produce immutable releases. A version manifest contains record IDs or content hashes, schema version, taxonomy and guideline versions, code revision, source window, filters, split policy, quality report, and parent dataset versions. Use semantic labels if helpful, but content-addressed manifests make silent changes detectable.

You should be able to answer:

Which raw traces produced this training record?
Which human or model supplied its label?
Which redaction and filters ran?
Which datasets and experiments consumed it?
Did any ancestor appear in evaluation?
Which deployed model was affected?

If those questions cannot be answered, the improvement cannot be reliably reproduced or reversed.

Dataset cards document intent, not just statistics

Dataset cards summarize why a dataset exists, how it was built, what it represents, and how it should not be used. Include:

intended and prohibited uses;
behaviour specification and unit of analysis;
sources, time ranges, and populations;
collection and annotation process;
taxonomy, guidelines, and agreement;
synthetic-data recipe and proportion;
filtering, deduplication, and split logic;
known gaps, biases, and long-tail limitations;
privacy, retention, licensing, and access controls;
contamination tests;
quality metrics and review owners;
version history.

Google's Data Cards research describes them as structured summaries for stakeholders across a dataset's lifecycle. The key idea is that facts such as intended use, rationale, and social context cannot be recovered from the data file alone.

Measure data quality by fitness for purpose

Data-quality metrics should cover at least:

Dimension	Example metrics
Validity	schema pass rate, executable tool-call rate
Label quality	agreement, adjudication rate, audited precision
Completeness	missing evidence, missing provenance, null-label rate
Coverage	populated taxonomy cells, slice deficits, cluster entropy
Uniqueness	exact duplicate and near-duplicate rates
Freshness	source age, policy-version lag, drift by time window
Representativeness	distance from production slice distribution
Safety and privacy	PII detection rate, redaction audit failures
Contamination	exact, fuzzy, semantic, and lineage overlap rates
Utility	downstream quality gain per 100 or 1,000 added examples

There is no universal “dataset quality score.” A complete, deduplicated dataset can still be useless if its labels do not correspond to the target behaviour.

9. Privacy, copyright, and retention belong at collection time

Minimize before you redact

Privacy filtering begins by not collecting fields the improvement loop does not need. Separate identity from content, restrict access by purpose, encrypt sensitive payloads, and log dataset access. Redaction is a secondary control, not permission to capture everything.

PII removal should combine deterministic patterns, named-entity detection, domain-specific rules, and human audits. Replace values with typed placeholders such as <EMAIL_1> or <ORDER_ID_1> so the example preserves structure. Maintain consistent placeholders within a record when relationships matter.

Test redaction for both false negatives and utility-destroying false positives. Free text can reveal identity indirectly through rare combinations even after names and phone numbers are removed. De-identification reduces risk; it does not automatically make re-identification impossible. NIST's Privacy Framework treats privacy as an ongoing risk-management problem and includes controls that limit identification and linkability.

For an India-based deployment, the Digital Personal Data Protection Act, 2023 and the official Digital Personal Data Protection Rules, 2025 are part of the current legal context. Exact obligations depend on the system, role, data, and commencement schedule, so product and legal review must be jurisdiction-specific.

Copyright is a provenance and rights problem

Copyright considerations apply to source documents, user submissions, public web data, teacher outputs, and the generated dataset. “Publicly accessible” does not mean “licensed for training.” Record the source, rights basis, licence, attribution requirements, usage restrictions, and removal path. Avoid reproducing protected material unnecessarily; store references or derived facts where that meets the purpose.

Do not rely on a generic claim that all training is fair use or that synthetic output is automatically free of protected expression. Copyright rules vary by jurisdiction and the facts of use. The U.S. Copyright Office's AI initiative separates questions about digital replicas, copyrightability of outputs, and use of copyrighted works in training—an indication that these are distinct legal issues, not one blanket answer.

Retention must be explicit

Data retention defines how long raw traces, redacted records, annotations, derived datasets, and audit logs remain available. They need not share one period.

A practical policy specifies:

purpose and owner for each data class;
retention duration and deletion trigger;
legal hold and incident exceptions;
whether deletion propagates to derivatives;
how training influence is handled when a source record is removed;
backup expiration;
proof of deletion;
periodic review.

Short-lived raw traces can feed longer-lived, strongly redacted annotations if the purpose and rights allow it. Lineage is what makes deletion propagation possible.

10. From online evidence to an offline improvement loop

An online-to-offline pipeline converts live events into reproducible datasets without letting mutable production state silently alter experiments.

The boundary should look like this:

Production emits immutable, versioned trace events.
Automated detectors attach weak signals; they do not rewrite the trace.
A snapshot job selects a time-bounded candidate pool.
Privacy and access controls run before broad analysis.
Deduplication and clustering organize candidates.
Sampling selects representative, high-impact, uncertain, and long-tail cases.
Humans annotate under a versioned guideline.
Adjudicated records and approved synthetic derivatives form an immutable release.
Experiments consume explicit dataset versions.
Deployment produces new traces, but old experiment inputs never change.

Feedback loops can improve or corrupt the system

Feedback loops arise when deployed behaviour changes the data later used to improve the system. Helpful loops surface genuine failures and verify corrections. Harmful loops reinforce the model's own preferences: a model generates data, another similar model approves it, the trained model produces more of the same data, and rare human patterns disappear.

Other feedback traps include:

only logging cases the current classifier flags;
training on accepted answers while ignoring users who abandon the product;
overfitting to vocal users;
using a model judge whose preferences become the training target;
deploying a policy that changes user behaviour, then comparing post-deployment traffic directly with old traffic.

Preserve random monitoring samples, independent human audits, real-data anchors, and time-based holdouts. Track data-source proportions and performance separately for real, synthetic, weakly labelled, and human-labelled records.

Continuous dataset improvement is therefore not continual ingestion into one growing file. It is a sequence of governed releases:

[ Failure \rightarrow Evidence \rightarrow Specification \rightarrow Dataset \rightarrow Experiment \rightarrow Deployment \rightarrow New\ Evidence ]

Each cycle should state what changed, why it changed, and whether the improvement generalized beyond the cases that triggered it.

11. Building the Agent Reliability Lab failure-to-dataset pipeline

The practical project is a pipeline that turns agent failures into decisions about prompts, retrieval, workflows, tools, or fine-tuning.

Stage 1: Ingest reconstructable traces

Instrument every agent run with:

user intent and redacted input reference;
model, prompt, tool, workflow, and policy versions;
retrieved document IDs, versions, scores, and passages;
planned and actual tool calls;
tool outputs and error classes;
state transitions, retries, budgets, and approvals;
final response;
deterministic assertions, evaluator scores, user feedback, cost, and latency.

Use append-only events and a stable trace_id. A trace is eligible for dataset work only if the system state can be reconstructed well enough to judge it.

Stage 2: Detect failed or low-confidence runs

Combine independent signals:

deterministic contract failures;
unsuccessful or contradictory tool outcomes;
missing required citations;
policy-grounding violations;
loop, timeout, or budget exhaustion;
calibrated low confidence or model disagreement;
low rubric score;
user correction, abandonment, or escalation;
random samples of apparent successes.

The random success sample is essential. Otherwise, detectors define reality and invisible failures never enter the dataset.

Stage 3: Redact before wider processing

Apply field allowlists, tokenize direct identifiers, run contextual PII detectors, quarantine high-risk records, and retain encrypted raw payloads only under narrower access. Record redactor version and audit outcome. Do not send unredacted traces to an external teacher model unless the approved data flow explicitly permits it.

Stage 4: Cluster failure types

Create a structured representation containing intent, failure stage, tool sequence, retrieval state, verifier signals, and a redacted summary. Generate candidate clusters using rules plus embeddings. Sample cluster centres and outliers for human naming.

The output is a versioned failure taxonomy and a queue of cases, not an unquestioned machine-generated label.

Stage 5: Select representative examples

Allocate the annotation budget across four buckets:

random production sample for unbiased measurement;
high-impact failures for risk reduction;
uncertain or disputed cases for learning taxonomy boundaries;
diverse representatives from new and long-tail clusters.

Cap repeated templates and dominant tenants. Preserve weights or sampling probabilities so analysts can estimate population metrics from a deliberately enriched sample.

Stage 6: Remove duplicates without erasing boundaries

Use exact canonical hashes, shingle-based near-duplicate retrieval, semantic neighbours, and behaviour signatures. Assign every duplicate family a lineage_group_id. Keep a representative record plus frequency metadata; frequency itself is useful evidence.

Review pairs near decision boundaries before merging. Two almost identical cases with opposite correct actions are high-value contrastive examples, not duplicates.

Stage 7: Route cases for annotation

Route by expertise and risk:

general annotators for clear taxonomy labels;
domain experts for policy correctness;
security reviewers for injection or permission cases;
adjudicators for disagreement or high-severity cases.

Run qualification and calibration batches. Blind annotators to the candidate system version when possible. Store individual decisions, confidence, rationale, time, guideline version, and adjudicated result.

Stage 8: Define corrections, not only failure labels

For each failed case, annotation should produce:

observable failure;
root-cause evidence;
corrected final outcome;
corrected retrieval evidence, if applicable;
expected tool calls and arguments;
required approval or abstention;
acceptable variants;
intervention recommendation;
evaluation assertions.

Corrections are behaviour specifications at record level. They can become few-shot examples, retrieval judgements, workflow tests, or fine-tuning data.

Stage 9: Generate targeted synthetic variants

Generate variants only for documented gaps. Parameterize intent, language, entities, state, permissions, document freshness, tool outcomes, ambiguity, and adversarial content. Keep seed and generator lineage. Never use evaluation records or their descendants as seeds.

Stage 10: Filter synthetic records

Apply schema, policy, simulator, contradiction, diversity, duplication, and judge filters. Human-review a stratified sample from every generator configuration and failure class. Estimate precision with confidence intervals; do not report only the fraction accepted by the same model that generated the records.

Stage 11: Create protected splits

Split by lineage_group_id, source document, session/user boundary, and time where appropriate. A useful layout is:

train: mutable input to model adaptation;
dev: prompt, retrieval, and workflow iteration;
test: frozen generalization measurement;
challenge: adversarial and high-severity cases;
temporal_holdout: newer production period;
monitor: weighted live sample, never reused automatically.

Run overlap checks after every transformation and again before release. If a test case is inspected to design a fix, mark it exposed and migrate it out of the protected set.

Stage 12: Version the dataset and publish its card

Create an immutable manifest containing record hashes, split, parent IDs, source windows, transformation code, redaction version, taxonomy and guideline versions, agreement report, synthetic proportions, contamination report, coverage matrix, known gaps, and retention policy.

Call the release something explicit such as agent-reliability-failures:2026.08.3, not final_v2_really_final.

Stage 13: Route the evidence to the smallest intervention

The dataset should trigger a diagnosis before training:

Evidence	First experiment
Correct evidence retrieved, model ignores it	Prompt/context change; then adaptation if persistent
Required evidence absent	Corpus, chunking, metadata, or ranking change
Wrong tool chosen due to overlapping schemas	Narrow tool descriptions or workflow routing
Correct tool, invalid arguments	Schema constraints, validators, corrective loop, or adaptation
Business sequence violated	Deterministic workflow/state-machine change
Stable behavioural gap across many varied cases	Fine-tuning or model change
Policy itself ambiguous	Product/policy clarification and user clarification path

Fine-tuning is justified only when the failure is genuinely in model behaviour and stronger non-training baselines have been tested.

Stage 14: Measure improvement before and after deployment

For every experiment, record:

dataset and system versions;
primary behaviour metric;
slice-level metrics;
regression suite results;
cost, latency, and trajectory length;
safety and permission violations;
confidence intervals and sample counts;
failure-cluster deltas;
comparison with the strongest existing baseline.

Require success on the frozen test set and no unacceptable regression. Then canary the change and measure a representative live sample. Compare equivalent cohorts and account for traffic or policy changes. A reduction in detector alerts is not sufficient if the detector itself changed.

The final measure is not “the new dataset contains 50,000 examples.” It is:

Dataset version 2026.08.3 reduced confirmed argument-validation failures from 7.2% to 3.1% on the protected temporal holdout and from 6.8% to 3.5% in the canary, without degrading tool-selection accuracy, permission compliance, latency SLOs, or high-severity challenge cases.

That statement closes the data-improvement lifecycle. It connects a production failure to evidence, data, intervention, and deployed outcome.

12. The governing principles

The most important lessons are simple, but demanding:

Specify behaviour before collecting examples. Data is useful only relative to a decision and a target behaviour.
Treat production logs as evidence, not labels. Feedback, heuristics, and model judges are noisy sensors.
Preserve ambiguity. Disagreement can reveal missing evidence or an underspecified product rule.
Separate optimization data from evaluation data. Once a case influences a fix, it cannot provide an unbiased test of that fix.
Generate synthetic data to fill measured gaps. Volume without diversity, verification, and provenance can reinforce blind spots.
Deduplicate by behaviour, not text alone. Near-identical boundary cases may be the most valuable records.
Mine hard cases without forgetting normal traffic. Keep representative samples and population weights.
Version every transformation. Reproducibility requires source, label, redaction, generator, filter, split, and consumer lineage.
Build privacy, rights, and retention into ingestion. Cleanup at export time is too late.
Judge data by downstream evidence. The best dataset is the smallest governed dataset that produces a real, repeatable improvement without hidden regressions.

An LLM system does not become reliable because it has seen more tokens. It becomes reliable when its builders can repeatedly convert observed failures into precise behavioural evidence—and can prove that the resulting change works on cases the system has never seen.

That is why the most important LLM optimization may be a better dataset.

Topic coverage map

This article covers the full syllabus as follows:

Sections 1–2: data as a system component, behaviour specifications, data requirements, data sources, production traces, and user feedback.
Section 3: human annotation, guidelines, taxonomies, ambiguity, and inter-annotator agreement.
Section 4: weak supervision and heuristic labels.
Section 5: synthetic data, teacher generation, self-instruct, diversity, and filtering.
Section 6: deduplication, near-duplicate detection, contamination, leakage, and train-evaluation overlap.
Section 7: hard-example mining, failure clustering, active learning, uncertainty sampling, curriculum design, balancing, distribution coverage, long-tail cases, and adversarial examples.
Section 8: data lineage, dataset versioning, provenance, dataset cards, and data-quality metrics.
Section 9: privacy filtering, PII removal, copyright, and retention.
Sections 10–11: feedback loops, online-to-offline pipelines, and continuous dataset improvement through the complete Agent Reliability Lab project.