Lifecycle & Ground Truth in RAG: Freshness and Evaluation from First Principles

One line: A RAG system is a stack of caches over a world that keeps moving — so freshness and evaluation aren't features you add, they're the price of the data staying true.

Why this exists / what it solves

Ship a RAG system and it starts decaying the moment it's live. The documents drift from the world, the index drifts from the documents, and your "golden" evaluation set drifts from both. The two questions that decide whether the thing stays trustworthy are: is the data still fresh? and how would I even know if it weren't? This post rebuilds both from the ground up — as a chain of questions, each pushed until it rests on something that can't be argued with.

Part 1 — Freshness

When a RAG system is "stale," what exactly is stale?

Not the source documents — those are the freshest thing you hold; they can't lag themselves. Staleness is always a derived layer lagging its source. The pipeline is a stack of caches:

world → documents → chunks → embeddings → index → answer

There are two independent lags, in two reference frames:

docs lag the world — a price changes; the doc updates only when a human edits it. (Outside the system.)
index lags the docs — a doc changes; its embeddings are behind until reprocessed. (Inside the system.)

The served answer is stale if either link lagged.

How do you actually detect it — instead of assuming it on a schedule?

You can't detect divergence between two things without a property you can compare on both sides. Store a content hash (or timestamp) at index time; compare it to the document's current hash.

The painful way — recompute everything on a timer, mostly wasting the work:

@cron("0 3 * * *")           # every night, no matter what
def rebuild_index():
    for doc in all_documents():
        reembed(doc)         # most of this is unnecessary

The clean way — let the diff drive it:

def is_stale(doc, index_record):
    return doc.content_hash != index_record.content_hash

for doc in changed_documents():        # only what actually moved
    if is_stale(doc, index.get(doc.id)):
        reembed(doc)

The marker is the whole game: without it, "fresh" is a guess on a cron schedule; with it, staleness is a fact you can query.

Is stale data always harmful? (If yes, freshness is theater; if no, what separates the two?)

It isn't. A query about a time-invariant fact — a definition, 1789, a frozen API version — returns the same correct answer from a stale index. So harm is conditional:

harm ≈ content_volatility × query_sensitivity

If either factor is zero, staleness is harmless. This is why a stock-price RAG and a docs-for-a-2019-library RAG have completely different freshness budgets. You tune freshness to the volatility of the facts being asked about — not to a calendar. When staleness does bite, the chain is: wrong answer → eroded trust, which is the actual cost you're paying to avoid.

Re-embed everything is the reflex. When is that wrong?

An embedding is a pure function of two inputs:

embedding = f(chunk_text, model)

A pure function's output changes only when an input changes — so the decision tree falls out for free:

if chunk_text_changed:    reembed(chunk)         # input 1 changed
elif model_changed:       reembed(everything)    # input 2 changed → all vectors incomparable
elif metadata_changed:    patch_metadata(chunk)  # not an input → no reembed at all

Two consequences worth burning in: editing a doc only re-embeds the chunks whose text actually changed, and swapping the embedding model invalidates the entire index — you cannot compare vectors produced by different models.

Part 2 — Ground Truth & Evaluation

What is "ground truth" in RAG — the right chunk, or the right answer?

Both, because they're two independent targets. Proof by the cases that come apart:

retrieval_correct  = retrieved_chunk_id == gold_chunk_id      # was the right evidence fetched?
generation_correct = faithful(answer, retrieved_chunk)        # given that evidence, is the answer grounded?

# They disagree all the time:
#   right chunk  + hallucinated answer    → retrieval ✅  generation ❌
#   wrong chunk  + lucky parametric answer → retrieval ❌  generation ✅

Score them separately or you'll mistake a retrieval bug for a generation bug.

Who certifies a label as correct — and what makes it authoritative, not just an opinion?

"The user" is too vague. Authority comes from direct access to the referent + reproducibility, and it differs per axis:

Faithfulness (answer vs. chunk) — a closed comparison. Any reader, even an LLM, can certify it.
Correctness (answer vs. world) — needs someone with world-access: a domain expert.
Relevance (answer vs. intent) — only the end user can certify; it's their goal.

A label two independent experts reproduce is, operationally, ground truth. That's the line between a label and an opinion.

Does ground truth stay true?

No. A golden answer is pinned to a fact; when the fact changes, the answer rots and your eval starts scoring against a lie. The golden set is itself a cache over the world — it decays exactly like the index, and needs the same versioned upkeep.

You retrieved the right chunk and generated faithfully — and the user still left unhappy. What did you miss?

A third axis. Retrieval and faithfulness are both internal (system-relative): "right evidence," "grounded in it." But the answer can be grounded and still useless — wrong granularity, incomplete, literal but off-intent. Usefulness is external (user-relative), and the system never fully holds the user's intent.

internal:  retrieval quality + faithfulness   ← system can verify alone
external:  usefulness to intent               ← only the user can verify

A system can be internally perfect and externally useless. That gap is irreducible.

Why trust a metric — or an LLM judge — enough to gate a release on it?

Both are proxies for truth, and a proxy earns trust exactly one way: measured agreement with a trusted reference on samples where the truth is already known.

# An LLM judge is a proxy. Validate it like any proxy.
kappa = cohen_kappa(llm_judge_labels, human_labels)   # on a held-out gold set

# Gate releases on the judge only if it agrees with humans
# about as much as humans agree with each other.

recall@k, faithfulness, an LLM judge — same move every time. The chain of proxies always bottoms out at human judgment with referent-access. You never escape needing one trusted anchor; you only justify the cheap proxy by how well it tracks the expensive one.

So why is RAG a lifecycle and not a build-once system?

Because the referent — the world, and the data mirroring it — changes continuously. Every layer below it is a cache, every cache decays, and the golden set you'd use to detect the decay is decaying too. The loop isn't a best practice; it's a consequence of building on something that moves.

The first principles it rests on

Verify these on revisit — everything above is built from them:

A cache lags its source. Every layer (docs → chunks → embeddings → index → answer) is a derivative of the one above; staleness is always relative to that source, in one of two frames (docs-vs-world, index-vs-docs).
Divergence is undetectable without a shared comparable marker. A hash/timestamp on both sides turns "assume on a schedule" into "detect on a diff."
Harm from staleness = volatility × query-sensitivity. Not universal; tune freshness to the facts being queried.
A pure function's output changes only when its inputs change. embedding = f(chunk_text, model) ⇒ re-embed on text change, re-embed all on model change, patch metadata otherwise.
Retrieval-truth and generation-truth are independent. Either can fail while the other holds; measure separately.
Certification authority = referent-access + reproducibility. Faithfulness is closed (checkable against the chunk); correctness needs world-access; relevance needs the user.
Internal correctness ≠ external usefulness. Usefulness is user-relative and irreducible to retrieval + faithfulness.
A proxy is validated only by measured agreement with a trusted reference on known-truth samples — and the regress terminates at human judgment with referent-access.
The referent moves, so the loop is mandatory. Build-once is impossible because every layer — including the golden set — decays against a changing world.

Acronyms

RAG — Retrieval-Augmented Generation
LLM — Large Language Model
LLMOps — Large Language Model Operations
recall@k — recall measured over the top k retrieved results
kappa — Cohen's kappa, an inter-annotator agreement statistic