Relevance Refinement in RAG, From First Principles

One line: The retrieval stack — query transformation → retrieval mode → reranking — exists because the user's raw query is the weakest possible proxy for what they actually need, and a single fast similarity score is too crude to fix on its own.

Why this exists / what it solves: A generator can only ground its answer in what you put in its context. So answer quality is upper-bounded by retrieval. Every stage below is one move to tighten that bound — by repairing the query, by measuring relevance two different ways, and by re-scoring the survivors with a model that's too expensive to run on everything. None of it is decoration; each stage either moves a metric or gets cut.

This post is built as a chain of questions, the same order the understanding was actually assembled. Read it top to bottom and the stack assembles itself.

Q1. What is "relevance" even measured against?

Not the query's words. Relevance is satisfaction of the information need — the answer the user would accept as correct. That's unobservable, so every system optimizes a proxy for it:

Proxy	What it is	Distance from truth
Lexical overlap	shared words (BM25)	far
Embedding proximity	learned semantic closeness	closer
Judgments / answer-correctness	labeled or downstream signal	closest

The query's literal words are the weakest proxy. That single fact is what licenses everything that follows.

Q2. So why not just retrieve on the raw query?

Because the raw query fails in two grounded ways:

Vocabulary mismatch. People pick different words for the same thing — two people choose the same term less than ~20% of the time. The relevant document rarely shares the query's words.
Intent underspecification. One embedding is one point in space. A need with several facets gets collapsed to a centroid that's relevant to nothing.

query:    "automobile fails to crank"
document: "how to fix a car that won't start"
lexical overlap: ~0    ← BM25 on the raw query misses the right doc

Q3. How do we repair the query, then?

You stop trusting the user's phrasing as the retrieval key. Three standard moves:

# Painful: one literal query, one shot
hits = retrieve(user_query)

# Clean: reshape the query into a better key before retrieving
# 1) Multi-query — paraphrase into several keys, union the hits
variants = llm(f"Rewrite this into 3 search queries:\n{user_query}")
hits = union(retrieve(q) for q in variants)

# 2) HyDE — retrieve with a *hypothetical answer*, not the question
#    (answers live near answers in embedding space, not near questions)
hyp = llm(f"Write a passage that answers:\n{user_query}")
hits = retrieve(hyp)

# 3) Decompose — split a multi-hop need into single-focus sub-queries
subs = llm(f"Break into independent sub-questions:\n{user_query}")
hits = [retrieve(s) for s in subs]

Each move targets a specific defect: paraphrase fights vocabulary mismatch, HyDE fights the question-vs-answer gap, decomposition fights the centroid problem.

Q4. The retriever scores by "closeness." But does closeness mean relevance?

Only conditionally — and this is the crux most people skip.

Dense models are trained with a contrastive objective: pull an anchor toward a positive, push negatives away. So "close" means whatever the positives were during training.

positives = co-occurring passages   → model learns TOPICAL SIMILARITY
positives = (question, answer) pairs → model learns RELEVANCE (asymmetric)

They are not the same model, and you can tell which one you have:

# Diagnostic: does your model prefer the paraphrase or the answer?
q       = "When did the French Revolution begin?"
paraphr = "What year did the revolution in France start?"  # looks like q
answer  = "The storming of the Bastille occurred in 1789." # shares few words

# Similarity-trained: ranks `paraphr` closest  (surface match)
# Relevance-trained: ranks `answer`  closest  (answer-bearing)

"Proximity ≈ relevance" is true exactly to the degree your positives were relevance pairs. That's the whole content of "because of the embedding model's training."

Q5. Why isn't one retrieval mode enough?

Because dense and sparse fail on opposite inputs, and the failures are derivable from what each one computes.

DENSE (cosine over learned vectors) — semantic, smoothed
  WINS: "automobile fails to crank"  ≈  "car won't start"
  FAILS: "error code P0420"  →  pulls P0430, P0440 (semantically adjacent, wrong)

SPARSE / BM25 (term frequency over an inverted index) — exact, literal
  WINS: "error code P0420"  →  exact rare token, nailed
  FAILS: "automobile fails to crank"  →  ~0 overlap with "car won't start"

Dense covers paraphrase and synonymy. Sparse covers identifiers, rare tokens, exact strings — anything where the precise string is the meaning. Hybrid isn't belt-and-suspenders; it's covering each mode's structural blind spot with the other's strength.

Q6. How do you combine two retrievers whose scores aren't comparable?

You don't compare scores — a BM25 score of 14.2 and a cosine of 0.81 live on different scales. You combine by rank. Reciprocal Rank Fusion:

def rrf(rankings, k=60):
    scores = {}
    for ranking in rankings:           # e.g. [dense_hits, sparse_hits]
        for rank, doc in enumerate(ranking):
            scores[doc] = scores.get(doc, 0) + 1 / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)

fused = rrf([dense_results, sparse_results])

A doc ranked high by either retriever floats up; a doc both rank well dominates. No score normalization, no tuning to make incomparable numbers comparable.

Q7. The retriever already ranked these. What can a reranker do that it structurally can't?

This is the sharpest distinction in the stack.

RETRIEVER = bi-encoder
  embed(query)            ─┐
  embed(doc)  [PRECOMPUTED]├─→  score = dot(qv, dv)
  query and doc NEVER meet ┘
  WHY forced: docs are embedded at INDEX TIME, before the query exists,
              so you can search millions with one fast ANN lookup.

RERANKER = cross-encoder
  score = transformer([query, doc])   ← both in one input, full cross-attention
  every query token attends to every doc token
  WHY expressive: it can model term interaction, conditioning, negation.
  WHY costly: depends on the query → CANNOT precompute → one forward pass PER doc.

The retriever is forbidden from letting query and document interact, because it must precompute document vectors with no query in sight. The reranker is allowed full interaction — but only affordable on the top-k the retriever already narrowed to. Cross-attention accuracy is mathematically incompatible with precomputing over millions of docs. That's why you can't just make the retriever "as accurate" — you'd lose the ability to search at scale.

Q8. Then why not skip retrieval and rerank everything?

Cost. The cross-encoder is one forward pass per document. Over a million docs that's a million passes per query — impossible. So the architecture is forced into two stages:

candidates = hybrid_retrieve(query, top_k=100)   # cheap, wide, approximate
top        = cross_encoder.rerank(query, candidates)[:5]  # expensive, narrow, precise
context    = top

Retrieve wide and cheap to narrow the field; rerank narrow and expensive to order what survived. Two stages because no single model is both scalable and precise.

Q9. Is the stage order forced, or just convention?

Forced, by data dependency:

transform → retrieve → rerank
   │           │          └── needs candidates to exist  ⇒ must come AFTER retrieve
   │           └── needs a key to search with            ⇒ transform output IS that key
   └── produces the retrieval key                        ⇒ must come FIRST

You can't rerank what hasn't been retrieved, and you can't retrieve without a key. The order isn't a style choice.

Q10. How do you know any given stage is earning its place?

Ablation. Freeze everything else, toggle one stage, measure the terminal metric on a fixed eval set with known-good answers.

base   = evaluate(pipeline)                       # all stages on
no_rer = evaluate(pipeline.without("reranker"))   # one stage off

delta = base.score - no_rer.score
# delta ≈ 0 (within noise)  →  reranker is dead weight FOR THIS DATA → cut it
# delta > noise             →  it's paying for its latency → keep it

Every stage is instrumental; only the final answer is terminal. A stage with no measurable marginal contribution is pure cost.

The first principles it rests on

Verify these on revisit — everything above is built from them:

Relevance = satisfaction of an information need, which is unobservable; systems optimize proxies (lexical < semantic < judgments). The query's words are the weakest proxy.
Vocabulary mismatch is real and measured — two people pick the same word for the same thing less than ~20% of the time, so the relevant doc rarely shares the query's words.
An embedding is one point. A multi-focus need collapses to a centroid relevant to nothing — hence decomposition.
Proximity equals relevance only insofar as the contrastive training positives were relevance pairs, not co-occurrence pairs. The paraphrase-vs-answer test distinguishes the two.
BM25 scores exact lexical overlap; dense scores learned semantic proximity. Each fails precisely where the other is strong (identifiers vs. paraphrase).
Scores from different retrievers aren't magnitude-comparable — fuse by rank (RRF), not by score.
A bi-encoder must precompute document vectors with no query present (to search millions), so query and doc never interact; relevance is compressed to one dot product.
A cross-encoder allows full query–doc attention — far more expressive, impossible to precompute, affordable only on the small candidate set the retriever already produced.
Stage order is forced by data dependency: transform's output is the retrieval key; rerank needs candidates.
A stage earns its place only if ablating it moves the terminal metric beyond noise.

Acronyms & terms

RAG — Retrieval-Augmented Generation: feeding retrieved documents into an LLM's context so its answer is grounded in real sources.
LLM — Large Language Model: the generator that produces the final answer.
IR — Information Retrieval: the field that defines relevance as satisfaction of an information need.
BM25 — Best Matching 25: the standard sparse/lexical ranking function; scores documents by term frequency over an inverted index.
TF — Term Frequency: how often a query term appears in a document; the core signal BM25 uses.
HyDE — Hypothetical Document Embeddings: retrieve using a generated hypothetical answer instead of the raw question, so the key lands near real answers.
RRF — Reciprocal Rank Fusion: merges two rankings by rank position rather than raw score, since scores from different retrievers aren't comparable.
ANN — Approximate Nearest Neighbor: the fast index that finds the closest precomputed document vectors without scanning all of them.
Bi-encoder — Retriever architecture: embeds query and document independently, scores by dot product; document vectors are precomputed at index time.
Cross-encoder — Reranker architecture: feeds [query, document] together through full cross-attention; far more accurate, cannot be precomputed.
Cranfield paradigm — The IR evaluation model that operationalizes relevance through human judgments on query–document pairs.