Bhaskar
← All writing
first principlesJun 26, 2026· 11 min read

Storage & Search Efficiency in RAG: Dimension, Quantization, Indexes, and Vector Stores

One line: Retrieval works fast and cheap because real embeddings don't fill the high-dimensional space they live in — they sit on a thin, structured surface, and every optimization below is a way of exploiting that.

Why this exists / what problem it solves. A naive RAG system stores full-precision vectors and compares the query against every one of them. At a few thousand chunks that's fine. At a few million it's slow, memory-hungry, and expensive. Dimension reduction, quantization, and approximate indexes exist to make retrieval survivable at scale — and they all work for the same underlying reason. This post rebuilds that reasoning as a chain of questions.

Part 1 — Dimensions: what they are and why there are 768 of them

Q: What does one dimension in an embedding encode, and what fixes the total count? A single dimension encodes nothing human-readable on its own — meaning lives in the whole vector's direction. The count (384, 768, 1536) is the width of the model's final projection layer. It's frozen at training time.

Q: So is 768 a necessity or a convention? Split it. That the count is fixed and unchangeable without retraining is necessity. That it's a round number (a power-of-two-friendly value aligned to hardware and attention heads) is convention. Meaning doesn't require exactly 768 dims; the GPU likes it.

Q: How would you get a model that emits 500 dims? Retrain with a 500-wide output head, or attach a learned projection on top. You can't just slice — see below for when slicing is and isn't allowed.

Part 2 — Truncation: when fewer dimensions is a free lunch

Q: People truncate 768→256 dims with little quality loss. What does that imply? If information were spread evenly across all dimensions, dropping two-thirds would cost two-thirds of the signal. Cheap truncation implies the information is front-loaded — concentrated in the early dimensions.

Q: Is that concentration automatic? No. It's trained in. Matryoshka Representation Learning (MRL) explicitly optimizes nested sub-vectors so each prefix is usable alone. Truncating a non-MRL model corrupts the vector.

# DON'T: blindly truncate a model not trained for it
vec = model.encode(text)          # 768-d, info spread across all dims
short = vec[:256]                 # may discard critical signal → recall drops

# DO: truncate a Matryoshka model (e.g. OpenAI text-embedding-3, nomic-embed)
vec = model.encode(text)          # 768-d, early dims carry the most
short = vec[:256]                 # trained to stay useful → ~free 3x storage cut
short = short / np.linalg.norm(short)   # renormalize after slicing

Rule of thumb: truncate only if the model card says it supports it. Otherwise reduce dimensions at training/projection time, not by slicing.

Part 3 — Quantization: storing the same vector in fewer bits

Q: Quantization swaps float32 for coarse codes. What's actually thrown away? Not "the decimals at the end." Quantization rounds every component to the nearest of a small set of levels: float32 (32 bits/dim) → int8 (256 levels) → binary (1 bit, just the sign).

Q: Why does retrieval survive that? Retrieval depends on relative ranking, not exact values. Unbiased rounding error is tiny compared to the gaps between distinct vectors, so the order of nearest neighbors barely changes.

Q: What does each quantizer assume, and where does it break?

Method What it does Assumes Breaks when Typical size cut
Scalar (SQ) Each dim → int8 Values are bounded per-dim Heavy outliers stretch the range 4×
Product (PQ) Split into sub-vectors, k-means codebook each Sub-spaces have cluster structure Dims are highly correlated 8–64×
Binary (BQ) Keep only the sign of each dim Angle (sign pattern) carries the meaning Magnitude matters 32× 
# Binary quantization: 768 floats (3072 bytes) → 768 bits (96 bytes), 32x smaller
binary = (vec > 0).astype(np.uint8)
packed = np.packbits(binary)              # 96 bytes
# Distance becomes Hamming (XOR + popcount) — extremely fast on CPU
dist = np.count_nonzero(packed_query ^ packed_doc)

The standard high-performance recipe: binary or PQ for a fast first pass, then rescore the top candidates with full-precision vectors. You get the speed of compression and the accuracy of floats.

Part 4 — Why distance means anything at all

Q: Why does geometric closeness correspond to semantic similarity? Because the embedding model was trained that way. Contrastive training pulls semantically related pairs together and pushes unrelated pairs apart, directly in vector space. Closeness ≈ meaning is the literal training objective, not a happy accident.

Q: Which distance metric, and what fixes the choice? Fixed by how the model was trained and whether vectors are normalized:

# Cosine: angle only — the default for normalized text embeddings
cos = (a @ b) / (norm(a) * norm(b))

# Dot product: angle AND magnitude — use only if the model was trained for it
dot = a @ b

# L2 (Euclidean): straight-line distance — equivalent to cosine after normalization
l2 = norm(a - b)

If vectors are L2-normalized, cosine and dot and (rank-wise) L2 agree. Match the metric to the model's training; don't pick by habit.

Part 5 — Why brute force stops working

Q: Exact search is trivially correct. What's its cost? O(N·D) per query. N distance computations, each costing D multiply-adds. With 10M vectors × 768 dims × many queries/sec, that product is what blows the latency and compute budget.

# Exact brute force — correct, but O(N·D) every single query
def search(query, corpus, k):
    sims = corpus @ query          # N×D matmul, recomputed per query
    return np.argsort(-sims)[:k]

That cost is why approximate nearest neighbor (ANN) indexes exist. They trade a sliver of recall for orders-of-magnitude speed.

Q: Why is accepting <100% recall acceptable in RAG specifically? Three cushions absorb the error: you over-fetch (ask for top-50, a single miss rarely matters), you can rerank the candidates exactly, and the generator tolerates an imperfect context set. End-task quality barely moves between 95% and 100% recall.

Part 6 — The index zoo: how each one cheats, and what it assumes

Q: If distances "concentrate" in high dimensions, why does NN search work at all? Concentration is a theorem about uniformly random points — and real embeddings are not uniform. Contrastive training forces them onto a thin, curved, low-dimensional manifold inside the high-D cube. Their intrinsic dimension is far below 768, so true neighbors stay separably close while random pairs stay far. This is the root fact the whole post rests on.

Every index is a different way to exploit that manifold:

Index Mechanism Exploits Knobs Best for
Flat (brute force) Compare to all — (exact) — Small N, ground truth
IVF k-means cells; search nearest nprobe cells Locality on the manifold nlist, nprobe Medium-large, batchable
IVFPQ IVF cells + PQ-compressed vectors Locality + sparsity + PQ params Billion-scale, RAM-bound
HNSW Layered small-world graph, greedy descent Manifold is navigable M, efConstruction, efSearch Low latency, high recall
ScaNN Anisotropic quantization + partitioning Direction-aware error tree/leaf params Google-scale, max recall/speed
LSH Hash so near points collide Locality via random projections #tables, #bits Streaming, simple, lower recall
DiskANN / Vamana Graph index living on SSD Manifold + cheap disk graph degree Billion-scale on one box
Annoy Forest of random projection trees Recursive partitioning #trees Static, mmap, read-only

Q: Why does HNSW's greedy walk land near the true neighbor? HNSW builds a layered graph: top layers have long-range "express" links, lower layers dense local ones. You start at the top, greedily step to whichever neighbor is closer to the query, and drop a layer when you can't improve. Each hop strictly reduces distance, and construction guarantees the graph is navigable — so greedy converges.

# HNSW intuition: descend coarse-to-fine, always stepping closer
node = entry_point
for layer in range(top, -1, -1):
    improved = True
    while improved:
        improved = False
        for nbr in graph[layer][node]:
            if dist(query, nbr) < dist(query, node):
                node, improved = nbr, True
# node is now a near-optimal neighbor

Q: Why does IVF's "search only a few clusters" speed things up? k-means carves space into Voronoi cells. You compare the query to the nlist centroids, then search only vectors in the nearest nprobe cells. Speedup ≈ nprobe / nlist. It assumes your true neighbors share a cell with the query (locality); neighbors stranded just across a border are missed — which is exactly why raising nprobe recovers recall.

# IVF: the locality/recall dial
index.nprobe = 1     # fastest, lowest recall — search 1 cell
index.nprobe = 32    # slower, higher recall — search 32 of nlist cells

Part 7 — Vector stores: a new thing, or Postgres with a column?

Q: What makes a "vector store" different from a normal DB with a vector column + index? Capability-wise: nothing fundamental. pgvector runs real HNSW/IVF inside Postgres and works.

Q: So is the distinction necessary or conventional? Operational, not fundamental. Dedicated stores win at scale and ops, not capability:

  • billion-vector sharding and horizontal scale
  • quantization and memory-mapped indexes built in
  • metadata-filtered ANN (filter + vector search in one pass, not filter-then-scan)
  • hot reindexing, replication, hybrid (keyword + vector) search out of the box
-- pgvector: fully capable at small/medium scale
CREATE INDEX ON items USING hnsw (embedding vector_cosine_ops);
SELECT * FROM items ORDER BY embedding <=> :query LIMIT 10;

The honest decision rule:

Situation Choice
< ~1M vectors, already on Postgres, need transactions/joins pgvector
Millions–billions, low-latency, metadata filtering, hybrid search Dedicated store (Qdrant, Weaviate, Milvus, Vespa, pinecone, etc.)
Embedded / in-process, no server FAISS, Chroma, LanceDB

The landscape moves quickly — treat specific product names as a snapshot, not gospel, and benchmark on your own data.

Part 8 — How do you know your configuration is good enough?

Q: What do you measure, and against what? Measure recall@k against an exact brute-force result on a frozen golden set of queries — that's your ground truth. Then track latency and memory at your target QPS. The final arbiter is end-task / user validation, because retrieval only matters insofar as it improves answers.

# recall@k against exact ground truth
def recall_at_k(approx, exact, k):
    return len(set(approx[:k]) & set(exact[:k])) / k
# Sweep nprobe / efSearch / quantization until recall@k clears your bar
# at acceptable latency. Then confirm with real users.

Tuning order that works in practice: pick the metric the model was trained for → pick an index for your scale → turn the recall knob (efSearch / nprobe) until recall@k passes → add quantization for memory → rescore top candidates with full precision → validate end-to-end.

First principles it rests on

The bedrock facts, so you can re-verify on revisit:

  1. Embeddings occupy a low-dimensional, structured manifold — not the full high-D volume. (The root fact. Everything below is a corollary.)
  2. Closeness ≈ meaning is a trained objective (contrastive learning), not an inherent property of vectors.
  3. The dimension count is the model's output width — fixed by necessity; its round value is convention.
  4. Truncation is cheap only when the model was trained to front-load information (Matryoshka); otherwise it's destructive.
  5. Quantization preserves rank, not value — retrieval depends on relative ordering, so coarse rounding survives.
  6. Each quantizer is a definition plus an assumption — scalar (bounded range), product (sub-space clusters), binary (sign carries angle).
  7. Brute force is O(N·D) per query — the product with QPS is what forces approximation.
  8. ANN's recall loss is absorbed by over-fetch + exact rerank + generator slack.
  9. HNSW works because a small-world graph is navigable — greedy descent provably reduces distance each hop.
  10. IVF works because of locality — neighbors share Voronoi cells; nprobe trades recall for speed.
  11. High-D distance concentration doesn't break search because real embeddings aren't uniform — the manifold keeps true neighbors separable.
  12. A "vector store" is an operational convenience, not a new capability — the distinction is scale and ops, not what's possible.
  13. "Good enough" = recall@k vs. an exact golden set, confirmed by user validation.

Acronyms

Acronym Full form
RAG Retrieval-Augmented Generation
ANN Approximate Nearest Neighbor
NN Nearest Neighbor
MRL Matryoshka Representation Learning
SQ Scalar Quantization
PQ Product Quantization
BQ Binary Quantization
IVF Inverted File (index)
IVFPQ Inverted File with Product Quantization
HNSW Hierarchical Navigable Small World (graph)
ScaNN Scalable Nearest Neighbors
LSH Locality-Sensitive Hashing
DiskANN Disk-based Approximate Nearest Neighbor (Vamana graph)
FAISS Facebook AI Similarity Search
L2 L2 / Euclidean (straight-line) distance
QPS Queries Per Second
RAM Random-Access Memory
SSD Solid-State Drive
CPU Central Processing Unit
GPU Graphics Processing Unit
DB Database
XOR Exclusive OR (bitwise operation)
SQL Structured Query Language