A first-principles guide to inference, memory, batching, caching, quantization, parallelism, scaling, reliability, and cost

Updated for the serving ecosystem as of 19 August 2026.

An LLM endpoint looks like an ordinary HTTP API, but its internal economics are unusual. A request can occupy a few CPU milliseconds for tokenization, many GPU milliseconds for prompt processing, then repeatedly re-enter the GPU once per generated token while retaining an expanding private memory allocation. Another request may arrive halfway through. A third may share the first 8,000 prompt tokens with the first. A fourth may ask for 2,000 output tokens and monopolize capacity long after the others finish.

The serving problem is therefore not simply “run a neural network behind FastAPI.” It is a scheduling and memory-management problem wrapped around a highly asymmetric computation.

This chapter derives that system from the smallest possible starting point: one model, loaded on one GPU, serving one request at a time.

1. The smallest possible inference server

Imagine a decoder-only transformer already trained and saved as a checkpoint. At startup, a server:

reads the model configuration and tokenizer;
allocates GPU memory;
transfers the model weights to the GPU;
initializes kernels, workspaces, and often compiled or captured execution graphs;
accepts a request;
tokenizes its text;
processes all prompt tokens;
samples one next token;
repeatedly processes the newest token until a stop condition is met;
detokenizes and streams text to the client.

In a current vLLM deployment, these responsibilities are separated: the API process handles HTTP, input processing, and streaming; an engine-core process schedules requests and manages the KV cache; GPU worker processes load weights and execute forward passes. That decomposition is useful, but it is an implementation of the lifecycle above—not a replacement for understanding it. See the current vLLM architecture overview.

flowchart TD
    A["HTTP request"] --> B["Tokenize and validate"]
    B --> C["Admission and scheduling"]
    C --> D["Prefill prompt"]
    D --> E["Allocate KV cache"]
    E --> F["Decode one or more tokens"]
    F --> G{"Stop condition?"}
    G -- No --> C
    G -- Yes --> H["Finish stream and free state"]

Inference is not training

Training performs a forward pass, stores intermediate activations needed for differentiation, computes a loss, runs backpropagation, calculates gradients, and updates optimizer state. Inference normally performs only forward computation. Model weights are read but not updated; gradient and optimizer memory disappear.

That makes inference much lighter than training, but not cheap. The weights must remain resident, transient activations and kernel workspaces still exist, and autoregressive generation creates a growing per-request KV cache. In production, the unused training memory is usually converted into serving capacity: more requests in flight, longer contexts, or a larger model.

2. What occupies GPU memory

A useful first-order memory budget is:

[ M_{GPU} \approx M_{weights} + M_{KV} + M_{activations} + M_{workspace} + M_{runtime} + M_{margin} ]

Model weights

For a dense model with (P) parameters stored at (b) bits per parameter:

[ M_{weights,raw} \approx \frac{P \times b}{8}\text{ bytes} ]

For a 7-billion-parameter model:

Storage	Raw weight memory
FP32	28 GB
FP16/BF16	14 GB
INT8/FP8	7 GB
INT4	3.5 GB

These are lower bounds, not deployment guarantees. Quantization scales, zero-points, unquantized layers, alignment, temporary dequantization buffers, CUDA graphs, and allocator reservations add overhead. A “4-bit model” does not imply that the whole server consumes exactly 3.5 GB.

GPU memory is not one undifferentiated pool

The serving engineer should reason about at least five consumers:

Persistent weights: mostly fixed after load.
KV-cache blocks: grow and shrink with active sequences.
Temporary activations: depend on batch shape and the number of scheduled tokens.
Kernel/compiler workspaces: attention kernels, collective communication, CUDA graphs, and compilation may reserve memory.
Allocator slack and fragmentation: free bytes may exist but not in a useful arrangement, or may be held by a caching allocator.

This is why “the checkpoint fits on the GPU” is weaker than “the workload fits.” A 14 GB BF16 checkpoint on a 24 GB card might load successfully yet leave too little KV-cache space to meet concurrency or context targets.

Compute-bound versus memory-bound

A kernel is compute-bound when execution is limited by arithmetic throughput. It is memory-bound when it is limited by moving data through the memory hierarchy. Arithmetic intensity is useful shorthand:

[ \text{arithmetic intensity} = \frac{\text{operations}}{\text{bytes transferred}} ]

Compare that value with the GPU's peak operations-per-byte ratio. NVIDIA's matrix multiplication performance guide uses exactly this roofline-style comparison.

This distinction explains much of LLM serving:

Prefill processes many prompt positions together. Its matrix multiplications are relatively large and often exploit GPU compute well.
Decode usually adds one token per sequence per iteration. At small batch sizes, its matrix multiplications are skinny, and the GPU repeatedly reads large weights and KV state to produce very little new output. It is commonly memory-bandwidth-bound.

Batching can increase the arithmetic intensity of decode because one weight read contributes to multiple sequences. Quantization can reduce the bytes that must be moved. These optimizations work for physical reasons, not because their flags are magical.

3. Tokenization, prefill, and decode

Suppose the request is:

Explain why continuous batching helps an inference server.

The server does not send characters directly into the transformer. A tokenizer converts the rendered prompt—including system instructions, roles, separators, and special tokens—into integer token IDs. Tokenization is typically CPU work. For short text it may be negligible; with high QPS, huge prompts, expensive chat templates, multimodal preprocessing, or insufficient API-server CPU, it can become visible in TTFT.

Token count, not character count, controls most model-side costs. Always benchmark the exact tokenizer and exact rendered prompt used in production.

Prefill

During prefill, the model processes the prompt tokens and constructs attention keys and values for every transformer layer. Causal attention prevents a position from seeing future tokens, but the positions can still be computed in parallel because all prompt tokens are already known.

Prefill produces:

the logits used to choose the first output token;
a KV cache representing the prompt's attention history.

Longer prompts increase prefill work and usually increase time to first token. Naive attention has quadratic work in prompt length, although optimized attention kernels reduce memory traffic and practical constants; they do not make arbitrary context free.

Decode

After the first output token is selected, the future is unknown. To produce token (t+1), the model needs token (t). Generation therefore advances through a dependency chain:

[ x_1 \rightarrow x_2 \rightarrow x_3 \rightarrow \cdots ]

At each decode iteration, the server:

embeds the newest token;
executes the transformer layers for that position;
reads prior keys and values from the KV cache;
appends the new keys and values;
computes logits;
samples or selects the next token;
checks stop sequences, EOS, and token budgets.

Without a KV cache, the server would recompute attention projections for the entire prefix on every step. Caching changes repeated compute into persistent memory consumption.

The first major trade-off

A long prompt hurts prefill latency and consumes KV memory. A long answer hurts decode duration, keeps the request resident longer, increases KV memory further, and delays capacity release. Two requests with the same total tokens can therefore behave differently:

8,000 input + 100 output tokens: prefill-heavy;
100 input + 8,000 output tokens: decode-heavy and long-lived.

Treating both as “8,100-token requests” hides the scheduling problem.

4. The metrics: latency, throughput, and goodput

Average response time is not enough. Instrument the lifecycle.

Time to first token

From the client’s perspective:

[ TTFT = T_{network} + T_{queue} + T_{tokenize} + T_{prefill} + T_{first\ decode} + T_{stream} ]

TTFT answers: How long did the user wait before anything useful appeared? Long prompts mainly affect the prefill term. Saturation mainly affects the queue term.

Inter-token latency and time per output token

Inter-token latency (ITL) is the observed gap between successive streamed chunks or tokens. Time per output token (TPOT) is commonly computed per request as:

[ TPOT = \frac{E2E - TTFT}{N_{output}-1} ]

They are similar under ordinary one-token-at-a-time streaming, but not identical. With speculative decoding, one engine step may yield several accepted tokens in one streamed chunk. The current vLLM benchmark documentation explicitly warns that metric names are not standardized and explains this difference.

End-to-end latency

For ordinary decoding, an approximation is:

[ E2E \approx TTFT + (N_{output}-1)\times TPOT ]

This shows why optimizing TTFT may barely change a 2,000-token answer, and why reducing TPOT may barely matter for a one-token classifier.

Throughput

Measure at least:

request throughput: completed requests per second;
input-token throughput: prompt tokens processed per second;
output-token throughput: generated tokens per second;
total-token throughput: input plus output tokens per second.

Requests per second is meaningless without the input/output length distribution. Tokens per second is more physical, but it still combines prompt and decode tokens that have different cost.

Goodput and successful-task throughput

Maximum throughput can be purchased by making every user slow. Production needs goodput: useful work completed within the service-level objective (SLO).

One practical definition is:

[ \text{goodput} = \frac{\text{successful requests meeting quality and latency SLOs}}{\text{second}} ]

Track p50, p95, and p99, not only means. Queueing systems become nonlinear near saturation; a modest arrival-rate increase can cause tail latency to explode while GPU utilization appears impressively high.

5. KV cache: trading computation for memory

At every attention layer, the model derives a key and a value vector for each token. During decode, old keys and values do not change, so the server stores them.

For a conventional decoder with (L) layers, (H_{KV}) key/value heads, head dimension (D), sequence length (T), and (s) bytes per cache element, an approximate per-sequence cache size is:

[ M_{KV} \approx 2 \times L \times H_{KV} \times D \times T \times s ]

The factor 2 represents keys and values. Across requests, sum over every resident sequence. Tensor parallelism and particular implementations may shard or duplicate parts of this state, so use the formula for planning and engine-reported capacity for deployment.

Worked example

Consider an architecture with 28 layers, 4 KV heads, head dimension 128, and BF16 KV entries:

[ 2 \times 28 \times 4 \times 128 \times 2 = 57{,}344\text{ bytes/token} ]

That is about 56 KiB per cached token. A 32,768-token sequence consumes roughly 1.75 GiB of KV cache. Ten such resident sequences would require roughly 17.5 GiB—before model weights and runtime memory.

Grouped-query attention (GQA) and multi-query attention (MQA) reduce (H_{KV}) relative to the number of query heads, directly shrinking this term. Architecture therefore affects serving economics even when parameter counts look similar.

Why naive allocation wastes memory

Suppose each request is allocated one contiguous region sized for its declared maximum sequence length. Most requests finish early, so the unused tail is wasted. Variable lengths also create holes as allocations come and go. A large request may fail to find a sufficiently large contiguous region even if total free memory is adequate.

This is allocator fragmentation layered on top of uncertain demand.

Paged KV-cache management

PagedAttention divides cache storage into fixed-size physical blocks and maps a sequence's logical token blocks to them. A sequence can grow one block at a time without occupying one giant contiguous allocation. Blocks can be reused and, in some cases, shared. The original PagedAttention paper reports near-zero KV-memory waste and couples block management with preemptive scheduling; current vLLM design documentation describes the paged-cache-compatible attention kernel.

Do not overextend the operating-system analogy. These are engine-managed GPU cache blocks, not ordinary CPU virtual-memory pages, and “paging” should not be read as permission to swap hot KV state to slow storage without cost.

The trade-offs are block-table lookups, metadata, kernel complexity, and internal waste in the final partially filled block. In exchange, the server can fit more useful tokens and therefore more concurrent sequences in the same memory.

Preemption and recomputation

If KV space is exhausted, the scheduler may evict or preempt a request. In current vLLM V1 behavior, recomputation is the default preemption approach: the request later rebuilds discarded state. This preserves liveness but damages latency and burns compute. Frequent preemption is a capacity-planning failure signal, not a harmless optimization. See vLLM's optimization guidance.

6. Why batching changes the machine

One decode request may underutilize a large GPU because it performs small matrix operations and streams a large amount of weight data. Put several independent sequences into the same forward pass, and the same weights serve more work.

Static batching

Collect a fixed set of requests, pad them to compatible shapes, and run the set until every request finishes.

This is simple and effective for offline workloads with known, similar lengths. For online generation it creates two wastes:

shorter prompts may be padded to the longest prompt;
shorter outputs finish but their batch slots remain trapped behind the longest output.

That second failure is head-of-line blocking at batch granularity.

Dynamic batching

Wait for a short time window, gather requests that arrived near one another, form a batch, and then run it. This improves formation under online arrivals but the batch may still be fixed for its lifetime. The waiting window also adds TTFT.

Continuous batching

Autoregressive generation already proceeds in iterations. A continuous scheduler can reconsider membership after each iteration:

remove completed or cancelled sequences;
admit new prompts when capacity becomes available;
batch decode tokens from many active sequences;
interleave or chunk prefill work.

This is also called iteration-level scheduling. It prevents a 50-token answer from remaining tied to a 2,000-token neighbor. The improvement comes from maintaining a useful token batch, not from increasing an HTTP framework's concurrency setting.

Scheduling is multi-dimensional

A scheduler does not merely ask “is there a free slot?” It reasons about:

available KV blocks;
maximum sequences per iteration;
maximum scheduled tokens per iteration;
prompt versus decode work;
request priority and age;
context and output limits;
fairness across tenants;
deadlines and SLO classes;
preemption cost;
prefix-cache locality.

A first-come-first-served policy is easy to explain but can let one giant prefill delay the next decode step for many interactive users.

Chunked prefill

A long prefill is compute-efficient but can monopolize an iteration. Chunked prefill splits it into bounded pieces that can share iterations with decode work. Current vLLM V1 scheduling prioritizes pending decodes, then uses the remaining token budget for prefill chunks. Smaller per-iteration token budgets can improve ITL; larger budgets can improve TTFT and throughput for long prompts. This is a workload-dependent frontier, not a universal setting. The behavior and tuning direction are documented in vLLM optimization and tuning.

7. Prefix caching, prompt caching, and KV reuse

“Prompt cache” is overloaded. Separate three ideas:

Result cache: same request and generation parameters return a previously generated answer. No model execution occurs, but stochastic freshness and authorization make invalidation difficult.
Tokenization/template cache: reuse CPU-side preprocessing.
KV-prefix cache: reuse the model's computed keys and values for a matching token prefix, skipping that portion of prefill.

If many requests begin with the same system prompt, tool definitions, retrieved document, or conversation history, their prefix computation is redundant. A KV-prefix cache indexes previously computed blocks—usually through hashes of token blocks and relevant model/cache identity—and lets a new request reference matching blocks.

The current vLLM automatic prefix caching guide makes the boundary explicit: it can accelerate shared-prefix prefill, but it does not make generation of new output tokens faster.

When prefix caching wins

a long, stable system prefix shared by many users;
repeated questions over the same long document;
multi-turn chat where prior history is unchanged;
agent calls carrying the same tool schemas;
cache-aware routing that sends related requests to the replica holding their blocks.

When it does not

prefixes differ early because of timestamps, random IDs, or per-user data;
outputs dominate total latency;
cache churn evicts blocks before reuse;
round-robin routing destroys replica locality.

Security and correctness

Cache identity must include everything that changes the computation: exact tokens, model revision, adapter, relevant multimodal inputs, and cache precision. Multi-tenant systems need collision-resistant keys and authorization-aware isolation. Current vLLM defaults to SHA-256 for prefix hashes and warns that non-cryptographic hashing creates a theoretical collision and leakage risk; see the current serve configuration.

Caching changes resource consumption, not the model's intended semantics. Yet floating-point scheduling and nondeterministic kernels can still cause tiny numerical differences, so validate reproducibility requirements rather than assuming bitwise identity.

8. Quantization: moving fewer bits, carefully

Quantization maps high-precision numbers into lower-precision representations. Its primary serving benefits are:

smaller persistent weights;
lower memory bandwidth per weight read;
more room for KV cache and concurrency;
sometimes faster matrix operations on hardware with suitable low-precision kernels.

The words “sometimes” and “suitable” matter. A compact checkpoint can be slower if the engine lacks a fused kernel and repeatedly dequantizes values inefficiently.

Weight-only quantization

Weights are stored at low precision while activations remain FP16/BF16 or another higher precision. Common notation includes W4A16 and W8A16. Weight-only methods are attractive for memory-bound decode because they reduce the dominant weight traffic while avoiding the difficulty of quantizing dynamic activations.

Activation quantization

Both weights and activations use lower precision, such as W8A8. This can unlock low-precision tensor-core compute and reduce activation traffic, but activation ranges vary by input and layer. Calibration, scaling granularity, outliers, and hardware support matter more.

KV-cache quantization

This is separate from weight quantization. An FP8 KV cache roughly halves cache storage relative to FP16/BF16, increasing resident-token capacity. It may introduce attention error and extra scaling/conversion overhead. Current vLLM KV-cache documentation describes FP8 cache support and its capacity motivation.

8-bit versus 4-bit

8-bit: generally easier to preserve quality, about half the raw weight memory of 16-bit, and often a natural target where INT8/FP8 hardware is strong.
4-bit: about one-quarter the raw weight memory of 16-bit, often decisive for fitting a model on one GPU, but more sensitive to outliers, group size, scales, and kernel implementation.

Bit width is not the complete method. “INT4” without format, grouping, calibration, kernel, and hardware is not a reproducible configuration.

GPTQ intuition

Naively rounding each weight minimizes a local number-representation error, not the layer's output error. GPTQ quantizes weights using approximate second-order information and compensates remaining weights as it proceeds. The original GPTQ paper presents it as one-shot post-training weight quantization for large generative transformers.

AWQ intuition

AWQ observes that a small fraction of weight channels are disproportionately important under representative activations. It searches for per-channel scaling that protects salient weights while retaining hardware-friendly weight-only quantization. The original AWQ paper motivates activation-aware protection rather than treating all weights equally.

Quality degradation is a system metric

Perplexity alone is insufficient. Quantization errors may appear in:

instruction following;
JSON/schema adherence;
tool argument accuracy;
code correctness;
multilingual text;
long-context retrieval;
rare facts or low-margin token choices;
calibrated probabilities and logprobs.

Keep decoding parameters and prompts fixed, evaluate the production task, and report confidence intervals. A useful evaluation compares exact task success and semantic/rubric scores, plus divergence indicators such as agreement under greedy decoding. Current vLLM supports many quantization paths, but hardware compatibility varies; consult the live vLLM quantization matrix before selecting a format.

9. When one GPU is not enough

There are four distinct reasons to use more GPUs, and each leads to a different parallelism strategy.

Tensor parallelism

Tensor parallelism shards individual layer parameters and computation across GPUs. Every layer step involves multiple devices and collective communication.

Use it when one model replica does not fit on one GPU, especially within a node with high-bandwidth interconnect. It can also free per-GPU memory for KV cache. The costs are synchronization, communication, kernel-shape changes, and a larger failure domain. Adding GPUs may reduce capacity per dollar if the model already fits comfortably on one.

Pipeline parallelism

Pipeline parallelism assigns groups of layers to stages. Hidden states flow through the stages.

It is useful when the model spans nodes or GPUs with less favorable all-reduce connectivity, and it can support uneven layer splits. It introduces pipeline bubbles and can increase per-token latency, especially at low batch size. Current vLLM parallelism guidance recommends a single GPU when the model fits, tensor parallelism within a node, and combined tensor/pipeline parallelism when a model spans nodes; it also notes cases where pipeline parallelism can be preferable without NVLink.

Data parallelism

Data parallel inference replicates the whole model and sends independent request batches to each replica. It scales aggregate throughput and provides a natural availability boundary. Each replica has an independent local KV cache, so naive routing can reduce prefix-cache hit rates. Queue-aware and cache-aware routing outperform blind round-robin under heterogeneous prompts. The current vLLM data-parallel deployment guide calls out queue and cache state as useful routing inputs.

Expert parallelism

Mixture-of-experts models activate only a subset of experts per token. Expert parallelism places experts on different GPUs and routes token representations to the selected experts. It saves each GPU from holding every expert but creates all-to-all communication and load imbalance when popular experts receive more tokens. The number of active parameters controls per-token compute; total expert parameters still influence weight memory. See the current vLLM expert-parallel deployment guide.

Selection rule

Requirement	First strategy to test	Main risk
Model fits; need more QPS	Data-parallel replicas	routing and duplicated weights
Dense model exceeds one GPU, fits one node	Tensor parallelism	collectives on every layer
Model spans nodes or split is uneven	Pipeline + tensor parallelism	bubbles and per-token latency
MoE expert weights dominate	Expert parallelism, often with DP/TP	all-to-all and expert skew

Choose the smallest parallelism degree that satisfies memory and SLO constraints. Parallelism is a tax paid to unlock a capacity or latency benefit.

10. Speculative decoding: spend cheap compute to avoid serial expensive steps

Autoregressive decoding is slow because (K) output tokens normally require (K) serial target-model forward passes. Speculative decoding introduces a fast proposer:

a small draft model proposes several future tokens;
the target model scores those proposed positions in parallel;
a verification algorithm accepts a prefix of proposals and corrects the first rejection;
generation repeats from the accepted state.

If five proposed tokens are accepted, one expensive verification step advances generation by several tokens. With the correct rejection-sampling algorithm, this can preserve the target distribution; the original speculative decoding paper derives that guarantee.

The speedup depends on:

draft latency;
acceptance rate;
number of proposed tokens;
target verification efficiency;
batch size and QPS;
sampling settings;
extra draft weights and KV memory.

A poorly aligned drafter adds work and memory but earns few accepted tokens. At high batch sizes the target GPU may already be efficient, so speculation can reduce throughput even if it helps low-QPS latency. Current vLLM speculative-decoding guidance explicitly positions the method around medium-to-low-QPS, memory-bound workloads and recommends benchmarking by model, traffic, hardware, and sampling setup.

Modern serving engines may also use n-gram, suffix, multi-token-prediction, or EAGLE-like proposers. The invariant is propose cheaply, verify correctly, and measure accepted tokens per target step—not “turn on speculation.”

11. Routing and multi-model serving

Once multiple replicas or models exist, routing becomes part of inference performance.

Request routing among equivalent replicas

Least-connections is better than random only when “connection” approximates work. For LLMs, a request with 32 input tokens and 16 output tokens is not equivalent to one with 32,000 input tokens and a 2,000-token budget.

A useful router considers:

queued and running tokens, not just request count;
free KV capacity;
prompt-prefix cache affinity;
requested adapter/model already loaded;
tenant priority and deadline;
hardware/model compatibility;
replica health and recent tail latency.

Kubernetes' Gateway API Inference Extension reflects this principle by using inference-specific endpoint data such as queues, memory, and loaded adapters rather than generic round-robin alone.

Model routing

Model routing selects a model class, not merely a replica. A small model may handle classification, extraction, or easy chat; a larger model handles difficult or high-risk requests.

This can reduce cost only if routing overhead and misroutes are controlled. Evaluate the complete policy:

[ \text{policy utility} = f(\text{task success},\ latency,\ cost,\ escalation) ]

A router that sends 90% of traffic to a cheap model but ruins 20% of valuable tasks is not optimized.

Multi-model serving

Loading several full checkpoints onto one GPU divides memory and can reduce KV capacity for all of them. Alternatives include:

one GPU pool per model;
dynamic loading and eviction, accepting cold starts;
shared base weights plus LoRA adapters;
memory partitioning where isolation matters;
routing requests to already-warm replicas.

The decision depends on model popularity, switching cost, SLOs, and whether peaks correlate. GPU utilization alone cannot tell you which layout is economical.

12. Protecting the server: admission control and load shedding

An unbounded queue turns overload into timeouts, wasted work, and cascading retries.

Admission control

Before accepting expensive work, estimate whether the server can honor it. Inputs include:

prompt token count;
maximum output tokens;
current queued/scheduled token debt;
available KV blocks;
tenant quota;
deadline and priority;
model and adapter readiness.

Possible actions are accept, queue with a bound, route elsewhere, downgrade, or reject quickly with a retry signal. Reserve capacity for small/interactive or high-priority work rather than allowing giant batch requests to consume every block.

Load shedding

When demand exceeds sustainable capacity, discard the least valuable work deliberately:

reject new low-priority requests;
cap output lengths;
disable optional n-way sampling or logprobs;
route to a smaller model if quality policy allows;
cancel expired queued requests;
reject work that cannot meet its deadline.

Do not start computation for a request whose client has already timed out. Propagate cancellation from the gateway into the engine.

Head-of-line blocking

It appears in several places:

one giant prefill ahead of short interactive prompts;
a static batch waiting for its longest sequence;
one tenant filling the global queue;
pipeline stages waiting on an imbalanced stage;
an overloaded model holding a generic gateway queue.

Chunking, size-aware queues, continuous batching, separate SLO pools, and bounded fairness are targeted remedies.

13. Autoscaling without fooling yourself

CPU-based autoscaling is a poor default for GPU inference. GPU utilization alone is also insufficient: a saturated GPU may be healthy and efficient, while a memory-bound workload can show misleading compute utilization.

Useful scaling signals include:

waiting requests and queued tokens;
queue delay and predicted deadline miss rate;
KV-cache pressure and preemptions;
TTFT and TPOT SLO burn;
per-replica token throughput;
arrival rate split by prompt/output class.

Scale on leading indicators such as queued work, with latency as confirmation. Add stabilization windows and hysteresis to avoid oscillation.

Cold starts

A new replica is not capacity until it can serve. Startup can include:

node provisioning and GPU scheduling;
container/image pull;
checkpoint download;
weight deserialization and GPU transfer;
kernel compilation/autotuning;
CUDA graph capture;
health and warm-up requests.

If startup takes five minutes and traffic doubles in thirty seconds, reactive autoscaling is too late. Keep warm headroom, pre-stage weights, use a fast model store, benchmark startup separately, and scale on forecasts or queue growth.

Scale-to-zero optimizes idle cost but maximizes cold-start exposure. It fits asynchronous work better than strict interactive SLOs.

14. Fragmentation, failures, and recovery

GPU fragmentation

Paged KV blocks address KV allocation waste, but they do not eliminate all fragmentation. PyTorch/CUDA allocators, graph pools, workspaces, model loading/unloading, adapters, and co-located processes can leave unusable layouts or reserved memory.

Monitor:

free versus reserved versus allocated memory;
KV-block utilization;
allocation failures;
preemption/recomputation;
memory after repeated model/adapter churn;
OOMs by batch shape and request class.

Use process isolation and stable memory budgets. Recycle a degraded worker gracefully if fragmentation accumulates; do not wait for random OOMs.

GPU and worker failure

The request has two kinds of state:

reconstructable state: prompt tokens, sampling parameters, generated token IDs, random seed, request metadata;
ephemeral accelerator state: weights resident on a particular worker and its KV blocks.

KV cache is generally not durable application state. After worker loss, reconstruct it from tokens on a healthy worker. This may require replaying the prompt and accepted output tokens.

Recovery design should include:

readiness removed before traffic is routed;
bounded restart and model reload;
replicas in separate failure domains;
request IDs and trace continuity;
cancellation-aware retry;
circuit breakers around failing pools;
enough spare capacity for failover.

Streaming complicates retries. Before the first byte, a gateway can often retry safely. After visible tokens have streamed, blindly retrying may duplicate or diverge. The client protocol needs resumable semantics or must surface a partial failure explicitly.

For multi-GPU tensor/pipeline replicas, one GPU failure usually invalidates the whole replica. Data-parallel replicas provide the failover boundary.

15. Serving observability: explain every slow request

Every request should be attributable across gateway, API process, scheduler, worker, and stream.

Request-level dimensions

Record:

request, tenant, model, model revision, adapter, and replica IDs;
arrival, admission, queue, first-token, and completion timestamps;
rendered input and output token counts;
requested maximum output;
finish reason and cancellation state;
cache-hit tokens;
preemptions/recomputations;
quantization and parallelism configuration;
speculative proposed/accepted token counts;
quality-evaluation version where sampled.

Server metrics

At minimum:

TTFT, TPOT/ITL, and E2E histograms;
request, input-token, and output-token throughput;
waiting/running requests and scheduled tokens;
KV-cache usage and prefix-cache hit ratio;
preemptions and OOMs;
GPU memory, compute, power, temperature, and memory-bandwidth indicators;
model-loading and readiness duration;
error, cancellation, retry, and rejection rates.

The current vLLM production metrics expose queue depth, KV usage, prefix-cache hits, preemptions, phase timings, E2E latency, and ITL via /metrics. Pair these with GPU telemetry such as NVIDIA DCGM metrics and gateway traces.

Diagnostic patterns

Observation	Likely pressure	Confirm with
TTFT rises, TPOT stable	queue or prefill saturation	queue delay, prompt lengths, prefill time
TPOT rises with concurrency	decode saturation	batch size, memory bandwidth, scheduled sequences
KV near full + preemptions	resident-token limit	KV blocks, sequence lengths, recompute count
Low GPU use + long queues	CPU/tokenizer, scheduler, I/O, tiny batches	CPU profiles, batch tokens, engine-step gaps
Prefix hit ratio falls after scale-out	cache-unaware routing	hits per replica and routing affinity
Throughput rises, p99 explodes	operation beyond knee	arrival sweep and queue-growth curve

An alert should link to a hypothesis and runbook, not merely announce “GPU 95%.”

16. Capacity planning from tokens, not hope

Memory capacity

Start with the physical budget:

[ M_{KV,available} = M_{GPU} - M_{weights} - M_{runtime} - M_{activations} - M_{margin} ]

Then estimate token capacity:

[ T_{resident,max} \approx \frac{M_{KV,available}}{M_{KV/token}} ]

Convert that into concurrency only after applying the workload's distribution of active sequence lengths. Maximum context length is a limit, not a realistic average—but admission must still prevent worst-case overcommit.

Compute capacity

Measure sustainable output tokens per second for each workload class at the latency SLO. Then use a queueing safety factor:

[ \text{replicas} \geq \frac{\lambda \times E[\text{work per request}]}{\text{sustainable work per replica} \times \rho_{target}} ]

where (\lambda) is arrival rate and (\rho_{target}<1) leaves headroom. Do not size at the absolute throughput peak; tail latency and failure recovery require slack.

Cost per token

For a GPU pool costing (C_h) per hour and delivering (R) billable tokens per second:

[ \text{cost per million tokens} = \frac{C_h}{3600R}\times 10^6 ]

Report input and output token costs separately when their processing differs. Include idle time, CPU/RAM, storage, networking, orchestration, and redundancy for a production number.

Cost per successful task

Tokens are an internal unit; users buy outcomes. If the service completes (Q) tasks per hour and fraction (p) pass the quality and operational success criteria:

[ \text{successful tasks per dollar} = \frac{Qp}{C_h} ]

or equivalently:

[ \text{cost per successful task} = \frac{C_h}{Qp} ]

This metric correctly punishes a fast quantized model that breaks tool calls, a cheap small model that requires repeated attempts, or an overloaded deployment that times out valuable requests.

Practical project: LLM Inference Benchmark Lab

The lab should produce an evidence package, not one leaderboard number. Never compare configurations while simultaneously changing model, prompts, decoding settings, and traffic.

17. Experimental target

Use one open instruct model with an official or trusted quantized variant. A practical starting point is:

target: Qwen/Qwen2.5-7B-Instruct;
quantized comparison: Qwen/Qwen2.5-7B-Instruct-AWQ;
optional speculative drafter: a tokenizer-compatible smaller Qwen2.5 model.

If your GPU cannot hold the BF16 baseline with useful KV capacity, choose a 3B-class model rather than comparing an OOM with a working quantized server. Record exact model revisions, container digest, engine version, CUDA/driver, GPU SKU, power limit, clocks, and host CPU/RAM.

18. Configuration ladder

Change one causal factor at a time.

ID	Configuration	Purpose
A	BF16, `max_num_seqs=1`, prefix cache off	serial reference
B	BF16, normal continuous batching, prefix cache off	isolate batching/scheduling
C	B + prefix caching	isolate repeated-prefix reuse
D	same scheduler as B, AWQ 4-bit weights	isolate weight quantization
E	best quality-approved base + speculative decoding	test decode acceleration

The max_num_seqs=1 case is a serial approximation inside the same serving engine, not a claim that its other optimized kernels disappear.

Example launch shapes follow. Pin the image by digest in actual experiments; flags evolve, so verify them against the installed version's CLI.

# A: serial reference
vllm serve Qwen/Qwen2.5-7B-Instruct \
  --dtype bfloat16 \
  --max-num-seqs 1 \
  --no-enable-prefix-caching \
  --port 8000

# B: continuous-batching candidate
vllm serve Qwen/Qwen2.5-7B-Instruct \
  --dtype bfloat16 \
  --max-num-seqs 64 \
  --max-num-batched-tokens 8192 \
  --no-enable-prefix-caching \
  --port 8000

# C: prefix reuse
vllm serve Qwen/Qwen2.5-7B-Instruct \
  --dtype bfloat16 \
  --max-num-seqs 64 \
  --max-num-batched-tokens 8192 \
  --enable-prefix-caching \
  --port 8000

# D: weight-only quantization
vllm serve Qwen/Qwen2.5-7B-Instruct-AWQ \
  --quantization awq \
  --max-num-seqs 64 \
  --max-num-batched-tokens 8192 \
  --port 8000

# E: illustrative draft-model speculation
vllm serve Qwen/Qwen2.5-7B-Instruct \
  --speculative-config \
  '{"method":"draft_model","model":"Qwen/Qwen2.5-0.5B-Instruct","num_speculative_tokens":5}' \
  --port 8000

Speculative compatibility is engine- and model-specific. Confirm tokenizer/vocabulary compatibility and treat failed or low-acceptance proposals as an experimental result. The current vLLM configuration schema uses --speculative-config; consult its live speculative-decoding page before running.

19. Workload matrix

Create a JSONL dataset from production-shaped prompts with these strata:

Axis	Values
Input tokens	128, 2K, 8K, 32K where supported
Output tokens	32, 256, 1K
Concurrency	1, 4, 16, 32, 64 or until overload
Arrival pattern	closed-loop and Poisson/open-loop
Prefix reuse	0%, 50%, 90% shared long prefix
Task type	extraction, JSON tool call, QA, code/reasoning

Use both traffic models:

Closed-loop: each simulated user sends the next request after the previous completes. Good for user concurrency, but the offered load falls when the server slows.
Open-loop: arrivals occur independently of completion. Required to expose queue buildup and the saturation knee.

For every cell, include a warm-up, run long enough to stabilize, repeat at least three times, and randomize configuration order if thermal or co-tenant effects are possible.

20. Measurements

Capture:

model loading and ready time;
TTFT p50/p95/p99;
TPOT and ITL p50/p95/p99;
E2E p50/p95/p99;
request, input-token, output-token, and total-token throughput;
GPU allocated/reserved memory and engine KV usage;
GPU compute, memory activity where available, power, and temperature;
queue depth, queue time, running sequences, preemptions, and errors;
prefix-cache queried and hit tokens;
speculative acceptance rate and accepted tokens per step;
exact task success and quality scores.

vllm bench serve reports successful requests, request throughput, token throughput, TTFT, TPOT, and ITL, and can save detailed results. Use its current benchmark CLI reference rather than copying an old command blindly.

The canonical result table is:

Config	Workload	TTFT p95	TPOT p95	E2E p95	out tok/s	req/s	peak GPU GiB	KV %	quality pass %	$/1M tok	successful tasks/$
A	2K/256, C=1	—	—	—	—	—	—	—	—	—	—
B	2K/256, C=32	—	—	—	—	—	—	—	—	—	—
C	8K shared/256, C=32	—	—	—	—	—	—	—	—	—	—
D	2K/256, C=32	—	—	—	—	—	—	—	—	—	—
E	2K/1K, C=1	—	—	—	—	—	—	—	—	—	—

The dashes are intentional. No benchmark result should be invented before the lab runs.

21. Quality-preserving evaluation

Freeze a versioned evaluation set before performance tuning:

deterministic extraction and schema tests;
tool-selection and argument assertions;
reference-based QA with evidence requirements;
code tests executed in a sandbox;
rubric evaluation for open-ended responses;
long-context needle and distractor cases;
safety and refusal regression cases relevant to the application.

Use identical prompt rendering, stop conditions, sampling parameters, maximum tokens, and seeds where supported. For stochastic evaluation, compare distributions or use repeated trials. Require non-inferiority bounds, for example:

no statistically meaningful drop in primary task success;
zero new critical schema/tool regressions;
long-context accuracy within an agreed tolerance;
latency SLO met at the target arrival rate.

Speculative decoding is intended to preserve the target distribution under the correct algorithm, but implementation numerics and reproducibility can still vary. Quantization has no such lossless guarantee. Both stay behind the same evaluation gate.

22. Prefix-cache experiment

Build prompts as:

[ \text{shared 8K-token document} + \text{unique question} ]

Run in this order:

cold cache, prefix caching off;
repeated requests, prefix caching off;
cold cache, prefix caching on;
warm repeated requests, prefix caching on;
the same requests spread randomly across two replicas;
the same requests routed with prefix affinity.

Expected causal signature: warm hits reduce computed prefill tokens and TTFT, while TPOT for long generation changes little. If both TTFT and TPOT fall dramatically, another variable changed.

23. Continuous-batching experiment

Mix output lengths: 32, 128, 512, and 1,024 tokens. Compare A and B at increasing offered load. Plot:

completed output tokens/s versus concurrency;
p95 TTFT and TPOT versus arrival rate;
active sequences over time;
the completion timeline for short and long requests.

Expected signature: B reuses freed sequence capacity, so short requests no longer remain bound to the longest fixed batch. Throughput rises until another resource becomes limiting. Stop increasing load when goodput falls or queues grow without bound.

24. Quantization experiment

Compare B and D on the same prompts and traffic. Record:

startup and checkpoint size;
weight/runtime/KV memory breakdown;
low-concurrency TPOT;
saturation throughput;
maximum concurrency before preemption/OOM;
every quality metric.

Possible outcomes:

memory and speed win: low-bit fused kernels match the GPU well;
memory-only win: more requests fit, but single-request speed is unchanged;
slowdown: dequantization or kernel shapes are inefficient;
quality failure: cost improvement is rejected.

All four are valid findings.

25. Speculative-decoding experiment

Use a decode-heavy workload and sweep proposed-token count, for example 2, 4, and 6. Measure low and high QPS separately. Track:

draft time;
target verification time;
acceptance rate by proposal position;
accepted tokens per target step;
TPOT and output tokens/s;
extra GPU memory;
equality/task quality.

Expect the best point to shift with load. A configuration that wins at concurrency 1 may lose at concurrency 32.

26. Failure-recovery experiment

Run a steady open-loop workload, then terminate one serving worker or GPU pod.

Measure:

detection time;
time until readiness is removed;
failed requests before and after first token;
retry success and duplicate-output behavior;
replacement cold-start duration;
p99 impact on surviving replicas;
time to restore spare capacity.

Pass criteria should distinguish requests that had not streamed from partially streamed requests. Verify that load balancers stop sending new traffic to an unhealthy replica and that the retry storm does not overload survivors.

27. Choosing the winner

Do not choose the configuration with the highest raw token throughput. Define constraints first:

quality pass rate ≥ target;
TTFT p95 ≤ interactive SLO;
TPOT p95 ≤ streaming SLO;
error and preemption rates ≤ limits;
survives one-replica failure at degraded SLO;
lowest cost per successful task among passing configurations.

The “winner” may be a policy: BF16 for sensitive tasks, AWQ for routine traffic, prefix-aware routing for document workloads, and speculation only in low-QPS decode-heavy periods.

The reconstructed production architecture

Once all requirements are introduced, the original one-process server becomes:

flowchart TD
    A["Clients"] --> B["Gateway: auth, limits, deadlines"]
    B --> C["Admission and model router"]
    C --> D["Queue/cache-aware replica router"]
    D --> E["API and tokenizer"]
    E --> F["Continuous scheduler"]
    F --> G["Paged KV manager"]
    F --> H["GPU workers: prefill and decode"]
    G --> H
    H --> I["Streaming response"]
    J["Metrics, traces, quality samples"] --> K["Autoscaling and capacity policy"]
    B --> J
    F --> J
    H --> J
    K --> C

The architecture exists because each added production condition created a failure:

Condition	Failure in the simple server	Derived mechanism
Longer prompts	TTFT and cache grow	chunked prefill, prefix reuse, context limits
Concurrent users	GPU underuse or queue explosion	continuous batching and bounded admission
Large weights	model does not fit	quantization or model parallelism
KV growth	OOM and fragmentation	paged cache, token-aware capacity
Variable outputs	fixed-batch blocking	iteration-level scheduling
Strict latency	throughput tuning starves users	SLO classes, decode priority, headroom
High throughput	single replica saturates	larger batches and data parallelism
Multiple models	memory duplication and cold swaps	model routing, warm pools, adapters
GPU failure	in-flight state disappears	replica failover and reconstructable state
Cost pressure	tokens cheap but tasks fail	quality gates and cost per successful task

Mastery gate: the reconstruction review

You have mastered this topic when you can produce a design review for the following system without reaching first for an engine flag:

Serve an open 32B model for an agent platform. Traffic is 60% short tool calls (1K input, 80 output), 30% document questions (16K shared prefix, 500 output), and 10% long-form generation (2K input, 2K output). Peak arrival rate is 12 requests/s. The SLOs are p95 TTFT below 800 ms for tool calls and p95 TPOT below 40 ms. One GPU worker may fail without losing more than 1% of requests. Quantization is allowed only if task success falls by less than 0.5 percentage points.

Your review must contain:

approximate BF16, 8-bit, and 4-bit weight memory;
KV bytes per token from the chosen model configuration;
a separation of prefill and decode demand by traffic class;
a latency-versus-throughput benchmark matrix;
the continuous-batching and chunked-prefill policy;
prefix-aware routing for the document class;
a quantified quantization non-inferiority test;
a justified TP/PP/DP/EP choice;
a speculative-decoding hypothesis for the decode-heavy class;
admission, load shedding, and cancellation rules;
autoscaling signals that account for cold-start delay;
single-worker failure behavior and streaming retry semantics;
dashboards connecting queue, KV pressure, TTFT, TPOT, quality, and cost;
cost per million tokens and per successful task;
a final architecture reconstructed from those constraints.

A passing answer does not need perfect capacity numbers before hardware tests. It must make assumptions explicit, derive approximate bounds, and specify which benchmark will replace each assumption with evidence.

Coverage map

Every syllabus topic appears in the connected derivation above.

Topics	Covered in
1–4: inference/training, weights, GPU memory, compute vs memory	§§1–2
5–7: tokenization, prefill, decode	§3
8–13: TTFT, TPOT, E2E, throughput, RPS, TPS	§4
14–15: KV cache and growth	§5
16–22: batching, scheduling, blocking, paged KV	§§6 and 5
23–25: prefix/prompt caching and KV reuse	§7
26–31: quantization, GPTQ/AWQ, quality	§8
32–35: tensor, pipeline, data, expert parallelism	§9
36–37: speculative decoding, draft/target	§10
38–40: model/request routing, multi-model serving	§11
41–42: admission and load shedding	§12
43–46: autoscaling, cold start, fragmentation, recovery	§§13–14
47: observability	§15
48–50: capacity, cost/token, cost/success	§16

Primary references

Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention, SOSP 2023.
Yu et al., Orca: A Distributed Serving System for Transformer-Based Generative Models, OSDI 2022.
Leviathan et al., Fast Inference from Transformers via Speculative Decoding, 2022/2023.
Frantar et al., GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers.
Lin et al., AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration.
vLLM current documentation: architecture, metrics, benchmarking, caching, quantization, speculation, and parallel deployment.
NVIDIA, Matrix Multiplication Background User's Guide.
Kubernetes, Gateway API Inference Extension.

The central principle is simple: optimize the bottleneck you measured, preserve the quality users need, and judge the system by successful work—not by how busy the GPU looks.