Deriving the Transformer from the Problem of Predicting Language

A transformer can look like a pile of matrices, arrows, and unfamiliar names. It is easier to understand if we do not start with the finished diagram. Instead, we will start with a modest goal:

Given some text, predict the next token.

Every major transformer component will appear because the mechanism built so far cannot yet solve some part of that problem.

1. Learning a function instead of writing every rule

Suppose we want software that receives an input (x) and produces an output (y). Ordinary software uses a function written by a programmer:

[ y = f(x) ]

But nobody can write complete rules for every valid continuation of every piece of text. We therefore choose a flexible function with adjustable values called parameters:

[ \hat{y} = f(x; \theta) ]

(x) contains the input information.
(\theta) contains the parameters learned from examples.
(\hat{y}) is the model's prediction.

Training is the process of finding parameter values that make useful predictions. The architecture decides how information may flow; training decides the numerical values used by that architecture.

The smallest useful learned function

A linear function computes a weighted combination:

[ z = w_1x_1 + w_2x_2 + b ]

If (x_1=2), (x_2=3), (w_1=0.5), (w_2=-1), and (b=1), then:

[ z=(0.5)(2)+(-1)(3)+1=-1 ]

The weights say how strongly each input affects the result; the bias shifts the result. A neuron is essentially this linear combination followed by an activation function:

[ h = \sigma(Wx+b) ]

Without the activation, stacking linear operations still produces only another linear operation. Such a network could not learn curved or conditional relationships. Activations such as ReLU introduce nonlinearity:

[ \operatorname{ReLU}(z)=\max(0,z) ]

A layer evaluates many neurons in parallel. A neural network stacks layers so later layers can construct useful patterns from representations produced by earlier ones.

import numpy as np

x = np.array([2.0, 3.0])
W = np.array([[0.5, -1.0],
              [1.0,  0.25]])
b = np.array([1.0, -0.5])

z = x @ W.T + b             # linear combinations
h = np.maximum(z, 0)        # ReLU activation
print(z, h)

The sequence of computations from input to prediction is the forward pass.

2. How a network learns

A prediction alone does not tell the network how to improve. We need a loss function: a number measuring the disagreement between the prediction and the desired answer.

For next-token prediction, the model produces a probability distribution over the vocabulary. If the correct next token is sat, cross-entropy loss becomes small when the probability assigned to sat is high and large when it is low:

[ L=-\log p(\text{sat}) ]

If (p(\text{sat})=0.8), the loss is about (0.22). If it is (0.01), the loss is about (4.61).

The gradient of the loss tells us how a small change in each parameter would change the loss. Backpropagation efficiently applies the chain rule backward through the operations of the forward pass to calculate those gradients. It does not decide what the model ought to represent; it assigns numerical credit or blame to parameters for the current error.

Gradient descent then adjusts parameters slightly in the direction that reduces loss:

[ \theta \leftarrow \theta-\eta\nabla_{\theta}L ]

Here (\eta), the learning rate, controls the update size. One example causes one tiny correction. Repeated over enormous collections of examples, these corrections make the network reproduce statistical structure found in the training data.

That gives us a general learning mechanism. It does not yet tell us how language should enter the network.

3. Turning language into model inputs

Language modelling

A language model assigns probabilities to token sequences. An autoregressive language model factors a sequence into next-token predictions:

[ P(t_1,\ldots,t_n)=\prod_{i=1}^{n}P(t_i\mid t_1,\ldots,t_{i-1}) ]

For The cat sat, training cases include:

The → predict cat
The cat → predict sat

This creates labels automatically from ordinary text: at every position, the following token is the target.

Why tokens are necessary

Neural networks operate on numbers, not raw strings. We first divide text into tokens using a tokenizer. A token may be a word, punctuation mark, character, or subword fragment. It is not inherently a word or a unit of meaning.

A tokenizer might encode:

"unbelievable!" → ["un", "believ", "able", "!"] → [91, 804, 37, 5]

The tokenizer's finite set of possible tokens is its vocabulary. Each token has an integer ID. The ID is merely an address; ID 804 is not numerically “more meaningful” than ID 37.

Subword tokenization is a compromise. Whole-word vocabularies become enormous and handle unseen words badly. Character sequences are flexible but long. Reusable fragments keep the vocabulary and sequence lengths manageable, though token boundaries may look unnatural and differ across models.

Why token IDs are not enough

Passing the raw integer 804 into a network would falsely suggest an ordered numeric relationship among token IDs. We instead use an embedding matrix:

[ E\in\mathbb{R}^{|V|\times d} ]

(|V|) is the vocabulary size.
(d) is the representation width.
Row (E_i) is the learned vector for token ID (i).

An embedding lookup selects a row:

[ x_i=E[t_i] ]

The vector contains learned features useful for reducing prediction loss. Similar usage can produce geometrically related vectors, but an embedding should not be treated as a little dictionary definition stored inside the model.

E = np.array([
    [0.2, 0.7, -0.1],   # "the"
    [0.9, 0.1,  0.3],   # "cat"
    [0.6, 0.4,  0.8],   # "sat"
])

token_ids = np.array([0, 1])
x = E[token_ids]        # shape: (sequence_length, embedding_dimension)

We now have one vector per token. But two new problems remain: order and relationships.

4. Language is ordered and contextual

dog bites person and person bites dog contain the same token set but mean different things operationally: they imply different likely continuations. A model therefore needs sequence order.

It also needs context. Here, context means the preceding tokens available when calculating the next-token distribution. The token bank should influence predictions differently after river than after central.

Earlier sequence models such as recurrent neural networks processed tokens one after another while maintaining a hidden state:

[ h_t=f(h_{t-1},x_t) ]

This gives them order and context, but creates important limitations:

Token positions must be processed sequentially, limiting training parallelism.
Information from distant tokens must repeatedly pass through an evolving fixed-width state.
Long dependency paths make learning distant relationships difficult, even though gated variants such as LSTMs improve it.

We want every token representation to retrieve relevant information from earlier tokens directly. That requirement leads to attention.

5. Deriving attention from information retrieval

Consider predicting the missing word in:

The animal did not cross the street because it was too tired.

To build a useful representation for it, the model should draw strongly from animal. In another sentence, a pronoun may need a different source. A fixed rule such as “always use the previous token” will fail.

For each token, we need a learned retrieval operation:

Describe what information the current position is looking for.
Describe what information each available position offers.
Compare those descriptions.
Mix the offered content according to the comparison.

These roles are called query, key, and value.

Queries, keys, and values

From each token representation (x_i), learned matrices produce three vectors:

[ q_i=x_iW_Q,\qquad k_i=x_iW_K,\qquad v_i=x_iW_V ]

A query contains features describing what the receiving position is seeking.
A key contains features used to decide whether a source position matches that query.
A value contains the information that will be transferred if the source is selected.
(W_Q,W_K,W_V) are parameters learned from prediction errors.

The names describe computational roles, not fixed linguistic meanings. A query does not literally store an English question.

Attention scores

We compare query (q_i) with every permitted key (k_j) using a dot product:

[ s_{ij}=\frac{q_i\cdot k_j}{\sqrt{d_k}} ]

A larger dot product means stronger alignment in the learned feature space. Division by (\sqrt{d_k}) keeps scores from growing excessively as vector width increases, which helps optimization.

Imagine one query and three keys produce scores:

[ [2,1,0] ]

Raw scores do not yet form mixture weights. Softmax exponentiates and normalizes them:

[ \operatorname{softmax}(s_i)j=\frac{e^{s{ij}}}{\sum_m e^{s_{im}}} ]

The resulting weights are approximately:

[ [0.665,0.245,0.090] ]

They are nonnegative and sum to one. Softmax is not merely selecting one token: it creates a differentiable weighted mixture.

The output for position (i) is:

[ z_i=\sum_j a_{ij}v_j ]

If the value vectors are ([1,0]), ([0,2]), and ([3,1]), the contextualized result is approximately:

[ 0.665[1,0]+0.245[0,2]+0.090[3,1]=[0.935,0.580] ]

The output now contains a learned mixture of information from relevant positions.

Matrix form

For a whole sequence:

[ Q=XW_Q,\quad K=XW_K,\quad V=XW_V ]

[ \operatorname{Attention}(Q,K,V)= \operatorname{softmax}\left(\frac{QK^T}{\sqrt{d_k}}+M\right)V ]

The mask (M) is crucial in a next-token model. It sets attention to future positions to an effectively impossible score. Otherwise, during training, a position could inspect the answer it is supposed to predict. With a causal mask, tokens can attend only to themselves and earlier positions.

This is self-attention because queries, keys, and values all come from the same sequence. Its output is a contextualized representation: the vector for a token now depends on other permitted tokens, not only on that token's embedding.

6. One relationship is not enough

A single attention calculation has one set of query, key, and value projections. Language contains different useful relationships: local syntax, reference resolution, delimiter matching, topic continuity, and many others. We therefore run several attention operations in parallel with different learned projections.

This is multi-head attention:

[ \operatorname{head}_r=operatorname{Attention}(XW_Q^{(r)},XW_K^{(r)},XW_V^{(r)}) ]

[ \operatorname{MHA}(X)=operatorname{Concat}(\operatorname{head}_1,\ldots,\operatorname{head}_h)W_O ]

Each head has the opportunity to learn a different retrieval pattern. This is capacity, not a promise that every head has a clean human-readable job.

Attention alone does not know order

Self-attention without position information treats its inputs like a set: rearranging tokens rearranges outputs but does not otherwise reveal which token came first. We must inject positional information.

One approach adds a position vector (p_i) to each token embedding:

[ x_i=E[t_i]+p_i ]

Positions may be fixed sinusoidal vectors, learned embeddings, or encoded through relative/rotary mechanisms. The implementation differs, but the requirement is the same: attention must be able to distinguish content at different positions and reason about their relative placement.

7. From attention to a transformer block

Attention moves information between token positions. After information reaches a position, the network still needs to transform it. A position-wise feed-forward network supplies that computation:

[ \operatorname{FFN}(x)=W_2,\sigma(W_1x+b_1)+b_2 ]

The same learned transformation is applied independently to every position. A useful mental split is:

Attention: retrieve and combine information across positions.
Feed-forward network: transform the information available at each position.

Deep networks introduce an optimization problem: every new transformation could overwrite useful representations, and gradients must travel through many operations. Residual connections preserve a direct path:

[ y=x+F(x) ]

The sublayer learns a modification to the existing representation rather than constructing everything again.

Normalization keeps activation scales controlled and improves training stability. Modern transformers commonly use LayerNorm or RMSNorm in arrangements that vary by architecture. Conceptually, normalization prevents the numerical scale of representations from drifting unpredictably across many blocks.

A simplified pre-normalized transformer block is:

[ H'=H+\operatorname{MHA}(\operatorname{Norm}(H)) ]

[ H''=H'+\operatorname{FFN}(\operatorname{Norm}(H')) ]

Stacking many blocks allows repeated rounds of retrieval and transformation. Early representations come directly from token and position encodings. Later representations can encode increasingly context-dependent features useful for prediction.

For a decoder-only language model, the end-to-end path is:

text → tokens → token IDs → embeddings + positions
     → repeated masked transformer blocks
     → vocabulary logits → probabilities for the next token

8. Turning the final representation into a next-token prediction

At each position, the final block produces a vector (h_i). A linear output projection converts it into one score, called a logit, for every vocabulary token:

[ \ell_i=h_iW_{\text{vocab}}+b ]

Softmax converts logits to probabilities:

[ P(t_{i+1}=v\mid t_{\le i})=operatorname{softmax}(\ell_i)_v ]

If the logits for [cat, dog, runs] are [2.0, 1.0, 0.0], the probabilities are approximately [0.665, 0.245, 0.090].

During training, the model sees many known sequences. It predicts the following token at every position, loss compares all predictions with the actual following tokens, backpropagation computes gradients, and an optimizer updates parameters. A causal mask prevents future-token leakage even though many positions can be trained in parallel.

During inference, the parameters are normally fixed. The model:

tokenizes the available context;
performs a forward pass;
obtains the next-token distribution;
selects one token;
appends it to the context;
repeats until a stopping condition is reached.

The model does not ordinarily write an entire response in one operation. It repeatedly conditions on its own generated tokens. One poor early choice can therefore redirect later probabilities.

9. From pretraining to an assistant

Pretraining

In pretraining, the model learns next-token prediction from a very large and varied text corpus. To lower loss, its parameters must capture recurring statistical structure: spelling, syntax, semantic associations, discourse patterns, common facts, code structure, and patterns resembling procedures and argumentation.

Operationally, saying that a model “understands” a pattern should mean something testable—for example, that it can use contextual information to predict, classify, transform, or generate appropriate continuations across novel inputs. It does not imply human experience, grounded awareness, or a symbolic database containing explicit beliefs.

Instruction tuning

A pretrained model is optimized to continue text, not necessarily to follow a user's request. Instruction tuning continues training on examples that pair instructions with desired responses. It changes the conditional behaviour: prompts formatted as requests become more likely to receive useful, direct responses.

Preference optimization

Several answers can be plausible while differing in usefulness, safety, style, or honesty. Preference optimization trains against comparative or scored feedback so preferred responses become more probable than disfavoured ones. Methods differ, but the purpose is behavioural shaping beyond raw continuation and supervised demonstrations.

Neither instruction tuning nor preference optimization replaces pretraining knowledge. They alter which behaviours are elicited and selected.

10. Sampling: probabilities become actual text

Always choosing the highest-probability token is greedy decoding. It is reproducible under fixed conditions but may become repetitive or settle into locally likely wording. Sampling instead draws from the predicted distribution, allowing different valid continuations.

Temperature rescales logits before softmax:

[ P(v)=\operatorname{softmax}\left(\frac{\ell_v}{T}\right) ]

Lower (T) sharpens the distribution, concentrating probability on leading candidates.
Higher (T) flattens it, giving lower-ranked tokens more chance.
Temperature does not add knowledge or directly measure creativity.

def softmax(x):
    x = x - np.max(x)
    e = np.exp(x)
    return e / e.sum()

logits = np.array([2.0, 1.0, 0.0])

for temperature in [0.5, 1.0, 2.0]:
    probabilities = softmax(logits / temperature)
    print(temperature, probabilities)

Production decoders may also use top-k or top-p filtering, repetition controls, and explicit stop tokens. These are inference policies around the model's distribution, not changes to the trained parameters.

11. The context window is the model's active input boundary

The context window is the maximum number of tokens the model can consider in one inference request, including instructions, retrieved material, conversation history, tool results, and generated tokens counted under the API's rules.

Information outside that window is not directly available during the forward pass. Information inside the window is available but not guaranteed to be used correctly: attention and downstream transformations are learned and imperfect. Longer context also consumes computation and can make relevant evidence harder to distinguish from noise.

This is why application-level context engineering matters. A system must decide what to retrieve, include, order, label, summarize, or omit. The window is not durable memory; durable state must live outside the model and be reintroduced when needed.

12. Hallucination without anthropomorphism

A language model is trained to produce a probable continuation, not to execute a built-in truth-verification procedure. A fluent false statement may receive high probability because it resembles patterns associated with a plausible answer.

Hallucination can therefore be described operationally as generated content that is unsupported by the available evidence or inconsistent with the relevant external facts. It can arise when:

the training signal rewards likely wording rather than verified truth;
the prompt lacks necessary evidence;
relevant parameterized knowledge is weak, conflicting, or outdated;
sampling selects an unfortunate continuation;
generated mistakes become context for subsequent tokens;
the task requires calculation, retrieval, or state the model cannot reliably perform internally.

The probability assigned to a token is not a calibrated probability that the claim containing it is true. A model can be locally confident about wording and globally wrong about the world.

Applications reduce this risk with retrieval, tools, explicit citations, validation, constrained outputs, uncertainty policies, human review, and evaluations. These mechanisms supply evidence or enforcement that next-token prediction alone does not guarantee.

13. How next-token prediction produces broader capabilities

At first, predicting one token sounds too narrow to produce summarization, translation, classification, code generation, or tool use. But the training objective applies to many kinds of text. To predict continuations across those texts, the network benefits from internal features that track syntax, entities, topics, relationships, formats, procedures, and common transformations.

Scale can produce emergent application capabilities: behaviours become useful or noticeable only after sufficient model capacity, data, training, and suitable prompting. The word “emergent” describes the observed capability curve; it does not identify a magical new mechanism. The runtime remains repeated forward passes producing token distributions.

Application behaviour comes from several layers working together:

Layer	What it contributes
Pretraining	Broad statistical patterns and reusable representations
Post-training	Instruction following, preferences, safety, and interaction style
Prompt and context	The task, evidence, examples, and current state
Decoding	How one continuation is selected from probabilities
Application code	Tools, permissions, validation, memory, retries, and enforcement

A prompt can elicit a learned behaviour, but it cannot guarantee facts, permissions, schema validity, or business invariants. Those require application mechanisms.

14. Mechanistic limitations that remain

The transformer solves important sequence-modelling problems, but its mechanism imposes boundaries:

Finite context: only represented input tokens can directly influence the current prediction.
Quadratic standard attention cost: comparing every token with every other token makes ordinary full self-attention scale roughly with the square of sequence length.
No native truth guarantee: likelihood of text is not factual verification.
No durable state by default: parameters and the current context are not a transactional application database.
Prompt sensitivity: small wording or ordering changes can alter internal activations and the resulting distribution.
Autoregressive error propagation: generated tokens become inputs to later steps.
Opaque distributed representation: knowledge and computation are spread across parameters and activations; they are not generally editable or inspectable as clean symbolic records.
Dependence on data and objective: biases, omissions, contradictions, and weak coverage in training signals affect behaviour.
Limited grounding: without tools or supplied evidence, the model has no automatic access to current external reality.
Compute and latency costs: larger models, longer contexts, and longer outputs require more resources.

These limitations explain why reliable LLM systems add retrieval, deterministic code, tool interfaces, state stores, security boundaries, observability, and evaluation around the model.

15. Tiny educational self-attention project

The following NumPy program is intentionally small. It is not a useful foundation model. Its purpose is to expose the complete data path:

import numpy as np

np.set_printoptions(precision=3, suppress=True)
rng = np.random.default_rng(7)

vocabulary = ["the", "cat", "sat", "dog", "ran"]
token_to_id = {token: i for i, token in enumerate(vocabulary)}
tokens = ["the", "cat", "sat"]
token_ids = np.array([token_to_id[token] for token in tokens])

vocab_size = len(vocabulary)
d_model = 4
d_key = 3

# Learned in a real model; random here so we can inspect the mechanics.
embedding_table = rng.normal(0, 0.5, (vocab_size, d_model))
position_table = rng.normal(0, 0.1, (len(tokens), d_model))
W_q = rng.normal(0, 0.5, (d_model, d_key))
W_k = rng.normal(0, 0.5, (d_model, d_key))
W_v = rng.normal(0, 0.5, (d_model, d_model))
W_vocab = rng.normal(0, 0.5, (d_model, vocab_size))

def softmax(x, axis=-1):
    x = x - np.max(x, axis=axis, keepdims=True)
    e = np.exp(x)
    return e / e.sum(axis=axis, keepdims=True)

def probabilities_at_temperature(logits, temperature=1.0):
    if temperature <= 0:
        raise ValueError("temperature must be positive")
    return softmax(logits / temperature)

# 1. IDs select token vectors; positions add order information.
token_embeddings = embedding_table[token_ids]
X = token_embeddings + position_table

# 2. Each position produces a query, key, and value.
Q = X @ W_q
K = X @ W_k
V = X @ W_v

# 3. Every query is compared with every key.
scores = Q @ K.T / np.sqrt(d_key)

# 4. Prevent each position from looking into the future.
causal_mask = np.triu(np.full(scores.shape, -np.inf), k=1)
masked_scores = scores + causal_mask
attention_weights = softmax(masked_scores, axis=-1)

# 5. Each row becomes a weighted mixture of value vectors.
contextualized = attention_weights @ V

# 6. Use the final position to predict the next token.
next_token_logits = contextualized[-1] @ W_vocab
next_token_probabilities = probabilities_at_temperature(
    next_token_logits, temperature=1.0
)

# 7. Sample one token from the distribution.
sampled_id = rng.choice(vocab_size, p=next_token_probabilities)

print("token IDs:\n", token_ids)
print("embeddings + positions:\n", X)
print("queries:\n", Q)
print("keys:\n", K)
print("values:\n", V)
print("masked attention scores:\n", masked_scores)
print("attention weights:\n", attention_weights)
print("contextualized representations:\n", contextualized)
print("next-token probabilities:\n", dict(zip(vocabulary, next_token_probabilities)))
print("sampled token:\n", vocabulary[sampled_id])

Run it several times with the random seed unchanged: the matrices and probabilities stay fixed, while sampling may vary. Then perform these experiments:

Change the temperature to 0.3 and 2.0; compare probability concentration and sampled tokens.
Remove position_table; observe that the mechanism has lost explicit order information.
Remove the causal mask; inspect how earlier positions can then use future values, which would leak training answers.
Replace tokens with another sequence and trace every resulting shape.
Set a query equal to one key by hand and observe how its score and attention weight change.

The program omits multi-head attention, normalization, residual connections, feed-forward layers, training, and batching. Add them one at a time only after you can explain every printed array.

16. Reconstructing the complete mental model

You can now rebuild the transformer without memorizing its diagram:

We need to learn a function too complicated to program manually, so we use parameterized neural layers.
Linear combinations need nonlinear activations to represent complex relationships.
A forward pass produces predictions; loss measures error; backpropagation and gradient descent adjust parameters.
Language must become numbers, so tokenization maps text to vocabulary IDs and embeddings map IDs to learned vectors.
Language depends on order and context, while recurrent processing creates long dependency paths and poor parallelism.
Attention lets each position retrieve information directly from relevant permitted positions using queries, keys, values, scores, softmax weights, and weighted sums.
Multiple heads offer several learned retrieval spaces; positional information makes order visible.
Feed-forward layers transform each position; residual connections and normalization make deep stacking trainable.
Repeated transformer blocks produce contextual representations, which an output projection converts into next-token probabilities.
Pretraining learns broad patterns; instruction tuning and preference optimization shape interaction behaviour.
Inference repeatedly selects and appends one token. Sampling policy and temperature affect which continuation is realized.
Context is finite, probabilities are not truth guarantees, and reliable applications must supply external evidence and deterministic controls.

That is the central transformer mental model: repeated learned retrieval and transformation of token representations, optimized to predict the next token. The remarkable application behaviour is real, but it remains behaviour produced by that mechanism together with post-training, context, decoding, tools, and application code.