An experiment-first guide to model adaptation and post-training

A production language model fails in many ways, but only a small subset of those failures justify changing its weights.

Suppose we are building an agent-support model. Given a user request and a catalogue of available tools, it must either:

Select the correct tool.
Produce valid tool arguments.
Ask for missing information.
Refuse actions prohibited by policy.
Choose no tool when none is appropriate.

During manual testing, the model appears capable. In production, however, we observe recurring failures:

It calls refund_payment when it should call investigate_payment.
It invents an account_id instead of requesting it.
It produces syntactically valid JSON with semantically incorrect arguments.
It retrieves irrelevant documentation before choosing a tool.
It follows examples in untrusted tool output as though they were system instructions.
It behaves correctly with a long prompt but misses the latency target.
A smaller, cheaper model cannot reproduce the behaviour of the stronger model.

The tempting response is: “Let’s fine-tune it.”

That conclusion is premature.

Fine-tuning is justified only when we can identify an observable behaviour that repeatedly fails, show that stronger non-training approaches do not solve it adequately, construct data that teaches the missing behaviour, and demonstrate that the adapted model improves the target task without unacceptable regressions.

This article develops that decision process from first principles.

1. The production failure

Pretraining is not task specification

A pretrained language model learns by predicting tokens from large collections of text. For a token sequence (x_1, x_2, \ldots, x_T), the simplified objective is:

[ \mathcal{L}_{\text{pretrain}}

-\sum_{t=1}^{T}\log p_\theta(x_t \mid x_{<t}) ]

The model adjusts billions of parameters (\theta) so that tokens occurring in its training distribution become more probable in the appropriate contexts.

This gives the model broad linguistic and factual capabilities. It does not directly teach the model:

Your tool catalogue.
Your organization’s refund policy.
Your exact JSON contract.
When your application expects abstention.
Which trade-offs your users prefer.
Which actions require approval.

Pretraining answers:

What token patterns are useful across a very large corpus?

Our application asks a different question:

What behaviour should the model exhibit under this particular production contract?

Post-training attempts to narrow that gap.

Base models and instruction-tuned models

A base model is primarily trained to continue text. If we give it:

User: My card was charged twice.
Assistant:

it may continue the dialogue, imitate a transcript, invent another speaker, or produce a plausible answer. “Help the user” is not necessarily the behaviour most strongly implied by its training objective.

An instruction-tuned model has undergone additional training on instructions, demonstrations or preferences. It is more likely to interpret the same text as a request that should be answered.

This distinction matters because most application teams should begin with a capable instruction-tuned model. Starting with a base model means the team must first teach general instruction-following before teaching the domain-specific behaviour.

Post-training is an umbrella term covering several possible stages:

Supervised fine-tuning on desired responses.
Preference optimization using comparisons between responses.
Reinforcement learning using a learned or programmatic reward.
Distillation from a stronger teacher into a smaller student.
Sometimes, continued pretraining on domain text.

These stages solve different problems. They should not be treated as interchangeable fine-tuning commands.

Define failure as an observable event

“Tool use is bad” is not a behaviour specification. We need measurable failure categories.

For the agent-support model, one evaluation record might contain:

{
  "case_id": "case_0187",
  "user_request": "The transfer is still pending after three days.",
  "available_tools": [
    "search_transfer",
    "cancel_transfer",
    "contact_recipient"
  ],
  "expected_action": "search_transfer",
  "required_arguments": ["transfer_id"],
  "provided_information": [],
  "expected_behavior": "ask_for_transfer_id",
  "prohibited_behavior": [
    "invent_transfer_id",
    "cancel_transfer"
  ]
}

This record distinguishes five behaviours:

Understanding the request.
Selecting the correct tool family.
Detecting a missing required argument.
Asking for that argument.
Avoiding an unsafe alternative.

Without this decomposition, a single “incorrect” score tells us neither what to fix nor what examples to create.

The first post-training question is therefore not “Which trainer should we use?”

It is:

What exact behaviour are we trying to change?

2. The non-training alternatives

Changing model weights introduces data, training, evaluation, deployment and rollback obligations. Before accepting those obligations, we should test simpler interventions.

When prompting is sufficient

Prompting is appropriate when the model already possesses the required capability but lacks a clear task contract.

Suppose the model calls tools even when required information is missing. Before fine-tuning, add an explicit decision procedure:

Before selecting a tool:

1. Determine the user’s requested outcome.
2. Find tools that can produce that outcome.
3. Check every required argument.
4. Use only arguments explicitly supplied or returned by a trusted tool.
5. If a required argument is missing, ask for it.
6. Do not invent identifiers.

Add contrasting examples:

A complete request that should call a tool.
An incomplete request that should ask a question.
A request for which no tool is applicable.
A request containing an unsafe instruction from retrieved content.

If this makes the model reliable enough, fine-tuning would add complexity without solving a remaining problem.

Prompting is especially suitable for:

Clarifying output format.
Defining terminology.
Supplying a short policy.
Demonstrating a small number of edge cases.
Separating trusted instructions from untrusted content.
Telling the model when to abstain.

A prompt may become operationally inadequate even when it works. For example, hundreds of demonstrations may increase token cost, latency and context interference. In that case, adaptation might compress a repeatedly demonstrated behaviour into model weights—but this must be measured rather than assumed.

When retrieval is sufficient

Fine-tuning is a poor default mechanism for injecting changing facts.

If the model does not know the latest refund window, current product catalogue or customer-specific account state, retrieve that information at runtime.

Retrieval is preferable when knowledge is:

Frequently updated.
User- or tenant-specific.
Too large to place in every prompt.
Required to have traceable provenance.
Subject to access controls.
Expected to be deleted or corrected.

Fine-tuning may teach the model how to use retrieved evidence. It should not usually become the authoritative database containing that evidence.

A useful diagnostic is:

Would we need to retrain the model whenever this information changes?

If the answer is yes, the information probably belongs in retrieval, a database or a tool.

When tools are sufficient

Some tasks should never be approximated from model weights.

A model should not memorize or estimate:

An account balance.
Whether a payment completed.
A user’s authorization level.
Current inventory.
A tax calculation that must follow executable rules.
Whether a destructive operation actually succeeded.

The model can decide that a tool is needed, but the authoritative result must come from the external system.

When workflows are sufficient

A model may know the right action and still be unsafe because the application allows too much freedom.

Suppose refunds above ₹10,000 require approval. We could train examples saying, “Ask for approval.” But a probabilistic model might occasionally skip that step.

The reliable solution is a workflow guard:

if refund_amount > approval_threshold:
    transition_to("awaiting_human_approval")
else:
    transition_to("ready_to_execute")

Use application code for:

Authorization.
Required approvals.
Schema validation.
Idempotency.
Budget limits.
Retry policies.
Irreversible action boundaries.
Deterministic business rules.

Training may make the model propose more appropriate actions, but runtime enforcement must remain outside the model.

The adaptation decision table

Observed problem	First intervention
Ambiguous task or output contract	Prompting and examples
Missing or changing knowledge	Retrieval
Need for authoritative external state	Tools
Deterministic business constraint	Code or workflow
Repeated behavioural error across diverse prompts	Consider fine-tuning
Subjective preference between otherwise valid responses	Preference optimization
Strong model works but is too expensive or slow	Distillation
Very long demonstrations work but violate latency or cost targets	Compare fine-tuning against prompt compression
Failure disappears with a stronger base model	Compare model upgrade against adaptation cost

Fine-tuning becomes plausible only after the strongest relevant non-training baseline has been measured.

3. Why adaptation may be necessary

Fine-tuning is justified when all of the following are true:

The target behaviour is precisely defined.
The model exhibits a repeated, representative failure.
The failure persists after reasonable prompt, retrieval, tool and workflow improvements.
The desired behaviour can be demonstrated in training examples.
There is enough high-quality data to separate learning from memorization.
The expected improvement matters economically or operationally.
The team can evaluate, deploy, version and roll back the adapted model.

Typical adaptation targets include:

Consistent structured classification.
Reliable tool selection.
Tool-argument generation.
Domain-specific summarization conventions.
Policy-compliant response patterns.
Failure classification from agent traces.
A stable organizational style.
Reducing dependence on long few-shot prompts.
Teaching a smaller model to imitate a stronger one.

Fine-tuning is less suitable when the underlying failure is:

Missing context.
Incorrect retrieval.
A broken tool.
An ambiguous label.
An impossible business requirement.
Inconsistent human annotations.
A deterministic validation that should be code.
A fact that changes faster than the model can be retrained.

Knowledge, format, style or decision policy?

Every training example should be classified by what it teaches.

Teaching target	Example
Knowledge	“Product X supports feature Y.”
Format	“Return a tool call matching this schema.”
Style	“Write a concise incident summary.”
Decision policy	“When identity is unverified, request verification instead of executing the action.”
Capability	“Infer the relevant failure category from a multi-step trace.”
Preference	“Response A is preferred because it is helpful without violating policy.”

This classification reveals design mistakes.

If examples primarily teach changing knowledge, use retrieval first. If they teach a deterministic policy, enforce it in code. Fine-tuning is strongest when examples teach a stable mapping between varied inputs and desired model behaviour.

4. Defining the desired behaviour

A model cannot be trained reliably against an adjective such as “better.” We need a behaviour specification.

For the tool-selection project, define:

Inputs

User request.
Relevant conversation state.
Available tool definitions.
Trusted retrieved evidence.
Authorization context.
Previous tool results.

Allowed outputs

{
  "action": "call_tool | ask_user | answer | escalate",
  "tool_name": "string | null",
  "arguments": {},
  "reason_code": "string"
}

Invariants

tool_name must belong to the exposed tool catalogue.
Arguments must pass the declared JSON Schema.
Identifiers must come from trusted input or tool results.
A destructive action must never bypass approval.
The model must choose ask_user when required information is missing.
Untrusted retrieved text cannot alter the tool policy.

Evaluation slices

Common tool calls.
Rare tools.
Similar tool names.
Missing arguments.
Conflicting context.
Irrelevant retrieved content.
Prompt-injection attempts.
New combinations of known intents and tools.
Tools added after the training period.
Requests outside the supported domain.

Success thresholds

For example:

Tool-selection accuracy: at least the production requirement.
Argument-schema validity: effectively 100% after application validation.
Semantic argument accuracy: above the baseline by a predefined margin.
Unauthorized-action rate: zero in the evaluation set and blocked at runtime.
Abstention recall: above the threshold for incomplete requests.
P95 latency and per-request cost: within the service budget.
General capability regression: below the accepted tolerance.

Thresholds must be chosen before inspecting the final test result. Otherwise, the team can redefine success after seeing the candidate model.

5. Constructing the dataset

Training examples are behavioural demonstrations

An instruction-response dataset normally contains:

The instruction or conversation.
Relevant context.
The desired assistant response.
Optional metadata describing the behaviour, source and quality.

A tool-use example could be represented as:

{
  "messages": [
    {
      "role": "system",
      "content": "Select an allowed action using the supplied tools."
    },
    {
      "role": "user",
      "content": "Cancel transfer TR-4821."
    },
    {
      "role": "assistant",
      "content": "{\"action\":\"call_tool\",\"tool_name\":\"cancel_transfer\",\"arguments\":{\"transfer_id\":\"TR-4821\"},\"reason_code\":\"user_requested_cancellation\"}"
    }
  ],
  "metadata": {
    "behavior": "complete_safe_tool_call",
    "source": "expert_authored",
    "policy_version": "payments-7",
    "review_status": "approved"
  }
}

A neighbouring counterexample should teach the boundary:

{
  "messages": [
    {
      "role": "user",
      "content": "Cancel my transfer."
    },
    {
      "role": "assistant",
      "content": "{\"action\":\"ask_user\",\"tool_name\":null,\"arguments\":{},\"reason_code\":\"missing_transfer_id\"}"
    }
  ]
}

If the dataset contains only successful tool calls, the model learns that every request should produce a tool call. Examples must cover the complete decision boundary, including abstention, escalation and missing information.

Dataset curation

High-quality curation includes:

Removing duplicates and near-duplicates.
Correcting contradictory labels.
Normalizing schemas and chat templates.
Checking whether the answer is derivable from the input.
Removing sensitive data or applying approved de-identification.
Tracking policy and tool versions.
Balancing important behavioural slices.
Preserving difficult, valid examples.
Removing impossible or under-specified cases.
Recording annotation provenance.

The LIMA study demonstrated strong alignment results from only 1,000 carefully curated examples, while also observing diminishing returns from adding quantity without sufficient diversity. It should not be interpreted as a universal “1,000 examples are enough” rule; it is evidence that data quality and diversity can matter more than raw volume for some adaptation tasks. See the LIMA paper.

Split before expanding

Create train, validation and test partitions before synthetic expansion or iterative hard-example mining.

Split by the unit most likely to leak:

Customer or tenant.
Incident.
Document family.
Time period.
Tool workflow.
Policy scenario.
Conversation template.

Randomly splitting near-identical support tickets can put paraphrases of the same case into both training and test sets. The resulting score measures memorization rather than transfer.

The test set must remain frozen. The validation set supports hyperparameter and checkpoint selection. Once a test set repeatedly influences development, it has become another validation set and should be replaced.

Synthetic training data

Synthetic examples are useful when they expand controlled variation:

Paraphrasing a valid request.
Generating rare but plausible argument combinations.
Creating adversarial distractors.
Varying which required fields are missing.
Producing policy-boundary cases.
Expanding underrepresented tools.

Self-Instruct demonstrated a pipeline that generates instructions and responses and then filters invalid or overly similar samples. The important idea is not “let the model generate unlimited data”; it is “generate candidates, then curate them.” See the Self-Instruct paper.

Synthetic examples should pass filters such as:

Schema validation.
Deterministic policy checks.
Deduplication.
Similarity checks against evaluation data.
Source-grounding verification.
Strong-model review.
Human review for high-risk examples.
Distribution and diversity analysis.

A teacher model can reproduce its own blind spots at scale. Synthetic data increases volume more easily than it increases truth.

Hard-example mining

After evaluating the baseline, collect cases where:

The model is confidently wrong.
Similar tools are confused.
Arguments are syntactically valid but semantically wrong.
A policy boundary is missed.
The answer changes across repeated runs.
Performance drops on a rare slice.
Retrieval distractors change the decision.

Annotate these failures and add them to the training pool—but not automatically.

Oversampling one visible failure can damage broad performance. Hard examples must remain representative, correctly labelled and balanced with ordinary cases.

6. Supervised fine-tuning

Supervised fine-tuning, or SFT, trains the model to assign higher probability to desired responses.

For an input (x) and desired response (y = (y_1,\ldots,y_T)):

[ \mathcal{L}_{\text{SFT}}

-\sum_{t=1}^{T} \log p_\theta(y_t \mid x, y_{<t}) ]

The mathematical form resembles pretraining, but the data distribution is different. Pretraining learns broad token regularities. SFT uses curated demonstrations to make desired application behaviour more probable.

For conversational training, loss is often applied only to assistant or completion tokens. Current TRL documentation supports completion-only loss for prompt-completion datasets and assistant-only loss when the chat template exposes the required generation mask. Chat-template compatibility therefore affects what the trainer actually optimizes. See the current TRL SFT documentation.

Full fine-tuning

Full fine-tuning updates all or nearly all model parameters.

Its advantages include:

Maximum adaptation capacity.
No separate adapter at inference.
Potentially better results when the task requires broad representational change.

Its costs include:

High GPU-memory use.
Large optimizer states and checkpoints.
Greater storage cost per variant.
More difficult multi-tenant customization.
Increased catastrophic-forgetting risk.
More expensive experimentation and rollback.

Full fine-tuning is reasonable when:

The model is small enough.
The dataset is large and diverse enough.
The adaptation is broad.
Parameter-efficient methods have been tested and are insufficient.
The deployment can support a separate full model.

For most initial domain-adaptation experiments, it is not the smallest useful intervention.

Training hyperparameters

Hyperparameters determine how strongly and how often the dataset changes the model.

Learning rate

The learning rate controls update magnitude.

Too high:

Training may become unstable.
The model may overfit quickly.
General capabilities may regress.
Adapter updates may dominate the base behaviour.

Too low:

The model may barely change.
Training may appear stable while producing no meaningful improvement.
More steps may be required.

Do not inherit a learning rate blindly from a tutorial. Sweep a small logarithmic range appropriate to the model, optimizer and adaptation method. Select it using validation behaviour, not training loss alone.

Batch size and gradient accumulation

The physical batch size is limited by accelerator memory. Gradient accumulation approximates a larger effective batch by accumulating gradients over several microbatches before an optimizer update.

A simplified effective batch size is:

[ B_{\text{effective}}

B_{\text{device}} \times N_{\text{devices}} \times N_{\text{accumulation}} ]

Larger batches can stabilize gradients but reduce the number of optimizer updates per epoch. They may also change generalization. Gradient accumulation saves memory relative to placing the full effective batch on the device, but it increases the time between optimizer updates.

Epoch selection

An epoch is one pass through the training dataset.

More epochs do not imply more useful learning. Small datasets can be memorized quickly. Evaluate checkpoints during training and stop based on held-out task metrics, safety tests and general-capability retention.

Sequence length

Longer sequences increase activation memory and computation. Truncation may silently remove the user request, tool definitions or target completion. Inspect actual tokenized examples and truncation rates.

Gradient checkpointing

Gradient checkpointing reduces activation memory by recomputing some activations during backpropagation, trading additional computation for lower memory consumption. Current TRL trainers enable it by default according to the TRL memory guide, but framework defaults can change and should be recorded in the training configuration.

Overfitting

Signs of overfitting include:

Training loss continues falling while validation performance stalls.
Exact training phrasings work, but paraphrases fail.
Output diversity collapses.
The model copies training responses.
Rare slices worsen while aggregate accuracy rises.
Small prompt-format changes cause large failures.

Mitigations include:

More diverse examples.
Deduplication.
Fewer steps or epochs.
A lower learning rate.
Early stopping.
Better regularization.
Stronger train-validation separation.
Removing noisy or contradictory examples.

Catastrophic forgetting

Catastrophic forgetting occurs when adaptation improves the target behaviour but damages previously useful capabilities.

For example, a model trained heavily on tool-call JSON might:

Produce JSON when a natural-language response is required.
Become less capable on general summarization.
Lose multilingual performance.
Overuse one domain’s terminology.
Refuse benign requests because safety examples were overly broad.

Detect forgetting using a capability-retention suite unrelated to the adaptation target.

Possible mitigations include:

Parameter-efficient fine-tuning.
Lower learning rates.
Fewer training steps.
Mixing carefully selected general examples into training.
Regularizing the candidate toward the original model.
Choosing an earlier checkpoint.
Routing only relevant requests to the adapter.

7. LoRA and QLoRA

Parameter-efficient fine-tuning

Parameter-efficient fine-tuning freezes most of the original model and trains a relatively small number of additional or selected parameters.

This reduces:

Trainable parameter count.
Optimizer-state memory.
Checkpoint size.
Storage cost for multiple domain variants.

It does not automatically make the base model smaller or cheaper to serve. The full base model is still required unless the adapted behaviour is distilled into a smaller student.

LoRA from first principles

Consider a pretrained weight matrix:

[ W_0 \in \mathbb{R}^{d \times k} ]

Full fine-tuning learns an unrestricted update:

[ W' = W_0 + \Delta W ]

LoRA assumes the useful update can be approximated by a low-rank product:

[ \Delta W = BA ]

where:

[ B \in \mathbb{R}^{d \times r}, \qquad A \in \mathbb{R}^{r \times k}, \qquad r \ll \min(d,k) ]

The base matrix (W_0) remains frozen. Only (A) and (B) are trained.

This works when the task-specific change lies in a much lower-dimensional space than the full parameter matrix. The original LoRA paper reported competitive adaptation while training far fewer parameters than full fine-tuning.

Rank

Rank (r) controls adapter capacity.

A small rank:

Uses less memory.
Produces a smaller adapter.
May be enough for a narrow format or classification behaviour.
May underfit complex adaptation.

A larger rank:

Increases capacity.
Increases trainable parameters and memory.
May fit more complex transformations.
Can increase overfitting risk.

Rank is not a direct “quality” control. Sweep it only after defining an evaluation that can show whether additional capacity helps.

Alpha

LoRA alpha controls the scale applied to the low-rank update. In the common formulation, the contribution is scaled using a value related to:

[ \frac{\alpha}{r}BA ]

Alpha affects update strength; rank affects representational capacity. They interact and should be versioned together.

Target modules

Target modules determine where adapters are inserted.

Depending on the architecture, candidates may include:

Query projections.
Key projections.
Value projections.
Attention output projections.
Feed-forward projections.
Other linear layers.

Module names vary across model families. Copying q_proj and v_proj from an unrelated tutorial can result in incomplete adaptation or a configuration that does not match the model.

Inspect the model architecture, confirm which parameters are trainable and record the result. Current PEFT documentation exposes r, lora_alpha, target_modules, per-layer rank patterns and related configuration through LoraConfig. See the PEFT LoRA documentation.

A sensible adapter experiment

Start with a deliberately small search:

Two plausible rank values.
One or two target-module strategies.
A small learning-rate sweep.
Frequent validation.
A fixed training budget.
At least two seeds for finalists.

Do not search dozens of configurations against the final test set.

QLoRA

LoRA reduces trainable parameters, but the frozen base model may still require substantial memory. QLoRA stores the frozen base model in a quantized representation—commonly four-bit—and backpropagates through it into higher-precision LoRA adapters.

The base model is quantized and frozen. The adapters are trainable.

The QLoRA paper introduced techniques including four-bit NormalFloat, double quantization and paged optimizers to reduce memory requirements.

Current Transformers documentation recommends NF4 for training four-bit base models and documents compute types and nested quantization through BitsAndBytesConfig. See the official bitsandbytes integration guide.

QLoRA should not be described as “training the model in four-bit.” More precisely:

The frozen base weights are represented in a quantized form.
Computation occurs using a configured compute type.
Gradients update the LoRA parameters.
The quantized base weights themselves are not updated.

QLoRA is attractive when memory is the main training constraint. It does not guarantee identical results to ordinary LoRA. Compare both if the expected value justifies the extra experiment.

8. Preference optimization

SFT teaches:

Produce this desired answer.

Preference training teaches:

For this prompt, answer A is preferable to answer B.

This distinction matters when there are many valid outputs but some are more helpful, concise, safe or policy-aligned than others.

Preference data

A preference record contains:

{
  "prompt": "...",
  "chosen": "...",
  "rejected": "...",
  "preference_reason": [
    "correct_tool",
    "does_not_invent_identifier",
    "asks_for_missing_information"
  ]
}

The rejected response should be plausible. Comparing a strong response with nonsense teaches little about subtle decision boundaries.

Preference labels should have an explicit rubric. Otherwise, annotators may optimize incompatible notions of quality.

Reward-model intuition

A reward model learns a scalar score:

[ r_\phi(x,y) ]

for prompt (x) and response (y).

Given chosen response (y^+) and rejected response (y^-), a simplified pairwise objective encourages:

[ r_\phi(x,y^+) > r_\phi(x,y^-) ]

One common loss is:

[ -\log \sigma\left( r_\phi(x,y^+) - r_\phi(x,y^-) \right) ]

The reward model does not prove that a response is objectively good. It approximates the preferences represented by its annotation data.

RLHF

A simplified RLHF pipeline is:

Supervised fine-tune a model on demonstrations.
Generate multiple responses.
Ask humans to rank them.
Train a reward model from those rankings.
Optimize the policy model to increase reward.
Constrain it from drifting too far from a reference model.

The InstructGPT work used demonstrations followed by preference rankings and reinforcement learning, showing that post-training can substantially change how models follow instructions. See the InstructGPT paper.

RLHF is operationally demanding because it introduces:

Response generation during training.
A separate reward model.
Reinforcement-learning stability concerns.
Reward hacking.
Reference-policy constraints.
More complicated monitoring and reproducibility.

RLAIF

Reinforcement learning from AI feedback uses a model to produce some or all preference labels.

This can scale annotation, but the evaluator model may:

Prefer its own style.
Miss domain-specific errors.
Reproduce shared misconceptions.
Be vulnerable to superficial features.
Apply a written policy inconsistently.

Constitutional AI demonstrated supervised self-critique and an RL phase using AI-generated preferences guided by principles. See the Constitutional AI paper.

Use RLAIF when the evaluation rubric is explicit and auditable. Retain human review for high-risk and ambiguous cases.

Direct Preference Optimization

DPO trains directly on chosen and rejected responses without first training a separate explicit reward model and then running a conventional reinforcement-learning loop.

A simplified DPO objective compares how the trainable policy and a reference policy score the preferred and rejected responses:

[ \mathcal{L}_{\text{DPO}}

-\mathbb{E} \left[ \log \sigma \left( \beta \left[ \log \frac{\pi_\theta(y^+ \mid x)} {\pi_{\text{ref}}(y^+ \mid x)}

\log \frac{\pi_\theta(y^- \mid x)} {\pi_{\text{ref}}(y^- \mid x)} \right] \right) \right] ]

The reference model anchors the update. The (\beta) parameter controls the strength of the preference-relative constraint.

DPO is simpler than the classic reward-model-plus-RLHF pipeline, but it does not remove the need for high-quality preferences, representative evaluation or safety testing. See the original DPO paper and current TRL DPO documentation.

Use SFT when you can demonstrate the correct response. Consider preference optimization when choosing among multiple plausible responses is the central problem.

9. Distillation

Fine-tuning adapts a model. Distillation transfers behaviour from a teacher to a student.

Suppose a large model achieves the required tool-selection reliability, but its latency or cost is unacceptable. A smaller model fails even with the same prompt and tools.

The stronger model can generate or score training targets for the smaller model.

Teacher and student

The teacher is the stronger model, ensemble or system.
The student is the smaller model being trained.
The distillation dataset covers the input distribution the student must serve.
The objective encourages the student to imitate teacher outputs, probabilities or intermediate behaviour.

Classical knowledge distillation uses softened output distributions to communicate more than the top prediction. See Distilling the Knowledge in a Neural Network.

For API-only language models, teacher logits may be unavailable. In that case, teams commonly use response distillation:

Sample representative prompts.
Generate teacher responses.
Verify or filter those responses.
Fine-tune the student on accepted outputs.
Evaluate the student against ground truth, not merely teacher agreement.

A controlled distillation experiment

Compare:

Student with strong prompting.
Student with retrieval and workflow improvements.
Student fine-tuned on human-authored examples.
Student fine-tuned on verified teacher outputs.
Teacher production system.

Measure:

Target-task success.
Safety.
Latency.
Cost.
Long-context performance.
Out-of-distribution transfer.
Failure correlation with the teacher.

Distillation is worthwhile only if the student approaches the required behaviour while materially improving the serving objective.

A LoRA adapter does not shrink its base model. Distillation can.

10. Evaluation

Training loss answers:

How well does the model predict the training targets?

Production evaluation asks:

Does the resulting system behave better on representative unseen cases?

These are not equivalent.

Controlled comparison

Use the same frozen test set to compare:

Arm	System
A	Original model with current production prompt
B	Original model with improved prompt and examples
C	Original model with improved prompt, retrieval and workflow
D	Adapted model with the improved prompt
E	Adapted model with the same retrieval and workflow as C
F	Smaller distilled model with the same runtime controls

Arm C is the strongest non-training baseline. Comparing only A with D exaggerates the value of training if simple application improvements were never tested.

Control:

Model revision.
Tokenizer and chat template.
Tool definitions.
Retrieval index.
Decoding parameters.
Hardware.
Concurrency.
Retry policy.
Evaluation code.
Random seeds where applicable.

Target metrics

For tool use:

Tool-selection accuracy.
No-tool accuracy.
Missing-information detection.
Argument-schema validity.
Argument semantic accuracy.
End-to-end execution success.
Unauthorized-action rate.
Unnecessary-tool-call rate.
Average steps per successful case.
Recovery after tool failure.

For summarization:

Required-field coverage.
Unsupported-claim rate.
Entity and numerical accuracy.
Compression ratio.
Human preference under a fixed rubric.

For policy-compliant generation:

Correct refusal or safe-completion rate.
Over-refusal rate.
Policy citation accuracy.
Prompt-injection resistance.
Leakage of protected information.

Reliability

Evaluate repeated runs under production decoding settings.

Two systems may have the same average accuracy while one fails unpredictably. Track:

Per-case pass rate over repeated samples.
Variance across runs.
Failure concentration.
Calibration or confidence where available.
Sensitivity to harmless prompt paraphrases.

Transfer beyond the training distribution

Create evaluation partitions that differ from training by:

Time.
Customer.
phrasing.
tool combination.
policy scenario.
document source.
input length.
language.
adversarial structure.

If improvement appears only on near-duplicates of the training examples, the model has not learned a transferable behaviour.

Safety and capability regressions

A release candidate should pass three suites:

Target-task evaluation — Did the intended behaviour improve?
Safety evaluation — Did prohibited behaviour increase?
Capability-retention evaluation — Did unrelated useful abilities decline?

Safety testing should include:

Direct prompt injection.
Indirect injection in retrieved documents.
Attempts to fabricate authorization.
Cross-tenant requests.
Sensitive-data extraction.
Requests for irreversible actions.
Conflicting system and user instructions.
Malformed tool results.

Latency and cost

Measure the whole system rather than only model-generation time:

Prompt construction.
Retrieval.
Queueing.
Time to first token.
Generation.
Tool execution.
Validation and retries.
Adapter loading or routing.

Record:

P50 and P95 latency.
Input and output tokens.
Accelerator utilization.
Requests per second.
Retry frequency.
Cost per successful task.
Cost per prevented failure.

The adapted model is worth its serving complexity only if the combined quality, reliability, latency and cost trade-off beats the alternatives.

11. Regression risk

Model versioning

An adapted model is not a single file. It is the result of an artifact graph.

Version:

Base-model identifier and exact revision.
Model license.
Tokenizer revision.
Chat template.
Behaviour specification.
Tool and policy versions.
Raw-data snapshot.
Curated dataset.
Train, validation and test split identifiers.
Synthetic-data generator and prompt.
Filtering rules.
Training code revision.
Library versions.
Random seed.
LoRA or QLoRA configuration.
Hyperparameters.
Checkpoints.
Adapter artifact.
Merged artifact, if any.
Evaluation report.
Deployment configuration.

A name such as payments-model-final-v2 cannot reproduce any of this.

Training checkpoints

A training checkpoint can contain:

Model or adapter weights.
Optimizer state.
Scheduler state.
Random-number-generator state.
Training progress.
Trainer configuration.

The “last” checkpoint is not necessarily the “best” checkpoint. Select release candidates using validation metrics and regression suites.

Keep enough state to resume interrupted training, but distinguish resumable training checkpoints from compact release artifacts.

Adapter merging

A LoRA adapter can be kept separate from the base model or merged into it.

Keeping it separate provides:

Smaller domain-specific artifacts.
Easier adapter switching.
Clearer rollback.
Multiple adaptations over one base model.

Merging can simplify serving and avoid adapter application overhead, but it creates a larger standalone checkpoint and removes some runtime flexibility.

Current PEFT tooling supports loading adapters independently and merging them through operations such as merge_and_unload(). The merged candidate must be evaluated again; do not assume numerical precision, quantization or serialization leaves behaviour perfectly unchanged. See the PEFT model-use documentation and PEFT LoRA guide.

Never delete the original base-model reference, adapter or pre-merge evaluation report merely because a merged artifact was created.

12. Deployment and rollback

Treat an adapted model as a production code change.

Pre-deployment gate

Require:

Frozen evaluation results.
Safety and capability-regression results.
Data and model cards.
Artifact hashes.
Reproducible configuration.
Serving load test.
Rollback procedure.
Monitoring thresholds.
Approval from relevant domain and safety owners.

Deployment stages

Offline evaluation Compare every experimental arm on the frozen suites.
Shadow deployment Send production inputs to the candidate without allowing its outputs to affect users or tools.
Canary deployment Route a small, controlled traffic percentage to the candidate.
Progressive rollout Increase traffic only while quality, safety and operational metrics remain within limits.
Full release Preserve the previous model as an immediately selectable deployment.

Rollback triggers

Rollback should be automatic or operationally simple when:

Safety violations exceed the threshold.
Tool execution errors rise.
Latency or memory consumption violates the SLO.
A protected slice regresses.
Output distribution changes unexpectedly.
The candidate depends on a missing base-model or tokenizer revision.
Monitoring cannot reliably distinguish candidate and control traffic.

Adapter-based deployments can make rollback as simple as switching the active adapter or routing back to the base model. That advantage disappears if artifacts and routing configuration are not versioned.

13. Continuous adaptation from production failures

A mature adaptation pipeline forms a controlled loop:

flowchart TD
    A["Production traces"] --> B["Failure taxonomy"]
    B --> C["Expert annotation"]
    C --> D["Curated training pool"]
    D --> E["Candidate training"]
    E --> F["Frozen evaluations"]
    F --> G{"Release gates pass?"}
    G -- No --> B
    G -- Yes --> H["Shadow and canary"]
    H --> A

The loop must not train blindly on every production interaction.

Production data may contain:

Sensitive information.
User mistakes.
Prompt injections.
Incorrect model outputs.
Tool failures.
Policy violations.
Duplicate incidents.
Unrepresentative bursts.
Feedback influenced by the existing model.

Use production traces to discover failure classes, not as automatically trusted labels.

For every adaptation cycle:

Define the newly targeted behaviour.
Confirm that application-layer fixes are insufficient.
Add correctly annotated examples.
Check for evaluation leakage.
Retrain from a known base or approved predecessor.
Re-run all target, safety and retention evaluations.
Compare against the latest non-training baseline.
Deploy through shadow and canary stages.
Monitor whether offline improvement transfers to production.
Roll back or continue based on predefined gates.

Continuous adaptation does not mean continuous weight mutation. It means continuous evidence collection with controlled, versioned releases.

The smallest useful project

For an agent-support adaptation project, begin with one narrow task: selecting an action and generating its arguments.

Phase 1: Baseline

Build an instruction-tuned-model baseline with:

A concise tool catalogue.
A strict output schema.
Clear abstention rules.
Deterministic validation.
A strong set of few-shot counterexamples.
No fine-tuning.

Run it against a representative evaluation dataset.

Phase 2: Failure collection

Classify failures into:

Wrong tool.
Unsupported tool.
Missing-argument oversight.
Fabricated argument.
Invalid schema.
Unsafe action.
Unnecessary tool call.
Failure to abstain.
Prompt-injection influence.
Correct tool call but failed execution.

Fix retrieval, tool descriptions, schema handling and workflow enforcement before training.

Phase 3: Behaviour specification

Define allowed actions, invariants, metrics and release thresholds. Freeze the test set.

Phase 4: Training data

Create expert-reviewed instruction-response examples from the training partition. Add synthetic variants only where they cover justified gaps. Filter, deduplicate and record provenance.

Phase 5: First adaptation

Fine-tune one suitable instruction model using LoRA or QLoRA.

Keep the first experiment intentionally small:

One base-model revision.
One dataset version.
A limited hyperparameter sweep.
Frequent validation.
No preference optimization yet.
No adapter merging until the candidate passes evaluation.

Phase 6: Fair comparison

Compare:

Current prompt.
Improved prompt.
Improved prompt plus RAG or workflow.
Adapted model.
Adapted model plus the same RAG or workflow.

Do not remove runtime safeguards from the adapted arm.

Phase 7: Regression evaluation

Test:

Safety.
General instruction following.
Summarization.
No-tool conversational responses.
Multilingual behaviour if required.
Long-context behaviour.
New tools and unseen tool combinations.

Phase 8: Preference optimization

Only after SFT has produced a competent model should you consider DPO or another preference method—and only if the remaining failures concern preference between plausible responses rather than missing capability or incorrect labels.

Phase 9: Distillation

If the successful model is too expensive, create a separate teacher-student experiment. Do not assume adapter training reduces serving cost.

Phase 10: Deployment

Version the dataset, adapter, configuration, evaluation report and serving manifest. Deploy through shadow traffic, canary traffic and progressive rollout with an immediate rollback path.

Final decision framework

Change the prompt when the capability exists but the instruction is unclear.

Add retrieval when the model lacks authoritative or changing information.

Add tools when the answer depends on external state or deterministic computation.

Add workflows when the system must enforce ordering, authorization, approval or safety boundaries.

Use supervised fine-tuning when a stable behaviour can be demonstrated repeatedly and stronger non-training baselines remain inadequate.

Use preference optimization when several responses are plausible but some are consistently preferred according to an explicit rubric.

Use distillation when a strong model behaves correctly but cannot meet deployment cost or latency requirements.

Use full fine-tuning only when the required adaptation exceeds what parameter-efficient methods can provide and the additional training and serving costs are justified.

The final test is not whether an adapted model scores higher than the first prompt you tried. It is whether it beats the strongest practical non-training system on representative unseen cases, retains its other important capabilities, satisfies safety requirements and produces enough operational value to justify becoming another model your team must train, evaluate, deploy, monitor and eventually replace.

If that evidence is absent, change the system—not the weights.