How to Know Whether an LLM Improvement Is Real

Suppose we evaluate two agent configurations on 200 tasks:

Configuration	Successful tasks	Success rate	Mean cost/task	p95 latency
Agent A	140/200	70%	$0.018	4.2 s
Agent B	152/200	76%	$0.029	7.8 s

Agent B appears better by six percentage points.

But this table does not yet tell us whether:

B is genuinely better on future tasks.
The difference arose from sampling variation.
B succeeds on tasks where A fails.
The improvement is large enough to matter.
The extra quality justifies the additional cost and latency.
The evaluator's labels are trustworthy.
The test set represents production traffic.

A trustworthy evaluation must answer all of these questions.

An observed success rate is not the true success rate

The 200 evaluated tasks are a sample from a larger population of possible production tasks.

The population might mean:

All support requests the agent will receive from Indian customers during the next three months.

That definition matters. If our sample contains only short English questions, it cannot tell us how the agent will perform on long conversations, regional languages or tool failures.

For each task, define a random variable:

$$ X = \begin{cases} 1 & \text{if the agent succeeds} \ 0 & \text{if the agent fails} \end{cases} $$

The sample success rate is simply the mean:

$$ \hat{p} = \frac{X_1 + X_2 + \cdots + X_n}{n} $$

For Agent B:

$$ \hat{p}_B = \frac{152}{200} = 0.76 $$

This means B succeeded on 76% of the evaluated sample. It does not prove that B's true future success rate is exactly 76%.

If we selected another 200 representative tasks, we would probably get a different percentage. This is sampling variation.

Sample size controls uncertainty

Imagine evaluating B on only ten tasks and observing eight successes. Its success rate would be 80%, but one additional failure would reduce it to 70%.

With 1,000 tasks, one additional failure changes the result by only 0.1 percentage points.

Larger samples produce more stable estimates because unusual individual tasks have less influence. Uncertainty generally decreases in proportion to:

$$ \frac{1}{\sqrt{n}} $$

This has an important consequence: halving the uncertainty usually requires approximately four times as many observations.

A large sample, however, cannot repair a biased sample. Evaluating 100,000 easy tasks can give a very precise answer to the wrong question.

Confidence intervals describe the remaining uncertainty

A confidence interval gives a range of population success rates compatible with the observed sample and statistical assumptions.

A rough 95% interval for a success rate is:

$$ \hat{p} \pm 1.96\sqrt{\frac{\hat{p}(1-\hat{p})}{n}} $$

For Agent B:

$$ 0.76 \pm 1.96\sqrt{\frac{0.76(0.24)}{200}} $$

This produces an approximate interval of:

$$ 70.1% \text{ to } 81.9% $$

For Agent A, the approximate interval is:

$$ 63.6% \text{ to } 76.4% $$

In practice, a Wilson interval is usually preferable for binary success rates, especially with small samples or percentages near zero or one.

A 95% confidence interval does not literally mean there is a 95% probability that the fixed true value is inside this particular interval. It means that if we repeatedly sampled and constructed intervals using this procedure, approximately 95% of those intervals would contain the true value.

The useful plain-English conclusion is:

The observed percentage is an estimate, and the interval communicates how imprecise that estimate remains.

Compare models on the same tasks

The individual confidence intervals overlap, but that does not settle whether B is better.

Because both configurations were tested on the same tasks, the observations are paired. We should compare their result on each task rather than treating the two sets of results as unrelated.

Suppose the paired results are:

Outcome	Number of tasks
Both succeed	132
Only A succeeds	8
Only B succeeds	20
Both fail	40

The totals are still:

A: $132 + 8 = 140$ successes
B: $132 + 20 = 152$ successes

The useful evidence comes from the 28 tasks on which they disagree:

B wins 20.
A wins 8.
B therefore has 12 additional successes.

For each task, define a paired difference:

$$ D_i = \begin{cases} 1 & \text{if only B succeeds} \ -1 & \text{if only A succeeds} \ 0 & \text{if their outcomes match} \end{cases} $$

The mean paired difference is:

$$ \bar{D} = \frac{20-8}{200}=0.06 $$

So B improves success by six percentage points on this sample.

Pairing removes much of the variation caused by task difficulty. A difficult task affects both agents, so the comparison concentrates on where their behaviour differs.

Bootstrap the difference

A bootstrap estimates uncertainty by repeatedly resampling the evaluated rows with replacement.

For each bootstrap sample:

Draw 200 paired task records from the original 200.
Calculate B's success rate minus A's success rate.
Store the result.
Repeat thousands of times.
Examine the resulting distribution.

A simplified implementation looks like this:

import numpy as np

# One value for every evaluated task:
#  0: both agents have the same outcome
#  1: only Agent B succeeds
# -1: only Agent A succeeds
paired_results = np.array(
    [0] * 172
    + [1] * 20
    + [-1] * 8
)

rng = np.random.default_rng(42)
bootstrap_means = []

for _ in range(10_000):
    sample = rng.choice(
        paired_results,
        size=len(paired_results),
        replace=True,
    )
    bootstrap_means.append(sample.mean())

lower, upper = np.percentile(bootstrap_means, [2.5, 97.5])

For these data, an approximate paired 95% interval is around:

$$ 0.9 \text{ to } 11.1 \text{ percentage points} $$

The exact bootstrap result varies slightly with resampling.

This permits us to say:

The experiment supports an improvement, but the plausible size ranges from roughly one to eleven percentage points.

It does not permit us to claim that the improvement is exactly six points.

Statistical significance answers a narrow question

A hypothesis test begins with a null hypothesis:

Agent A and Agent B have the same underlying probability of success.

For paired binary outcomes, McNemar's test focuses on the tasks where the agents disagree. Here, B wins 20 disagreements while A wins eight.

The corresponding two-sided p-value is approximately:

$$ p = 0.036 $$

If the null hypothesis were true, results this imbalanced or more extreme would occur about 3.6% of the time under the test assumptions.

The p-value is not:

The probability that the null hypothesis is true.
The probability that the result happened "by chance."
The probability that B will remain better in production.
A measurement of how valuable the improvement is.

Using a conventional threshold of 0.05, this result is statistically significant. But it is also close enough to the threshold that replication would be valuable.

The threshold should be selected before examining the result. Otherwise, it is easy to move the decision boundary until the preferred model wins.

Statistical significance is not practical significance

With enough data, even a tiny improvement can become statistically significant.

We therefore need an effect size: a description of how large the improvement is.

Absolute improvement

$$ 76%-70%=6 \text{ percentage points} $$

Relative success-rate improvement

$$ \frac{76-70}{70}=8.6% $$

Relative error reduction

A's failure rate is 30%, while B's is 24%:

$$ \frac{30-24}{30}=20% $$

These statements describe the same data from different perspectives. "Twenty percent error reduction" sounds much larger than "six percentage-point improvement," so a responsible report should provide the underlying rates as well.

Before running the experiment, the team should define a minimum worthwhile improvement. For example:

We will deploy B only if its improvement is plausibly at least four percentage points without violating cost and latency limits.

Our interval extends below four points. The experiment suggests that B is better, but does not establish that it meets this stronger business requirement.

Quality must be evaluated alongside cost

Agent B costs an additional:

$$ $0.029-$0.018=$0.011 $$

per task.

Across 200 tasks, that is:

$$ 200 \times $0.011=$2.20 $$

B produced 12 additional successes, so the observed incremental cost per additional success is:

$$ \frac{$2.20}{12}=$0.183 $$

At one million tasks, B would add approximately $11,000 in model costs if the average remained stable.

Whether this is worthwhile depends on the business consequence of success and failure. Spending eighteen cents to prevent an incorrect medical escalation could be excellent. Spending it to improve the formatting of a low-value summary might not be.

Means also hide distributions. Cost and latency should include values such as:

Median cost and latency
p90, p95 and p99 latency
Maximum tool calls
Frequency of extreme token usage
Timeouts and incomplete runs

A configuration that improves average quality but doubles p95 latency may still be unacceptable for an interactive product.

The evaluator is part of the measurement system

Agent success is often determined by an LLM judge. The judge can introduce noise and bias just like any other measurement instrument.

Suppose an LLM judge and a trusted human evaluator label 50 responses:

	Human success	Human failure
Judge says success	32	4
Judge says failure	3	11

The judge agrees with the human on:

$$ \frac{32+11}{50}=86% $$

Its success-label precision is:

$$ \frac{32}{32+4}=88.9% $$

Its recall is:

$$ \frac{32}{32+3}=91.4% $$

Precision asks:

When the judge declares success, how often is that declaration correct?

Recall asks:

Of the responses humans consider successful, how many does the judge recognize?

The appropriate trade-off depends on the cost of each error. In a safety evaluation, incorrectly passing a failed response may be more damaging than incorrectly rejecting a good response.

Raw agreement can also be inflated when one label is common. Cohen's kappa adjusts agreement by estimating how much agreement would be expected from the evaluators' label frequencies:

$$ \kappa = \frac{\text{observed agreement}-\text{expected agreement}} {1-\text{expected agreement}} $$

For this example, kappa is approximately 0.66.

That indicates meaningful agreement beyond what would be expected from label frequencies alone. It does not prove that either evaluator is correct. Two evaluators can agree because they share the same bias.

Judge reliability should therefore be tested using:

Human-reviewed examples
Clear scoring rubrics
Multiple judges or repeated judgments
Blind evaluation without model identity
Agreement broken down by task category
Manual inspection of disagreements

Bias cannot be fixed with a confidence interval

Statistical uncertainty describes what happens because we observed a limited sample. Bias occurs when the evaluation systematically measures the wrong thing.

Common sources include:

Selection bias

If we include only tasks that both agents completed, we remove crashes, timeouts and malformed outputs. The resulting success rate describes survivors rather than all attempted tasks.

Survivorship bias

If production logs retain only completed runs, failed runs may disappear from the evaluation dataset.

Data leakage

A model may have seen benchmark answers during training, prompt development or previous evaluation rounds. It can score well without generalizing to new tasks.

Judge bias

An LLM judge might prefer verbose answers, familiar formatting or outputs generated by the same model family.

Repeated benchmark tuning

If we repeatedly change prompts based on one test set, that test set becomes part of development. Performance on it becomes increasingly optimistic.

Multiple comparisons

If we test 20 prompt variations and report only the best one, some apparent improvement may come from random variation. The final candidate needs confirmation on untouched data.

Confidence intervals and p-values do not remove these problems. A biased experiment can produce an extremely precise but misleading conclusion.

A trustworthy comparison starts with the research question

Before executing the evaluation, write the question in operational terms:

For production support requests from the target customer population, does Agent B improve task success over Agent A by at least four percentage points, without exceeding $0.03 mean cost or eight seconds p95 latency?

Then design the experiment around that question:

Define the production population.
Select representative tasks without examining model outcomes.
Run both configurations on the same tasks.
Keep model identity hidden from evaluators.
Define success before running the experiment.
Record failures, timeouts, costs and latency.
Choose one primary metric and practical threshold.
Measure judge agreement against human labels.
Report raw counts, effect size and uncertainty.
Confirm promising results on untouched data or in a randomized production A/B test.

Offline paired evaluation tells us how the agents compare on the chosen dataset. A randomized production A/B test provides stronger evidence about how a deployment change affects real users.

So, is Agent B genuinely better?

The correct conclusion is conditional:

On this paired sample of 200 tasks, Agent B improved success from 70% to 76%. The estimated improvement is six percentage points, with an approximate 95% paired interval of 0.9 to 11.1 points. The paired result is statistically significant at the 5% level, but the interval does not establish that the improvement exceeds a four-point business threshold. B also costs $0.011 more per task and has substantially higher p95 latency. Deployment therefore depends on the value of an additional successful task, validation of judge reliability and confirmation that the sample represents production traffic.

That conclusion is less exciting than "B is 20% better."

It is also far more useful.

A trustworthy LLM evaluation does not produce a winning percentage. It produces a bounded decision: what changed, how uncertain the change remains, what it costs, which assumptions support it and what the experiment does not allow us to conclude.