Why AI Agents Must Be Designed as Distributed Systems

An ordinary HTTP handler is expected to do something small: validate a request, read or write a database, and return a response. If the process dies, the client can often try again.

An AI agent is different. A single run may call a model several times, retrieve documents, invoke tools, wait for a human, create external side effects, and consume a meaningful budget. It may run for minutes or hours. During that time, processes restart, networks time out, providers throttle requests, and users cancel work.

The important boundary is this:

When a run must survive beyond one process, coordinate with remote systems, or preserve side effects across retries, it is a distributed job—not an HTTP request.

This article derives the infrastructure for such a job from the failures of a simple synchronous LLM API.

The innocent synchronous design

Imagine a FastAPI endpoint that asks a model to research a topic and write a report:

@app.post("/reports")
async def create_report(request: ReportRequest):
    sources = await search(request.topic)
    draft = await llm.generate(request.topic, sources)
    url = await documents.publish(draft)
    return {"url": url}

For a demo, this is excellent. The control flow is visible, there are few moving parts, and the caller immediately receives the result.

Its implicit invariant is much stronger than it looks:

The client connection, API process, network, model provider, search service, and document service must all remain available until the entire run finishes.

Production breaks that invariant routinely.

Failure 1: the work lasts longer than the request

Suppose report generation grows from 10 seconds to 12 minutes. A load balancer closes the HTTP connection after 60 seconds. Did the run stop? Usually not. The API process may still be generating the report even though the client sees a timeout.

Increasing the HTTP timeout is a naive fix. It makes the connection wait longer, but it does not make the computation durable. A deployment, process crash, machine failure, or lost connection can still erase progress. Long timeouts also tie application capacity to slow external work.

We need to separate accepting work from performing it.

The API becomes a control plane. It validates the request, creates a durable run record, enqueues work, and returns a run ID quickly:

POST /runs       -> 202 Accepted { "run_id": "run_123" }
GET /runs/run_123
POST /runs/run_123/cancel

A worker later executes the run. The new invariant is:

Once the API accepts a run, the system retains enough durable information to execute it or report why it cannot.

The queue is not merely a performance optimization. It is the ownership transfer between the short-lived request and the long-lived job.

Failure 2: one component fails while the rest remain healthy

In a single process, failure often appears total: the function returns or raises. Across a network, failure is partial. The worker may be healthy while the model provider is unreachable. The provider may finish a request while the response is lost. PostgreSQL may be available while Redis is not.

This creates an uncomfortable fact:

A timeout tells us that we stopped waiting. It does not tell us whether the remote operation failed.

Consider a tool that sends an email. The worker sends the request, the email service accepts it, and the response disappears. Retrying may send the email twice; not retrying may leave it unsent. No timeout value can remove this ambiguity.

Distributed systems therefore treat failure handling as four different concerns:

Concern	Question	Example mechanism
Failure prevention	Can we reduce how often it happens?	Validation, capacity headroom, connection pools
Failure detection	How do we suspect something failed?	Timeouts, heartbeats, health checks
Failure recovery	How do we make progress afterward?	Retries, reassignment, checkpoints, compensation
Correctness guarantee	What outcomes are still allowed?	Idempotency, state invariants, delivery semantics

Operational convenience—dashboards, deployment tools, queue viewers—helps humans operate the system, but does not itself make execution correct.

Failure 3: transient errors make retries necessary

Remote calls fail for temporary reasons: a connection resets, the provider returns 429, or a gateway returns 503. Immediately failing the whole agent wastes completed work.

The first naive fix is to retry every error immediately. At scale, this creates a retry storm. When a provider is struggling, thousands of workers add more traffic precisely when it has the least spare capacity.

A safe retry policy needs five decisions:

1. Which errors are retryable? Retry transport failures, selected 5xx responses, and throttling. Do not retry invalid prompts, failed authorization, or deterministic schema errors without changing something.
2. How long should one attempt wait? Every remote call needs a bounded timeout.
3. How should attempts be spaced? Use exponential backoff, capped at a maximum delay.
4. How do workers avoid synchronizing? Add jitter so they do not all retry on the same schedule.
5. When must retrying stop? Enforce an attempt limit, elapsed-time limit, and cost budget.

A common delay is based on:

[ d_n = \min(d_{max}, d_0 2^n) ]

and then randomized. If a provider sends Retry-After, respect it. The exact jitter algorithm matters less than preventing synchronized retries.

The protected invariant is:

A temporary dependency failure may delay progress, but retries remain bounded and cannot consume unlimited traffic, time, or money.

Retries improve availability. They do not guarantee correctness. For that, we must handle duplicate execution.

Failure 4: reliable delivery creates duplicates

Suppose a worker receives a queue message, completes the model call, and crashes just before acknowledging the message. The queue cannot know that the work completed, so it delivers the message again.

This is at-least-once delivery: the message should eventually be processed, but it may be processed more than once.

The alternatives have different trade-offs:

Guarantee	Meaning	Failure consequence
At-most-once	Deliver zero or one time	Work may be lost
At-least-once	Deliver one or more times	Work may be duplicated
Exactly-once	Produce one externally visible effect	Requires cooperation from storage and side-effect boundaries

“Exactly once” is usually an end-to-end property, not something a queue can provide alone. Even if a broker delivers a message once, a worker can time out after a remote side effect and repeat it. In practice, durable systems combine at-least-once delivery with idempotent processing.

Idempotency turns repetition into safety

An operation is idempotent when repeating it with the same logical identity has the same externally visible result as performing it once.

The client supplies an idempotency key when creating a run. PostgreSQL enforces uniqueness per tenant:

CREATE UNIQUE INDEX uq_run_idempotency
ON agent_runs (tenant_id, idempotency_key);

If the API receives the same key again, it returns the existing run rather than creating another one.

That protects run creation, but each side effect needs its own protection. A send_invoice step might use (run_id, step_id) as the idempotency key accepted by the billing service. If the external service does not support idempotency, record an intent and result locally, reconcile ambiguous outcomes, or make the action require human confirmation. A database flag set after an unprotected external call cannot close the crash window between the two operations.

The invariant is:

Retrying the same logical operation cannot create an additional business effect.

This is more precise than “the function can run twice.” Model inference itself may be repeated and produce a different answer. What must be deduplicated is the committed step result or external side effect.

Failure 5: workers crash while owning messages

A queue must decide when a message stops belonging to one worker and becomes eligible for another.

One design removes the message as soon as a worker receives it. That gives at-most-once behavior: a crash loses the job. The durable design keeps the message until the worker acknowledges successful completion.

While a worker is processing, the message is hidden using a visibility timeout or lease. If the worker does not acknowledge before the lease expires, the message becomes available again. Long steps extend the lease with heartbeats.

This produces three rules:

• Acknowledge only after durable progress has been committed.
• Make processing safe to repeat because leases can expire prematurely.
• Size or extend the visibility timeout; do not assume every LLM call finishes within a fixed duration.

After repeated failures, the queue moves the message to a dead-letter queue. A DLQ is not a recovery strategy by itself. It is durable quarantine: it prevents a poison job from retrying forever and preserves evidence for inspection, repair, replay, or terminal failure.

Failure 6: a restarted worker no longer knows what happened

If the entire agent lives in local memory, reassignment starts it from the beginning. That repeats model calls, tools, costs, and possibly side effects.

We need durable execution state. Model the run as a state machine rather than an arbitrary Python stack:

stateDiagram-v2
    [*] --> queued
    queued --> running
    running --> waiting_retry
    waiting_retry --> running
    running --> succeeded
    running --> failed
    running --> cancelling
    cancelling --> cancelled

Only defined transitions are legal. Terminal states do not return to running. PostgreSQL is the source of truth for run status; queue messages are delivery signals, not the authoritative state.

Useful records include:

agent_runs
  id, tenant_id, status, current_step, version
  cancel_requested_at, attempt_count
  token_budget, cost_budget, cost_used
  trace_id, created_at, updated_at

run_steps
  run_id, step_id, status, attempt
  input_ref, output_ref, side_effect_key
  started_at, completed_at, error_code

Workers update state with optimistic concurrency:

UPDATE agent_runs
SET status = 'running', version = version + 1
WHERE id = :run_id
  AND status IN ('queued', 'waiting_retry')
  AND version = :expected_version;

If zero rows change, another worker or cancellation request won the race. This protects the invariant:

Every run follows legal state transitions, and concurrent actors cannot silently overwrite each other’s decisions.

Distributed locks can sometimes enforce exclusive ownership, but locks expire, holders pause, and networks partition. A lock is a lease—not proof that no other worker exists. Fencing tokens or version checks are still required when stale workers could write.

Failure 7: completed progress is too expensive to repeat

State tells us where the run is. A checkpoint preserves enough validated output to resume from there.

After search completes, store the source artifact and mark the search step complete. After the draft passes validation, store the draft and mark that step complete. On recovery, the worker loads committed checkpoints and resumes from the first incomplete step.

A checkpoint should contain references to durable artifacts, model and prompt versions, tool inputs and outputs, and any information needed to decide whether reuse is valid. Large documents belong in object storage; PostgreSQL stores their metadata, hashes, and provenance.

Checkpoint order matters. If a step performs an external side effect, the system must not mark it complete before the effect is committed. But performing the effect first creates an ambiguous crash window. That is why side-effect idempotency, reconciliation, or a transactional outbox is required; checkpointing alone does not solve dual writes.

The invariant is:

Recovery reuses only durable, validated progress and does not skip an uncommitted operation.

Checkpoints trade storage and schema complexity for lower recovery time and lower repeated model cost. Checkpoint every step when work is expensive or side-effectful; checkpoint less often when steps are cheap and deterministic.

Failure 8: cancellation races with execution

Deleting a queue message is not cancellation. A worker may already own it. Killing the worker is also insufficient: a remote model or tool may already have accepted the request.

Cancellation is a state transition and a cooperative protocol:

1. The API sets cancel_requested_at using a conditional update.
2. The worker checks cancellation before each new step, during long loops, and after remote calls.
3. The worker stops scheduling new work and attempts to cancel in-flight operations when the provider supports it.
4. It preserves completed checkpoints and moves through cancelling to cancelled.
5. Late results are recorded or discarded according to the state machine; they cannot turn a cancelled run into succeeded.

Some actions cannot be cancelled once committed. Sending a message, charging a card, or deploying code may require a compensating action. This is saga-style recovery: undo what can be undone, record what cannot, and escalate when human judgment is required.

The invariant is:

After cancellation is accepted, the system initiates no new avoidable work or side effects, and late completions cannot revive the run.

Failure 9: healthy workers can still overload the system

Adding more workers improves throughput only until a constrained dependency saturates. If 500 workers call a model provider that permits 100 requests per minute, scaling workers creates throttling, retries, higher queue age, and possibly an outage.

This is an overload problem, not merely a retry problem.

Backpressure makes producers or dispatchers slow down when downstream capacity is exhausted. Useful signals include queue depth, age of the oldest runnable job, worker saturation, provider latency, throttle rate, and remaining tenant budgets.

Related controls solve different problems:

Control	Protects against	AI-system example
Rate limit	Too much usage over time	Requests or tokens per tenant per minute
Concurrency limit	Too much simultaneous work	Maximum in-flight calls for one provider
Backpressure	Admission exceeding drain capacity	Reject or defer new runs when queue age is unsafe
Circuit breaker	Repeated calls to an unhealthy dependency	Pause calls after a high provider failure rate
Bulkhead	One workload consuming every resource	Separate pools for interactive chat and batch evals

Provider limits may apply to requests, input tokens, output tokens, or concurrent streams. A single request counter is therefore insufficient. Redis can coordinate fast token buckets or semaphores, while PostgreSQL preserves durable tenant entitlements and budget records.

The queue absorbs short bursts; it cannot create capacity. If arrival rate remains above completion rate, backlog grows without bound. The system must eventually reject, delay, degrade, or provision more capacity.

The protected invariant is:

Accepted work cannot grow without a bounded relationship to processing capacity, dependency limits, and promised completion time.

Failure 10: a run succeeds technically but exceeds its budget

An agent can make progress forever: refine the plan, call another tool, ask another model, and retry a low-quality answer. Technical success is not enough if one run consumes $200 or blocks a worker for hours.

Treat cost as a correctness constraint. Store maximum tokens, model calls, tool calls, wall-clock time, and currency cost with the run. Before an expensive operation, reserve an estimated amount atomically; afterward, reconcile it with actual usage.

This avoids a concurrency race where several parallel steps each observe enough remaining budget and collectively overspend it.

When the budget is nearly exhausted, graceful degradation may mean using a smaller model, reducing retrieval breadth, skipping optional critique, returning partial results, or asking the user to authorize more work. The choice must be explicit in product semantics.

The invariant is:

No run or tenant can consume more than its authorized bounded resources, apart from a defined estimation tolerance.

Failure 11: provider fallback changes behavior

Failing over from one model provider to another sounds like replacing one HTTP endpoint. Models differ in context windows, tool-call formats, safety behavior, latency, price, and output quality. A fallback can return a syntactically valid but materially worse result.

Use an adapter that normalizes capabilities and errors, then define fallbacks per task—not globally. Validate structured outputs, enforce the same tool permissions, record the selected model, and evaluate fallback quality on representative datasets.

A circuit breaker can temporarily stop calls to an unhealthy provider. Bulkheads prevent its slow requests from occupying every worker slot. Fallback occurs only when the run’s quality, privacy, region, and cost constraints permit it.

The invariant is:

Provider failure may reduce service quality in a declared way, but cannot silently violate the run’s contract.

Failure 12: without propagation, an incident becomes guesswork

One run now crosses an API, PostgreSQL, a queue, multiple workers, models, and tools. Separate logs such as “request timed out” or “tool failed” do not reconstruct causality.

Assign a run_id for business identity and a trace_id for one end-to-end execution trace. Propagate them through queue-message metadata and every remote call. Each attempt becomes a span with attributes such as:

• tenant, run, step, attempt, worker, model, and provider;
• queue wait time and execution time;
• input and output tokens, cost, and rate-limit delay;
• timeout, retry decision, error class, and checkpoint used;
• prompt and tool versions, with sensitive content redacted.

Correlation IDs make events searchable. Tracing records causal timing across boundaries. Metrics reveal aggregate behavior. All three are needed.

Measure the service from the user’s perspective with service-level indicators (SLIs). Possible objectives include:

• 99% of accepted interactive runs reach a terminal state within 5 minutes;
• 99.9% of accepted runs are never lost;
• fewer than 0.1% of side-effecting steps create an unintended duplicate effect;
• 95% of cancellation requests stop new work within 10 seconds;
• 99% of terminal runs contain a complete trace and cost record.

These targets must be defined by workload class. A one-minute chat run and a two-hour evaluation batch cannot share a useful latency SLO.

Observability detects and explains failures. It does not repair them. Recovery still depends on durable state, repeatable operations, and explicit ownership.

The resulting durable architecture

Every component now has a reason to exist:

flowchart TD
    C["Client"] --> A["FastAPI control plane"]
    A --> P["PostgreSQL: runs and steps"]
    A --> Q["Durable queue"]
    Q --> W["Python workers"]
    W --> P
    W --> R["Redis: limits and leases"]
    W --> X["Models, tools, artifact storage"]

The execution flow is:

1. The API authenticates the caller, validates the request, and atomically creates a run under a tenant-scoped idempotency key.
2. A transactional outbox or equivalent mechanism ensures that a committed run is eventually published to the queue. This closes the crash window between the database write and queue publish.
3. A worker receives the message under a visibility lease and conditionally claims the run.
4. It loads the latest checkpoint, verifies cancellation and budgets, and executes the next incomplete step.
5. Every remote call has a timeout, classified retry policy, exponential backoff, jitter, trace context, and applicable idempotency key.
6. The worker durably stores validated artifacts and step state before acknowledging completed work.
7. If it crashes, the lease expires and another worker resumes. At-least-once delivery may repeat computation, but idempotency prevents duplicate business effects.
8. Repeatedly unprocessable jobs enter the DLQ with enough context for diagnosis and controlled replay.

The source-of-truth rule is important:

PostgreSQL says what the run is. The queue says what work may need attention. Redis coordinates fast, disposable control state. Artifact storage holds large outputs.

Redis loss may temporarily reduce throughput or force limits to rebuild, but it should not erase the authoritative run history.

A compact failure model

The architecture is easier to review by asking what happens at each crash boundary:

Failure	Detection	Recovery	Correctness mechanism
API dies after accepting request	Missing outbox publication	Outbox relay publishes later	Atomic run + outbox transaction
Worker dies before acknowledgement	Visibility lease expires	Another worker receives message	At-least-once + idempotency
Provider completes but response is lost	Client timeout; outcome ambiguous	Query status, retry with key, or reconcile	Provider idempotency or durable intent
Two workers claim the same run	Conditional update conflict	Loser stops or reloads	State machine + version check
Retryable outage persists	Attempt/elapsed budget exhausted	Delay, fallback, or DLQ	Bounded retry policy
Cancellation races with completion	State/version conflict	Apply legal winning transition	Monotonic terminal states
Queue grows faster than workers drain it	Queue age and depth rise	Throttle admission, scale, or degrade	Backpressure and capacity limits
Primary model is unavailable	Error-rate threshold opens circuit	Approved fallback handles eligible tasks	Capability and quality contract

This table also exposes “fixes” that are only operational conveniences. A queue dashboard may show a stuck job, but only lease expiry and safe replay recover it. A log may reveal a duplicate email, but only idempotency prevents the duplicate.

Capacity is part of correctness

If runs arrive at rate (\lambda) and workers complete them at rate (\mu), a sustained (\lambda > \mu) means unbounded backlog. Average estimates are not enough; LLM latency and token counts have long tails. Capacity planning must use workload distributions, provider quotas, retry amplification, tenant mix, and desired queue-age SLOs.

Partition work when different classes require isolation—for example, by tenant tier, region, provider, or interactive versus batch traffic. Replicate durable data according to recovery objectives, but remember that replicas introduce lag and consistency choices. Leader election is needed only for singleton coordination tasks such as an outbox relay shard or scheduled sweeper; ordinary worker execution should remain horizontally distributed.

Caching can reduce cost and dependency load for reusable deterministic inputs, but model output caches require prompt, model, parameters, tool versions, permissions, and tenant scope in the key. Invalidation is a product-correctness decision, not just a TTL choice.

The design test

A durable agent platform should answer these questions without relying on “that probably will not happen”:

• What durable fact proves that a run was accepted?
• Who owns the run now, and when can ownership expire?
• What happens if the worker crashes after every individual side effect?
• Which operations are safe to retry, and under what logical key?
• Which state transitions are legal when completion, retry, and cancellation race?
• From which checkpoint can a new worker resume?
• What stops one tenant, provider, or workload from consuming all capacity?
• What bounds attempts, elapsed time, tokens, and money?
• When is fallback allowed, and how is degraded quality made visible?
• Which SLI tells us that users are receiving the promised service?

If these answers live only in Python control flow, the system is still process-shaped. If they are encoded in durable state, delivery rules, invariants, and observable recovery paths, the system is job-shaped.

Conclusion

AI agents do not require distributed-systems design because models are mysterious. They require it because useful agent runs are long-lived, remote, stateful, expensive, and side-effecting.

Starting from one synchronous endpoint, production failures force each architectural step:

• long-running work forces separation of request and execution;
• process crashes force durable queues and acknowledgements;
• acknowledgements force at-least-once delivery;
• at-least-once delivery forces idempotency;
• expensive multi-step work forces state machines and checkpoints;
• concurrent control forces conditional transitions;
• user intent forces cooperative cancellation;
• finite downstream capacity forces rate limits, bulkheads, and backpressure;
• external outages force bounded retries, circuit breakers, and deliberate fallback;
• cross-service execution forces correlation, tracing, SLOs, and tested recovery.

The goal is not to prevent every failure. That is impossible in a distributed system. The goal is to make failures detectable, recovery repeatable, costs bounded, and externally visible outcomes correct.

That is the real foundation of a production agent runtime.