Bhaskar
← All writing
first principlesAug 05, 2026· 21 min read

Running AI Agents as Durable Distributed Systems

An agent looks simple in development:

@app.post("/run")
async def run_agent(request: Request):
    state = initial_state(request)
    while not state.done:
        decision = await model_call(state)
        state = await execute(decision, state)
    return state.output

This works while execution is short, one process owns the run, dependencies are healthy, and retrying the HTTP request is harmless. Production removes every one of those assumptions.

An agent may run for minutes or hours. A worker can crash after sending an email but before recording success. A model provider can return 429, time out after accepting a request, or change behavior behind the same model name. Two workers can receive the same job. A deployment can replace a pod halfway through a run. One tenant can consume the provider quota needed by everyone else.

The important shift is this:

A production agent is not a long HTTP request. It is a durable, versioned state machine that performs expensive and sometimes irreversible distributed side effects.

Once we accept that framing, most of the infrastructure follows from ordinary distributed-systems reasoning.

1. Deriving the requirements from failure

Start with synchronous request-response execution and add one failure at a time.

New condition What breaks Infrastructure requirement Invariant
The run outlives the request Client disconnects or gateway times out Asynchronous submission and durable execution An accepted run survives the API request
The process crashes In-memory state disappears Checkpoints and recoverable workers A run resumes from durable state
Multiple workers run concurrently The same step can execute twice At-least-once-aware processing and idempotency Duplicate delivery does not duplicate protected effects
A dependency fails The run stops or retries uncontrollably Classified retries, backoff, timeouts and dead-letter handling Only retryable failures are retried, within a finite budget
A cancel arrives mid-step Work and cost continue after cancellation Durable cancellation state and cooperative interruption Cancellation is eventually observed and terminal state is consistent
Several tenants share capacity One tenant starves the others Tenant quotas, fair scheduling and isolation Every expensive operation is charged and constrained to one tenant
Models, prompts and tools change Old runs cannot be explained or reproduced Immutable version manifests Every run is tied to its exact execution configuration
A deployment regresses behavior Error and cost rise before humans notice Evaluation gates, canaries, SLOs and rollback New versions earn production traffic

These are not product features. They are correctness conditions.

2. The architecture of the Agent Reliability Lab

The lab will use the following production shape:

flowchart TD
    C["Client"] --> API["FastAPI control plane"]
    API --> PG["PostgreSQL<br/>runs, budgets, versions, ledgers"]
    PG --> O["Transactional outbox"]
    O --> T["Temporal task queues"]
    T --> W["Versioned workers"]
    W --> M["Models and tools"]
    W --> S3["Object storage<br/>artifacts and checkpoints"]
    W --> PG
    W --> R["Redis<br/>ephemeral coordination"]
    W --> E["Trace event stream"]
    E --> CH["ClickHouse<br/>trace analytics"]

Why each component exists

Component Problem it solves What it must not become
FastAPI Authentication, admission, validation, status, cancellation and replay APIs The place where long agent execution lives
Temporal Durable orchestration, task delivery, timers, retry state and recovery after worker loss The business database or artifact store
PostgreSQL Authoritative run admission, tenant ownership, budgets, version manifests and side-effect ledger A high-volume trace warehouse
Object storage Large, immutable inputs, outputs, checkpoints and tool artifacts A transactional coordination database
ClickHouse Cheap analytical queries over high-volume spans, token usage, latency and evaluations A correctness dependency on the execution path
Redis Fast rate counters, short leases, cache entries and concurrency coordination The only copy of run state or money usage
Kubernetes Isolation, deployment, worker scaling and resource management A workflow engine; restarting a pod is not resuming a step

Temporal is used here because the project contains long-running state, timers, cancellation, retries and versioned workers. A basic queue is sufficient when jobs are short and stateless. If the lab used RabbitMQ or Kafka instead, we would have to build the execution state machine, timers, leases, checkpoint recovery, retry scheduling and workflow version compatibility ourselves.

Temporal workflow code should contain deterministic orchestration; model calls, database queries and tool execution belong in retryable Activities. Temporal explicitly requires deterministic workflows for replay and recommends idempotent Activities because an Activity can execute more than once.

There are two authorities, with a deliberate boundary:

  • Temporal history is authoritative for orchestration progress.
  • PostgreSQL is authoritative for admission, tenant ownership, reserved and spent budget, version manifests, and protected side effects.

PostgreSQL also contains a query-friendly run-status projection. That projection may lag and must be rebuildable from execution events; it must not silently become a second competing orchestration truth.

3. Accepting a run without a dangerous dual write

Suppose the API inserts a PostgreSQL row and then starts a workflow. It can crash between those operations. Reversing the order merely reverses the inconsistency.

Use a transactional outbox:

  1. Authenticate the tenant.
  2. Validate the request and calculate the maximum permitted budget.
  3. In one PostgreSQL transaction:
    • insert agent_runs;
    • persist the immutable version manifest;
    • reserve the initial budget;
    • insert run_requested into outbox_events.
  4. Return 202 Accepted with run_id.
  5. An outbox dispatcher starts the workflow using run_id as the workflow ID.
  6. Mark the outbox event delivered only after the durable execution system accepts it.

The dispatcher may send the same event twice. That is expected. Starting the same logical workflow with the same identifier must be idempotent.

The acceptance invariant is:

If the API returns 202, either the run eventually starts or an operator-visible reconciliation process identifies and repairs it.

Monitor the age of the oldest undelivered outbox row. Queue depth alone cannot detect a run stranded before queue publication.

4. The execution state machine

Do not encode run state as scattered booleans such as is_running, is_failed, and cancelled. Use explicit states and validated transitions.

stateDiagram-v2
    [*] --> QUEUED
    QUEUED --> RUNNING
    QUEUED --> CANCEL_REQUESTED
    RUNNING --> WAITING
    WAITING --> RUNNING
    RUNNING --> CANCEL_REQUESTED
    WAITING --> CANCEL_REQUESTED
    RUNNING --> SUCCEEDED
    RUNNING --> FAILED
    RUNNING --> DEAD_LETTERED
    CANCEL_REQUESTED --> CANCELED

WAITING covers retry timers, provider throttling, scheduled wakeups and human approval, with a separate wait_reason. Terminal states are SUCCEEDED, FAILED, CANCELED, and DEAD_LETTERED.

Every state transition should:

  • declare allowed source states;
  • increment state_version;
  • record who or what caused it;
  • append an immutable event;
  • occur with a compare-and-swap condition such as WHERE state_version = :expected.

This prevents a late worker from overwriting a newer cancellation or completion.

Minimal metadata model

agent_runs(
  run_id uuid primary key,
  tenant_id uuid not null,
  status text not null,
  state_version bigint not null,
  workflow_id text not null unique,
  version_manifest_id uuid not null,
  parent_run_id uuid null,
  replay_mode text null,
  cancel_requested_at timestamptz null,
  created_at timestamptz not null,
  started_at timestamptz null,
  finished_at timestamptz null
);

run_steps(
  step_id uuid primary key,
  run_id uuid not null,
  tenant_id uuid not null,
  logical_step_key text not null,
  attempt int not null,
  status text not null,
  idempotency_key text not null,
  input_hash text not null,
  output_artifact_id uuid null,
  error_class text null,
  started_at timestamptz not null,
  finished_at timestamptz null,
  unique(run_id, logical_step_key, attempt),
  unique(tenant_id, idempotency_key)
);

side_effects(
  tenant_id uuid not null,
  idempotency_key text not null,
  effect_type text not null,
  request_hash text not null,
  status text not null,
  provider_reference text null,
  response_artifact_id uuid null,
  primary key(tenant_id, idempotency_key)
);

budget_ledger(
  entry_id uuid primary key,
  tenant_id uuid not null,
  run_id uuid not null,
  step_id uuid null,
  kind text not null,             -- reserve, charge, release, correction
  amount_microusd bigint not null,
  tokens bigint null,
  created_at timestamptz not null
);

version_manifests(
  version_manifest_id uuid primary key,
  manifest_json jsonb not null,
  manifest_hash text not null unique,
  created_at timestamptz not null
);

The full schema also needs artifacts, checkpoint_refs, run_events, evaluations, outbox_events, and tenant quota tables. Large payloads belong in object storage; PostgreSQL stores their URI, content hash, size, media type, encryption metadata and retention class.

5. At-least-once execution and the exactly-once illusion

A worker performs a remote call, receives success, and crashes before acknowledging the task. The queue cannot know whether the side effect happened, so it redelivers the task.

This is why production workers should assume at-least-once execution. “Exactly once” usually means:

  1. work may execute more than once;
  2. a stable idempotency key identifies one logical effect;
  3. a unique constraint or downstream idempotency facility exposes the effect once.

For a tool call, derive the key from stable identity, not the attempt number:

idempotency_key =
  SHA256(tenant_id + run_id + logical_step_key + effect_type + canonical_input)

The worker protocol is:

  1. Insert or load the side_effects row by idempotency key.
  2. If it is SUCCEEDED, return the recorded result.
  3. If another live owner is executing it, wait or fail with a retryable conflict.
  4. Call the external system with the same idempotency key when supported.
  5. Record the provider reference and result.
  6. Mark the effect complete and checkpoint the next workflow state.
  7. Acknowledge completion.

There is still an unavoidable gap if the external provider performs an effect but offers neither an idempotency key nor a queryable operation ID. No database transaction can atomically cover your PostgreSQL database and an unrelated API. In that case, choose explicitly among:

  • an idempotent downstream API;
  • a reconciliation query before retry;
  • an outbox consumed by a side-effect service;
  • an effectively-once business rule;
  • human resolution for ambiguous outcomes.

Model calls deserve the same honesty. A retry can duplicate cost even if only one response becomes visible. If the provider cannot deduplicate requests, budget for bounded duplicate execution and record the ambiguity.

6. Checkpoints, crashes and recovery

A checkpoint is not a Python object dump. It is a durable boundary after which completed work does not need to be recomputed.

A useful checkpoint contains:

  • current logical node and next eligible transitions;
  • compact agent state;
  • references and hashes for large artifacts;
  • completed side-effect keys;
  • accumulated token and monetary usage;
  • immutable version manifest ID;
  • checkpoint sequence number and schema version.

Write a checkpoint only after all effects it claims as complete are durably recorded. Otherwise recovery may skip work that never happened.

After a worker crash:

  1. Temporal redelivers the unfinished Activity.
  2. The new worker loads the run and latest checkpoint.
  3. It checks durable cancellation and budget state.
  4. It inspects the side-effect ledger.
  5. It returns a recorded result or safely retries unfinished work.
  6. The workflow continues from durable history.

Use frequent checkpoints around expensive or irreversible boundaries, not after every token. More checkpoints reduce recovery work but increase storage, write amplification and schema-evolution burden.

7. Timeouts, cancellation, retries and dead letters

These controls solve different problems:

  • A timeout limits how long one operation may remain incomplete.
  • A retry policy decides whether and when another attempt is useful.
  • A cancellation says the result is no longer wanted.
  • A dead-letter state says automatic recovery has stopped and explicit inspection is required.

Classify failures before retrying:

Failure Retry? Typical action
Provider 429 or transient 5xx Yes Exponential backoff with jitter; respect Retry-After
Network timeout with unknown remote outcome Carefully Reconcile by idempotency key or provider operation ID
Invalid tool arguments Usually no Return to planner or fail validation
Authentication or revoked secret No automatic storm Open circuit, alert and require repair
Context-length violation No identical retry Compact, switch strategy, or terminate
Budget exhausted No Finish with explicit budget terminal reason
Worker process crash Yes Redeliver from durable state

Retries need four bounds: maximum attempts, maximum elapsed time, maximum backoff, and remaining run budget. Add jitter so many failed runs do not wake together.

Cancellation is cooperative. The API durably records cancel_requested_at and signals the workflow. Workers check cancellation before every expensive action and heartbeat during long tools. A completed external email or payment is not rolled back by marking a run canceled. Cancellation means “stop future work,” not “erase history.”

After retries are exhausted, place the run in DEAD_LETTERED with its failure class, last safe checkpoint, owning version, affected tenant and operator action. Redriving creates a new attempt or child run; it does not edit history until the old run looks successful.

8. Concurrency, provider limits and cost budgets

Horizontal scaling can make an outage worse. If a provider starts throttling, adding workers can produce more retries, longer queues and greater cost.

Control concurrency at several levels:

  • per worker, to protect CPU, memory and file descriptors;
  • per model/provider, to respect requests-per-minute and tokens-per-minute;
  • per tenant, to prevent noisy neighbors;
  • per tool, to protect fragile downstream systems;
  • globally, to cap total spend and blast radius.

Redis is useful for fast distributed semaphores and rate counters, but durable budget truth belongs in PostgreSQL. A lost Redis key may briefly reduce coordination quality; it must not erase money already spent.

Use budget reservation:

  1. Estimate the maximum cost of the next operation.
  2. Atomically reserve it against the tenant and run budgets.
  3. Reject or choose a cheaper path if reservation fails.
  4. Execute the call.
  5. Reconcile the reservation with actual tokens and cost.
  6. Release unused capacity.

Hard limits should exist for money, input tokens, output tokens, wall-clock time, steps, tool calls and retries. Model fallback is a policy decision, not an exception handler: record the ordered fallback set, quality floor, price ceiling and errors that permit switching.

Scale workers from queue backlog and oldest-task age, while respecting provider headroom. Backlog says how much work exists; age says whether users are waiting too long.

9. Caching, artifacts and trace storage

Caching an agent response is safe only if the cache key captures everything that can change the answer:

tenant + normalized input hash
+ prompt version + model identifier + sampling parameters
+ tool contract/version + retrieval snapshot
+ policy version + orchestration version

Never share sensitive cached content across tenants. Do not cache side effects. Treat semantic caches as approximate optimizations with evaluation and invalidation policies, not as correctness mechanisms.

Use object storage for immutable blobs:

  • original inputs;
  • model requests and responses, after redaction;
  • generated files;
  • tool stdout/stderr;
  • retrieval snapshots;
  • checkpoints too large for PostgreSQL.

Content-address artifacts with a SHA-256 hash, encrypt them, and apply tenant-aware retention. Store metadata and access policy in PostgreSQL.

Execution must not wait for ClickHouse. Emit trace events through an outbox or buffered collector and ingest them asynchronously. ClickHouse stores spans, attempts, model usage, evaluator results and cost events for analytical queries. If analytics ingestion fails, execution continues and the lag alert fires.

10. Version everything required for replay

“We used model X” is not enough. Each run needs an immutable manifest containing:

  • orchestration Git commit and container image digest;
  • workflow and checkpoint schema versions;
  • prompt template content hash;
  • model provider, model identifier and exposed revision;
  • sampling, reasoning and token parameters;
  • tool code image, API schema and dependency versions;
  • retrieval corpus or snapshot identifiers;
  • safety, permission, routing and fallback policy versions;
  • evaluator versions;
  • environment feature flags.

Secrets are referenced by identity and rotation version, never copied into the manifest.

There are three different meanings of replay:

  1. Workflow replay: reconstruct orchestration state from recorded events without repeating external calls.
  2. Diagnostic replay: execute orchestration against recorded model and tool outputs; no external side effects.
  3. Live re-execution: create a new run linked by parent_run_id, pin the old manifest where possible, and perform real calls in a sandbox.

Never “replay” by resetting an old row to QUEUED. Preserve the original evidence and create a linked run. Exact behavioral reproduction may still be impossible because hosted models and external data can change. The goal is traceable reproducibility, not a false promise of bit-for-bit determinism.

11. Tenant isolation and secret management

Every durable record, cache key, trace, artifact prefix and idempotency key must carry tenant_id. Enforce isolation twice:

  • application authorization on every request and worker operation;
  • database-level protection such as row-level security or tenant-scoped repositories.

For stronger boundaries, high-risk or regulated tenants may receive separate queues, worker pools, encryption keys, databases or clusters.

Workers receive least-privilege service identities. Fetch short-lived secrets at execution time from a secret manager; do not store them in queue payloads, traces, prompts, checkpoints or container images. Redaction happens before telemetry leaves the worker. Tool permissions should be scoped by tenant, run and operation, not merely by which worker binary is running.

12. Observability that explains behavior

Logs alone cannot explain an agent trajectory. Represent a run as correlated spans:

run
├── planning step
├── model call
├── tool call
├── checkpoint write
├── retry delay
└── evaluation

Every event should include tenant_id, run_id, step_id, attempt, trace/span IDs, workflow ID, version manifest ID, model/tool version, latency, token usage, cost, cache status and error class. Never use raw prompt text as a metric label.

OpenTelemetry provides shared conventions for traces, metrics and logs. Add agent-specific attributes without losing the ordinary HTTP, database and messaging spans that reveal infrastructure causes.

Useful dashboards:

  • submission rate, queue depth and oldest-task age;
  • run start and completion latency by tenant and version;
  • success, cancellation, retry and dead-letter rates;
  • model/tool latency, 429s and timeouts;
  • tokens and cost per successful run;
  • duplicate-effect conflicts;
  • checkpoint age and recovery duration;
  • trace ingestion lag;
  • evaluator scores by canary and control versions.

Initial SLIs and SLOs

SLI Example SLO
Accepted submissions / valid submissions 99.9% monthly
Runs started within 30 seconds / accepted runs 99%
Recovered within 2 minutes / runs interrupted by worker loss 99%
Cancellation observed within 30 seconds / cancellable runs 99%
Runs with complete core trace / terminal runs 99.9%
Known duplicated protected side effects 0
Runs exceeding enforced hard budget 0

Do not define “agent success” as one infrastructure SLO. Separate availability, latency, cost and behavioral correctness. Alert on error-budget burn, oldest queue age, provider saturation, budget violations and trace gaps—not merely CPU.

13. Safe deployments and evaluation gates

Agent behavior can regress while HTTP error rate remains flat. Deployment therefore needs two independent gates.

Offline gate

Before production:

  • run deterministic unit and state-transition tests;
  • replay representative workflow histories;
  • test checkpoint migrations;
  • run golden, adversarial and failure-injection cases;
  • compare task success, side-effect correctness, cost and latency against the current version;
  • block releases that cross explicit regression thresholds.

Online gate

Release a new version to a small tenant-safe cohort:

  • pin each run to one worker and manifest version;
  • compare canary and control on operational and behavioral metrics;
  • prevent a long-running workflow from silently switching orchestration code;
  • expand traffic only after minimum sample size and observation time;
  • stop routing new work and drain or retain compatible workers during rollback.

Temporal worker versioning is designed for the problem of changing workflow code while old executions remain active. Ordinary rolling deployment is insufficient when a run can span multiple releases.

14. Kubernetes deployment and capacity planning

Run separate Deployments for:

  • FastAPI control-plane replicas;
  • outbox dispatchers;
  • workers grouped by workload and permission class;
  • telemetry collectors;
  • dead-letter and reconciliation processors.

Prefer managed PostgreSQL, object storage, ClickHouse and Temporal for the first production version. Stateful systems inside Kubernetes add backup, upgrade and quorum responsibilities that do not teach agent reliability cheaply.

For each worker Deployment:

  • use readiness, liveness and startup probes for their distinct purposes;
  • stop polling before shutdown, then drain active work;
  • set a termination grace period longer than the normal checkpoint/heartbeat interval;
  • use PodDisruptionBudgets and anti-affinity for availability;
  • assign resource requests and limits from measured workloads;
  • autoscale from task-queue age/backlog with a provider-quota ceiling;
  • isolate dangerous tools with separate service accounts, networks and worker pools.

Kubernetes Jobs retry pods until completion, but a pod retry does not understand agent steps, remote side effects, prompt versions or checkpoints. Use Jobs for bounded batch administration, not as the primary durable agent runtime.

Capacity planning begins with demand and service time:

required concurrency ≈ arrival rate × average execution time

That is only the starting point. Split capacity by model-bound, I/O-bound and CPU-bound steps; include peak traffic, retry amplification, provider quotas, tenant fairness and recovery headroom. Load-test with realistic token sizes and dependency latency, not empty mock calls.

15. Production debugging and incident response

For any bad run, reconstruct one timeline:

  1. admission and version manifest;
  2. queue wait and worker assignment;
  3. every attempt and timeout;
  4. model/tool request identity and protected result;
  5. budget reservations and charges;
  6. checkpoints and state transitions;
  7. cancellation or deployment events;
  8. evaluator outcome.

During an incident:

  1. Contain: pause affected versions, tenants, tools or providers.
  2. Preserve: retain histories, traces, manifests and artifacts.
  3. Classify: infrastructure, dependency, orchestration, data, policy or model behavior.
  4. Recover: resume from a safe checkpoint, use fallback, or redrive from dead letter.
  5. Reconcile: detect ambiguous and duplicated external effects.
  6. Learn: add the incident to offline evaluations and failure-injection tests.

Consider a provider slowdown:

provider latency rises
→ activity timeouts rise
→ retries multiply
→ queue age rises
→ autoscaler adds workers
→ provider throttling worsens
→ cost and latency accelerate

The earliest causal failure may be the provider slowdown, but the system-level outage is amplified by retry and scaling policies that ignored provider headroom. A useful postmortem fixes both.

16. A practical build order

Build the Agent Reliability Lab in increments, and require every component to justify itself with a failed invariant:

  1. Synchronous baseline: one FastAPI request, one worker function, full trace.
  2. Durable submission: 202, PostgreSQL run row, transactional outbox, Temporal workflow.
  3. Execution state: explicit transitions, immutable events and status projection.
  4. Crash recovery: Activities, checkpoints, worker termination tests.
  5. Safe effects: idempotency ledger, unique constraints and reconciliation.
  6. Failure control: timeouts, classified retries, jitter and dead-letter processing.
  7. Human control: durable cancellation, approval waits and operator redrive.
  8. Resource control: tenant concurrency, provider limits and budget reservation.
  9. Evidence: object artifacts, OpenTelemetry, ClickHouse dashboards and alerts.
  10. Reproducibility: version manifests and three replay modes.
  11. Deployment safety: offline evaluation gate, canary routing and rollback.
  12. Kubernetes: separated deployments, graceful draining and quota-aware autoscaling.
  13. Chaos tests: kill workers at every persistence boundary; inject 429, timeouts, partial writes, Redis loss, trace lag and deployment changes.

The final demonstration should prove behavior, not merely show architecture:

  • kill a worker after a protected side effect and show that retry does not duplicate it;
  • cancel a run during a long tool and show bounded cancellation latency;
  • exceed a tenant budget and show that the next model call never starts;
  • deploy incompatible workflow code while an old run is active and show safe version routing;
  • replay an old run diagnostically without external calls;
  • overload one tenant and show that another tenant retains its SLO;
  • break ClickHouse ingestion and show that execution continues while observability lag alerts.

Conclusion

Reliable agent infrastructure is mostly the disciplined application of distributed-systems fundamentals.

Queues provide delivery, not exactly once. Workers are disposable, so state must be durable. Checkpoints define recovery boundaries. Idempotency protects side effects from redelivery. Budgets and concurrency limits turn cost into a controlled resource. Version manifests turn mysterious behavior into attributable behavior. Traces explain trajectories. Offline evaluations and canaries protect deployments from regressions that ordinary uptime checks cannot see.

The production question is not, “Can the agent complete this task?”

It is:

Can the system preserve its invariants when processes crash, messages repeat, providers fail, tenants compete, models change and deployments happen during execution?

If the answer is measurable and testable, the agent is becoming a production system.

Primary references