An AI system becomes difficult to control long before it becomes an “agent.” The trouble begins as soon as a useful request needs more than one model call, database query, external API call, or human decision.

Consider a customer-support operation:

Receive request
→ classify
→ retrieve policy
→ inspect account
→ generate recommendation
→ validate policy
→ request approval
→ execute action
→ notify customer

At first, this looks like a sequence of functions. That appearance is misleading. In production, a process can stop between any two steps. A model can return an invalid category. A database can time out. A reviewer can wait for two days. A worker can crash after an external action succeeds but before the success is recorded. A customer can cancel while the workflow is paused. A deployment can change the state schema while old executions are still alive.

The solution is not a more elaborate prompt. It is an explicit workflow: durable state, legal transitions, guarded actions, recorded evidence, and defined recovery behavior. Only after that foundation exists should a model be allowed to choose among actions.

This chapter derives that foundation from first principles.

1. Begin with sequential procedures

The smallest implementation is ordinary sequential code:

def handle_support_request(request):
    category = classify(request)
    policy = retrieve_policy(category)
    account = inspect_account(request.customer_id)
    recommendation = generate_recommendation(request, policy, account)
    validate_policy(recommendation, policy)
    approval = request_approval(recommendation)
    action_result = execute_action(recommendation, approval)
    notify_customer(request.customer_id, action_result)
    return action_result

This is a procedure: control flow is implicit in the program counter. If the process is currently calling inspect_account, that fact exists only in memory and perhaps in logs.

For a short-lived, deterministic, all-local calculation, this is ideal. It is readable and has little machinery. It fails when the operation becomes long-running or externally observable:

If the worker crashes after execute_action, where should execution resume?
If the account service times out, which call should be retried, how often, and when?
If approval takes a day, should a worker remain alive for a day?
If a duplicate request arrives, can the refund be issued twice?
If the customer cancels, which states permit cancellation?
If policy validation fails, is that a technical failure, a business rejection, or a request for human escalation?
If code is redeployed halfway through, can the old execution be interpreted safely?

Sequential code hides the answers. Its call stack is being used as the workflow database.

2. A workflow is a procedure whose progress is explicit

A workflow represents a multi-step business process whose progress matters independently of any single process or worker. It does not merely say which functions exist. It records:

what has happened;
what is allowed to happen next;
what data was produced;
what is waiting;
what failed;
whether execution may resume, retry, cancel, or compensate.

The decisive design move is to replace “the next line of code” with state plus transitions.

That gives us a durable answer to the question: Where is this request now?

3. Deriving states from requirements

A state is a meaningful condition that changes what events are legal next. Do not create a state merely because a function exists. Create one when the system must remember a distinction across time, failure, observation, or human intervention.

For the support workflow, useful states include:

REQUEST_RECEIVED
CLASSIFYING
RETRIEVING_POLICY
INSPECTING_ACCOUNT
GENERATING_RECOMMENDATION
VALIDATING_POLICY
AWAITING_APPROVAL
EXECUTING_ACTION
NOTIFYING_CUSTOMER
RETRY_WAIT
COMPENSATING
MANUAL_REVIEW
COMPLETED
REJECTED
CANCELLED
FAILED
FAILED_COMPENSATED

Why make AWAITING_APPROVAL a state? Because time may pass, no worker should remain allocated, and only approval-related events should be legal. Why make RETRY_WAIT a state? Because “temporarily failed and scheduled to try again” is operationally different from “permanently failed.” Why distinguish FAILED from REJECTED? Because infrastructure failure and a valid business refusal have different owners, metrics, and customer messages.

A useful test is:

If the process stops here, do we need to remember this condition in order to decide safely what may happen later?

If yes, it probably deserves explicit representation.

4. Events explain why state changes

A state is a noun-like condition. An event is a fact that may cause a transition:

START
CLASSIFICATION_SUCCEEDED
POLICY_RETRIEVED
ACCOUNT_INSPECTED
RECOMMENDATION_GENERATED
POLICY_VALID
POLICY_INVALID
APPROVED
REJECTED_BY_REVIEWER
ACTION_EXECUTED
CUSTOMER_NOTIFIED
RETRYABLE_ERROR
PERMANENT_ERROR
RETRY_DUE
CANCEL_REQUESTED
COMPENSATION_SUCCEEDED
COMPENSATION_FAILED
ESCALATE

Events should describe observed facts or explicit commands, not vague intentions. POLICY_RETRIEVED is better than GO_NEXT. The former gives audit evidence and can carry a policy identifier and version.

An event normally contains:

event type;
workflow ID;
unique event ID;
timestamp;
actor or source;
causation and correlation IDs;
payload or references to artifacts;
schema version.

Unique event IDs make duplicate delivery detectable. Causation IDs allow us to say that a model call produced a classification event, which caused a retrieval step, which eventually caused an approval request.

5. Transitions define legal change

A transition maps a current state and event to a new state:

(current state, event) → next state

Examples:

(REQUEST_RECEIVED, START) → CLASSIFYING
(CLASSIFYING, CLASSIFICATION_SUCCEEDED) → RETRIEVING_POLICY
(VALIDATING_POLICY, POLICY_VALID) → AWAITING_APPROVAL
(AWAITING_APPROVAL, APPROVED) → EXECUTING_ACTION
(NOTIFYING_CUSTOMER, CUSTOMER_NOTIFIED) → COMPLETED

Anything absent from the transition table is forbidden. APPROVED cannot move a request from REQUEST_RECEIVED directly to EXECUTING_ACTION. A late approval event cannot revive a cancelled workflow. The engine, not the prompt, enforces this.

This converts control flow into data that can be inspected, tested, and audited.

6. Guards decide whether a legal-looking transition is allowed

State and event type are sometimes insufficient. A transition may also require a guard: a deterministic predicate that must be true.

For example:

AWAITING_APPROVAL + APPROVED → EXECUTING_ACTION

may require:

the approver has the support_refund_approver role;
the approval refers to the current recommendation version;
the amount is within the approver’s limit;
the request has not expired or been cancelled;
mandatory policy evidence exists.

Guards answer “may this transition occur?” They must not perform side effects. They should be deterministic over explicit inputs so their decisions can be reproduced.

def approval_guard(state, event) -> bool:
    return (
        event.actor_role == "support_refund_approver"
        and event.recommendation_version == state.recommendation_version
        and event.approved_amount <= event.actor_limit
        and state.policy_validation.passed
        and not state.cancel_requested
    )

A model may recommend approval. It should not be the final authority for authorization, monetary limits, tenant boundaries, or irreversible actions.

7. Actions perform work because a transition occurred

An action is a side effect or computation associated with entering a state or accepting an event. Examples include calling a model, querying an account service, persisting a checkpoint, scheduling a retry, or issuing a refund.

Keep these concepts separate:

Guard: may the state change?
Transition: how does state change?
Action: what work is performed?

If an action fails, the workflow does not pretend the transition completed. The failure becomes an event and enters the failure model.

Actions should receive explicit inputs and return explicit results. In AI steps, store the model name/version, prompt or template version, input artifact references, structured output, validation outcome, token usage, latency, and trace ID. Do not treat a prose completion as an invisible local variable.

8. Initial state establishes a single entry point

Every workflow needs an initial state. Here it is REQUEST_RECEIVED.

Creation should validate the request, assign a stable workflow ID, record the initiator and tenant, attach a request deduplication key, and persist the first checkpoint and audit event atomically.

The initial state is not merely a default enum value. It is evidence that the request was accepted into the workflow boundary. If request creation is retried, the same deduplication key should return the existing workflow rather than create a second execution.

9. Terminal states end transition processing

A terminal state has no ordinary outgoing transitions. The workflow has reached a final outcome and late events are recorded but do not mutate it.

Our terminal states are:

COMPLETED: required business work and notification completed;
REJECTED: the business request was validly refused;
CANCELLED: cancellation was accepted before an unsafe boundary;
FAILED: the workflow could not complete and no compensation was needed or possible;
FAILED_COMPENSATED: work failed after a side effect, and the compensating action succeeded.

MANUAL_REVIEW is intentionally not terminal. A human may resolve, retry, reject, cancel, or complete the workflow through a controlled event.

10. Success is a business outcome, not “no exception”

A function returning without throwing is not enough to declare success. Define success operationally.

For this workflow, COMPLETED means:

a policy-valid recommendation was approved by an authorized actor;
the external action has a recorded provider result tied to an idempotency key;
the customer was notified, or the product explicitly defines notification as non-blocking and records a separate delivery workflow;
the terminal result contains stable references to evidence.

If notification is best-effort, then action success and notification success should be separate business processes. Do not silently redefine “complete” inside exception handling.

11. Failure states must preserve meaning

Not every undesirable outcome is the same:

Outcome	Meaning	Example
Business rejection	System worked; request was not allowed	Policy forbids the refund
Validation failure	Produced data violated a contract	Model returned an unknown category
Transient technical failure	Retry may succeed	Account API timed out
Permanent technical failure	Same attempt should not be repeated	Account no longer exists
Authorization failure	Actor lacked permission	Reviewer exceeded approval limit
Safety escalation	Automated decision is too uncertain or risky	Conflicting policy sources
Compensation failure	Side effect cannot be safely reversed automatically	Refund issued but reversal endpoint failed

If all of these become one FAILED state with an error string, operators cannot route them correctly, metrics become misleading, and retry behavior becomes dangerous.

12. Retry states make waiting durable

A retry is not an exception handler that sleeps. It is explicit workflow state:

RETRY_WAIT
  retry_target = INSPECTING_ACCOUNT
  attempt = 2
  next_attempt_at = 2026-08-17T10:02:00Z
  error_class = ACCOUNT_SERVICE_TIMEOUT

When the timer fires, a RETRY_DUE event returns the workflow to the stored target. A worker can die during the wait without losing the schedule.

Retry policy belongs to the operation:

Operation	Retry?	Example policy
Model classification	Yes, for transport/rate-limit errors	3 attempts, exponential backoff + jitter
Invalid structured model output	Limited	one repair attempt, then escalation
Policy not found	Usually no blind retry	business failure or human review
Account API timeout	Yes	bounded exponential backoff
Approval rejection	No	terminal business outcome
Execute refund	Only with idempotency	reconcile provider result before retry
Notify customer	Yes	independent delivery retries, then dead-letter/manual review

Retry only errors likely to be transient. Every retry consumes time, money, provider quota, and possibly duplicate-effect risk.

13. Conditional branching makes decisions visible

Sequential if statements are fine until their results must be durable and explainable. A workflow expresses branches through events and guarded transitions:

VALIDATING_POLICY + POLICY_VALID   → AWAITING_APPROVAL
VALIDATING_POLICY + POLICY_INVALID → REJECTED
VALIDATING_POLICY + POLICY_UNCLEAR → MANUAL_REVIEW

The validator returns a typed result with evidence, not merely True or False. The transition table makes all supported outcomes discoverable.

For model classification, the model may produce billing, access, or technical. Deterministic code validates that the value is in the allowed schema and chooses the corresponding workflow branch. The model proposes structured data; code owns the state mutation.

14. Loops require explicit bounds

Loops occur when a recommendation is revised after validation or human feedback:

GENERATING_RECOMMENDATION
→ VALIDATING_POLICY
→ GENERATING_RECOMMENDATION

An unbounded AI loop can spend indefinitely while appearing productive. Store a loop counter and budget:

maximum revisions;
token and monetary budget;
deadline;
repeated-output detector;
escalation condition.

After two failed revisions, for example, transition to MANUAL_REVIEW. The stopping condition is code-enforced. “Try until correct” is not a policy.

15. Workflows form directed graphs

Once branches, retries, approval waits, and compensation exist, the procedure is naturally a directed graph:

nodes represent states;
directed edges represent allowed transitions;
event types label edges;
guards constrain edge traversal;
actions produce effects and new events.

The graph reveals unreachable states, cycles, missing failure exits, and accidental paths around approval. A diagram is useful, but the executable transition table remains the source of truth.

flowchart TD
    R["Request received"] --> C["Classify"]
    C --> P["Retrieve policy"]
    P --> A["Inspect account"]
    A --> G["Generate recommendation"]
    G --> V["Validate policy"]
    V -->|valid| H["Await approval"]
    V -->|invalid| X["Rejected"]
    V -->|unclear| M["Manual review"]
    H -->|approved| E["Execute action"]
    H -->|rejected| X
    E --> N["Notify customer"]
    N --> D["Completed"]

Retries and compensation are omitted from the diagram for readability, not from the state model.

16. A finite-state machine supplies enforceable semantics

A finite-state machine (FSM) consists of:

a finite set of states;
a finite set of events;
one initial state;
a transition function;
optional guards and actions;
a set of terminal states.

An FSM is valuable because it rejects undefined transitions. It does not infer that approval “probably means” execution should start. This becomes a security boundary: even a compromised prompt or malformed tool output cannot jump from classification to refund execution if no such transition exists.

Real workflow systems also carry data, timers, history, parallelism, and external effects, so they are richer than a textbook FSM. The finite-state discipline still gives the control plane its core integrity.

17. Workflow data is not the same as state

The word “state” is overloaded. Separate five concepts:

Concept	Question answered	Example
Business state	Where is the customer’s request in the business process?	`AWAITING_APPROVAL`
Execution state	What is the runtime doing or waiting for?	attempt 2, retry due at 10:02, lease owner `worker-7`
Model context	What evidence and instructions are supplied to a model call?	request text, policy excerpts, account summary, schema
Persisted checkpoint	What durable snapshot lets execution resume safely?	business + execution data at version 12
Terminal result	What stable outcome can downstream systems consume?	approved refund ID, notification ID, evidence references

Workflow data is the typed collection of durable artifacts produced along the way: classification, policy references, account snapshot, recommendation, validation report, approval record, action receipt, and notification receipt.

Do not place an ever-growing prompt transcript into the workflow row. Store raw artifacts separately and keep stable references plus hashes in workflow data. Build model context afresh for each model call from authorized, relevant artifacts.

18. Execution state belongs to the runtime

Execution state includes operational details that are not business facts:

attempt counts;
current operation;
worker lease and lease expiry;
retry target and next attempt time;
timeout deadlines;
cancellation flag;
state version for optimistic concurrency;
last processed event IDs;
workflow definition version.

Suppose the account inspection times out. The business request has not become invalid. Its execution is waiting to retry. Mixing these layers often leads to status fields like FAILED_RETRYING_APPROVAL_PENDING, which encode unrelated dimensions into an impossible enum.

A practical schema can retain one primary business state plus structured execution metadata.

19. Persistence removes the worker from the correctness boundary

Persistence is needed as soon as the workflow must survive process loss, delayed events, or deployment. The database, not the worker’s stack, becomes the durable source of current progress.

At each accepted transition, commit atomically:

the new state and state version;
workflow data changes;
an immutable audit event;
any outgoing job or message through an outbox record.

The outbox matters. If the database commit succeeds but publishing the next job fails, an outbox relay can publish it later. Without this pattern, the workflow can be durable yet permanently stuck.

Use optimistic concurrency:

UPDATE workflows
SET state = :next_state,
    version = version + 1,
    data = :data
WHERE id = :id AND version = :expected_version;

If zero rows change, another worker or event already advanced the workflow. Reload and reconcile instead of overwriting it.

20. A checkpoint is a resumable boundary

A checkpoint is a persisted, self-consistent snapshot from which the workflow can decide what to do next without reconstructing volatile call-stack state.

Checkpoint after every externally significant transition, especially:

before waiting for human input;
before and after an irreversible or costly side effect;
after a model produces a validated artifact;
when scheduling a retry;
when entering a terminal state.

Persisting “some variables occasionally” is not enough. A checkpoint should include the workflow definition version, state version, business state, execution metadata, durable artifact references, budgets, and relevant causation IDs.

21. Resumption means deciding, not blindly continuing

On resume, the engine loads the checkpoint and asks:

Is the workflow terminal?
Is it waiting for an external event or timer?
Was an operation started but its result not recorded?
Has its lease expired?
Is the workflow definition/schema compatible?
Is cancellation pending?
What is the next legal action?

The dangerous case is an ambiguous side effect. If a worker crashed after sending a refund request but before recording the response, never issue a fresh refund under a new key. Query the provider using the original idempotency key, reconcile the result, then record either success or a safe retry decision.

Resumption therefore depends on idempotency and reconciliation, not only serialization.

22. Timeouts are events, not just client settings

There are several distinct timeouts:

operation timeout: one HTTP or model call took too long;
state timeout: approval has been pending for 24 hours;
workflow deadline: the whole support request exceeded its service-level objective;
lease timeout: a worker stopped heartbeating.

Each timeout should produce an event such as OPERATION_TIMED_OUT or APPROVAL_EXPIRED. The transition table then decides whether to retry, escalate, cancel, or fail. An HTTP timeout alone does not tell us whether the remote side effect occurred.

23. Cancellation is a controlled transition

Cancellation is not deleting the workflow row or killing a worker. It is a request that may be accepted, delayed, or denied depending on current state.

Before external execution, cancellation can usually transition to CANCELLED. While a side effect is in flight, cancellation becomes cancel_requested = true; the engine must first reconcile the action. After a refund has completed, “cancel” may require a compensating transaction rather than a state flip.

Define cancellation semantics per state:

Current state	Cancellation behavior
`AWAITING_APPROVAL`	cancel immediately
`RETRY_WAIT` before action	cancel and remove/supersede timer
`EXECUTING_ACTION`	mark requested, reconcile in-flight action
`NOTIFYING_CUSTOMER`	business action remains; decide whether notification may be suppressed
terminal state	reject as a late command; do not mutate outcome

24. Human approval is a durable external event

Human-in-the-loop design does not mean pausing a thread until someone clicks a button. Enter AWAITING_APPROVAL, persist the checkpoint, issue an approval task, and release the worker.

The approval request should show:

proposed action and parameters;
policy evidence and account evidence;
uncertainty and model rationale as supporting information, not proof;
amount or risk level;
expiration time;
recommendation version;
approve, reject, or request-changes choices.

The returned event must identify the actor and recommendation version. Otherwise, someone may approve version 2 while the engine executes a later version 3.

Approval is both a business control and an authorization boundary. The UI is not the enforcement layer; the transition guard is.

25. Escalation routes ambiguity to a different capability

Escalation is appropriate when automation cannot safely classify an outcome as success, retryable failure, or rejection. Examples include conflicting policies, high-value actions, repeated invalid model outputs, suspected prompt injection, missing account evidence, or exhausted budgets.

MANUAL_REVIEW should preserve:

why escalation occurred;
evidence gathered so far;
exact unresolved questions;
allowed reviewer commands;
deadline and owner;
whether the workflow may still be cancelled.

“Send to a human” without a defined state and resolution protocol merely moves ambiguity elsewhere.

26. Compensation addresses committed side effects

Database rollback cannot undo an email, refund, account mutation, or shipment. Distributed workflows use compensating actions: explicit business operations that semantically counter earlier effects.

Examples:

refund issued incorrectly → create a reversal or recovery case;
account credit applied → post an offsetting debit, if policy permits;
reservation created → cancel the reservation;
external notification sent → it cannot be unsent; send a correction.

Compensation is not guaranteed restoration. It can fail, and some effects are irreversible. Model it explicitly:

EXECUTING_ACTION + POST_ACTION_INVARIANT_FAILED → COMPENSATING
COMPENSATING + COMPENSATION_SUCCEEDED          → FAILED_COMPENSATED
COMPENSATING + COMPENSATION_FAILED             → MANUAL_REVIEW

Do not compensate merely because customer notification failed. The correct compensating boundary is a business decision. Reversing a valid refund due to an email outage would compound the failure.

27. Idempotency makes retries safe enough

An operation is idempotent when repeating the same logical request produces no additional business effect.

For action execution, derive a stable key from workflow identity and logical operation, not attempt number:

refund:{workflow_id}:{recommendation_version}

All retries use the same key. The action service stores or honors the key and returns the existing result for duplicates.

Idempotency is required at several layers:

workflow creation deduplicates the incoming support request;
event ingestion deduplicates event IDs;
transition persistence uses state-version concurrency;
external writes use stable idempotency keys;
notifications use a stable message/delivery key.

“Exactly once” is rarely a property of distributed transport. A more honest design is at-least-once delivery plus deduplication, idempotent effects, and reconciliation.

28. Parallel branches reduce latency but create new state

Policy retrieval and account inspection may be independent after classification. They can run in parallel:

flowchart TD
    C["Classification complete"] --> P["Retrieve policy"]
    C --> A["Inspect account"]
    P --> J["Join evidence"]
    A --> J
    J --> G["Generate recommendation"]

The workflow must track each branch independently:

{
  "branches": {
    "policy": {"status": "SUCCEEDED", "attempts": 1},
    "account": {"status": "RETRY_WAIT", "attempts": 2}
  }
}

Parallelism is not “start two promises.” It needs branch-level retries, timeouts, cancellation, artifacts, and ownership.

29. Joining defines what completion means across branches

A join is a guard over branch outcomes. Common policies are:

all branches must succeed;
any one successful branch is enough;
a quorum must succeed;
one branch is mandatory and another is optional;
proceed after a deadline with explicitly degraded evidence.

For recommendations affecting customer accounts, both policy and account evidence are mandatory. The join fires only when both have succeeded. If one fails permanently, the parent transitions to failure or manual review and cancels any unnecessary unfinished branch.

Store which artifact version each branch produced so the recommendation consumes a consistent evidence set.

30. Error propagation is a policy, not automatic exception bubbling

In sequential code, exceptions bubble up the stack. In a workflow graph, decide how branch or child-workflow failures affect the parent.

Possible policies include:

fail fast and cancel siblings;
wait for all branches and aggregate errors;
continue with optional branch failure;
retry only the failed branch;
escalate with partial evidence;
compensate already-completed sibling effects.

The chosen policy should be visible in workflow definition and audit events. A generic except Exception: FAILED discards business meaning.

31. State versioning keeps old executions interpretable

Long-running workflows outlive deployments. Record two versions:

state row version, incremented for optimistic concurrency;
workflow definition/schema version, identifying the meaning of states and data.

Never silently reinterpret an old state using new semantics. Use one of these strategies:

keep the old workflow definition executable until existing instances finish;
migrate checkpoints through an explicit, tested migration;
route old instances to manual review;
introduce additive changes that preserve old meaning.

Model prompts and policies need versions too. The terminal result should say which workflow, prompt, model, policy, and validation versions produced it.

32. Observability reconstructs causality

Logs are not enough if they cannot reconstruct the workflow. Emit an immutable audit event for every accepted and rejected transition.

A useful event includes:

{
  "workflow_id": "wf_123",
  "event_id": "evt_456",
  "event_type": "APPROVED",
  "from_state": "AWAITING_APPROVAL",
  "to_state": "EXECUTING_ACTION",
  "actor": {"type": "human", "id": "user_42"},
  "timestamp": "2026-08-17T10:15:00Z",
  "workflow_version": 1,
  "state_version": 12,
  "causation_id": "approval_task_91",
  "correlation_id": "support_case_88",
  "guard_results": ["role_allowed", "amount_within_limit"],
  "artifact_refs": ["recommendation:v2", "policy:returns:v7"]
}

Observe at least:

workflow counts and duration by terminal outcome;
time spent in each state;
retry counts by operation and error class;
approval wait time and rejection rate;
model cost, latency, schema failures, and confidence distribution;
invalid transition attempts;
compensation and manual-review rates;
stuck workflows and expired leases;
idempotency conflicts and reconciliation outcomes.

Tracing should connect each model, retrieval, database, and external API span to the workflow and event that caused it.

33. Workflow testing targets paths and invariants

Function tests are insufficient. Test the graph.

Important test classes include:

Transition-table tests: every allowed pair reaches the expected state.
Invalid-transition tests: undefined state/event pairs are rejected without mutation.
Guard tests: unauthorized or stale approval never executes an action.
Path tests: success, rejection, transient failure, permanent failure, escalation, cancellation, and compensation.
Retry tests: backoff, maximum attempts, budget exhaustion, and stable idempotency keys.
Crash tests: crash before a call, after the side effect, and before result persistence; resume safely.
Concurrency tests: two workers consume the same event; only one state version commits.
Duplicate-event tests: replay does not duplicate transitions or effects.
Timer tests: early, late, and duplicate timer delivery.
Version-migration tests: old checkpoints retain meaning.
Property/invariant tests: execution never occurs without valid policy evidence and current approval.
Model-boundary tests: arbitrary structured model outputs cannot create an illegal transition.

The strongest tests assert invariants across all paths, for example:

Every path to EXECUTING_ACTION must previously contain a successful policy validation and a current authorized approval.

34. Deterministic and model-controlled transitions are different

An LLM is useful for interpreting language, classifying ambiguous text, synthesizing evidence, and drafting recommendations. It is not a reliable authority for runtime control, permissions, financial limits, or state integrity.

Use three levels of control:

Decision	Model’s role	Code’s role
Classify request	propose an enum plus confidence/evidence	validate schema; route or escalate
Select policy passages	rank candidates	enforce tenant/filter scope and allowed sources
Generate recommendation	propose typed action and explanation	validate policy, limits, and supported action types
Decide whether approval is required	perhaps estimate risk	deterministic policy decides
Approve action	no authority	authorized human/system guard decides
Execute action	supply already-validated parameters at most	capability-scoped tool performs idempotent write
Mark complete	no authority	engine verifies required receipts and transitions

Even when a model appears to “choose the next step,” it should choose only from capabilities exposed for the current state. Its proposal becomes an event candidate. Code validates the event, arguments, budget, permissions, and transition.

The workflow engine remains the reference monitor.

35. Hybrid AI workflows combine probabilistic judgment with deterministic control

A hybrid AI workflow uses models inside selected nodes while preserving deterministic orchestration around them.

The pattern is:

deterministic code assembles authorized context;
the model produces a schema-constrained proposal;
deterministic validation checks structure and invariants;
an evaluator or policy engine checks quality and safety;
the state machine accepts one of a finite set of events;
risky writes require approval and capability-scoped execution;
every result becomes durable evidence.

This provides controlled flexibility. The model handles ambiguity where rules are brittle; the workflow handles authority and recovery where probabilities are unsafe.

Practical project: a controlled support workflow

We will now turn the design into a small framework-free Python implementation. It is intentionally plain: enums, dataclasses, a transition table, a repository boundary, and an engine.

Explicit state schema

from __future__ import annotations

from dataclasses import dataclass, field, replace
from datetime import datetime, timezone
from enum import StrEnum
from typing import Any, Callable
from uuid import uuid4


class State(StrEnum):
    REQUEST_RECEIVED = "REQUEST_RECEIVED"
    CLASSIFYING = "CLASSIFYING"
    RETRIEVING_POLICY = "RETRIEVING_POLICY"
    INSPECTING_ACCOUNT = "INSPECTING_ACCOUNT"
    GENERATING_RECOMMENDATION = "GENERATING_RECOMMENDATION"
    VALIDATING_POLICY = "VALIDATING_POLICY"
    AWAITING_APPROVAL = "AWAITING_APPROVAL"
    EXECUTING_ACTION = "EXECUTING_ACTION"
    NOTIFYING_CUSTOMER = "NOTIFYING_CUSTOMER"
    RETRY_WAIT = "RETRY_WAIT"
    COMPENSATING = "COMPENSATING"
    MANUAL_REVIEW = "MANUAL_REVIEW"
    COMPLETED = "COMPLETED"
    REJECTED = "REJECTED"
    CANCELLED = "CANCELLED"
    FAILED = "FAILED"
    FAILED_COMPENSATED = "FAILED_COMPENSATED"


TERMINAL_STATES = {
    State.COMPLETED,
    State.REJECTED,
    State.CANCELLED,
    State.FAILED,
    State.FAILED_COMPENSATED,
}


class EventType(StrEnum):
    START = "START"
    CLASSIFIED = "CLASSIFIED"
    POLICY_RETRIEVED = "POLICY_RETRIEVED"
    ACCOUNT_INSPECTED = "ACCOUNT_INSPECTED"
    RECOMMENDATION_GENERATED = "RECOMMENDATION_GENERATED"
    POLICY_VALID = "POLICY_VALID"
    POLICY_INVALID = "POLICY_INVALID"
    APPROVED = "APPROVED"
    REJECTED_BY_REVIEWER = "REJECTED_BY_REVIEWER"
    ACTION_EXECUTED = "ACTION_EXECUTED"
    CUSTOMER_NOTIFIED = "CUSTOMER_NOTIFIED"
    RETRYABLE_ERROR = "RETRYABLE_ERROR"
    PERMANENT_ERROR = "PERMANENT_ERROR"
    RETRY_DUE = "RETRY_DUE"
    CANCEL_REQUESTED = "CANCEL_REQUESTED"
    ESCALATE = "ESCALATE"
    COMPENSATE = "COMPENSATE"
    COMPENSATION_SUCCEEDED = "COMPENSATION_SUCCEEDED"
    COMPENSATION_FAILED = "COMPENSATION_FAILED"


@dataclass(frozen=True)
class Event:
    type: EventType
    event_id: str
    actor_id: str
    actor_role: str
    payload: dict[str, Any] = field(default_factory=dict)
    occurred_at: datetime = field(
        default_factory=lambda: datetime.now(timezone.utc)
    )


@dataclass(frozen=True)
class ExecutionState:
    attempts: dict[str, int] = field(default_factory=dict)
    retry_target: State | None = None
    next_attempt_at: datetime | None = None
    cancel_requested: bool = False
    lease_owner: str | None = None
    lease_expires_at: datetime | None = None


@dataclass(frozen=True)
class WorkflowData:
    request: dict[str, Any]
    classification: dict[str, Any] | None = None
    policy_ref: str | None = None
    account_ref: str | None = None
    recommendation: dict[str, Any] | None = None
    recommendation_version: int = 0
    validation: dict[str, Any] | None = None
    approval: dict[str, Any] | None = None
    action_receipt: dict[str, Any] | None = None
    notification_receipt: dict[str, Any] | None = None
    compensation_receipt: dict[str, Any] | None = None


@dataclass(frozen=True)
class Workflow:
    workflow_id: str
    tenant_id: str
    state: State
    data: WorkflowData
    execution: ExecutionState
    version: int = 0                 # optimistic-concurrency version
    definition_version: int = 1      # workflow/schema meaning
    processed_event_ids: frozenset[str] = frozenset()


@dataclass(frozen=True)
class TerminalResult:
    workflow_id: str
    outcome: State
    action_receipt: dict[str, Any] | None
    notification_receipt: dict[str, Any] | None
    evidence_refs: tuple[str, ...]

Notice that model context is not a field. A context builder constructs it for a particular model call from request, authorized policy data, account data, and prompt version. The checkpoint stores durable references, not an uncontrolled transcript.

Allowed transitions

Guard = Callable[[Workflow, Event], bool]


def always(_: Workflow, __: Event) -> bool:
    return True


def valid_approval(wf: Workflow, event: Event) -> bool:
    return (
        event.actor_role == "support_approver"
        and wf.data.validation is not None
        and wf.data.validation.get("passed") is True
        and event.payload.get("recommendation_version")
            == wf.data.recommendation_version
        and event.payload.get("amount", 0)
            <= event.payload.get("actor_limit", 0)
    )


TRANSITIONS: dict[tuple[State, EventType], tuple[State, Guard]] = {
    (State.REQUEST_RECEIVED, EventType.START):
        (State.CLASSIFYING, always),
    (State.CLASSIFYING, EventType.CLASSIFIED):
        (State.RETRIEVING_POLICY, always),
    (State.RETRIEVING_POLICY, EventType.POLICY_RETRIEVED):
        (State.INSPECTING_ACCOUNT, always),
    (State.INSPECTING_ACCOUNT, EventType.ACCOUNT_INSPECTED):
        (State.GENERATING_RECOMMENDATION, always),
    (State.GENERATING_RECOMMENDATION, EventType.RECOMMENDATION_GENERATED):
        (State.VALIDATING_POLICY, always),
    (State.VALIDATING_POLICY, EventType.POLICY_VALID):
        (State.AWAITING_APPROVAL, always),
    (State.VALIDATING_POLICY, EventType.POLICY_INVALID):
        (State.REJECTED, always),
    (State.AWAITING_APPROVAL, EventType.APPROVED):
        (State.EXECUTING_ACTION, valid_approval),
    (State.AWAITING_APPROVAL, EventType.REJECTED_BY_REVIEWER):
        (State.REJECTED, always),
    (State.EXECUTING_ACTION, EventType.ACTION_EXECUTED):
        (State.NOTIFYING_CUSTOMER, always),
    (State.NOTIFYING_CUSTOMER, EventType.CUSTOMER_NOTIFIED):
        (State.COMPLETED, always),
    (State.COMPENSATING, EventType.COMPENSATION_SUCCEEDED):
        (State.FAILED_COMPENSATED, always),
    (State.COMPENSATING, EventType.COMPENSATION_FAILED):
        (State.MANUAL_REVIEW, always),
}

Retry, cancellation, escalation, and permanent errors apply to multiple nonterminal states. Treat them as explicit policy rules rather than duplicating every edge:

RETRYABLE_STATES = {
    State.CLASSIFYING,
    State.RETRIEVING_POLICY,
    State.INSPECTING_ACCOUNT,
    State.GENERATING_RECOMMENDATION,
    State.NOTIFYING_CUSTOMER,
}

CANCELLABLE_STATES = {
    State.REQUEST_RECEIVED,
    State.CLASSIFYING,
    State.RETRIEVING_POLICY,
    State.INSPECTING_ACCOUNT,
    State.GENERATING_RECOMMENDATION,
    State.VALIDATING_POLICY,
    State.AWAITING_APPROVAL,
    State.RETRY_WAIT,
    State.MANUAL_REVIEW,
}

EXECUTING_ACTION is excluded from blind retry and immediate cancellation because an ambiguous external write must be reconciled first.

Repository and audit boundary

The following interfaces state the transactional requirement without tying the example to a database library:

class VersionConflict(Exception):
    pass


class InvalidTransition(Exception):
    pass


class GuardRejected(Exception):
    pass


class Repository:
    def load(self, workflow_id: str) -> Workflow:
        raise NotImplementedError

    def commit_transition(
        self,
        previous_version: int,
        workflow: Workflow,
        audit_event: dict[str, Any],
        outbox_jobs: list[dict[str, Any]],
    ) -> None:
        """Atomically update state and insert audit/outbox rows."""
        raise NotImplementedError

In production, implement commit_transition in one database transaction. The WHERE version = previous_version check prevents two workers from advancing the same checkpoint.

Transition engine

class WorkflowEngine:
    def __init__(self, repository: Repository):
        self.repository = repository

    def apply(self, workflow_id: str, event: Event) -> Workflow:
        current = self.repository.load(workflow_id)

        if event.event_id in current.processed_event_ids:
            return current  # duplicate delivery

        if current.state in TERMINAL_STATES:
            raise InvalidTransition(
                f"terminal workflow {current.state} rejects {event.type}"
            )

        next_state, next_execution = self._resolve(current, event)
        updated_data = self._merge_payload(current, event)

        updated = replace(
            current,
            state=next_state,
            data=updated_data,
            execution=next_execution,
            version=current.version + 1,
            processed_event_ids=(
                current.processed_event_ids | {event.event_id}
            ),
        )

        audit = {
            "audit_id": str(uuid4()),
            "workflow_id": current.workflow_id,
            "event_id": event.event_id,
            "event_type": event.type,
            "from_state": current.state,
            "to_state": updated.state,
            "actor_id": event.actor_id,
            "actor_role": event.actor_role,
            "state_version": updated.version,
            "definition_version": updated.definition_version,
            "occurred_at": event.occurred_at.isoformat(),
        }

        jobs = self._jobs_for(updated)
        self.repository.commit_transition(
            previous_version=current.version,
            workflow=updated,
            audit_event=audit,
            outbox_jobs=jobs,
        )
        return updated

    def _resolve(
        self, wf: Workflow, event: Event
    ) -> tuple[State, ExecutionState]:
        if event.type == EventType.CANCEL_REQUESTED:
            if wf.state not in CANCELLABLE_STATES:
                raise InvalidTransition(
                    f"cannot cancel safely from {wf.state}"
                )
            return State.CANCELLED, replace(
                wf.execution, cancel_requested=True
            )

        if event.type == EventType.ESCALATE:
            return State.MANUAL_REVIEW, wf.execution

        if event.type == EventType.PERMANENT_ERROR:
            return State.FAILED, wf.execution

        if event.type == EventType.RETRYABLE_ERROR:
            if wf.state not in RETRYABLE_STATES:
                raise InvalidTransition(
                    f"blind retry is unsafe from {wf.state}"
                )
            attempts = dict(wf.execution.attempts)
            attempts[wf.state] = attempts.get(wf.state, 0) + 1
            if attempts[wf.state] > self._max_attempts(wf.state):
                return State.MANUAL_REVIEW, replace(
                    wf.execution, attempts=attempts
                )
            return State.RETRY_WAIT, replace(
                wf.execution,
                attempts=attempts,
                retry_target=wf.state,
                next_attempt_at=event.payload["next_attempt_at"],
            )

        if event.type == EventType.RETRY_DUE:
            if wf.state != State.RETRY_WAIT:
                raise InvalidTransition("retry timer is not currently expected")
            if wf.execution.retry_target is None:
                raise InvalidTransition("retry target is missing")
            return wf.execution.retry_target, replace(
                wf.execution,
                retry_target=None,
                next_attempt_at=None,
            )

        if event.type == EventType.COMPENSATE:
            if wf.data.action_receipt is None:
                raise GuardRejected("nothing has been committed to compensate")
            return State.COMPENSATING, wf.execution

        transition = TRANSITIONS.get((wf.state, event.type))
        if transition is None:
            raise InvalidTransition(
                f"{event.type} is invalid from {wf.state}"
            )

        next_state, guard = transition
        if not guard(wf, event):
            raise GuardRejected(
                f"guard rejected {event.type} from {wf.state}"
            )
        return next_state, wf.execution

    def _max_attempts(self, state: State) -> int:
        return {
            State.CLASSIFYING: 2,
            State.RETRIEVING_POLICY: 3,
            State.INSPECTING_ACCOUNT: 4,
            State.GENERATING_RECOMMENDATION: 2,
            State.NOTIFYING_CUSTOMER: 5,
        }[state]

    def _merge_payload(self, wf: Workflow, event: Event) -> WorkflowData:
        data = wf.data
        if event.type == EventType.CLASSIFIED:
            return replace(data, classification=event.payload)
        if event.type == EventType.POLICY_RETRIEVED:
            return replace(data, policy_ref=event.payload["policy_ref"])
        if event.type == EventType.ACCOUNT_INSPECTED:
            return replace(data, account_ref=event.payload["account_ref"])
        if event.type == EventType.RECOMMENDATION_GENERATED:
            return replace(
                data,
                recommendation=event.payload,
                recommendation_version=data.recommendation_version + 1,
            )
        if event.type in {EventType.POLICY_VALID, EventType.POLICY_INVALID}:
            return replace(data, validation=event.payload)
        if event.type == EventType.APPROVED:
            return replace(data, approval=event.payload)
        if event.type == EventType.ACTION_EXECUTED:
            return replace(data, action_receipt=event.payload)
        if event.type == EventType.CUSTOMER_NOTIFIED:
            return replace(data, notification_receipt=event.payload)
        if event.type == EventType.COMPENSATION_SUCCEEDED:
            return replace(data, compensation_receipt=event.payload)
        return data

    def _jobs_for(self, wf: Workflow) -> list[dict[str, Any]]:
        if wf.state in TERMINAL_STATES | {
            State.AWAITING_APPROVAL,
            State.RETRY_WAIT,
            State.MANUAL_REVIEW,
        }:
            return []
        return [{
            "job_id": str(uuid4()),
            "workflow_id": wf.workflow_id,
            "state": wf.state,
            "expected_version": wf.version,
        }]

The engine has no model-specific control loop. A worker consumes the outbox job for the current state, performs that state’s action, and submits one typed event. For example:

def run_classification(job, workflow, model_client, engine):
    context = build_classification_context(
        request=workflow.data.request,
        tenant_id=workflow.tenant_id,
        prompt_version="support-classifier-v3",
    )
    raw = model_client.classify(context)
    result = validate_classification_schema(raw)

    if result.confidence < 0.75:
        event_type = EventType.ESCALATE
        payload = {"reason": "low_classification_confidence"}
    else:
        event_type = EventType.CLASSIFIED
        payload = result.to_dict()

    engine.apply(
        workflow.workflow_id,
        Event(
            type=event_type,
            event_id=str(uuid4()),
            actor_id="classification-worker",
            actor_role="system",
            payload=payload,
        ),
    )

The model controls the proposed classification content. Deterministic code controls schema acceptance, the confidence threshold, available event types, and the actual state transition.

Idempotent write action

def execute_support_action(workflow, action_client, engine):
    recommendation = workflow.data.recommendation
    assert recommendation is not None

    key = (
        f"support-action:{workflow.workflow_id}:"
        f"v{workflow.data.recommendation_version}"
    )

    # The provider must return the original receipt for a repeated key.
    receipt = action_client.execute(
        action_type=recommendation["action_type"],
        parameters=recommendation["parameters"],
        idempotency_key=key,
    )

    engine.apply(
        workflow.workflow_id,
        Event(
            type=EventType.ACTION_EXECUTED,
            event_id=str(uuid4()),
            actor_id="action-worker",
            actor_role="system",
            payload={
                "provider_operation_id": receipt.operation_id,
                "idempotency_key": key,
                "status": receipt.status,
            },
        ),
    )

If the worker crashes after action_client.execute, resumption first calls a provider reconciliation endpoint with key. It does not invent a new key or assume failure from the missing local receipt.

Retry policy

Use exponential backoff with jitter and a hard cap:

import random
from datetime import timedelta


def next_retry_time(attempt: int) -> datetime:
    base_seconds = min(300, 2 ** attempt)
    jitter = random.uniform(0, base_seconds * 0.2)
    return datetime.now(timezone.utc) + timedelta(
        seconds=base_seconds + jitter
    )

The calculated time is persisted in RETRY_WAIT, and a durable timer produces RETRY_DUE. Unit tests should inject the clock and jitter source rather than depend on real time.

Invalid-transition tests

import pytest


def event(kind, **payload):
    return Event(
        type=kind,
        event_id=str(uuid4()),
        actor_id="test",
        actor_role="system",
        payload=payload,
    )


def test_cannot_execute_before_approval(repo, engine, received_workflow):
    repo.seed(received_workflow)

    with pytest.raises(InvalidTransition):
        engine.apply(
            received_workflow.workflow_id,
            event(EventType.ACTION_EXECUTED, provider_operation_id="x"),
        )

    assert repo.load(received_workflow.workflow_id).state \
        == State.REQUEST_RECEIVED


def test_stale_approval_is_rejected(repo, engine, awaiting_approval):
    repo.seed(awaiting_approval)  # recommendation_version == 3

    stale = Event(
        type=EventType.APPROVED,
        event_id=str(uuid4()),
        actor_id="approver-7",
        actor_role="support_approver",
        payload={
            "recommendation_version": 2,
            "amount": 100,
            "actor_limit": 1000,
        },
    )

    with pytest.raises(GuardRejected):
        engine.apply(awaiting_approval.workflow_id, stale)


def test_duplicate_event_has_no_second_effect(repo, engine, classifying):
    repo.seed(classifying)
    classified = event(
        EventType.CLASSIFIED,
        category="billing",
        confidence=0.94,
    )

    first = engine.apply(classifying.workflow_id, classified)
    second = engine.apply(classifying.workflow_id, classified)

    assert first.version == second.version
    assert repo.audit_count(event_id=classified.event_id) == 1


def test_cannot_blindly_retry_external_write(repo, engine, executing):
    repo.seed(executing)

    with pytest.raises(InvalidTransition):
        engine.apply(
            executing.workflow_id,
            event(
                EventType.RETRYABLE_ERROR,
                next_attempt_at=datetime.now(timezone.utc),
            ),
        )


def test_terminal_workflow_rejects_late_approval(repo, engine, cancelled):
    repo.seed(cancelled)

    with pytest.raises(InvalidTransition):
        engine.apply(
            cancelled.workflow_id,
            event(EventType.APPROVED),
        )

Additional integration scenarios should simulate:

process death before and after each checkpoint;
duplicate jobs delivered to two workers;
an external action that succeeds while the local response is lost;
approval arriving after cancellation;
approval for an obsolete recommendation version;
retry budget exhaustion;
policy and account branch failures under a parallel implementation;
compensation success and compensation failure;
an old definition-version checkpoint resumed by new code;
arbitrary model output attempting to name an unavailable action or transition.

Reconstructing the design from requirements

When facing a new AI workflow, do not begin by drawing boxes named after prompts. Use this derivation sequence:

Write the happy-path procedure. This exposes the intended business steps.
Mark every waiting and failure boundary. Network calls, model calls, human waits, timers, and writes deserve attention.
Derive states from durable distinctions. Ask what must still be known after a crash.
Name events as facts. Avoid meaningless NEXT events.
Enumerate allowed transitions. Default-deny everything else.
Add deterministic guards. Enforce authorization, versions, limits, evidence, and invariants.
Separate the five layers. Business state, execution state, model context, persisted checkpoint, terminal result.
Classify every failure. Decide retry, rejection, escalation, compensation, or terminal failure.
Bound retries and loops. Store attempts, budgets, deadlines, and stopping conditions.
Define cancellation per state. Especially around irreversible effects.
Make external writes idempotent and reconcilable. Assume duplicate delivery and lost responses.
Persist transitions, audit events, and outbox jobs atomically. Remove worker memory from correctness.
Version the state model. Old executions must retain meaning.
Test forbidden paths and crash boundaries. Happy-path tests prove very little.
Only then grant the model choices. Expose a finite, state-specific capability set and validate every proposal.

Mastery gate

You understand the subject when you can reconstruct the support workflow without memorizing its enum names.

Given only the business requirements, you should be able to answer:

Which conditions must survive a worker crash, and therefore deserve explicit state?
Which data is a business fact, which is runtime metadata, and which belongs only in model context?
What events can legally leave every state?
Which guards prevent bypassing validation, approval, permissions, and version checks?
Which failures are retryable, and why will retry not duplicate effects?
What exact checkpoint allows safe resumption after each external call?
What happens when an approval arrives after cancellation or for an old recommendation?
What cancellation means before, during, and after action execution?
Which side effects have valid compensations, and what happens if compensation fails?
Which judgments may be proposed by a model, and which transitions remain deterministic?
Which audit evidence reconstructs the complete causal path?
Which invalid transitions and crash points must tests attempt?

The final exercise is to rebuild the graph and transition table from those answers, then prove this invariant:

No customer-affecting action can execute unless authorized evidence, current policy validation, current human approval, an idempotency key, and a legal transition are all present.

That is the deeper reason to build the state machine before the agent. An agent can generate proposals. A trustworthy system must decide which proposals are legal, remember what happened, recover when reality interrupts the happy path, and prove why an action occurred.