An Agent Does Not Need More Memory; It Needs Better Information Selection

A naive research agent often begins with a simple design:

1. Receive the user’s request.
2. Call the model.
3. Execute the requested tool.
4. Append the tool result to the conversation.
5. Send the entire history back to the model.
6. Repeat until the task is complete.

This works surprisingly well for short tasks. The history contains the goal, earlier decisions and tool results, so the model appears to “remember.”

But the design eventually fails.

Imagine an agent researching whether a company is a good investment. It searches the web, downloads annual reports, extracts financial tables and compares competitors. After several iterations, every search result, failed tool call, raw document and outdated conclusion is passed back to the model.

The agent becomes slower and more expensive. Important evidence is buried beneath irrelevant text. Old conclusions conflict with newer evidence. A malicious instruction inside a downloaded page may reappear in every model call. Eventually, the conversation exceeds the model’s context window.

The obvious response is: “The agent needs better memory.”

That diagnosis is incomplete.

The agent already retained too much information. What it lacks is a system for deciding:

• What information exists?
• Where should it live?
• How long should it live?
• Which information is relevant now?
• Which version should be trusted?
• What should enter the next model call?

This is the real problem of state, memory and context engineering.

Four Concepts That Must Not Be Blurred

These terms describe different roles.

Concept	Precise meaning	Example
State	Information the application needs to continue operating correctly	Current step, completed actions and remaining budget
Memory	Information retained from the past because it may help future decisions	A user’s preferred report format
Storage	The system that persists information	PostgreSQL, object storage or Redis
Context	The information supplied to the model for one specific call	Goal, current step and three relevant evidence excerpts

Storage is a mechanism. Memory is a purpose.

A preference can be memory stored in PostgreSQL. An annual report can be stored in an object store without being agent memory. Execution state may be persisted in a database but should not automatically be called long-term memory.

Most importantly:

Information may exist in state, memory or storage without belonging in the current model context.

Context is a temporary, selected view—not the agent’s entire knowledge.

The First Failure: The Agent Cannot Continue Reliably

Suppose the research agent has completed these actions:

• Found the company’s latest annual report.
• Extracted its revenue.
• Failed to extract one competitor’s margin.
• Scheduled another search.
• Used 7 of its 15 allowed tool calls.

If this information exists only inside the conversation, the application must reconstruct operational progress from natural language. After a crash, it may not know which actions finished, whether an external operation succeeded or how much budget remains.

This is an execution-state failure.

Execution state represents the agent’s position inside a run:

goal
current step
plan
completed actions
pending action
tool results
errors
remaining step, token, cost and time budgets
run status

Execution state belongs primarily to the runtime. A small portion may enter the model context when the model needs it to choose the next action.

For example, the model may need to know that competitor-margin extraction failed twice. It probably does not need database transaction IDs or heartbeat timestamps.

The Second Failure: The Agent Loses the Thread of the Current Task

During a long investigation, the agent discovers dozens of facts. Only a few matter to its current reasoning:

• Revenue increased.
• Operating margin declined.
• The decline appears related to infrastructure spending.
• One source remains unverified.

Continuously replaying every observation is wasteful. Removing all earlier observations, however, makes the agent repeat work or lose intermediate conclusions.

This creates the need for working memory.

Working memory is short-lived, task-specific information that supports the current run. It can contain:

• Current hypotheses
• Important intermediate findings
• Unresolved questions
• Temporary constraints
• A compact summary of progress

Working memory should change as the task changes. Once the research run ends, much of it can expire.

A working-memory summary is not a perfect historical record. It is a compressed representation optimized for continuing the task. Raw evidence must remain elsewhere so important claims can be checked again.

The Third Failure: The Agent Forgets What the Conversation Means

A user might say:

Compare it with the previous company, but keep the same format.

Understanding this request requires conversational information:

• What “it” refers to
• Which company was previously discussed
• What “the same format” means
• Whether any earlier instruction was corrected

This is conversation memory: retained conversational information required to preserve continuity across turns.

Conversation memory is not identical to the raw message transcript. The transcript is an artifact. Conversation memory is the useful information derived from it, such as active references, decisions, corrections and unresolved requests.

Recent messages may be included directly. Older conversation can be summarized or retrieved selectively.

The Fourth Failure: The Agent Repeatedly Learns the Same Durable Facts

Suppose the user regularly requests research reports and has stated:

• Use concise explanations.
• Show monetary values in INR and USD.
• Separate verified facts from estimates.

Relearning these preferences in every conversation is inefficient.

This motivates long-term memory: information retained beyond a single run because it may help future runs.

Long-term memory is an umbrella, not one database table. It contains several different kinds of retained information.

Semantic memory

Semantic memory contains general facts believed to remain useful:

• The user works as a full-stack engineer.
• A company changed its name.
• A project uses FastAPI and PostgreSQL.

These facts are not tied primarily to remembering one particular event. They represent what the system currently believes to be true.

Episodic memory

Episodic memory records events or experiences:

• On July 20, the user rejected a highly verbose report.
• A previous research run failed because a source required authentication.
• The agent used a particular dataset and obtained a specific result.

Semantic memory says what is believed. Episodic memory says what happened.

Several episodes may support one semantic belief. For example, repeated requests for concise explanations may justify the semantic memory that the user prefers concise writing.

User preferences

Preferences deserve explicit treatment because they are scoped and revisable.

“Use TypeScript” might be:

• A global preference
• A preference only for web projects
• A constraint for one current task
• An outdated preference the user later reversed

A useful preference record therefore needs scope, provenance, confidence and validity—not just a key-value pair.

The Fifth Failure: Raw Artifacts Consume the Context Window

Research agents work with PDFs, webpages, datasets, images and generated reports. Passing an entire 150-page annual report into every model call is unnecessary.

These objects are raw artifacts.

Artifacts should live in an artifact store, such as object storage or a document repository. Application state should keep references to them:

artifact_id
tenant_id
content type
source URL
checksum
creation time
access policy
derived chunk IDs

Only relevant excerpts should enter model context.

An artifact is not automatically memory. It becomes a source from which useful context or retained knowledge may be derived.

This separation also preserves evidence. Summaries can be regenerated when better models or extraction methods become available.

The Sixth Failure: A Crash Causes Repetition or Corruption

Consider an agent that creates a paid external report and crashes immediately afterward. On restart, the runtime cannot tell whether the request completed. Repeating the call may charge the user twice.

Conversation history does not solve this.

The system needs persistent checkpoints: durable snapshots of execution state from which a run can resume.

A checkpoint may contain:

• Run and tenant identifiers
• Current plan and step
• Completed actions
• Pending action
• Idempotency key
• Working-memory summary
• Artifact references
• Remaining budgets
• Checkpoint version

A checkpoint is not a transcript and not long-term user memory. Its purpose is crash recovery.

For side-effecting operations, the runtime should:

1. Generate an idempotency key.
2. Save a checkpoint marking the action as pending.
3. Execute the action using that key.
4. Persist the result.
5. Save another checkpoint marking the action as complete.

After a crash, the runtime checks the pending action and external result before deciding whether to retry.

This is resumability: continuing from a known durable state without repeating completed work or losing constraints.

The Context Window Is a Budget, Not a Memory Store

A model’s context window has a fixed capacity. That capacity must accommodate more than retrieved information:

• System instructions
• Tool definitions
• User request
• Current execution state
• Working memory
• Retrieved evidence
• Conversation excerpts
• Expected model output

A basic budget is:

[ B_{\text{available}} = W - B_{\text{output}} - B_{\text{instructions}} - B_{\text{tools}} - B_{\text{safety}} ]

Where (W) is the model’s total context window.

Selected information must satisfy:

[ \sum_{i=1}^{n} tokens(item_i) \leq B_{\text{available}} ]

Using every available token is not necessarily beneficial. More context can reduce answer quality by introducing distraction, duplication and conflicting instructions.

The goal is not maximum context. It is sufficient, relevant and trustworthy context.

Context Engineering Is Information Selection

Before each model call, the application should assemble context deliberately.

flowchart TD
    A["Current goal and execution state"] --> D["Context assembler"]
    B["Conversation and retained memory"] --> D
    C["Artifact and knowledge retrieval"] --> D
    D --> E["Filter and rank"]
    E --> F["Resolve conflicts"]
    F --> G["Compress to token budget"]
    G --> H["Model call and context manifest"]

A selection policy can score candidates using:

• Relevance to the current decision
• Authority of the source
• Confidence in the extracted information
• Recency
• Validity period
• Specificity
• Novelty relative to already selected information
• Token cost

Mandatory information—such as the current goal and safety constraints—should be added first. Optional evidence competes for the remaining budget.

For every selected item, the system should record:

what was selected
why it was selected
where it came from
how many tokens it consumed
which version was used
what other candidates were rejected

This creates traceable context assembly.

When the model produces a poor decision, developers can inspect not only the model output but also the exact information-selection process that preceded it.

Retrieval Does Not Mean Vector Search

Embeddings are helpful when the system must find semantically similar text despite different wording. They are not the universal solution to memory.

Many retrieval problems are better solved without embeddings.

Need	Suitable retrieval method
Load the current run	Exact lookup by run_id
Find active preferences	SQL filters by user, scope and validity
Resume from a crash	Latest checkpoint by version
Retrieve a known report	Exact artifact ID
Find recent failures	Time-range query
Match an error code	Keyword or structured lookup
Follow entity relationships	Graph traversal
Find conceptually related passages	Embedding or hybrid search

A production retrieval system often combines:

1. Structured filters
2. Exact or lexical search
3. Semantic search when needed
4. Reranking
5. Context-budget selection

Vector search is one retrieval technique inside a larger information-selection system.

Chunking and Indexing Must Follow the Retrieval Need

Large artifacts usually need to be divided into retrievable units. Arbitrary fixed-size chunks are easy to implement but can separate a claim from its heading, table or qualification.

Useful chunk boundaries often follow document structure:

• Section
• Paragraph
• Table
• Code unit
• Conversation turn
• Tool result
• Event

Each chunk should retain metadata:

artifact_id
chunk_id
section
page or location
tenant_id
source timestamp
checksum
access policy

Indexing should support the questions the agent will ask. Financial tables may require structured extraction. Error logs may need exact matching. Natural-language reports may benefit from hybrid lexical and semantic search.

Compression and Summarization Are Lossy

Context compression reduces representation size. Summarization is one form of compression, but they are not identical.

Compression may include:

• Removing duplicated text
• Dropping irrelevant fields
• Converting verbose tool output into structured values
• Selecting only relevant excerpts
• Summarizing multiple observations

Summarization creates a shorter interpretation of source information. It can remove qualifications, merge conflicting claims or turn uncertainty into apparent certainty.

A reliable summary should therefore retain:

• Important conclusions
• Unresolved questions
• Contradictions
• Source references
• Confidence
• Links to raw evidence

Summaries should be replaceable, not treated as the only surviving record.

For high-stakes claims, the agent should retrieve the original evidence rather than repeatedly summarizing an earlier summary.

Provenance and Confidence

A retained fact without provenance is difficult to trust.

A memory or retrieved claim should record:

• Original source
• Extraction method
• Time observed
• Supporting artifact or event
• Confidence
• Validity period
• Whether it was explicitly provided or inferred

Confidence should describe the basis of a belief, not imitate mathematical certainty.

For example:

• Explicit user statement: high confidence
• Repeated behavioral inference: medium confidence
• One ambiguous interaction: low confidence

Low-confidence information may still be useful, but the context assembler should label it as uncertain.

Expiration and Forgetting

Information does not remain useful forever.

Different categories require different lifetimes:

Information	Typical lifetime
Pending tool call	Until resolved
Working-memory hypothesis	Current run
Checkpoint	Until run retention expires
Temporary conversation reference	Current conversation
Project preference	Until project ends or it is changed
Stable user preference	Until corrected or revalidated
Raw compliance artifact	Defined by retention policy

Expiration can be based on:

• Fixed time-to-live
• End of a run or project
• Source validity
• Replacement by a newer value
• User deletion
• Confidence decay
• Legal retention rules

Expired information may be deleted, archived or retained outside retrieval. It should not silently re-enter model context.

Conflicting Memories Must Remain Visible

Suppose two records exist:

May 1: User prefers Python.
July 10: User prefers TypeScript for this project.

Blindly overwriting the first record loses history. Blindly retrieving both without scope creates confusion.

Conflict handling should consider:

1. Scope: global, project, task or conversation
2. Time: which record is newer?
3. Authority: explicit user statement or inferred behavior?
4. Specificity: does a project preference override a global default?
5. Confidence: how strongly is each record supported?

If the conflict cannot be resolved reliably, the system should preserve both records and request clarification when the decision matters.

Memory updates should be versioned. “Current value” can be a derived view, while prior records remain available for audit.

Privacy and Tenant Isolation

A highly relevant memory is still invalid context if it belongs to another user or tenant.

Isolation must be enforced during storage and retrieval—not left to the model.

Every record should carry ownership and access metadata. Retrieval should apply tenant and user filters before semantic ranking. Encryption, deletion policies and access logs should cover raw artifacts, derived chunks, summaries and embeddings.

Embeddings can leak sensitive semantic information. They require the same isolation and retention discipline as the source data.

The model should receive the smallest amount of private information required for the current task.

A Minimal Production Design

flowchart TD
    U["User request"] --> R["Agent runtime"]
    R --> S["Execution-state store"]
    R --> C["Checkpoint store"]
    R --> M["Memory store"]
    R --> A["Artifact store"]
    S --> X["Context assembler"]
    C --> X
    M --> X
    A --> Q["Retrieval layer"]
    Q --> X
    X --> L["Model call"]
    X --> T["Context manifest and trace"]
    L --> R

The stores may share physical infrastructure, but their logical responsibilities should remain separate.

A compact typed model might look like this:

from dataclasses import dataclass, field
from datetime import datetime
from typing import Literal

@dataclass
class EvidenceRef:
    artifact_id: str
    chunk_id: str | None
    source: str
    observed_at: datetime
    confidence: float

@dataclass
class MemoryRecord:
    memory_id: str
    tenant_id: str
    subject: str
    value: str
    kind: Literal["semantic", "episodic", "preference"]
    scope: str
    evidence: list[EvidenceRef]
    valid_from: datetime
    expires_at: datetime | None
    supersedes: str | None = None

@dataclass
class AgentState:
    run_id: str
    tenant_id: str
    goal: str
    status: Literal["running", "waiting", "completed", "failed"]
    current_step: int
    plan: list[str]
    completed_actions: list[str]
    pending_action: str | None
    working_summary: str
    artifact_ids: list[str]
    remaining_tool_calls: int
    remaining_tokens: int
    checkpoint_version: int

@dataclass
class ContextItem:
    content: str
    category: str
    source_ref: str
    selection_reason: str
    token_count: int
    confidence: float | None = None

The context assembler can then follow a deterministic policy:

def assemble_context(state, user_message, token_budget):
    selected = []

    add_required(selected, state.goal, reason="Defines the active objective")
    add_required(selected, user_message, reason="Contains the current request")
    add_required(
        selected,
        relevant_execution_state(state),
        reason="Needed to choose the next valid action",
    )

    candidates = retrieve_conversation_context(state)
    candidates += retrieve_active_preferences(state.tenant_id)
    candidates += retrieve_evidence_for_current_step(state)

    candidates = remove_expired(candidates)
    candidates = enforce_tenant_access(candidates, state.tenant_id)
    candidates = detect_and_label_conflicts(candidates)
    candidates = rerank(candidates, current_step=state.current_step)

    for candidate in candidates:
        if candidate.token_count <= remaining_budget(selected, token_budget):
            candidate.selection_reason = explain_selection(candidate)
            selected.append(candidate)

    persist_context_manifest(state.run_id, selected)
    return selected

The important feature is not the particular ranking formula. It is that inclusion is explicit, budgeted, access-controlled and traceable.

Evaluating Context and Memory

A memory system should not be evaluated only by whether retrieval returns semantically similar text.

Evaluation should cover the complete lifecycle.

Selection quality

• Did the model receive the information required for the decision?
• How much selected context was irrelevant?
• Was critical evidence omitted?
• Did duplicate information waste tokens?

Memory quality

• Were durable facts stored correctly?
• Were temporary facts allowed to expire?
• Were preferences assigned the correct scope?
• Were uncertain inferences marked with appropriate confidence?
• Were conflicts detected rather than hidden?

Recovery quality

• Can a run resume from every checkpoint?
• Are completed side effects executed only once?
• Does the resumed agent preserve budgets and pending work?
• Can corrupted or incompatible checkpoint versions be detected?

Grounding quality

• Can every important claim be traced to evidence?
• Does the final answer cite the correct artifact and chunk?
• Can the system reconstruct the exact context used for a model call?

Security quality

• Can cross-tenant retrieval ever occur?
• Are deleted records removed from indexes and summaries?
• Does private information enter context unnecessarily?

Useful metrics include context precision, required-information recall, stale-memory rate, conflict-detection rate, provenance coverage, recovery success rate and tokens consumed per successful decision.

The Information Lifecycle

A production agent should be able to reconstruct how information moved through the system:

flowchart LR
    A["Observe"] --> B["Classify"]
    B --> C["Store"]
    C --> D["Retrieve"]
    D --> E["Select"]
    E --> F["Model context"]
    F --> G["Decision"]
    G --> H["Update or expire"]

For each piece of information, the system should answer:

• Where did it originate?
• Was it raw evidence, execution state or retained memory?
• Where was it stored?
• What scope and lifetime were assigned?
• Why was it retrieved?
• Why was it included or excluded?
• Did it influence a decision?
• Was it later corrected, superseded or deleted?

Without this lifecycle, “memory” becomes an opaque collection of text that grows indefinitely.

The Practical Rule

When new information appears, do not immediately ask, “How do we put this in memory?”

Ask four separate questions:

1. Does the runtime need this to continue correctly? Put it in application or execution state.

2. Must the exact original information be preserved? Put it in an artifact store.

3. Could this past information improve a future decision? Store it as scoped memory with provenance, confidence and expiration.

4. Does the model need it for this specific decision? Include it in the current context.

One item may have representations in several places. A PDF can remain in artifact storage, produce an indexed chunk, support a semantic memory and contribute one excerpt to the current context. Those representations are connected, but they are not interchangeable.

An effective agent does not remember everything. It preserves what may matter, retrieves what could matter and selects what matters now.

That is why context engineering is primarily an information-selection problem.