Topic 11 · Agents
Context, memory & persistence
What the model can see right now, and what the system remembers for later.
A model has no memory between calls. Everything it knows about the task has to be in the context window on every request: instructions, tool definitions, the conversation so far, tool results and anything retrieved from storage. The window is large but finite, and filling it costs money and latency.
Context management is the art of choosing what goes in. Persistence is the other half: saving the agent's state durably, so a long task can survive a crash, be replayed for debugging or pick up again next week.
In this topic
25
terms, each defined below
Filling a context window
A 128K-token window split between instructions, tools, retrieved knowledge, conversation history and room for the reply. Add turns and the history grows until it no longer fits. Choose a strategy to see what happens: truncation drops the oldest turns, compaction folds them into a short summary.
Turns
Everything fits. Each request still resends all of it, so prefix caching matters.
Glossary
25 terms
Context window
The maximum number of tokens a model can attend to at once, prompt and output together, such as 128K or 1M tokens.
Context length
The number of tokens actually in a given request. Cost and latency grow with it, and quality can slip well before the window is full.
Context management
Deciding what goes into the limited context on each model call: which instructions, history, tool results and retrieved knowledge.
Context assembly
Building the final prompt from its parts in a stable order: instructions, tool definitions, memory, retrieved documents and recent messages. A stable order also keeps prefix caching effective.
Short-term memory
Information kept within the current context or session, mainly the recent conversation and working notes.
Long-term memory
Information kept across sessions in external storage, such as files, databases or vector stores, and brought back when relevant.
Persistent memory
Memory that survives restarts and sessions, such as user preferences or project facts written to durable storage.
Conversation history
The sequence of past user, assistant and tool messages in a session.
Message history
The raw list of messages sent to the model on each turn. Because the model is stateless, the whole history is sent again on every call, ideally with caching.
Memory retrieval
Selecting the stored memories relevant to the current step, by recency, keywords or embedding similarity.
Retrieval systems
Search infrastructure, such as BM25, vector search, hybrid search and rerankers, that finds documents to place into context, as in retrieval-augmented generation (RAG).
Context compaction
Shrinking the context while keeping what matters: summarising old turns, dropping stale tool output, holding on to key decisions.
Context summarization
Replacing a span of history with a model-written summary. It saves tokens but can lose details.
Context truncation
Dropping the oldest or least important content when the context exceeds its budget. Simple, but information disappears abruptly.
Token budgets
Allocations of the window to each part (instructions, tools, history, retrieval, output), enforced when the context is assembled.
Persistent agent state
Agent state saved to durable storage so a run can pause, survive crashes and resume.
Execution history
A record of every step an agent took: model calls, tool calls, results and decisions.
Checkpointing
Saving the agent's state at defined points so it can resume from there instead of starting over.
State snapshots
Point-in-time copies of the full state, useful for debugging, branching and rolling back.
State recovery
Restoring an agent from its last checkpoint after a failure.
Execution replay
Re-running a recorded execution, reusing the recorded model and tool results, to debug it or to recover.
Deterministic replay
Replay that follows the same path every time because nondeterministic results, such as model outputs, clock reads and tool responses, are read from the log instead of recomputed. Workflow engines like Temporal depend on it.
Nondeterministic execution
Steps that can give a different result each time: sampled model output, network calls, the current time. They must be recorded to make replay safe.
Durable state
State stored where it survives process and machine failures, such as a database or an event log.
Persistent workflows
Long-running workflows whose progress is recorded durably, so they can run for hours or days and survive restarts.