In progressO'Reilly Media

Agent
Memory

Building Stateful AI Agents That Remember, Adapt, and Work Across Time

Agent Memory — O'Reilly book by Ben Labaschin

Who This Book Is For

Engineers building agents

You’re past the prompt-and-pray stage. You need memory that works across sessions, handles corrections, and doesn’t rot as it grows.

Teams shipping AI products

Your agent works in demos but breaks in production. This book covers the operational patterns that make memory reliable at scale.

Anyone designing stateful AI

If you’re deciding how an AI system should remember, forget, and share information, this is the architectural reference.

Questions This Book Answers

How do I stop my agent from forgetting everything between sessions?

The book separates context from memory and shows how to build retention that survives session boundaries without polluting future interactions.

Chapters 1 – 2

What should my agent actually remember?

Not everything. The book defines practical rules for what to keep, what to compress, and what to deliberately forget.

Chapter 3

How do I write memory without filling it with noise?

Write triggers, thresholds, restraint. The book covers when to write, when to wait, and how to avoid memory pollution.

Chapter 4

My retrieval keeps pulling the wrong memories. Now what?

The book covers query formation, ranking, context loading, semantic caching, and how to evaluate whether retrieval is helping or hurting.

Chapter 6

How do I keep memory useful as it grows?

Rollups, summaries, corrections, versioning, duplicate detection, and maintenance strategies that prevent memory rot.

Chapter 7

What happens when memory goes wrong?

Poisoning, leakage, staleness, over-trust. The book closes with detection, repair, rollback, and designing memory to fail safely.

Chapter 10

What You’ll Learn to Build

Design a memory API

Give your agent the verbs it needs: write, retrieve, update, forget, consolidate

Decide what to keep

Build practical retention rules instead of storing everything and hoping

Retrieve without hallucinating

Form better queries, rank candidates, and measure whether recall is helping

Resume interrupted work

Checkpoints, durable state, and handoff patterns for long-running agents

Maintain memory at scale

Prevent rot, resolve duplicates, and keep memory useful as it grows

Share memory safely

Ownership, boundaries, and coordination across agents and teams

Design for failure

Poisoning, leakage, rollback, and guardrails that limit blast radius

Evaluate memory quality

Baselines, regressions, and knowing when memory is making things worse

What the Book Covers

10 chapters across 3 parts.

Part IAgents, Memory, and the Act of Remembering

01

The Work of Remembering

What memory means for an agent, how it differs from context, and why continuity changes everything.

02

What the Agent Can Read and Write

Where memory lives, the five working verbs, and giving the agent a memory API.

Part IIBuilding and Managing Agent Memory

03

Choosing What Becomes Memory

Rules for retention, compression, forgetting, and knowing when memory helps.

04

How Memory Gets Written

Write triggers, normalization, correction, and failure modes in the write path.

05

Where Memory Lives and Who Controls It

Client vs server, portability, inspectability, and the control model.

06

Finding the Right Memory

Retrieval, ranking, context loading, caching, and evaluating quality.

07

Keeping Memory Useful Over Time

Rollups, corrections, versioning, duplicates, and maintenance strategies.

08

More Than Memory: State and Resumability

Checkpoints, durable work, resume, replay, and human pause points.

Part IIIShared Memory, Risk, and Recovery

09

Shared Memory: Coordination, Boundaries, and Conflict

Ownership, leakage, multi-agent coordination, and provenance.

10

Dangerous Memory: Risk, Failure, and Recovery

Poisoning, detection, repair, rollback, and designing memory to fail safely.