The model never remembers. Something around it does.
Spend a week with a coding agent and it feels like it knows you. But when the context window closes, the weights don’t change. A memory architecture stored that information, pulled it back at the right moment, and fed it into the next prompt.
filing cabinet → context builder → the desk · click to inspect
The chain of limitations — 6 sheets
- 01Working MemoryThe active context the model can see right now — the desk. Every token in the prompt is active state, and it runs on a budget.
Working memory breaks two ways: it ends when the session ends, and it fills up. Nothing survives the boundary on its own.
- 02Episodic MemoryWhat the agent remembers happening — a specific investigation, a specific debugging session. The key property: it has a when.
The past session says what happened last time. It cannot say what is true right now — or whether last time still holds.
- 03Semantic MemoryStanding facts that float free of time: repo layout, project rules, user preferences, name-to-ID mappings. The agent just knows.
Facts go stale and contradict. Without conflict handling, an old truth sits next to a new one as an equal — and memory becomes friction.
- 04Procedural MemoryThe how-to layer: runbooks, checklists, skills, tool schemas, orchestration code. Knowing how to ride the bike, not just that it has two wheels.
Durable stores are just a filing cabinet. None of it matters until the right slice reaches the context window at the right moment.
- 05Context AssemblyThe engineering problem at the heart of memory: retrieve, rank, and assemble slices of every store into the working context. The model only ever works at the desk.
A system that remembers everything eventually remembers too much. Perfect recall with no forgetting is the coworker who reads out their entire diary.
- 06Forgetting & ConflictsMemory hygiene: temporal decay, contradiction handling, compression, manual curation. Old memories can stay — they just shouldn't outrank what's true.
If you can answer the five questions — what's in working memory, which session matters, which facts are current, which workflow applies, what should be forgotten — you have an architecture. If not, you have scattered state.
Four kinds of remembering
Borrowed from cognitive science, and mapped one-to-one onto agent systems. Hover to see what each one holds.
Working memory
the deskWhat is active right now?
The current request, the visible conversation, tool results on this turn.
Episodic memory
the diaryWhat happened — and when?
“Last Friday's check found no approval comment on issue #9.”
Semantic memory
the fact sheetWhat is simply true?
“The app root is meteor/league. Buddy maps to GitHub ID buddy-b.”
Procedural memory
the runbookHow should the work be done?
“Fetch issue → confirm approval comment → work only in these two folders → draft PR.”