Context Assembly
The engineering problem at the heart of memory: retrieve, rank, and assemble slices of every store into the working context. The model only ever works at the desk.
Here is the key mental model of the whole pillar: memory is context assembly over time. Episodic memory can live in a session database, semantic memory in a fact store, procedural memory in skill files — none of it helps until some slice of it reaches the context window and shapes the model's next step.
The durable stores are the filing cabinet. Working memory is the desk. The model only ever works at the desk.
filing cabinet → context builder → the desk · click to inspect
The context builder
Before the model responds, the harness collects material from several places: the current request, the visible conversation, retrieved past sessions, project facts, loaded skills, tool results, open files. It retrieves the candidates, ranks them, and assembles the prompt.
Storing things is the easy part. The hard questions all live in the builder:
- What should we store in the first place?
- When should we retrieve it — and how much should we inject?
- What should outrank what?
- How do we handle contradictions?
- When should an old memory stop helping?
The end-to-end run
Watch every memory type fire on the request from earlier — "Pick up issue #9, the old-club-on-profile bug assigned to Buddy. Use the same approval-gated workflow as last time."
WORKING the current request itself, in the window
EPISODIC finds the prior issue-9 session: earlier run was blocked,
no approval comment existed at the time
SEMANTIC supplies repo layout, allowed folders, Buddy → buddy-b,
and the approval policy
PROCEDURAL loads the approval-gated implement workflow
And the response that good memory produces:
"I found issue #9 and the prior automation history. The earlier run was blocked because approval was missing; the later thread now includes a proposal. I'll read the onboarding doc, verify the approval comment, work inside the expected project roots, run focused tests, open a draft PR, and comment back with the result."
When you evaluate an agent product's "memory," get specific about which job it actually does. Session search, memory files, and skills line up almost one-to-one with episodic, semantic, and procedural memory. A vector database on its own is only a retrieval mechanism — the architecture around it decides whether the right slice reaches the desk.
Remembering everything is its own failure
The builder solves getting memory in. That leaves the opposite problem: a system that remembers everything eventually remembers too much — and the useful stuff drowns.