Context Assembly
The engineering problem at the heart of memory: retrieve, rank, and assemble slices of every store into the working context. The model only ever works at the desk — and when the task already fits on it, a memory harness buys nothing but cost. Past that threshold the recall policy is the system: ranked beats similar, a perfect retriever still misses the ceiling, and bad memory is expensive twice over.
Here is the key mental model of the whole pillar: memory is context assembly over time. Episodic memory can live in a session database, semantic memory in a fact store, procedural memory in skill files — none of it helps until some slice of it reaches the context window and shapes the model's next step.
The durable stores are the filing cabinet. Working memory is the desk. The model only ever works at the desk.
filing cabinet → context builder → the desk · click to inspect
The context builder
Before the model responds, the harness collects material from several places: the current request, the visible conversation, retrieved past sessions, project facts, loaded skills, tool results, open files. It retrieves the candidates, ranks them, and assembles the prompt.
Storing things is the easy part. The hard questions all live in the builder:
- What should we store in the first place?
- When should we retrieve it — and how much should we inject?
- What should outrank what?
- How do we handle contradictions?
- When should an old memory stop helping?
The end-to-end run
Watch every memory type fire on the request from earlier — "Pick up issue #9, the old-club-on-profile bug assigned to Buddy. Use the same approval-gated workflow as last time."
WORKING the current request itself, in the window
EPISODIC finds the prior issue-9 session: earlier run was blocked,
no approval comment existed at the time
SEMANTIC supplies repo layout, allowed folders, Buddy → buddy-b,
and the approval policy
PROCEDURAL loads the approval-gated implement workflow
And the response that good memory produces:
"I found issue #9 and the prior automation history. The earlier run was blocked because approval was missing; the later thread now includes a proposal. I'll read the onboarding doc, verify the approval comment, work inside the expected project roots, run focused tests, open a draft PR, and comment back with the result."
When you evaluate an agent product's "memory," get specific about which job it actually does. Session search, memory files, and skills line up almost one-to-one with episodic, semantic, and procedural memory. A vector database on its own is only a retrieval mechanism — the architecture around it decides whether the right slice reaches the desk.
Memory is a loop, not a store
Those five questions usually get answered by naming a product: a vector database, a memory file, a session log. That is the wrong shape of answer, and the second talk this sheet stands on says why. Memory is a write–manage–read control loop wrapped around the model — not a database attached to it. Storage is one third of one of those three verbs.
Drawn as a loop, the builder stops being a retrieval step and becomes the part of the harness with a policy. And a policy can be varied, measured, and got wrong.
That experiment splits the loop into three blocks, which map cleanly onto the layers this pillar has already walked through:
01Core
What it holds
Always in front of the agent, every single turn. The traces it must not be allowed to lose sight of, regardless of what else is competing for room.
Where it lands on this pillar
Working memory — the desk itself, and the part of it that is spoken for before the turn begins.
02Recall
What it holds
Pulled forward per turn, by a policy, from everything the run has produced so far. The only block the experiment varied.
Where it lands on this pillar
This sheet. Retrieve, rank, assemble — episodic and semantic material arriving on demand.
03Archival
What it holds
Kept across sessions, outliving the run that wrote it. Read rarely, but the reason anything survives a restart at all.
Where it lands on this pillar
Semantic and procedural memory — and whatever gets through the hygiene of the next sheet.
The agents underneath were chosen for having zero durable memory of their own, so that every remembered thing had to come from the harness. That is what makes the next result mean anything: hold the model fixed, hold the agent fixed, hold the task fixed, and change one block.
When the harness earns nothing
The first task was a literature review. The corpus contained a Nature paper announcing 742,000 promising new materials — a very large claim, later retracted. The retraction is a far smaller needle than the headlines and the citations that repeat the claim approvingly, which makes it exactly the sort of thing a memory system is supposed to save you from.
The harness changed nothing. Same accuracy with recall and without it, at higher cost — because every paper already fit inside the context window. There was nothing for recall to recover that attention could not reach on its own.
The second task came from xbench, an established benchmark for long-horizon memory, and it is built to make that distance unavoidable. The answer is written at step 124. The question is asked at step 500. Nothing about a bigger model, a better prompt, or more careful attention closes 376 steps — the material is simply not there to attend to. Past that threshold the recall policy is not a component of the system. It is the system.
The recall ladder
So the recall block was set four ways, from remembering nothing to being told the answer.
Four recall policies
Same model, same agent, same task — only this block changed
04The oracle
The harness is handed the correct memory for every loop. Retrieval solved by fiat — the upper bound, not a system anyone could ship.
The bound that isn’t
Did not reach the ceiling, and did not beat the ledger below it. The right memory in the window is not the right memory used: the model can still read past it, misread it, or be confused by it.
03Ranked decisions ledger⚑ Winner
Every turn writes down what was decided. Recall pulls the highest-priority decisions — not the nearest neighbours, and not the most recent.
Best measured
Best of everything tested, and cheaper than the alternatives. Held under ablation, on a second model, and on a second benchmark.
02Vector recall
Embed the trace, retrieve by similarity, inject whatever comes back. The default that ships in most memory features.
Between the two
Better than remembering nothing. Similarity is not priority — it has no way to know which turn was the one that mattered.
01No recall
The window and nothing else. The baseline every memory claim has to clear before it has earned its cost.
Baseline
Fails any question whose answer left the window. Cheapest to run, and useless on the far side of the gap.
Off-ladderOne further condition gated the harness — ask the model whether it needs memory this turn, and skip recall when it says no. It lost to simply ranking every turn. Asking a model what it has forgotten is itself a memory question.
Across 68 questions, with multiple seeds each, the ranked decisions ledger won. Not the embedding search, and not the ground truth. Writing down what was decided on every turn, and then recalling by priority rather than by resemblance, beat every other setting — and it held up under ablation, on a second model, and on a second benchmark.
The ledger's advantage is that it stores the right kind of thing. Similarity search retrieves text that looks like the question; a decisions ledger retrieves the turns that changed the state of the work. Those are rarely the same turns, and only one of them is what step 124 actually was.
Why the oracle misses the ceiling
The oracle rung is the one worth sitting with. It hands the harness the correct memory on every loop — retrieval solved, perfectly, by cheating. It still did not reach full marks, and it did not beat the ledger.
The reason is plain once stated: the oracle supplies the right memory, it does not force the model to use it. The right slice can land on the desk and be read past, misread, outweighed by something louder, or simply ignored. Retrieval accuracy is a ceiling on performance, not a substitute for it.
That should recalibrate how much of the memory problem you think retrieval represents. Even a perfect retriever leaves a gap — and everything in that gap is context management's problem, not memory's: where in the window the recalled material sits, what it is competing with, and whether the agent is told why it is there.
Bad memory is expensive
The last finding is the one that changes budgets. Ranked recall was not only more accurate, it was cheaper. That is not the trade-off anyone expects, and it follows directly from what bad recall does: it spends tokens on material that does not help, and then sends the agent in a direction it has to come back from. You pay twice, once for the tokens and once for the detour.
Which gives the heuristic this sheet ends on: treat the recall policy as a first-class metric. Not the store, not the embedding model, not the size of the window — the policy. What kinds of memory do you write? How do you rank them? What does the recall function actually return on the turn that matters?
Remembering everything is its own failure
The builder solves getting memory in. That leaves the opposite problem: a system that remembers everything eventually remembers too much — and the useful stuff drowns. Ranking is only half an answer, because it decides what wins this turn without ever deciding what should stop competing.