Project Engram
en·gram /ˈenɡram/ · noun
a unit of cognitive information imprinted in a physical substance, theorized to be the means by which memories are stored.
I spent a few weeks building a memory system for long-running AI agents, and most of what I learned is that I was asking the wrong question. I want to walk through how that happened, because the way it went wrong is more useful than the thing I set out to build.
The setup is simple to articulate. When an agent talks to you across hundreds of turns, it can’t keep the whole conversation in its context window — it costs too much and eventually it doesn’t fit. So something has to decide which past turns to carry forward and which to drop. The standard answers are summarize the old stuff or retrieve the relevant stuff. I wanted to try something fancier.
The Idea#
There’s a decades-old model of human memory from cognitive science called ACT-R. Its core claim is elegant: whether you recall something is governed by a single number — an activation — that goes up when a memory is recent, goes up when it’s frequently used, and goes up when something in your current situation points at it. Forgetting isn’t a bug in this model; it’s the system correctly betting that old, unused things probably won’t be needed.
That maps almost suspiciously well onto the agent-memory problem. So I built it. Each past turn gets an activation score: a decay term for how old it is, a similarity term for how well it matches what’s being asked right now, and — my one non-standard addition — an importance term, because some facts matter out of all proportion to how often they come up. You mention a shellfish allergy exactly once. It should never fall out of memory.
The bet was that this three-part score would keep the rare-but-critical facts alive, surface the relevant ones, and let everything else fade — beating both summarization and plain retrieval.
A Pressure-Testing Harness#
Here’s the part I’m actually proud of, and it has nothing to do with the memory system.
Before testing my idea, I built the experiment to disprove it. The standard benchmark for long-context recall is “needle in a haystack” — hide one fact in a wall of filler, see if the model finds it. It’s too easy, and it’s too easy in a specific way: a single vivid fact in bland filler is findable by accident. Real conversations are harder because they correct themselves. The deadline moves from Friday to Monday. A priority changes. A good memory system has to know not just what was said, but what’s still true.
So I wrote scenarios with interference, with facts that get superseded, with facts that should be forgotten on purpose. And I wrote controls whose entire job was to embarrass me. My favorite: a baseline that ranks turns purely by how “important-sounding” they are, ignoring the actual question. If that dumb baseline scored well, it meant my needles were secretly shiny — self-announcing with words like “critical” and “remember this” — and the whole eval was measuring nothing. It scored well. So I rewrote every planted fact to state itself plainly, and the dumb baseline dropped back to where it belonged.
That guard mattered more than I knew at the time, because the discipline behind it — build the thing that could kill your finding, then run it — is what eventually caught me making a much more embarrassing mistake.
Death Upon First Contact#
The fancy memory system lost. Badly. Plain retrieval — just grab the turns most similar to the question — beat it at every cost budget, hitting perfect recall at a fraction of the token cost. My ACT-R system landed near the bottom, keeping company with the summarizer.
The cause was almost funny once I found it. ACT-R adds its terms together because, in the original theory, they’re all in the same units — they’re all log-odds of needing a memory. My decay term was unbounded and got more negative the older a turn was, while my similarity term was a cosine score capped between 0 and 1. Adding them is a units error, like summing a temperature in Fahrenheit with a distance in miles. And it got worse the longer the conversation ran: the older a critical fact got, the more its age penalty buried its (correct, high) relevance score. The one fact I most wanted to protect — the stated-once allergy — was exactly the one the math worked hardest to forget.
So the headline idea was wrong. But sitting in the wreckage was something more interesting.
Recall?#
Plain retrieval got perfect recall. It found the answer every time. So why did it still feel inadequate?
Because of what else it dragged along. When I asked for the current deadline, retrieval cheerfully pulled in both “the deadline is Friday” and “actually, moved to Monday” — about 73% of the time it brought back the dead value right alongside the live one. To a similarity score, “deadline is Friday” and “moved to Monday” look almost identical. They’re near-paraphrases. Relevance simply cannot see which one is current.
That’s the thing I’d been missing. Recall — can you find the fact — is easy on this kind of task. The hard property is what I started calling liveness: is this fact still true? And liveness is exactly what decay and supersession encode, the stuff relevance is blind to. My broken system was useless for recall, but maybe it was the right instrument for precision — for keeping dead facts out.
That was the surviving hypothesis. A retriever that finds 85% of facts cleanly beats one that finds 90% while stuffing the window with useless information. Good story. So I went to test it, and it died too.
Does Stale Context Matter?#
Before building a whole precision-optimizing apparatus, I ran the cheapest possible test of its premise — that stale-but-present context actually degrades the answer. I took real retrieved windows, put both the dead value and the live value in front of a current model, and asked the question.
It answered correctly. Every single time. Zero out of fifteen, on two different model tiers, did the stale value drag the answer off course. Put the correction in the window and the model routes to it on its own. The 73% “stale rate” I’d been so worried about turned out to be cosmetic — a number the model simply ignores. The precision pivot died the same cheap way the recall idea had, on a control I ran before building the cathedral.
But — same as before — the autopsy was the interesting part. Stale facts present in the window are harmless. The damage is when the live fact is absent.
The Failure#
When I evicted the correction and left only the stale value, the model asserted it confidently, every time. And here’s the thing — that’s not the model being overconfident or making something up. Given a window that contains “deadline is Friday” and nothing contradicting it, “Friday” is the correct answer to the evidence it was handed. The wrongness only exists from my omniscient view outside the context window. From inside, the model has no way to know the window is incomplete.
I started calling this context blindness: the model can’t see the edge of its own context, so it treats a partial window as the whole truth and answers from it without hesitation. It doesn’t know what it doesn’t know — and nothing inside the window can tell it. It’s the genuinely dangerous failure mode, because it passes every spot-check. Picture an agent asked to draft a catering menu for a client whose shellfish allergy was mentioned once, thirty turns ago, and has since aged out of memory. The client’s stated love of seafood is still in the window. The model drafts a beautiful shrimp menu. Nobody fabricated anything, and the result could put someone in the hospital.
So the last question became the practical one: can you fix this from inside the window? Can you inject a generic “heads up, this retrieval might be incomplete” marker and get the model to check itself?
The Mistake#
My first runs said yes. The marker worked! I had a finding.
Then I made myself do the thing the whole project was built on. I re-ran it as a rate instead of a single example — forty samples per condition, with confidence intervals, on both models. The finding evaporated.
It turned out the model already self-checks the forgotten dimension some fraction of the time, entirely on its own — around 60% of the time on the stronger model, around 20% on the weaker one. That base rate is noisy. A single sample lands somewhere in that wide range, and if you run your “with marker” version once and your “without” version once, you can get any answer you want. My clean, legible, headline-shaped result had been pure noise.
This is the part I most want to pass along, because it’s not specific to memory systems. A single generation is a single scalar, and it hides the variance. The same convenience that makes a one-shot result so clean and quotable is exactly what makes it unfalsifiable. The dominance gaps earlier in the project were huge — perfect recall versus 47% — so they cleared the noise floor even from single samples. But the marker effect was small, and I chased it on single samples anyway.
What’s Left?#
So both versions of my idea are dead. Here’s what I picked out of the rubble:
Relevance solves recall on this kind of task. It cannot see liveness. Stale context that’s present alongside the truth doesn’t drag the output. Suppressing it optimizes a number the model ignores.
The real catastrophe is context blindness — and it fires at a stochastic, model-dependent rate. The strong model ships a forgotten safety-critical fact silently around 40% of the time; the weak one around 75%. That ~3× gap is, honestly, the most robust new thing I found. No generic in-window “this might be incomplete” signal reliably fixes it. That’s measured-dead, not assumed-dead.
Which points the engineering somewhere specific. You can’t reliably flag the absence of a fact from inside a window that doesn’t contain it — there’s nothing to point at. So the leverage is upstream: a retention policy that simply never evicts the never-reinforced critical fact in the first place. The cost of dropping a fact should be a function of its consequence, not its recency or frequency.
And that, strangely, is where my original instinct ends up vindicated. The “importance” term I bolted onto the ranker was in the wrong place. It doesn’t belong in the scoring function that picks what information to return. It belongs in the eviction policy that decides what you’re allowed to forget at all.
I set out to build a smarter way to remember. I ended up with a sharper definition of what’s worth never forgetting.
The full write-up, with the methodology, frontier curves, and complete results, is in the Project Engram whitepaper.