Existential Crisis: My agent's memory had no embeddings, and I had no idea
Existential Crisis: My agent’s memory had no embeddings, and I had no idea.
I’ve been running agentic memory for my AI coding agents for probably the last 1.5 years. I started with Zap as my first exposure to the concept. I immediately found value but noticed it was missing retrieval and duplicating the same lessons repeatedly. Then I switched to Mem0. Used Mem0 for another 6 months. Then I tried engram (https://github.com/OscillateLabsLLC/engram), written a colleague of mine. Claude quickly saw the change in memory retrieval quality after running some tests. Then, I started moving my mem0 memories into engram.
So, with Engram, the major design behind it is that it had a hybrid approach that allowed using embeddings for vector search. I tested a few settings, then settled on the results and continued on my happy way for 6 months.
While discussing agentic memory at work, I revisited the dynamics. I initially thought I was using a graph setup, with a vector as backup.
Turned out, I really didn’t understand exactly what I was running. Engram is written in Go, and uses DuckDB. It performs a hybrid search by combining BM25 keyword scoring with cosine similarity. Graph part wasn’t being used at all. There were no trees or linking. What was more interesting once I understand the settings is that I was already weighing the keyword search heavier than the vector search.
Given all the hype around using graph databases and vector databases as well, why did I have such a hight result in engram. To make matters worse, there was a silent bug in Engram resulting in the embeddings (not all but many) simply not getting populated.
So I did a spike and started comparing. I wrote a test query: “nightly data extraction job hangs and times out with no error message.” The symptom description versus the causal description. We also write queries or Claude based on unknown symptoms instead of known causes. For example, “I’m having a heart attack” is different from “I have pain in my left arm, difficulty breathing, and chest pains.” Same event, but completely different words.
Results:
- Hybrid search missed it.
- Pure vector search missed it
- Embeddings didn’t bridge the vocabulary gaps
To fix it, I added a line to the stored memory. It was a simple “symptom as experience” sentence. To be fair, I had Claude do it. Results - query went from zero hero.
Now that the enrichement worked, I needed to know which half of the hybrid searce did the heavy lifting. To test this, I re-ran the same query in keyword-only mode and vector-only mode (Engram can do that).
- Keywords: ranks 1 and 2
- Vector: still nowhere in the top ten, before enrichment and after it. I was completely shocked.
The fix was 100 percent BM25. The embedding missed the target before and after.
The process is simple. A whole-document embedding takes two sentences and averages them into a two-thousand-word vector. This averaging process itself is what I suspect causes the cue to vanish. BM25 does not average so a near-verbatim phrase match scores high no matter how long the document is.
The Big Reveal:
If you don’t read any other sentence in this blog, remember this - write-time enrichment beats query-time cleverness.
The agent storing the memory has all the context that a given session produced. All you ask of the agent is a simple: how would someone ask for this when they do not know the answer’s vocabulary? It’s cheap, and far more deterministic than you might think.
Does a knowledge graph solve this?
I spiked another fashionable answer - the knowledge graph. This was the pitch of Zap using NeoJ.
I took a copy of the live DuckDB store. The link expansion added one useful neightbor and six pieces of noise across ten test queries. Here is where I have some egg on my face, because i found the published findings very convincing so I mostly took it for granted.
My faith in those findings were shaken and I really do wonder if the published wins for graph memory are real but conditional and concentrated in multi-hop temporal reason, which admittedly, my workload barely contains.
Two days after my spike, Mike Gray, the author of the original project, retired the knowledge graph from his system.
So whats next:
Given what I discovered, I wanted to see if this could work in practice and what the results would be. So I built ‘Ecphory’. It comes from Semon’s 1904 term for memory retrieval.
What is Ecphory?
Ecphory is my expermental version of AI memory built in Rust for speed, uses redb for storage and tantivy for BM25. No model dependencies for this project.
The paraphrase-cue field is the schema. The design centers on the flight recorder. Every search and fetch is logged providing retrieval quality measured against the real query stream.
But what if we took this one step further. Have the agent that is retrieving the data, test the results and then improve the search with an enrichment cycle. Now you have a search-rate-heal cycle that continuously improves your agents memory over time.
