A word list beats embeddings at knowing a fact was reversed
A memory store holds a fact. Later the fact changes and the new version is written. Ask which one holds and a retriever will hand back both, because both match — and nothing malfunctioned to make that happen.
Similarity has no opinion about which of two contradictory statements is current, because currency is not a property of the text. It is a property of the history, and the history is what an append-only store discards. So the question is not how to retrieve better; it is where in the lifecycle something decides that one of the two is dead.
TL;DR
- Conflict resolution belongs at write time, because that is the only moment both records are in hand and comparable.
- The obvious implementation is a similarity threshold. We built the harness and it lost: the reversal that must be caught scores 0.8593, a hand-labelled negative that must not scores 0.8575. Eighteen ten-thousandths apart.
- Five derived statistics — margin, ratio, margin-over-sd, z-score, exponential tail — all discriminated worse than the raw cosine.
- What we shipped instead is lexical and deterministic: a title token-subset test, a polarity guard, and a list of phrases that assert reversal.
- On MemoryAgentBench FactConsolidation single-hop @262K that scores 0.90. On multi-hop it scores 0.07, against an all-time ceiling of 0.14 — nobody has solved chained supersession, including us.
Where everyone else puts the decision
In the two systems I read closely, the detection half is genuinely built. What neither does by default is act on it.
Graphiti, the graph engine underneath Zep, genuinely computes temporal validity — it sets
valid_at and invalid_at on fact edges during ingest, and has since its January 2025 paper. Then
look at the read path. In graphiti_core/search/search_filters.py, both invalid_at and
expired_at are declared Field(default=None), and the SQL predicate for each is emitted only
inside an if filters.invalid_at is not None branch. Across the two thousand lines of
search_utils.py, expired_at appears exclusively as a projected output column —
r.expired_at AS expired_at — and never once in a WHERE clause.
Their MCP server, the path an agent actually takes, constructs
SearchFilters(node_labels=entity_types). No temporal argument. The tool description the model
reads says a fact "can be superseded by newer information while its history is preserved" — which
tells the model history is kept, and never tells it that superseded facts are in the result set by
default.
mem0 states the same design in its own documentation: superseded memories still appear in search by
default, and you opt out with latest_only=true. Letta's is a paragraph of instructions in a
reflection prompt, honoured if the model complies.
So the expiry is computed, stored, returned as a field, and left for the model to notice. Read at getzep/graphiti on 2026-08-25; the file and field names above are so you can check rather than take our word for it.
Why write time
Conflict resolution can live in three places.
At read time you rank and hope. This cannot work, and the reason is structural: a superseded fact and its replacement are maximally similar. Same subject, same words. Any ranker good enough to retrieve one retrieves the other.
In the model you instruct it to notice contradictions. This degrades silently under exactly the conditions where it matters — long context, many candidates, a contradiction split across two retrieved chunks.
At write time you have what the other two never have: both records present at the moment one is arriving. That is the only point in the lifecycle where the comparison is cheap and total.
So the check runs inside knowl_store, in code, on every write, whether or not the agent asked for
it. An agent may pass an explicit supersedes: <id>, and that path exists. It is not the path
that does the work.
What we shipped
The primary test is sameSubjectTitle, a token-subset comparison rather than a similarity score.
Titles tokenise on [^a-z0-9_]+, stop words drop, and if one title's token set is a subset of the
other's, the newer atom retires the older.
A subset rather than a threshold, because thresholds have to be tuned per store and subsets do not. "Rate limit on the public API" and "Rate limit on the public API is 500/min" stand in a subset relation on a two-item store and a ten-thousand-item store alike.
The negation bug
None of the words that flip a claim are stop words. So an affirmative title is a strict subset of its own negation, and the subset test fires in the wrong direction. Measured by running the real function against our own store:
"Push gate blocks default branch" SUPERSEDES "Push gate no longer blocks default branch"
"Reranker is the right call" SUPERSEDES "Reranker is not the right call"
The survivor then asserts the opposite of what it retired, silently. The first pair is this project's own push-gate reversal: a title shape which, written twice across the reversal, retires its own predecessor in whichever order the two writes arrive.
The fix is a polarity guard — not, no, never, none, nor, without, cannot, longer,
unable, plus the bare contraction stems isn, aren, doesn, didn, hasn, couldn,
wouldn. If two titles differ by a polarity token, the subset relation is not a supersession.
It is deliberately narrow. can is excluded because "Can the gate block the branch" is a real
title. The asymmetry is intentional: a missed negation costs what today already costs, whereas a
false positive costs a supersede that should have happened, which the caller then has to make
deliberately.
We also rejected the numeric version of this guard, which is what the published prior art
(muninndb's dedup_separation.go) is built around — "$99" merging with "$149". That hole does not
exist here. Because the tokeniser splits on [^a-z0-9_]+, digits survive as tokens, 768 is simply
absent from a title containing 1024, the subset test already fails, and the two atoms already
coexist. Only polarity was unguarded, because polarity words are the only ones short, common, and
absent from the stop list.
The reversal no title mentions
A harder case, reproduced through the MCP path a real agent uses:
"Database choice: Postgres for everything"
"We are moving persistence to SQLite"
content: "The Postgres-for-everything plan is abandoned."
The titles share no subset relation. Whole-text token overlap is 0.33 — just under our 0.35
duplicate gate. The write lands in silence: both decisions active, both returned fresh by the next
query, nothing for knowl conflicts to surface.
What distinguishes this pair is not that the texts are close. It is that one of them says the
other is over. That is a lexical fact, so we detect it lexically: no longer, abandoned,
superseded, deprecated, reversed, obsolete, replaced by, overturned, rescinded,
retracted.
The list is short for a measured reason. "instead of", "dropped", "rejected" and "retire" were all candidates and all were cut after running against a real 831-item store, where they fire constantly in ordinary engineering prose. The survivors fired on 148 items there, and on 129 of 1,033 in this repository's own store. That rate is stated rather than tuned away — it is a cue for closer inspection, not an automatic retirement.
Why not embeddings
The obvious implementation is a similarity model: embed both atoms, threshold the cosine, call anything above the line a contradiction. We built the measurement harness for exactly this.
| pair | must | cosine |
|---|---|---|
| the reversal above | catch | 0.8593 |
| hand-labelled negative | not catch | 0.8575 |
Eighteen ten-thousandths apart. No threshold separates them.
We then tried five derived statistics on the same labelled set, on the theory that raw cosine was the wrong readout: margin, ratio, margin-over-standard-deviation, z-score, and an exponential tail. All five discriminated worse than the raw cosine. No gate over any of them reaches this case at a fire rate anyone would ship.
The reason is not that the embedding model is bad. Similarity is answering a different question. "These two texts are about the same thing" and "this text declares that thing over" are orthogonal properties, and a reversal is maximally similar to what it reverses — that is what makes it a reversal rather than a new topic. Asking a similarity score to separate them is asking for information it does not encode.
There is a second, more practical reason. Similarity gates need a distribution to calibrate against. On a fresh two-item store there is no distribution, so a statistical guard abstains by construction — precisely when a new user is deciding whether the tool works at all. A word list behaves identically on the second write and the ten-thousandth.
We reached the same conclusion from the opposite direction when we measured a competitor's similarity threshold: a Jaccard threshold of 0.7 lets the clearest contradictions through, because the more completely a fact is contradicted the lower it scores. Different statistic, same failure — string distance is not contradiction.
Retired, never deleted
Supersession marks the old atom superseded and records superseded_by_id pointing at its
replacement. It does not remove the row. knowl_query defaults to status: active, so the retired
version leaves the default result set but stays queryable, and the lineage is walkable both ways.
"What did we think before, and what changed it" is the question a wiki can never answer, because a
wiki is edited in place. It is also the audit surface: every retirement carries a timestamp and a
provenance — observed, user_stated or inferred — so six months on, "who settled this" still
has an answer.
There is no delete verb in the product. The store is bounded by a garbage collection pass that moves
cold atoms to archived after a staleness threshold; archived atoms stay queryable, and
knowl gc --preview shows exactly what would move before anything does.
What it measures
MemoryAgentBench has a task built for precisely this: FactConsolidation stores a fact, changes it, and asks which one holds. Single-hop, at 262K context:
The dotted line is the best published figure in the table — GPT-4o long-context at 60%. The two bars above it, Knowl and agentmemory, are my own runs on the benchmark's own harness; everything else is from the MemoryAgentBench paper.
| system | SubEM |
|---|---|
| Knowl | 0.90 |
| agentmemory | 0.79 |
| GPT-4o, full context | 0.60 |
| HippoRAG-v2 | 0.54 |
| BM25 | 0.48 |
| GPT-4o-mini, full context | 0.45 |
| Cognee | 0.28 |
| MemGPT | 0.28 |
| Mem0 | 0.18 |
| Zep | 0.07 |
Knowl and agentmemory were run by us on the benchmark's own harness at temperature 0.7. The remainder are the published MemoryAgentBench figures — a different runner and a different day, which is a caveat we would rather state than bury.
The row worth dwelling on is not ours. Zep markets temporal invalidation harder than anyone in the category and scores 0.07 on the task that measures it — below BM25, below plain long-context prompting. That is not an extraction-quality gap; their extraction works, we read it. It is the read-time filter that never gets applied.
And the part that does not flatter us: on multi-hop FactConsolidation we score 0.07, against an all-time ceiling across all systems of 0.14. Chaining two supersessions is unsolved, by us and by everyone. Raw results, harness configuration and the splits where we do worse are in the repository.
Run it
Knowl is Apache-2.0 and runs entirely on your machine — SQLite on disk in your repository, no API keys. It speaks MCP, so it attaches to Claude Code, Codex, Cursor, Copilot, Zed, Windsurf, Cline and anything else that speaks the protocol, one command each.
npm install -g @dat999zx/knowl
knowl init
Every rule, threshold and rejected alternative above is in github.com/dat999zx/knowl — the argument is only as good as the code under it.
The decision we are least sure of is the first one: retirement happens automatically at write time, with no confirmation step. We chose it because a prompt nobody answers is just a slower way to go stale. The argument against is real, and if you have hit the case where it goes wrong, the issue tracker is the right place for it.