A similarity threshold of 0.7 lets the clearest contradictions through
If a memory system decides whether a new fact replaces an old one by measuring how alike the two strings are, it has a gradient problem: small corrections score high and get caught, total reversals score low and get through.
TL;DR
- agentmemory supersedes a stored fact when Jaccard token similarity over the whole content
exceeds
0.7(src/functions/remember.ts:135). - Jaccard over whole content is length-sensitive, and the sensitivity runs backwards for this job: the larger the swapped value, the lower the similarity.
- A realistic contradicting pair scores exactly 0.7000, and the threshold is a strict
>. Both records stayed live, neither recorded a supersession, and the stale one ranked first. - Keying on subject-and-relation instead removes the length sensitivity: the size of the changed value stops affecting whether a conflict is detected.
- On FactConsolidation the two approaches separate by 12 points at 6k and 10 at 262k — with caveats stated below, because these are single runs.
The rule
Both systems compared here resolve conflicts when the fact is written rather than when it is read, which is the right moment. They differ in what they compare.
agentmemory computes Jaccard similarity — shared tokens over union of tokens — across the whole
content of the new fact and the candidate, and supersedes when that exceeds 0.7.
Jaccard is a reasonable near-duplicate detector. The problem is that "this fact was contradicted" and "this string is nearly identical" are different questions, and only one of them is being asked.
The worked example
Two statements that cannot both be true:
goaltender is associated with the sport of ice hockey
goaltender is associated with the sport of pesapallo
| shared tokens | 7 |
| union of tokens | 10 |
| Jaccard | 0.7000 |
| threshold | > 0.7 |
| result | no supersession |
It lands on the boundary and the comparison is strict, so it falls through. Checked against a
running server, both records came back with isLatest: true and supersedes: [], and search
returned both — with the stale record ranked first.
Nothing here is adversarial. It is a four-word predicate with one word swapped, which is the most ordinary shape a changed fact takes.
Why the gradient runs the wrong way
Hold the sentence frame constant and vary only how much the value changed:
| old → new | overlap | caught? |
|---|---|---|
ice hockey → ice hockey (men's) |
high | yes |
ice hockey → field hockey |
medium | maybe |
ice hockey → pesapallo |
low | no |
The clearer the contradiction, the fewer tokens survive, the lower the score, the likelier it slips past. A threshold tuned to catch more of these has to be lowered — which starts superseding facts that merely resemble each other and were both true.
That is the structural bind: one number is being asked to separate "this replaced that" from "this looks like that", and those are not the same axis.
Keying on the subject instead
Knowl keys conflicts on subject and relation — the atom's title — rather than on content overlap.
goaltender / sport holds one current value by construction. Whether the new value is one
character different or shares no tokens at all does not enter into it.
That removes the length sensitivity entirely, which is the whole point. It is not a better threshold; it is not a threshold.
What it costs on a benchmark
FactConsolidation single-hop from MemoryAgentBench, substring exact match, 100 questions,
retrieve_num: 10, scored by the benchmark's own code with gpt-4o-mini as the reader for every
row:
| system | 6k SubEM | 262k SubEM |
|---|---|---|
| Knowl, supersession on | 95.0 | 89.0 |
| agentmemory v0.9.29 | 83.0 | 79.0 |
| Knowl, supersession off | 75.0 | 73.0 |
And the conflicts each system actually resolved on the same corpus in the same order:
| system | 6k | 262k |
|---|---|---|
| Knowl | 149 of 455 | 6,761 of 18,332 |
| agentmemory | 48 | 3,913 |
| Knowl, supersession off | 0 | 0 |
agentmemory lands between Knowl's two arms at both sizes. That ordering holding across a 40× corpus change is the main reason to believe the measurement — a system scoring below the disabled arm would point at a broken adapter rather than a real result.
Every setting except the supersession pair was copied from the benchmark's own Simple_rag_bm25
config rather than chosen by us: same model, same temperature, same input length limit, same buffer
length.
What we are not claiming
These are single runs at temperature: 0.7 and should be read as directional. The ablation gap
moved 4 points between two runs of the same 6k cell. No figure above is publication-grade until it
has been repeated at least three times with the spread reported, and we would rather say that than
have someone else discover it.
Also: the ratio of conflicts resolved does not generalise. agentmemory resolved 32% of the conflicts Knowl did at 6k but 58% at 262k, because the mix of fact shapes changes what clears the threshold. Do not quote the 6k ratio as if it were a constant.
The mechanism finding does not depend on any of that. The threshold is in the source, the Jaccard arithmetic is deterministic, and the worked pair scores 0.7000 whoever runs it.
Method, raw per-question records, the exact configs, and the splits where we do worse: the full findings.