Benchmarks

Most suites below are retrieval-level and use no answer-generating reader. MemoryAgentBench supplies an external conflict fixture, and is additionally run end-to-end inside its own harness with an LLM reader; the other two suites are internal regression data.

MemoryAgentBench conflict resolution

MemoryAgentBench (ICLR 2026 paper) defines a Conflict Resolution track for retrieving the newest valid fact after updates. The Knowl harness uses the factconsolidation_sh_6k row: 455 facts, 156 detected conflict groups, 100 questions, active-only top-5 vector+BM25 retrieval, and no LLM reader.

Supersession retired 149 facts at write time; both rows otherwise use the same corpus, path, and metric. The instance covers dynamic single-hop latest-fact conflicts, not static, conditional, multi-hop, or reader behavior.

End-to-end, in MemoryAgentBench's own harness

The figures above are retrieval-level and scored by this repository. The same claim was also run inside MemoryAgentBench's harness and scored by its own code, with an LLM reader consuming what Knowl returned. Metric is substring_exact_match, the field the benchmark specifies for Conflict Resolution; reader is gpt-4o-mini at temperature 0.7, top-k 10, 100 questions per cell.

Each cell is a single run, not a median of several. At temperature 0.7 with n = 100, binomial standard error is roughly 3 points, so treat each individual figure as carrying a few points of uncertainty. The comparisons are wider than that — the ablation gap is about five standard errors and holds at two corpus sizes — but a single figure quoted alone should not be read to the point.

Knowl's row was measured here. Every other row is from the paper's Table 2, which states that "all RAG agents and commercial memory agents use GPT-4o-mini as the backbone" — the same reader used for Knowl, so the rows are like-for-like. Long-context rows use their own named model. Systems the paper does not evaluate on this task, including Zep, Graphiti and Letta, are not listed rather than carried over from secondary sources.

The ablation in that harness:

input_len stays within 294–410 on every row of every cell, which is the cheapest contamination check available: rows left behind by an earlier code path carry a visibly different context size, and MemoryAgentBench's --force flag does not discard old results — it only bypasses the context-level skip, so stale rows are otherwise silently averaged into a new score.

Multi-hop is retrieval-bound, not reader-bound. Only 14 of 100 multi-hop questions have the gold answer anywhere in the retrieved set, so no reader could exceed 14; Knowl scores 7. Quote it with that ceiling attached — the fix is chained retrieval, which Knowl does not do, not prompt or supersession tuning.

Method and full write-up: independent rebuild.

Re-measured 2026-08-06 with the granite-small-en-r2 default, stable across two runs. The previous figures — 96.0% and 40.0% top-1 with 3 stale returns — were taken on the minilm-l6-en default this repository shipped before 2026-08-02. The retrieval path here is vector + BM25, so the embedding model is part of the result and is named rather than left to the reader's config. Both arms moved; the gap narrowed from 56 to 51 points.

The two result JSON files are checked in: supersession ON and supersession OFF. The fetched dataset fixture is ignored by Git and is not checked in. fetch --row 4 reads the current Hugging Face row without a pinned dataset revision, so future fetches depend on that upstream row remaining unchanged. The stored results contain no hardware metadata; their timing fields are run artifacts, not portable latency claims.

See the protocol and interpretation. Reproduce from a source checkout with the repository-only harness:

npm run bench:cr -- fetch --row 4

# --json prints the full run result.
npm run bench:cr -- run \
  --instance benchmarks/memoryagentbench/data/cr-sh-6k.json \
  --top-k 5 --json
npm run bench:cr -- run \
  --instance benchmarks/memoryagentbench/data/cr-sh-6k.json \
  --top-k 5 --no-supersede --json

# --out writes a result snapshot.
npm run bench:cr -- run \
  --instance benchmarks/memoryagentbench/data/cr-sh-6k.json \
  --top-k 5 --out ./memoryagentbench-result.json

These npm run bench:cr -- ... commands are repository scripts, not published knowl CLI commands.

Internal governance regression suite

The checked-in retrieval-governance.json has 56 decision fixtures: 22 current decisions, 22 stale predecessors, and 12 rejected decisions. It defines 44 top-3 cases.

MRR is reciprocal rank, not top-1 accuracy, and at 100% it means the expected current decision ranked first in every one of the 44 cases. Stale predecessors remain active, so 21 stale traps still failed and produced 27 stale-active returns; rejected items test a separate status filter and never appeared. Recall@3 means every top three contained the expected current decision, not that every result was current.

This suite is now saturated and cannot show an improvement, only a regression. Every ranking metric is at ceiling, so the only figures left that can move are the two stale counts. Use semantic-suite.json for anything that has to discriminate.

Measured on the granite-small-en-r2 default. Two earlier sets of figures are worth knowing about, because both differed by something other than the ranker: MRR 94.3182% with 43 stale-active returns was taken on the minilm-l6-en default this repository shipped before 2026-08-02, and MRR 98.86% with 28 stale-active returns predates the semantic rescale in scoreCandidates. The model and the scoring are both part of the result.

knowl eval \
  --dataset docs/evals/retrieval-governance.json \
  --vector --json

Internal retrieval regression suite

The checked-in retrieval-suite.json contains 500 cases over 168 fixtures. It is a repository regression suite with expected items, stale traps, and forbidden items; it is not third-party evidence.

Measured on the granite-small-en-r2 default, byte-identical across two runs; no result snapshot is checked in. The run passed 497 of 500 evaluator cases, including expected, stale, and forbidden conditions rather than only search hits.

This suite is near its ceiling too, and the numbers moved slightly down when the semantic rescale in scoreCandidates landed — Recall@3 98.8% to 98.6% and Recall@10 100% to 99.8%, one case each. That change was kept because it wins where these suites cannot discriminate: on semantic-suite.json it moved Recall@3 89.6% to 91.9% and the hardest tier's MRR 33.3% to 38.1%, and it left the external MemoryAgentBench result unchanged. A suite at ceiling reports the cost of a change and not its benefit, which is the reason not to tune against this one.

Fresh BM25-only runs varied under equal-score ordering, including their failure counts. Exact BM25 outcome and rank values are therefore not published; rerun the command below in the target environment. No cross-hardware latency is claimed.

knowl eval --dataset docs/evals/retrieval-suite.json --vector --json
knowl eval --dataset docs/evals/retrieval-suite.json --json