89.6%, end to end,
and here's the method.

The headline

By category

Six question types, weighted as the benchmark defines them. Where Kemory is strong and where it isn't:

Run history

The same harness, run six times. We publish all six — including the runs that scored lower than the headline — because a single number without its variance is not a result.

Run-to-run variance is roughly ±2 points on most categories. Multi-session is the most prompt-sensitive and has swung about 7 points between runs. Treat 89.6% as the centre of a range, not a precision measurement — anyone quoting a memory benchmark to a tenth of a point is overstating what these harnesses can resolve.

Relative to the April canonical run, two further things are different, and neither makes it a like-for-like re-run: the harness gained self-consistency voting (a method change), and the target moved from a local instance to the production deployment. What did not change across all six runs: the dataset, the judge, the prompts, and the memories being retrieved against.

The dataset

LongMemEval (ICLR 2025) tests long-term memory in chat assistants: a haystack of prior sessions, then a question whose answer is buried somewhere in it. We use the oracle configuration of LongMemEval-S — the sessions containing the evidence, without the full distractor corpus.

A subset carries an abstention marker — questions whose answer is not in the history, where the correct behaviour is to decline rather than invent. A memory system that confabulates scores badly on these by design.

Which configuration, and why it matters

LongMemEval ships in more than one size. The distinction is how much irrelevant history surrounds the evidence:

Oracle is a legitimate and widely-reported configuration, and it is the harder-to-fake one in that retrieval still has to pick the right turn out of a real conversation. But it is not the full-haystack test, and a number produced on it should only be set against other oracle numbers. We have not yet published a full-haystack run.

Provenance: the corpus is the official release accompanying the LongMemEval paper (ICLR 2025). We hold it alongside the harness rather than fetching it at run time, so a result stays reproducible even if the upstream mirror moves — and so the exact bytes behind a published number are pinned rather than assumed.

How we ran it

Three phases, each independently resumable — a 500-question run is long enough that restartability matters.

Total: about 115 minutes for a full run from an empty vault; about 25 minutes for the published run, which reused the seeded vault. All three phases take --resume; the run output records the hypothesis and the judge's verdict for every one of the 500 questions.

What retrieval actually did

The number depends as much on what reached the reader as on the reader itself.

Two honest observations about those numbers.

Depth is tuned per question type, and that flatters the result. Multi-session gets three times the retrieval budget of a single-session question. That is a defensible engineering choice — a question whose evidence is spread across five conversations genuinely needs a wider window than one answered by a single turn — but it is a choice made with knowledge of the question type, which a live system does not always have. A flat top-k 10 across all types would score lower.

Turn-level granularity cuts the other way. Storing each turn as its own memory makes retrieval harder, not easier: the system has to locate the specific exchange rather than being handed a whole session that happens to contain the answer. That is also how Kemory stores real conversations, so it is the honest configuration rather than a benchmark convenience.

Caveats

Four things a careful reader should know before comparing this number to anything else. We would rather state them than have someone find them.

1. The harness is tuned to this benchmark

This is the most important one. The answer prompts are written per question type, and they were refined using failure analysis from earlier runs on this same dataset. Retrieval depth is likewise set per question type, and the self-consistency voting on counting and duration questions was added after failure analysis showed those questions dominated the misses. None of it was arrived at blind.

That makes the figure a measure of Kemory retrieving well when the harness around it has been tuned for the task — not a zero-shot result on unseen data. It is the normal way these numbers are produced across the field, and the per-type judge protocol in the benchmark itself invites per-type handling. But anyone comparing our number to a system evaluated with a single flat prompt is not comparing like with like, and we would rather say so than let the number imply otherwise.

What is not tuned: the storage and retrieval path. Ingestion goes through the public write API, retrieval through ordinary hybrid search, and every question is isolated in its own namespace.

2. A different judge

The paper judges with GPT-4o. We judge with Llama-3.3-70B, using the paper's prompt templates verbatim. The prompts are identical; the model grading them is not. LLM judges disagree at the margins, so our figure is not strictly interchangeable with a GPT-4o-judged one. We use the same model as reader and judge, which is a further deviation worth knowing about.

3. The oracle configuration

We run the oracle set — the evidence-bearing sessions, without the full distractor haystack. That is a legitimate configuration of the benchmark and the one many published numbers use, but it is an easier retrieval problem than the full corpus. A number on the oracle set should be compared with other oracle-set numbers.

4. Variance

±2 points run to run, ~7 on multi-session. The 86.2% reproduce run above is the same system on the same data. We publish the canonical run as the headline and the spread alongside it, rather than quietly reporting our best day.

On comparing this to other numbers

The most common mistake with memory benchmarks is comparing two figures that measure different things.

89.6% is the second kind. Recall figures are usually higher than end-to-end figures on the same system, because finding the evidence is easier than using it. A published R@5 in the mid-90s and an end-to-end score in the high-80s can describe systems of similar quality — they are not on the same axis, and setting them side by side is a category error rather than a result.

Reproduce it

The harness, the oracle dataset and the raw run output are preserved together — hypotheses and judge results for every one of the 500 questions, so any individual answer can be inspected rather than trusted.

The three phases run against a live Kemory instance with your own credentials:

# Phase 1 — ingest the haystack into Kemory
python3 ingest.py --data longmemeval_oracle.json \
                  --out output/ingestion_state.json --resume

# Phase 2 — retrieve and generate
python3 run.py --data longmemeval_oracle.json \
               --state output/ingestion_state.json \
               --out output/hypotheses.jsonl --topk 10 --resume

# Phase 3 — judge against the reference answers
python3 score.py --hyp output/hypotheses.jsonl \
                 --ref longmemeval_oracle.json \
                 --out output/results.json --resume

Every phase takes --resume, so an interrupted run continues rather than restarting. Phase 1 is the expensive one and only needs running once per vault.

Re-score our answers with a different judge

Phase 2 writes hypotheses.jsonl in the format the official LongMemEval evaluator expects. That means our answers can be graded by the paper's own scoring script, with GPT-4o, without re-running anything:

python3 evaluate_qa.py gpt-4o hypotheses.jsonl longmemeval_oracle.json

This is the direct answer to the judge caveat above. If you do not trust a Llama-judged score, you do not have to take ours — take our raw answers and grade them yourself, with whatever judge you consider authoritative.