Judging the LoCoMo judges

1. Q1

   How often does each judge accept a known-wrong answer, and does the type of wrong answer matter?

   A great deal. Acceptance ranged from 2.5% to 28.8% for specific wrong answers and from 32.5% to 88.8% for vague, on-topic wrong answers. Every judge was substantially more forgiving of vagueness.

2. Q2

   What does a stricter judge cost in rejected right answers?

   Very little when the reference answer is correct. Across 479 scored right answers, there was one rejection. The tradeoff appears when the reference answer itself is wrong.

3. Q3

   Does changing the grader change the LoCoMo score?

   Yes. On identical answers, agreement with the labels ranged from 60.8% to 87.9% depending only on the grader.

4. Q4

   Does showing the judge the source evidence matter?

   Yes. In this setup, evidence made the judge more permissive, not more accurate. The evidence-fed configuration accepted 7.7 percentage points more answers overall, with nearly all of the increase coming from vague wrong answers. Its overall agreement was lower, not higher.

5. Q5

   When the answer key is wrong, does the judge follow the key or the truth?

   It depends on the grader. Against a true answer, broken-key following ranged from 10% to 70%. Every grader accepted all 20 true answers once the key was corrected.

A separate consistency test found that three of five applicable graders did not treat equivalent list-answer cases consistently.

This is a benchmark of graders, not memory systems. It does not rank or re-score any memory product, and it cannot establish which published system is better.

Six actions for more reliable LoCoMo judging

1. Audit answer keys before tightening the judge. Strict grading caused only one false rejection across 479 right answers when the key was correct, but the strictest grader rejected 70% of true answers paired with broken keys. Adopt the audit's corrected keys, verify them, then standardize a stricter judge.
2. Require answers to state the requested fact. Every judge struggled more with vague, on-topic answers; even the strictest accepted 32.5%. Test a judge that first identifies the exact value an answer gives, then compares it with the reference. If no value is given, the answer fails.
3. Add a third verdict for a likely broken key. Every grader accepted all 20 true answers once the key was corrected. When an answer contradicts the reference but is supported by evidence, send it to review instead of automatically marking it wrong.
4. Test consistency with both accept and reject cases. The two graders that passed the current test accepted every probe, so it rewarded leniency as well as consistency. A replacement should include cases where rejection is the only consistent decision.
5. Spend evaluation effort on more items and better keys. Repeat disagreement was 1.6% overall and 2.9% at worst, while grader choice moved agreement by 27.1 points. On this bank, broader coverage and key auditing are more valuable than additional repeated calls.
6. Tell the judge how to use evidence. Simply adding source evidence raised acceptance by 7.7 points and lowered agreement from 78.3% to 70.0%; almost all extra acceptances were wrong. A reference prompt should require explicit comparison with the evidence, not merely include it as context.

The practical recommendation is one versioned LoCoMo reference judge, used with audited answer keys.

It should identify the fact an answer commits to, compare that fact with the reference, and allow review when the key appears wrong. These design changes remain proposals for a follow-up benchmark, not findings of this run.

How to use LoCoMo scores today

• Compare scores only when the grading configuration matches; otherwise, do not treat the difference as a memory-system effect.
• Require the judge model, prompt, evidence input, parser behavior, and repeated-call rule.
• Publish error rates separately for specific wrong and vague wrong answers.
• Treat small score gaps cautiously when grader choice can move agreement by 27.1 points.

A LoCoMo score is not fully specified by the memory system's name and final percentage. It is the product of an answer pipeline and a grading pipeline. A comparable result must disclose the six choices that define those pipelines.

The six variables

Whether the judge receives source evidence belongs in the judge-prompt entry because it changes the judge's input. Parser behavior and repeated-call aggregation should be reported alongside the judge configuration.

Three ways to report each variable

Each of the six variables must have one of three states:

Unknown information must be reported as not reported, not omitted or inferred from a repository name, model family, or surrounding context. Evidence belongs only with a choice measured in the benchmark. A publisher statement may be cited, but it should not be presented as an independent measurement.

Why the benchmark supports this standard

This benchmark directly measures the importance of the grading variables. Holding the answer bank and judge model fixed while changing the judge prompt moved agreement from 60.8% to 87.9%. Holding the Mem0 prompt family fixed while adding source evidence changed acceptance by 7.7 percentage points. The judge prompt and the judge's input therefore change the meaning of the resulting score.

The experiment did not vary the first four variables. They still need disclosure because they determine which candidate answers reach the grader. Without them, a reader cannot distinguish a memory-system effect from a retrieval, generation, or answer-style effect. The benchmark establishes the grader side of the problem; the complete reporting standard covers the whole measurement pipeline.

A LoCoMo score without all six entries is an incompletely described measurement. Scores produced under different disclosed configurations should not be treated as directly comparable.

This report's disclosure

Four entries are marked not reported. That is the honest description of a grader benchmark that used candidate answers produced elsewhere. This report makes no claim about upstream choices that its source materials cannot establish.

How the additional tests were run

The consistency test used 12 constructed list-answer probes, five applicable grading configurations, and three calls per probe, for 180 calls. The wrong-key test used 20 questions under both the official and corrected answer keys, six configurations, and three calls per case, for 720 calls. All planned calls completed.

• It does not evaluate or rank memory systems.
• It tests one dated judge-model snapshot on a balanced 240-answer diagnostic bank. Rates may differ on another model, snapshot, or answer distribution.
• The correctness labels were model-screened and selectively reviewed by the authors, not independently labelled by multiple human annotators.
• The 20 broken-key items and 12 consistency probes demonstrate mechanisms; they do not estimate how common those problems are in ordinary benchmark runs.
• The authors operated the experiment themselves. Colophon verifies the published files and their internal consistency, but not the provider's internal execution or the authors' organizational independence.

Download the full result data, grading outputs, prompts, and analysis files from the published evidence package. Colophon packages these materials and checks that the files used by this report have not changed since publication.

File verification confirms the integrity of the published materials. It does not prove that the correctness labels are right or independently verify the model provider's internal execution. Independent timestamping is pending.

The bundle, its IDs and the check command are in the evidence row at the top of this page.

This benchmark builds on the published materials in dial481/locomo-audit, with the author's stated permission, and on judge prompts posted by Backboard, Mem0, and community contributors in the LoCoMo discussion. The audited judge prompt derives from EverMemOS, and the revised rubric from mem0ai/memory-benchmarks, both under Apache-2.0. The consistency examples originated with @AnitaLeungxx, and the strict variant and audit materials with @dial481.

The underlying dataset is LoCoMo. LoCoMo and the audit annotations are CC BY-NC 4.0. This report and the selected derived materials are released under the same non-commercial license, with source pointers rather than full conversation data.

All 4,320 planned judge calls completed. Twenty-two responses could not be parsed, which excluded seven evidence-fed items from that comparison.

The record's reason for the exclusions: “Unparseable responses were recorded as neutral rather than converted into rejections, all of them in the evidence-fed configuration. The evidence-fed items consequently left without a valid majority appear as explicit exclusions.”

The method's pre-set minimum for a complete run: 99.5% of planned calls judged. This run judged 4,320 of 4,320; its recorded outcome is complete.

Repeat stability. Repeat disagreement was 1.6% overall and 2.9% in the least stable grading configuration.