Judging the LoCoMo judges
A controlled benchmark of the grading prompts behind published LoCoMo scores. Six grading configurations judged the same 240 answers on the same model snapshot.
Changing only the grader moved agreement with the same labels from 60.8% to 87.9%.