How Colophon turns a benchmark result into a claim others can check.
You lock the method before the run. Colophon seals it, accounts for every planned result, and publishes the answer with its evidence and limits attached.
A benchmark score is easy to publish. The difficult part is keeping the question, method, missing results, and limitations attached when the number travels.
Anyone can check a published claim, free. Colophon is for results that will inform a release, a customer claim, a choice between agent setups, or a technical argument that someone else needs to inspect.
When Colophon is useful
Use it when the answer needs to survive outside the person who produced it.
You need to support a performance claim
Before a result appears in a release, sales conversation, research note, or README, publish the basis for it.
You need to choose between agent setups
Compare them on work that represents the decision, using a method locked before the official run.
You expect the answer to be challenged
Give the reader the method, the full accounting, the evidence, and the limits without asking them to trust a screenshot.
From claim to published evidence
The order matters. The official method is fixed before the official result exists.
- 01
Start with the claim
Name the exact performance question and the decision the answer needs to support.
- 02
Choose the benchmark
Use an established suite when its official method fits, or a benchmark you already have.
- 03
Lock the method
Lock the tasks, setups, repetitions, grading, exclusions, and limits before the official run begins.
- 04
Run and account
Run the plan on the harness you already use, and keep every expected outcome, including failures, timeouts, and missing results.
- 05
Publish the evidence
Release a readable report, the exact supporting bundle, and the limits on the answer.
The report is readable. The evidence stays attached.
The page explains the answer. The bundle remains the exact source behind it.
A readable report
The question, result, method, accounting, and limitations in one public reading surface.
The evidence bundle
The exact records and artifacts behind the report, listed by digest and available for download.
A permanent URL
An append-only public origin. A correction becomes a new report; the old published bytes remain available.
Check a report
The public verifier answers a narrow, useful question: do the downloaded records belong together, and do they support the result that was published?
It does not run a benchmark, judge whether the benchmark was well chosen, or make an evaluator's decision correct.
npx @colophon-claims/verify@0.2.1 ./bundleFor the published report's format, the checker runs these seven checks.
manifest: Every file listed by the bundle is present and unchanged.evidence-closure: The evidence records cover the result the report presents.trust: The public keys in the bundle are the ones its signed records name, one key per signer.matrix-rederivation: The result table can be rebuilt from the evidence.report-verification: The signed report covers this result table, and its numbers recompute from it.claim-consistency: The machine-readable claim is rebuilt from the signed records and matches them exactly.integrity-anchors: Any timestamp proofs the bundle carries match the bytes they claim to cover.
The verifier is a convenience, not a requirement. A reader can inspect the JSON, recompute the listed SHA-256 digests, and verify the signatures with other tools.
Established methods or your own benchmark
You run the benchmark. Colophon seals and publishes the claim.
You run the benchmark on Harbor, Inspect or your own harness and bring the finished run back. Colophon does not run benchmarks for claimants today. Publishing your own claim from this site is not available yet; until it is, write to ritsu.kai2000@gmail.com.
When an official method specifies the agent, engine, tasks, repetitions, or grading, those choices remain fixed. Change them and the work becomes a custom comparison using that task source.
The runner or harness
Executes and grades the benchmark work.
Colophon
Seals the method you locked, records what happened, accounts for the complete plan, and publishes the result.
What a Colophon report does not prove
These are limits on the claim, not fine print to remove from it.
A method lock does not prove honesty against the run owner.
It disciplines the official process and makes the published run inspectable. It cannot show that another run was never withheld.
Distinct identities are not necessarily distinct parties.
A signature identifies the key that signed a record. It does not prove that unrelated people or organizations controlled those keys.
A valid record can still contain a wrong judgment.
Verification checks the signed bytes and declared reduction. Evaluators can still be mistaken, including together.
A report is not a certification or ranking.
It answers one declared question. Colophon does not certify or accredit a result, or rank unrelated reports.
The scope remains part of the result.
The model, tasks, method, failed checks, exclusions, and other limitations travel with the answer.
Publishing your own claim
Checking a published claim is free, and needs nothing from Colophon. Publishing your own claim from this site is not available yet. Until it is, write to ritsu.kai2000@gmail.com to publish.