Do you need a Skill, or is CLAUDE.md enough?
We put the same instructions in a native Skill and root CLAUDE.md, then ran 492 executions to see whether the loading path changed Claude Haiku 4.5's performance.
Not by enough to settle the question. Only 14 of 41 tasks informed the paired estimate, and the uncertainty includes outcomes favoring either setup. That makes this a useful Colophon demonstration: the method, all 492 runs, failed checks, and limits remain attached to an inconclusive answer.
The answer
The point estimate favored root CLAUDE.md by 0.047. The uncertainty still spans outcomes favoring either setup.
- 95% confidence interval
- −0.223 to 0.129
- Informative tasks
- 14 of 41
- What the method allowed
- An estimate only—no selection, ranking, or certification.
Read this as a lead, not a verdict. The 95% interval runs from −0.223 to 0.129, so the data remain compatible with either setup performing better. This run points toward root CLAUDE.md; it does not settle the choice.
Lock the question first
The Skill and CLAUDE.md setups used the same instruction bytes; a third setup received no instructions. The comparison changed where instructions loaded while holding the other task resources constant.
Task source. The report used SkillsBench v1.1 packages built for BenchFlow 0.6.3, pinned at commit b63b7b285022. The curated Skill bundles, task environments, oracles, and verifiers stayed unchanged. BenchFlow's standard modes could not hold those resources constant across all three setups, so the runner used jinn.demo1.claude-md-flatten@1 to change only instruction loading.
| Setup | What changed | Runs per task | Total runs |
|---|---|---|---|
| Native Skill | Loaded as a native Skill. | 5 | 205 |
| Root CLAUDE.md | Placed in root CLAUDE.md. | 5 | 205 |
| No instructions | No instructions; the control setup. | 2 | 82 |
Why the lock matters. Colophon fixed the task set, three setups, replicate counts, and informative-subset rule before execution. That kept the method from shifting toward a preferred result; the declaration digest makes any later edit detectable.
- Model
- claude-haiku-4-5-20251001
- Benchmark
- SkillsBench v1.1
- Tasks
- 41, all admitted before the run
- Skill runs
- 205
- CLAUDE.md runs
- 205
- No-instructions runs
- 82
Account for every run
Failed or inconvenient runs can change a result when they disappear. All 492 expected executions remain visible here.
Not all 41 tasks entered the paired estimate. The predeclared rule required every no-instructions run to score zero and at least one instructed setup to produce a positive mean. 14 tasks qualified.
| How the tasks divided | Tasks |
|---|---|
| All tasks run | 41 |
| No-instructions control scored above zero | 4 |
| Neither instructed setup produced a positive mean | 23 |
| Tasks that informed the paired estimate | 14 |
| What happened to every expected run | Runs |
|---|---|
| Expected | 492 |
| Kept in the record | 492 |
| Excluded | 0 |
| Unavailable | 0 |
| Silently dropped | 0 |
What the denominator shows. All 492 expected executions have a visible outcome: none were excluded, unavailable, or silently dropped. The 14 informative tasks fall below the registered confirmatory floor of 21 units across 13 clusters. The estimate can shape the next test; it cannot close the argument.
Both tasks failed closed in the 492-run denominator. Nothing was removed after the result was known.
pddl-airport-planningon hostw5pddl-tpp-planningon hostw6
Keep the limits attached
A headline can travel farther than its caveats. Colophon publishes the boundary with the result so the public record keeps both visible.
What it applies to
- This is one run of claude-haiku-4-5-20251001 on SkillsBench v1.1, not a general result about Skills,
CLAUDE.md, SkillsBench as a whole, or other models. - The paired answer comes from 14 of 41 tasks and falls below the confirmatory floor.
- The report does not rank or certify either loading path, and it does not show that Skills do not work.
How it ran
- The agent ran on the host. Grading ran in the pinned task container, so the agent-side environment was not the task image.
- Two host task checks failed. Their runs remained in the fail-closed 492-run denominator.
Who ran it
- One operator designed, ran, graded, and sealed this comparison. The published evidence makes the process inspectable; it cannot prove honesty against that operator.
- Separate run records and cell keys are not evidence of separate real-world parties.
Publish the evidence, not just the answer
The signed report, locked method, result matrix, 984 evidence records, 1003 artifacts, and limitations are published together.
Why publication matters. The manifest binds every file to a digest, and the signed envelope identifies the canonical report. A reader can inspect the record behind the conclusion instead of taking this summary on trust.
npx @colophon-claims/verify@0.1 ./bundleIt checks the manifest, evidence closure, artifact integrity, signature, result matrix, report, and claim consistency. Protocol identifiers under https://spec.jinn.network/ are names; that origin is not hosted yet. Verification uses the exact platform bytes installed from npm.
sha256:c66199af9e86dd9701f40c0a9d1fd8648dde4e50821f99335ce4052cfbb93583Bundle identitysha256:0e5bba90cbd732a47572bde54c1377fa30f19ee26ff4bf97c8fc2099eb78cf2e| Canonical file | Bytes | SHA-256 |
|---|---|---|
Complete bundle manifestbundle.json | 345.8 KB | 0e5bba90cbd732a47572bde54c1377fa30f19ee26ff4bf97c8fc2099eb78cf2e |
Bundle reading noteREADME.md | 1.4 KB | 9aaad26c4cca3aa750d224e78305094e5e82735aacde1e6c86e868d8166460b9 |
Public reading recordpresentation.json | 5.1 KB | 67475c9174d1fc1826103c535632a6110a7a15ad264e33bffb69e06fa69db2f7 |
Machine-readable claim packageclaim-package.json | 365.3 KB | 09d750e382445897ca6a696337b26d7d3209259082437800649354b4035f63f5 |
Signed report envelopereport-envelope.json | 6.2 KB | c66199af9e86dd9701f40c0a9d1fd8648dde4e50821f99335ce4052cfbb93583 |
Signed report payloadreport.json | 4.5 KB | 64cc54cc10ccf192267f0ffc5793612c71ec0f586c34643687b6cdacaf14ccdd |
Declared analysisanalysis-manifest.json | 2.8 KB | 822b2f7469dc2e58a3e72eee32688614d296ba20fc381d9a074e3935a68622b3 |
Evidence cohortcohort.json | 693.7 KB | f279e3fe7de1308f8177bea06d316b0f6c5d546900f75033396fe55a6cf0d001 |
Re-derivable result matrixmatrix.json | 668.8 KB | 1bd2dc6cd59df4784e78fca37907000b2f102ec69448a9a233acc9ca7c834470 |
Sealed human reportsource/demo1-report.md | 8.0 KB | 4f0ec783b5398985812430313a4673e0d1d161cff1e554bc3944339def6dadbf |
Have a claim that needs to hold up?
If you're preparing to ship or defend a skill, harness, loadout, or review-agent claim, Colophon can lock the method, account for the run, and publish the evidence. Bring the claim.
- Report envelope
- sha256:c66199af…3583
- Bundle
- sha256:0e5bba90…cf2e
- Subject
- claude-haiku-4-5-20251001
- Method
- Paired A−B on the pre-declared informative subset
- Venue
- Self-run; one operator designed, ran, graded, and sealed
- Sealed
- 2026-08-18 00:00:00 UTC
- Attribution
- Built on Jinn, by Jinn contributors.