The research · Act V, Where it stands · Chapter 18
Reproduce it (appendix)
Everything that can be rerun is public, and what cannot be rerun is named.
Yes, most of it. The benchmark questions, reference answers, judge prompt and scoring code of the first exam are public, and you can run them on your own app with a few commands. What you cannot rerun: the sealed question sets (sealed by design), the new exam (sealed until it is graded), and the exact judge outputs (the judge is a closed model, so reruns agree closely but not bit for bit).
Use it: bench, research record, packs, engine
Plug your app into the same answer key
Everything you need to measure your own assistant is public: 60 v0.1 questions (57 judged), frozen Opus-plus-web references with provenance, the judge, the scorers and the floor metric. Write one trace per question with itemId, answer.text and error: null, run the judge with your own Claude Code login, and compute the three scales. The research record holds frozen criteria, arm reports, aggregates, the negative-results registry and pack cards; the engine is a public upstream commit plus each arm's patch.
10 What can be rerun, by whom, at what cost
The engine is a public upstream commit (0cfd4a77) plus per-arm patches; the benchmark (v0.1 questions, frozen references with provenance, judge, scorers, floor metrics) and the v0.3 plan are public; models and packs are listed with licences and revisions in the bill of materials. Every judgment row records judge model, CLI version, judge version and the sha256 of traces, questions and references, and everything after the judge is deterministic given the judgment rows. A full v0.1 judge pass is 114 calls (≈ 5 min at four shards). Not reproducible from public material: the held-out sets (private by design; hashes only), bit-identical judge outputs (closed model; σ_judge ≈ 0.003), and the emergency-detector runner profile (base-model licence unknown; not released).
Badge rows the 2 public fact rows this depth cites; a number here without a chip comes from a row that is not public yet (see /data/)
| Claim and value | Set | Status | Judge | n | Seeds | 95 % CI | Date |
|---|---|---|---|---|---|---|---|
| Resume safety: a cached judgment row is reused only if traces, questions and references sha256, judge version and model all match; else exit 7.exit 7 | not recorded | not recorded | not recorded | not recorded | not recorded | 2026-10-13 | |
| One full v0.1 judge pass is 114 judge calls (57 items × 2 orders).114 calls | v0.1 | not recorded | Opus judge.v1 | not recorded | not recorded | not recorded |
On the calendar
4 dated events · 12 Oct → 15 Oct
- 12 Oct · 22:46Pack catalogue + authoring workflow (licence policy PD/CC0/CC BY/CC BY-SA only)
- 15 Oct · 01:03Public repos staged: scrub, mechanical gate, independent review (41/44 findings applied)
- 15 Oct · 01:18Public bench and research repos staged and verified (scrubbed, reviewed)
- 15 Oct · 10:07Public repos live: boar-bench (139 files) and boar-research (411 files)