Concept
ruler (bench v0.4)
A fixed set of 75 test questions, frozen on 2026-10-20 before any system answered them, meant to give the final score of the submitted app.
From the record
- In technical terms
- 75 items = 71 headline + 4 anchor questions taken from Vitalik Buterin's public post (scored outside the headline); written half by Claude and half by OpenAI gpt-5.6-sol via the Codex CLI, cross-fact-checked; sha f9ad2f2f. Not written outside the Claude family (a stated deviation). First exposed to two engines on 2026-10-24.
Linked facts · 3
- Bench v0.4 frozenBench v0.4 frozen75 items (71 headline + 4 anchors); 329 checklist points supported on the open web (at least one family, none contradicting; the 32 balance-round points by both families); sha f9ad2f2fSetv0.4 (held-out ruler)Statusheld-out ruler (v0.4; exposed to two engines 2026-10-24)Date2026-10-20Kinda record, not a measurement (judge, n, seeds: not recorded)Caveatwritten by Claude + OpenAI gpt-5.6-sol (the judge families)Sourcenot in the public repositories yetOpen the data file
- v0.4 ruler item set FROZEN (75 items, sha f9ad2f2f...) before any system answeredv0.4 ruler item set FROZEN (75 items, sha f9ad2f2f...) before any system answeredDate2026-10-20Kinda record, not a measurement (set, status, judge, n, seeds: not recorded)Sourcenot in the public repositories yetOpen the data file
- Bench v0.4 exposureBench v0.4 exposurefirst answers by two engines 10-24; any later engine change = developed after exposureSetv0.4 (held-out ruler, now exposed)Statusheld-out ruler (v0.4; exposed to two engines 2026-10-24)JudgeClaude Opusnnot recordedSeeds2Date2026-10-24Later2026-10-24 20:34: a third engine (BoarLite b4ef94f3) answered v0.4 in the final ruler run; its numbers are internal pending the author.Sourcenot in the public repositories yetOpen the data file