Decision ·
References = Claude Opus + web search, frozen once, hashed; never regenerated
From the record
- Context
- Frozen frontier-plus-web reference answers for the public v0.1 set start being generated (09-30 → 10-02)
- Why
- a fixed denominator makes runs comparable
- What followed
- same refs used for every v0.1 number; regenerating = not comparable
- Reversible
- no (by design)
- Status
- kept
- Alternatives considered
- live references per run
Linked facts · 1
- References = Claude Opus + web search, frozen once, hashed; never regenerated (why: a fixed denominator makes runs comparable)References = Claude Opus + web search, frozen once, hashed; never regenerated (why: a fixed denominator makes runs comparable)Date2026-09-30 → 10-02Kinda record, not a measurement (set, status, judge, n, seeds: not recorded)Caveatoutcome: same refs used for every v0.1 number; regenerating = not comparableSourcebenchmark
README.md:48-50benchmarkTRACE-FORMAT.md:79Open the data file