Concept
thinking budget
How many hidden "reasoning" words the model may write before answering. 512 was as good as 1024 at about half the time.
From the record
- In technical terms
- Cap on reasoning tokens; budget curve on V-para floor-adjusted: 0 → 0.33, 256 → 0.39, 512 → 0.44, 1024 → 0.42; 3072 no gain (model stops ≈ 1.3k tokens).
Linked facts · 3
- Thinking is the largest single measured gain on the bounty-style set.Thinking is the largest single measured gain on the bounty-style set.thinking (1024) vs no thinking: raw +0.075 (+0.040…+0.110); floor-adjusted +0.093 (+0.050…+0.137)SetV-paraStatusdev-styleJudgeClaude Opus (judge.v1, both orders)n51Seeds2Date2026-10-08Sourceresearch record
reports/BAR-TABLE.md:106Open the data file - A 512-token thinking budget matches 1024 in quality at about half the time.A 512-token thinking budget matches 1024 in quality at about half the time.512 vs 1024 on V-para: raw +0.011 (−0.010…+0.031), floor-adjusted +0.014 (−0.012…+0.039), ×0.55 time; on v0.2 −0.005 (−0.020…+0.009), ×0.62 timeSetV-para; v0.2Statusdev-style / devJudgeClaude Opus (judge.v1, both orders)n51 / 110Seeds2 (V-para) / 1 (v0.2)Date2026-10-09CaveatVerdict label: CHEAPER-PARITY (an equivalence, not a gain).Sourceresearch record
reports/CLOSED.md:39research recordreports/BAR-TABLE.md:107research recordreports/CLOSED.md#R20Open the data file - A 256-token budget is cheaper but measurably worse.A 256-token budget is cheaper but measurably worse.raw 0.514 (0.483–0.544); floor-adjusted 0.387 (0.349–0.426); vs 1024 raw −0.028 (−0.056…−0.003); p50 ≈ 55 sSetV-paraStatusdev-styleJudgeClaude Opus (judge.v1, both orders)n51Seeds2Date2026-10-09CaveatKeeps ≈ 63 % of the thinking gain at 40 % of the time.Sourceresearch record
reports/BAR-TABLE.md:41research recordreports/CLOSED.md:40Open the data file