Experiment ·
DEEPTHINK
Does a 1024-token thinking budget on the deep tier (arm T: cut 6 + Qwen3.6-35B, thinking on) lift V-para vs the same deep tier without thinking (D, DEEPVPARA), and at what time cost?
From the record
- What was tested
- DEEPTHINK
- Result
- thinking (1024) vs no thinking: raw +0.075 (+0.040…+0.110); floor-adjusted +0.093 (+0.050…+0.137)
- Verdict
- adopted: shipped or recommended by its frozen rule
- Detail
- lever-adopted (thinking on the deep tier adopted; budget later moved 1024 → 512 by DT512); L1 PROMOTED on the Δ clause
- Set
- V-para (V-para); v0.1 smoke 2 items
- Judge
- Claude Opus, Claude Haiku (L1)
- n
- 51 judged ids (58 requests) per seed
- Seeds
- 42, 43
- Pre-registered
- yes, sha256-frozen
Linked facts · 4
- Thinking is the largest single measured gain on the bounty-style set.Thinking is the largest single measured gain on the bounty-style set.thinking (1024) vs no thinking: raw +0.075 (+0.040…+0.110); floor-adjusted +0.093 (+0.050…+0.137)SetV-paraStatusdev-styleJudgeClaude Opus (judge.v1, both orders)n51Seeds2Date2026-10-08Sourceresearch record
reports/BAR-TABLE.md:106Open the data file - Letting the deep tier think for up to 1024 tokens before answering.Letting the deep tier think for up to 1024 tokens before answering.raw 0.542 (0.513–0.571); floor-adjusted 0.422 (0.385–0.459); win 0/51; desktop p50 ≈ 143 sSetV-paraStatusdev-styleJudgeClaude Opus (judge.v1, both orders)n51Seeds2Date2026-10-08Sourceresearch record
reports/BAR-TABLE.md:38Open the data file - Process review: 65/65 criterion shas verify; DEEPTHINK yielded 0.30 floor-adj points per 1k judge calls (×10 the deep-tier entry arm)Process review: 65/65 criterion shas verify; DEEPTHINK yielded 0.30 floor-adj points per 1k judge calls (×10 the deep-tier entry arm)65/65Date2026-10-10Kinda record, not a measurement (set, status, judge, n, seeds: not recorded)CaveatVerification note: Line refs are:13 (65/65) and:18 (DEEPTHINK). Unit: 0.30 floor-adj per 1k calls = 30.5 "points" (100×Δ) per 1k calls in public BAR-TABLE:106.Sourceresearch record
reports/BAR-TABLE.md:106-109Open the data file - Large levers survive multiplicity correctionLarge levers survive multiplicity correctionX2b z ≈ 5.5, LITE5-180 z ≈ 6, DEEPTHINK z ≈ 4.2Date2026-10-16Kinda record, not a measurement (set, status, judge, n, seeds: not recorded)Caveatcitation fix; ≈ 60 judged comparisons, Bonferroni over 100; the bigger risk is selection on the same dev setsSourcenot in the public repositories yetOpen the data file