Concept
winner's curse
If you pick the best of many setups on the same questions, it will look a bit better than it really is.
From the record
- In technical terms
- The recommended deep configuration was the best of ≥ 8 deep configurations on V-para; expect a ≈ 0.01–0.02 discount on new data.
Linked facts · 1
- Recommended deep configuration (35B mixture-of-experts + 512-token thinking budget) on the bounty-style set.Recommended deep configuration (35B mixture-of-experts + 512-token thinking budget) on the bounty-style set.raw 0.553 (0.524–0.581; with seed noise 0.513–0.592); floor-adjusted 0.436 (0.401–0.472; with seed noise 0.387–0.486); win 0/51; desktop p50 ≈ 79 sSetV-paraStatusdev-styleJudgeClaude Opus (judge.v1, both orders)n51Seeds2Date2026-10-09CaveatChosen as the best of ≥ 8 deep configurations on this same set: expect a winner's-curse discount of about 0.01–0.02. QUEUE prints raw 0.552; the two-seed aggregate is 0.553. Desktop only.Sourceresearch record
reports/BAR-TABLE.md:39research recordreports/BAR-TABLE.md:126research recordREADME.md:28Open the data file