Experiment ·
GEMMATHINK
Does Gemma 4 26B-A4B UD-Q2_K_XL + 1024-token thinking as the deep tier (arm G) match or beat Qwen3.6-35B-A3B + thinking (control T, DEEPTHINK) on V-para, and is it faster (BETTER / PARITY-FASTER / NO)?
From the record
- What was tested
- GEMMATHINK
- Result
- −0.028 (−0.057…+0.003)
- Verdict
- closed: rule failed, CI excluded the hoped-for gain, a step-0 gate failed, or the mechanism was falsified
- Detail
- closed (verdict NO: G is faster than T but not at parity. PARITY-FASTER failed because CI low −0.057 < −0.03, and BETTER failed)
- Set
- V-para (V-para); v0.1 (smoke 2 items)
- Judge
- Claude Opus, Claude Haiku (L1)
- n
- 51 judged ids (58 generated) per seed; slice 20 ids at stage 1
- Seeds
- 42, 43
- Pre-registered
- yes, sha256-frozen
- Reopen if
- none stated
Linked facts · 2
- R19 Gemma + thinking vs Qwen + thinking: 2 seeds OpusR19 Gemma + thinking vs Qwen + thinking: 2 seeds Opus−0.028 (−0.057…+0.003)Kinda record, not a measurement (set, status, judge, n, seeds, date: not recorded)Caveatwhy closed: parity-faster failedSourceresearch record
reports/CLOSED.md:38Open the data file - Gemma 4 26B + thinking (GEMMATHINK): faster deep model → −0.028 raw (parity clause failed)Gemma 4 26B + thinking (GEMMATHINK): faster deep model → −0.028 raw (parity clause failed)Kinda record, not a measurement (set, status, judge, n, seeds, date: not recorded)Sourcenot in the public repositories yetOpen the data file