The research · Act II, Pushing the engine · Chapter 05
The walls
Oracle tests showed where the small model's ceiling was: handing it the perfect passage helps, but getting that passage, and knowing what isn't in the kit, was the wall; bigger dense models, better prompts and better retrieval variants all failed.
The cemetery of ideas
Most ideas died here: 215 of 387 ended without a gain (187 closed, 28 too small to see), 57 adopted
Every experiment in the open registry, one mark each, placed on the day it was decided and filed by verdict. Point at, tap or focus a mark for its name and recorded result.
Data table
| verdict | meaning | count |
|---|---|---|
| Adopted | shipped or recommended | 57 |
| Parity | same quality, cheaper | 10 |
| Measurement | measured a property | 57 |
| Tooling | method or infrastructure | 32 |
| Open | parked or running | 16 |
| Not detected | too small to see; not a no | 28 |
| Closed | the idea failed its rule | 187 |
| (undated, not plotted) | 2 |
Hand it the right page
We tried cheating on purpose: give the small model the exact passage that holds the answer. Its score rose from 0.507 to 0.606 on the practice set. So the reader can read. The question is whether the right page ever reaches it. Oracle ladder: closed-book 0.487 · engine 0.507 · question-chosen passages 0.567 · gold passages 0.606 · 1500-token gold 0.626 · Haiku no-web 0.653see valueSetv0.1 EN devStatusdevJudgeOpusnnot recordedSeedsnot recordedDate2026-10-03Caveatgold passages chosen with knowledge of the reference → upper boundSourcenot in the public repositories yetOpen the data file
The library doesn't have the book
Often it doesn't, because the downloaded libraries simply don't contain the fact. A smarter search that pulled the right sentence into the prompt more than twice as often still did not raise the score.
Bigger small models, longer answers: no
Swapping in other models of similar size did not help. Asking for longer answers made the model invent more.
An audit, and a burnt exam
On 6 October an internal audit said the process was rigorous but going in circles. The same day we found that one sealed exam's questions had sat in a private code history the builders could have seen. We declared it burnt: from then on it counts as practice.
What did not work for the 4B (and why)
Retrieval, prompts and bigger small models: the walls
Oracle runs set the ceiling. On v0.1 (dev, Opus, 1 seed, n 57): closed-book 0.487, engine 0.507, question-chosen passages 0.567, gold passages 0.606, Claude Haiku with no web 0.653. A retrieval rewrite put the gold sentence in context 7 → 18 times out of 56 and still lost 0.021. A Wikipedia top-100k pack bought nothing for the 4B and made retrieval 3.3× slower. Recall@5 was 0.333 and coverage was the largest failure bucket (49 %). Other 4–9B models were no better closed-book (Qwen3.5-9B 0.488 vs Qwen3-4B 0.487). Prompts that asked for longer or more complete answers raised confident-wrong answers. The lesson for builders: on this hardware the small model's reading is not the bottleneck; knowledge the kit does not hold is.
§6.4 The small model's walls (v0.1 dev)
Oracle runs on v0.1 (n = 57, 1 seed, Opus, raw) bracket the 4B: closed-book 0.487, engine 0.507, question-chosen passages 0.567, gold passages 0.606, Claude Haiku without web 0.653. The 35B reads the same gold passages only slightly better (0.634, +0.029, −0.004…+0.063), so its advantage is stored knowledge. A retrieval rewrite increased gold-sentence presence 7 → 18 of 56 yet lowered quality (−0.021); 25–35B MoEs form a closed-book plateau at ≈ 0.60 (0.596–0.613); 4–9B alternatives do not beat Qwen3-4B (0.437–0.495 vs 0.487).
8 A registry of what did not work
We keep a public registry of 76 lever families (61 closed, 13 not detected, 2 measurements; a private registry continues to 98 rows) with, for each: what was tried, set and judge, effect with CI, minimum detectable effect where the test had one, and the condition that would reopen it. Families include larger closed-book models, prompt and persona changes, sampling presets, best-of-2 and self-verification, routing between tiers, re-ranking, query rewriting, corpus size, Wikipedia variants, computed verdicts, fine-tuning data, speculative decoding and quantization. Five closures based on n ≈ 20 slices with CI upper bounds ≥ +0.03 are relabelled here as "not detected at MDE 0.08" (R31, R83, R84, R85, R97), following an outside critique.
Badge rows the 7 public fact rows this depth cites; a number here without a chip comes from a row that is not public yet (see /data/)
| Claim and value | Set | Status | Judge | n | Seeds | 95 % CI | Date |
|---|---|---|---|---|---|---|---|
| Oracle ladder: closed-book 0.487 · engine 0.507 · question-chosen passages 0.567 · gold passages 0.606 · 1500-token gold 0.626 · Haiku no-web 0.653see value | v0.1 EN dev | dev | Opus | not recorded | not recorded | 2026-10-03 | |
| Retrieval v2 put the gold sentence in context 7 → 18/56 times but lowered qualityΔ −0.021 | v0.1 EN dev | dev | Opus | not recorded | 2 | CI90 −0.054…+0.009, 2 seeds | 2026-10-03 |
| Mixture-of-experts models in the 25–35B class form one closed-book plateau, well above the 4B.closed-book raw: Gemma 4 26B-A4B 0.613; Qwen3.5-35B-A3B 0.603; Qwen3.6-35B-A3B 0.596; vs 4B 0.487 | v0.1 | public dev | Claude Opus (judge.v1, both orders) | 57 | 1 | 2026-10-07 | |
| Deep model reads gold passages like the 4BR450 0.634; paired vs 4B +0.029 (−0.004…+0.063) | v0.1 dev | dev | Opus | 57 | 1 | 2026-10-06 | |
| Bigger closed-book models do not help the 4B tierQwen3.5-4B 0.437, Gemma E4B 0.495, Qwen3.5-9B 0.488, Ministral 8B 0.457 vs 0.487 | v0.1 dev | dev | Opus | 57 | 1 | 2026-10-05/06 | |
| Registry size98 rows private (R1–R98); 76 rows public | not recorded | not recorded | not recorded | not recorded | not recorded | 2026-10-16 | |
| Closed-family registry sizepublic 76 rows (61 closed, 13 not detected, 2 measurement); private 98 rows by 10-16 | not recorded | not recorded | not recorded | not recorded | not recorded | 2026-10-15/16 |
In one sentence
The small model reads well enough; what it lacks is knowledge the phone doesn't hold.
Turning point
2026-10-06 14:06, the pipeline audit; same day v0.2 declared burnt because its plaintext sat in a private repo's git history.
On the calendar
12 dated events · 1 Oct → 17 Oct
- 1 OctFirst frozen criteria on fresh sets (hm-fresh2/3) before any reading
- 2 Oct · 01:00Two-seed baseline: 4B raw 0.512 (0.461–0.567), 1.5B 0.379
- 2 Oct · 04:30Longer-answer persona makes the 4B hallucinate (Δ −0.044, confident-wrong 10 → 20)
- 4 OctORACLE: reader is not the bottleneck; retrieval is (+0.10 ceiling)
- 17 Oct · 20:52Claim v4.2.3 / v4.2.4: wording as of 2026-10-17 (the posted claim follows the standing at closure); submitted app ports retrieval and prompts, not the health path or deterministic answers
All 12 events
- 2026-10-01First frozen criteria on fresh sets (hm-fresh2/3) before any reading
- 2026-10-02 01:00Two-seed baseline: 4B raw 0.512 (0.461–0.567), 1.5B 0.379
- 2026-10-02 04:30Longer-answer persona makes the 4B hallucinate (Δ −0.044, confident-wrong 10 → 20)
- 2026-10-044B loss anatomy: gap to Haiku +0.121 net; biggest buckets have no live lever
- 2026-10-04ORACLE: reader is not the bottleneck; retrieval is (+0.10 ceiling)
- 2026-10-05Model line closed for the 4B tier (Qwen3.5-4B, Gemma 4 E4B, 9B, 8B)
- 2026-10-06Retrieval scorecard: coverage is the largest failure bucket (49 %); recall@5 0.333
- 2026-10-09 06:35GEMMATHINK: Gemma 27 % faster but −0.028 raw → NO
- 2026-10-13Determinism rule (SEEDGEN): pair runs on prompt identity; the desktop 4B path is byte-deterministic
- 2026-10-13 23:28RIVALSET: on a competitor team's public 72 questions (our port of their rubric) 4B 3.12 vs competitor 5.53 vs Haiku 7.17
- 2026-10-14 10:56SFT TEACHER-PROBE NO-GO (deep teacher 6/120 faithful); fine-tuning line closed
- 2026-10-17 20:52Claim v4.2.3 / v4.2.4: wording as of 2026-10-17 (the posted claim follows the standing at closure); submitted app ports retrieval and prompts, not the health path or deterministic answers