The research · Act II, Pushing the engine · Chapter 04
Seven frozen builds
Every improvement that survived its frozen test went into a numbered, hashed build ("cut"), and seven cuts later the small model's quality on held-out data had not moved beyond noise; what improved was safety, speed and instant answers.
Snapshots, not tweaks
Every idea that passed its rule went into a numbered snapshot of the app, a frozen build (cut). There were seven. Cut 7 is the one we stand behind today.
What actually got better
Safety got better: a small detector spots emergencies and shows a short reviewed first-aid card instead of letting the model improvise. Instant answers got better: a calculator, a travel-distance estimator and a table of 4,233 facts answer in a blink. Phones got faster after we found our own bugs.
AI-reviewed, not clinically reviewed. Nothing here is medical advice.
What did not move
On the sealed questions, the small model's quality barely moved through all seven builds: from 0.506 to 0.512 on the flattering scale, a difference smaller than the noise between two identical runs.
v0.2 · Claude Opus · 110 questions · 2 runs
The system: a question's journey
Detector → instant tier → health path → retrieval → keep → reader
Every question takes a fixed path. A small learned emergency detector (an mmBERT-small encoder, q8_0, 106 MB, threshold 0.2) and hand-written rules look first. Some questions never reach a language model: emergency numbers, unit conversions, arithmetic, travel distances, nearby places, a 4,233-entity Wikidata fact table (761 KB), and health and first-aid questions, which get a reviewed card (32 conditions) or a verbatim cited excerpt of at most 700 characters. On the default tier the model never writes health advice. Everything else goes to retrieval: SQLite FTS5 full-text search over 1,000-character chunks plus 384-d bge-small article-lead embeddings over the installed knowledge packs (k 6). A keep step drops off-topic and unsafe passages and fits the rest into a 1,200-token context. The reader of the frozen claim build, Qwen3-4B-Instruct (Q4_K_M, 2.5 GB), writes a short cited answer. The deep tier swaps in Qwen3.6-35B-A3B (UD-Q2_K_XL, 12.3 GB file) with a 400-token context, 200 answer tokens and a 512-token thinking budget (desktop only; not delivered). From 2026-10-20 the main 8 GB reader is Qwen3-1.7B (Q4_K_M, ≈ 1.1 GB file) and the delivered app keeps one Succinct style (T14, T18).
Seven frozen builds
Cut 1 → cut 7: what each build added and which test let it in
We never shipped tweaks, only numbered, hashed builds (cuts) that bundle the arms that passed their frozen criteria. Cut 3 added a self-contradiction guard, an arithmetic tool and a snakebite text; cut 4 fixed our own KV-cache save path and blocked fragment instant answers; cut 5 skipped the route gate where safe (32/54 gate calls per seed, 120/120 identical answers); cut 6 made the detector score the window and the current turn; cut 7 added the Wikidata fact table. The fact table fires 0/110 on v0.2 and 0/58 on V-para, so cut 7's held-out numbers are cut 6's. Small-model quality on the held-out set did not move beyond noise across all of it: main 0.506 → cut 6 0.512 raw on v0.2 (two seeds, Opus): +0.006.
5 Systems: a frozen 4B build, a 35B deep tier, knowledge packs and a lite tier
The frozen claim build (cut 7) routes each question through a learned emergency detector and deterministic layers (calculator, unit and distance estimators, places, a 4,233-entity Wikidata table, first-aid cards and verbatim cited health excerpts); remaining questions are answered by Qwen3-4B-Instruct-2507 (Q4_K_M) from passages retrieved by SQLite FTS5 and 384-d bge-small lead vectors over offline packs (k = 6, 1,200-token context), after a keep step that removes off-topic and unsafe passages. The deep tier replaces the reader with Qwen3.6-35B-A3B (UD-Q2_K_XL; ≈ 3B active parameters per token; 400-token context; 200 answer tokens; thinking budget 512). Seven cuts were frozen between 2026-10-03 and 2026-10-07; cut 7 differs from cut 6 only by the fact table, which fires on 0/110 v0.2 and 0/58 V-para items, so cut 7's held-out numbers are cut 6's. Desktop runs execute the same TypeScript engine in Node with native components shimmed; desktop/phone parity is tested directly (§6.6). From 2026-10-20 the main 8 GB reader is Qwen3-1.7B (Q4_K_M) with refuse-by-default and a warning line (§6.8); the closed research engine ENGINE-RC (2026-10-23) adds the SEFIX-124 search fixes, a compact per-pack index (W78) and a shaped factbook (§6.10). The submitted system is a different implementation (§6.11).
Badge rows the 6 public fact rows this depth cites; a number here without a chip comes from a row that is not public yet (see /data/)
| Claim and value | Set | Status | Judge | n | Seeds | 95 % CI | Date |
|---|---|---|---|---|---|---|---|
| The fast-tier reader is Qwen3-4B-Instruct-2507 Q4_K_M.2,497,281,120 B (≈ 2.5 GB), Apache-2.0 | not recorded | not recorded | not recorded | not recorded | not recorded | 2026-10 | |
| Cut lineage.cut 1 → 1b → 2 → 3 (VG v3 + AR + snakebite) → 4 (KVFIX + INSTANT-FRAG + INSTANT-SUBJ) → 5 (KV no-source prefix + gate skip) → 6 (DETMAX) → 7 (FACTS r6) | not recorded | not recorded | not recorded | not recorded | not recorded | 2026-10-03..07 | |
| Health questions on the default tier never reach the 4B: they get a fixed card or a verbatim source excerpt (0 model tokens).modelId grounding-guard / extractive | not recorded | not recorded | not recorded | not recorded | not recorded | 2026-10-15 | |
| FACTS asset size.4,233 entities, 760,970 B | not recorded | not recorded | not recorded | not recorded | not recorded | 2026-10-06 | |
| Fast-tier constants in cut 7.retrieve k 6, context 1,200 tokens (lookup 3/450; other 4/700 quick budget), fast thinking budget 256 (unused by the instruct 4B) | not recorded | not recorded | not recorded | not recorded | not recorded | 2026-10-07 | |
| Deep-tier constants.retrieve k 6, context 400 tokens, max 200 answer tokens; thinking budget 1024 in code, 512 recommended | not recorded | not recorded | not recorded | not recorded | not recorded | 2026-10-10 |
In one sentence
Seven builds made the app safer and quicker, not smarter.
Turning point
10-06: the audit counts 15 answer-identical arms and 0 ratio movement on the small model; "stop polishing the 4B".
On the calendar
42 dated events · 1 Oct → 15 Oct
- 1 Octexp-06: the app's real answer engine runs on the desktop (Mac mini), 54 questions in 2 min 34 s
- 1 Oct · 21:30First frozen criterion before opening a blind set (embedding danger detector; sha frozen, "no tuning after this point")
- 2 Oct · 06:10Rubric-floor risk found: a blank system scores ~0.20–0.31 of the reference
- 3 OctGold-passage oracle: 4B + gold passages 0.606 vs engine 0.507
- 3 OctRetrieval v2 killed: gold-in-context 7 → 18 yet quality −0.021
All 42 events
- 2026-10-01exp-06: the app's real answer engine runs on the desktop (Mac mini), 54 questions in 2 min 34 s
- 2026-10-01 21:30First frozen criterion before opening a blind set (embedding danger detector; sha frozen, "no tuning after this point")
- 2026-10-01 21:30Embedding danger detector criterion frozen before its blind set exists (start of learned-detector line)
- 2026-10-01 23:40First 4B baseline on the shared engine: raw 0.521 (s42), "first point above 0.50, CI contains it"
- 2026-10-02Direction: "eval + pipeline technique" (better answers from the same model)
- 2026-10-02 02:15Danger detector + no-source card (A1) confirmed on both profiles; first safety win
- 2026-10-02 06:10Rubric-floor risk found: a blank system scores ~0.20–0.31 of the reference
- 2026-10-03Several health blind sets declared dev after traces sat in readable paths
- 2026-10-03Learned emergency gate generalizes on blind sets (90/90, 71/72); lexical does not
- 2026-10-03English only
- 2026-10-03Gold-passage oracle: 4B + gold passages 0.606 vs engine 0.507
- 2026-10-03Cut 1 / 1b built; cut 2 frozen (cff2f2bc…)
- 2026-10-03Retrieval v2 killed: gold-in-context 7 → 18 yet quality −0.021
- 2026-10-03Final cut 2 (cut 1b + exposure fix + citation fix)
- 2026-10-04Cut 3 adopted (cut 2 + verdict guard v3 + arithmetic repair + snakebite text); L1 Δ +0.029 [+0.007, +0.058]
- 2026-10-044B loss anatomy: cut 2 gains over old main come only from non-model paths (cards, calculator); "thin" and "knowledge" buckets have no 4B lever
- 2026-10-04Cut 3 confirmed: v0.1 EN 0.566 vs cut 2 0.534, Δ +0.032 CI90 [+0.010, +0.062]
- 2026-10-04Computed-verdict line (CV/CVS/CVH/CVO) and safety-card trigger v1–v3 all closed at gates
- 2026-10-04 01:03Overnight generation: cut 2 v0.1 EN ≈ 0.533 over 4 seeds
- 2026-10-05Phone latency root cause: our own KV-cache save-path bug (desktop shim hid it)
- 2026-10-05Claim runs on cut 3 approved and run: v0.2 held-out raw 0.513 (0.479–0.550); V-para 0.413 FAIL
- 2026-10-05First independent claim-text review: 21 issues
- 2026-10-06Cut 5 (speed-only), cut 6 (detector fix)
- 2026-10-06Claim/device build frozen at cut 6; later only routing/safety fixes
- 2026-10-06Cut 6 claim runs: v0.2 raw 0.512, V-para raw 0.413 / floor-adj 0.261
- 2026-10-06FACTS r6 on blind set 4: recall 70 % < 75 % → FAIL; kept as head by decision
- 2026-10-06Cut 5 = cut 4 + no-source KV prefix + route-gate skip (CUT5-LITE adopted; CUT5-SPEED rejected on desktop gate)
- 2026-10-06Cut 6 = cut 5 + DETMAX (detector scores window and current turn)
- 2026-10-06 14:06Pipeline audit: "rigorous process, looping direction"; honest metric adopted
- 2026-10-07Fine-tune data line closed (teacher 16/20 → +rejection 11/20 → human 6/20 unsupported)
- 2026-10-07 09:23Cut 7 = cut 6 + FACTS r6; v0.3 to be written outside the Claude family; "no device exists"
- 2026-10-07 14:26Autonomous research mandate
- 2026-10-09DT512: thinking budget 512 = 1024 quality at 0.55× time → recommended config
- 2026-10-10 13:30R&D process review: strong instrument, wrong target; deliverables unbuilt
- 2026-10-10 21:50Author's direction: research on the Mac mini only until the bar is met; device and claim work after
- 2026-10-12 23:4xKnowledge Pack catalogue and authoring workflow (D-K1..D-K7)
- 2026-10-14 22:31PR-1 (cut 7 rebased on upstream + chat wiring) engineering complete, review "mergeable as is"
- 2026-10-15 15:24MATHROUTE: math-through-the-model router does not recover the calculator (20/26 vs 23/26)
- 2026-10-15 19:5xNative iOS port started (SwiftUI + Swift port of the Kotlin engine)
- 2026-10-15 20:36NUMGUARD r1→r3: lexicon guard rejected as unsafe by review; calculator-only r3 PASS
- 2026-10-15 21:34Independent reviewer REJECTs a numeric guard as unsafe; scope cut to calculator-only
- 2026-10-15 22:51HEALTHGAP: 15 health probes reach the model unguarded; predicate v1.1/v1.2, implementation PASS