The research · Act IV, From the bar to a product · Chapter 14
Closing the research: what a small model needs
On 2026-10-23 the author closed the research lines; on 2026-10-25 he stated the thesis the record presents: small models need external knowledge, and the project, having learned that, built a harness around a small model's real capabilities, tuned for the app user's experience (a health gate, QuickMath, pack search and delivery, the Succinct persona, a fixed check-the-source line, refusal behaviour) The author restated the thesis on 2026-10-25: small models need external knowledge; the project learned that and built a harness around a small model's real capabilities, tuned for the app user's experience (health gate, QuickMath, pack search and delivery, the Succinct persona, a fixed check-the-source line, refusal behaviour). It replaces the 2026-10-23 sentence "right whenever the content is in a pack" as the thesis statement; the conditional closing-test numbers are its evidence and its limitsthesis statement (author, 2026-10-25)Date2026-10-25Kinda record, not a measurement (set, status, judge, n, seeds: not recorded)Caveatwho: author; a framing statement, not a measurement. Evidence: with the matching pack the 1.7B is right on 48 % vs 26 % of pack-written self-tests; 65 % when the item's own article reaches the model vs 28 %. Limits: the article is delivered on 54 %; no pack reached the 0.75 bar; no gain on real development items. Never "the harness makes it half as good".Sourcenot in the public repositories yetOpen the data file.
Full thesis
The closing tests are its evidence and its honest limits: with the matching pack the 1.7B is right on 48 % of pack-written self-tests vs 26 % without, and on 65 % when the item's own article reaches the model vs 28 % otherwise; but the article arrives only 54 % of the time, no pack reached the 0.75 bar fixed in advance, and the gain did not transfer to real development items. The sentence the author first wrote on 2026-10-23, "right whenever the content is in a pack", is kept as the hypothesis those tests checked: they support it only in that conditional form. The afternoon turned to content (reshaping packs), the night to validating one closed engine, and the next day's judge kept the shortest answer style.
The model is one station of seven
- Routerreads the question
- Calculatorsums and unit conversions in code, no model
- Pack searchlooks through the libraries on the phone
- Deliverybottleneckthe right page gets through on 54 % of questions
- Health gatecan swap in a fixed safe text
- Small modelQwen3-1.7B writes from the page
- Answerwith its sources and “May contain errors; check the source.”
Knowledge packs
The pack helps when its article reaches the model, and search delivers it on about half the questions
Share of answers judged correct on 180 questions written from the nine packs, with and without each condition. Questions written from the packs favour the packs; the right-hand panel shows what happened on real practice questions.
Data table
| measure | value |
|---|---|
| Correct, pack installed vs not (180 pack-written) | 48 % vs 26 % |
| Correct, own article delivered vs not | 65 % vs 28 % |
| Own article delivered | 54 % |
| Packs at the 0.75 mark | 0 of 9; best 0.60, worst 0.25 (factbook) |
| fact already in the pack: pack-written vs real | +0.139 vs −0.010 |
| in-pack split: pack-written vs real | +0.056 vs −0.023 |
Research closed, a sentence tested
On 23 October the author closed the research and wrote down what he expected: "the app is right whenever the answer is in one of its libraries". Then we tested that sentence.
Right two times in three, when the page arrives
On each library's own questions, the small model was right 48 % of the time with the library installed and 26 % without. When search delivered the right article, it was right 65 % of the time, but search delivered it only about half the time. No library reached the 75 % mark we had fixed in advance. PACKPROOF pooled correct-any with pack installed vs without0.48 vs 0.26, Δ +0.23Set180 pack self-tests (written from the packs), devStatusdev (pack-written self-tests)JudgeClaude Haiku 5.5-low labels; correct-anyn180Seeds4295 % CICI +0.16..+0.31; n 180; seed 42Date2026-10-23Caveatself-tests favour the pack; hosts differ by arm; 1 seed Metric: outcome-label shares of n items (Claude Haiku 5.5-low labels, UXMATRIX protocol); not the judge.v1 ratio.Sourcenot in the public repositories yetOpen the data file
What we learned, and what we built
The tests support that first sentence only in a narrower form: right about two times in three when the right page reaches the model. On real questions, the libraries did not help in a measurable way. What the author takes from it, and states as the thesis (25 October 2026): small models need outside knowledge, so the app is a harness built around what a small model can really do: a safe text instead of a guess on health, sums done by code, a search that brings the right page, one short answer style, a line asking you to check the source, and a fixed refusal text where it should not guess. The numbers above are its evidence and its limits; they do not make the app half as good as the frontier. The author restated the thesis on 2026-10-25: small models need external knowledge; the project learned that and built a harness around a small model's real capabilities, tuned for the app user's experience (health gate, QuickMath, pack search and delivery, the Succinct persona, a fixed check-the-source line, refusal behaviour). It replaces the 2026-10-23 sentence "right whenever the content is in a pack" as the thesis statement; the conditional closing-test numbers are its evidence and its limitsthesis statement (author, 2026-10-25)Date2026-10-25Kinda record, not a measurement (set, status, judge, n, seeds: not recorded)Caveatwho: author; a framing statement, not a measurement. Evidence: with the matching pack the 1.7B is right on 48 % vs 26 % of pack-written self-tests; 65 % when the item's own article reaches the model vs 28 %. Limits: the article is delivered on 54 %; no pack reached the 0.75 bar; no gain on real development items. Never "the harness makes it half as good".Sourcenot in the public repositories yetOpen the data file
Short answers win
Longer answer styles were right just as often as the short one, but shown wrong without a warning about twice as often. The short style stayed. Succinct on ENGINE-RC (page numbers)ON CWS 0.30, correct-any 0.43, WU 0.12; OFF correct-any 0.22, WU 0.10SetUXMATRIX 60, devStatusdevJudgeClaude Haiku labelsn60Seeds42Date2026-10-24CaveatPASS vs frozen CWS ≥0.28, WU OFF ≤0.10 Metric: outcome-label shares of n items (Claude Haiku 5.5-low labels, UXMATRIX protocol); not the judge.v1 ratio.Sourcenot in the public repositories yetOpen the data file
Closing tests and the engine release candidate
Acceptance tests on a desktop twin, one closed engine, a judge pass the next morning
After the author closed the research lines (2026-10-23 11:53), the remaining work was acceptance tests on the app's desktop twin, scored with outcome labels (Claude Haiku 5.5-low; correct-any = right with the right source shown + right from memory; WU = wrong with no warning). The pack test (180 pack-written self-tests of 9 packs, 1.7B Succinct, seed 42): correct-any 0.48 with the pack vs 0.26 without, 0.65 when the item's own article is kept (54 % of items), 0 of 9 packs at 0.75. The answer-style test (60 dev items): Succinct beats Detailed with and without the pack; with the pack off every honest "unknown" is an engine decline. Pack shaping (factbook rewritten as short sentences that name the country) doubled fact delivery (7 → 15 of 20) and correct answers (5 → 10) on its own self-tests, but missed its frozen rule by one new warned confident error. Overnight one closed tree, the engine release candidate (the search fixes + shaped factbook + a reworded banner), passed every 0-LLM suite (identity 97/97 and 180/180, routes 103/110, the one letter-level failure pre-existing) and generated 2,855 answers with no judge call; the next day's judge kept Succinct (correct-any 0.43 vs 0.42 for longer styles, wrong-unwarned 0.12 vs 0.23–0.25).
Thesis box
"Small models need external knowledge. We learned that, and built a harness around a small model's real capabilities, tuned for the app user's experience." The harness in the shipped app: the health gate (fixed safe text when no health source is kept), QuickMath (conversions and arithmetic in code), pack search and delivery (the ported bench search, off on health turns), the Succinct persona, the fixed line "May contain errors; check the source." and fixed-text refusals (T18). Evidence: 48 % vs 26 % with the matching pack and 65 % vs 28 % when the article reaches the model (pack-written self-tests, Haiku labels, one seed). Limits: delivered on 54 %; 0 of 9 packs at 0.75; no gain on real dev items Pack self-test gain does not reproduce on real itemsself-test fact-added cell +0.139 (+0.075..+0.201) n 27 vs real −0.010 (−0.077..+0.050) n 10Setself-tests (pack-written) vs MAIN keyed (dev)Statusdev (pack-written self-tests)JudgeClaude Haikunnot recordedSeeds1Date2026-10-21Caveatpost hoc; small n on realSourcenot in the public repositories yetOpen the data file; the app's desktop-twin ruler reading is below the bar (T15). The 2026-10-23 sentence ("right whenever the content is in a pack") is the hypothesis these tests checked. Never "the harness makes it half as good".
6.10 Small models need external knowledge: what the closing tests support
The author closed the research lines on 2026-10-23 with the hypothesis "right whenever the content is in a pack"; on 2026-10-25 he stated the thesis as a design statement: small models need external knowledge, and the project built a harness around a small model's real capabilities, tuned for the app user's experience The author restated the thesis on 2026-10-25: small models need external knowledge; the project learned that and built a harness around a small model's real capabilities, tuned for the app user's experience (health gate, QuickMath, pack search and delivery, the Succinct persona, a fixed check-the-source line, refusal behaviour). It replaces the 2026-10-23 sentence "right whenever the content is in a pack" as the thesis statement; the conditional closing-test numbers are its evidence and its limitsthesis statement (author, 2026-10-25)Date2026-10-25Kinda record, not a measurement (set, status, judge, n, seeds: not recorded)Caveatwho: author; a framing statement, not a measurement. Evidence: with the matching pack the 1.7B is right on 48 % vs 26 % of pack-written self-tests; 65 % when the item's own article reaches the model vs 28 %. Limits: the article is delivered on 54 %; no pack reached the 0.75 bar; no gain on real development items. Never "the harness makes it half as good".Sourcenot in the public repositories yetOpen the data file. The numbers below are its evidence and its limits; they do not establish a quality level. Closing tests ran on the desktop twin with outcome labels (Claude Haiku 5.5-low; §2.13 of the claims page): on 180 pack-written self-tests of 9 packs (1.7B Succinct, seed 42), correct-any is 0.48 with the pack vs 0.26 without (+0.23, +0.16…+0.31); 0.65 when the item's own article is kept (n 98) vs 0.28; the own article is kept on 54 % and cited on 28 % (article level); 0 of 9 packs reach the 0.75 bar fixed in advance PACKPROOF pooled correct-any with pack installed vs without0.48 vs 0.26, Δ +0.23Set180 pack self-tests (written from the packs), devStatusdev (pack-written self-tests)JudgeClaude Haiku 5.5-low labels; correct-anyn180Seeds4295 % CICI +0.16..+0.31; n 180; seed 42Date2026-10-23Caveatself-tests favour the pack; hosts differ by arm; 1 seed Metric: outcome-label shares of n items (Claude Haiku 5.5-low labels, UXMATRIX protocol); not the judge.v1 ratio.Sourcenot in the public repositories yetOpen the data file. The hypothesis therefore holds only in the conditional form, and on real development items the pack gain does not transfer (fact-added cell −0.010, −0.077…+0.050, n 10, vs +0.139 on self-tests) Pack self-test gain does not reproduce on real itemsself-test fact-added cell +0.139 (+0.075..+0.201) n 27 vs real −0.010 (−0.077..+0.050) n 10Setself-tests (pack-written) vs MAIN keyed (dev)Statusdev (pack-written self-tests)JudgeClaude Haikunnot recordedSeeds1Date2026-10-21Caveatpost hoc; small n on realSourcenot in the public repositories yetOpen the data file. Pack shaping (factbook) doubled delivery and correct answers on 20 self-tests and moved Opus U 0.316 → 0.436 (+0.121, +0.030…+0.221), NOT-CONFIRMED by one new warned confident error Shaped factbookfact chunk kept 7→15/20; correct 5→10/20; Opus U 0.316→0.436 (+0.121, +0.030..+0.221)Setfactbook self-tests (pack-written)Statusdev (pack-written self-tests)JudgeClaude Haiku; Claude Opusnnot recordedSeeds2Date2026-10-23CaveatNOT-CONFIRMED (1 new warned CE); does not transfer to the native engineSourcenot in the public repositories yetOpen the data file. The closed engine passed every 0-LLM validation suite and cut data read per question 125.09 → 21.26 MB ENGINE-RC passes the 0-LLM validation suitesidentity 97/97 & 180/180; MAIN delivered 42; ST fact 84 vs 77; routes 103/110; 21.26 MB/qSetdevStatusdevJudgereplaynnot recordedSeedsnot recordedDate2026-10-23CaveatV1 letter-FAIL pre-existingSourcenot in the public repositories yetOpen the data file. The 2026-10-24 validation judge kept Succinct: correct-any 0.43 vs 0.42 for longer styles, wrong-unwarned 0.12 vs 0.23–0.25 (UXMATRIX 60, seed 42) Succinct on ENGINE-RC (page numbers)ON CWS 0.30, correct-any 0.43, WU 0.12; OFF correct-any 0.22, WU 0.10SetUXMATRIX 60, devStatusdevJudgeClaude Haiku labelsn60Seeds42Date2026-10-24CaveatPASS vs frozen CWS ≥0.28, WU OFF ≤0.10 Metric: outcome-label shares of n items (Claude Haiku 5.5-low labels, UXMATRIX protocol); not the judge.v1 ratio.Sourcenot in the public repositories yetOpen the data file; the Succinct reading moved from 0.35 on 10-23 to 0.43 on 10-24 on the same items after engine changes, with no isolated cause. Subject carry restored the turn-1 source on follow-ups (0.083 → 0.708) without changing accuracy (0.458) Follow-ups on ENGINE-RC, with and without subject carrycorrect-any 0.458 both; on-referent 1.00; turn-1 source 0.083→0.708; WU 2→4 of 48SetF8 devStatusdevJudgeClaude Haikun48Seeds42Date2026-10-24Caveathealth follow-ups 0.19 both Metric: outcome-label shares of n items (Claude Haiku 5.5-low labels, UXMATRIX protocol); not the judge.v1 ratio. AI-reviewed, not clinically reviewed.Sourcenot in the public repositories yetOpen the data file.
Badge rows the 10 public fact rows this depth cites; a number here without a chip comes from a row that is not public yet (see /data/)
| Claim and value | Set | Status | Judge | n | Seeds | 95 % CI | Date |
|---|---|---|---|---|---|---|---|
| The author restated the thesis on 2026-10-25: small models need external knowledge; the project learned that and built a harness around a small model's real capabilities, tuned for the app user's experience (health gate, QuickMath, pack search and delivery, the Succinct persona, a fixed check-the-source line, refusal behaviour). It replaces the 2026-10-23 sentence "right whenever the content is in a pack" as the thesis statement; the conditional closing-test numbers are its evidence and its limitsthesis statement (author, 2026-10-25) | not recorded | not recorded | not recorded | not recorded | not recorded | 2026-10-25 | |
| PACKPROOF pooled correct-any with pack installed vs without0.48 vs 0.26, Δ +0.23 | 180 pack self-tests (written from the packs), dev | dev (pack-written self-tests) | Claude Haiku 5.5-low labels; correct-any | 180 | 42 | CI +0.16..+0.31; n 180; seed 42 | 2026-10-23 |
| Pack self-test gain does not reproduce on real itemsself-test fact-added cell +0.139 (+0.075..+0.201) n 27 vs real −0.010 (−0.077..+0.050) n 10 | self-tests (pack-written) vs MAIN keyed (dev) | dev (pack-written self-tests) | Claude Haiku | not recorded | 1 | 2026-10-21 | |
| Shaped factbookfact chunk kept 7→15/20; correct 5→10/20; Opus U 0.316→0.436 (+0.121, +0.030..+0.221) | factbook self-tests (pack-written) | dev (pack-written self-tests) | Claude Haiku; Claude Opus | not recorded | 2 | 2026-10-23 | |
| ENGINE-RC passes the 0-LLM validation suitesidentity 97/97 & 180/180; MAIN delivered 42; ST fact 84 vs 77; routes 103/110; 21.26 MB/q | dev | dev | replay | not recorded | not recorded | 2026-10-23 | |
| Succinct on ENGINE-RC (page numbers)ON CWS 0.30, correct-any 0.43, WU 0.12; OFF correct-any 0.22, WU 0.10 | UXMATRIX 60, dev | dev | Claude Haiku labels | 60 | 42 | 2026-10-24 | |
| Follow-ups on ENGINE-RC, with and without subject carrycorrect-any 0.458 both; on-referent 1.00; turn-1 source 0.083→0.708; WU 2→4 of 48 | F8 dev | dev | Claude Haiku | 48 | 42 | 2026-10-24 | |
| Without the pack: wrong-confident / wrong-warned / honest-unknown0.17 / 0.45 / 0.12; refused-wrongly 0 | 180 pack self-tests (written from the packs), dev | dev | Claude Haiku | 180 | not recorded | 2026-10-23 | |
| ENGINE-RC reading cost (S8)21.26 MB/question vs 125.09; ctx p50 0.978x; disk 1.06x | not recorded | not recorded | 0-LLM | not recorded | not recorded | 2026-10-23 | |
| S+ / two-paragraph personascorrect-any ON 0.42 / 0.42; WU ON 0.23 / 0.25 (vs 0.12) | UXMATRIX 60, dev | dev | Claude Haiku | 60 | 42 | 2026-10-24 |
In one sentence
A small model needs a library: right about two times in three when the page arrives; the page arrives about half the time.
Turning point
2026-10-23 11:53, the author closes the research lines; at 12:19 a first thesis is stated, and by 14:2x the tests have reduced it to its conditional form; on 2026-10-25 the author restates the thesis as what was learned and built, with those numbers as its evidence and limits.
On the calendar
33 dated events · 18 Oct → 24 Oct
- 23 Oct · 11:53Author closes research lines; CLOSING-TESTS plan
- 23 Oct · 12:19Author states the thesis ("right whenever the content is in a pack"); PACKPROOF launched
- 23 Oct · 13:20UXMATRIX result
- 23 Oct · 14:19PACKPROOF result (48 % vs 26 %; 65 % when delivered)
- 23 Oct · 14:39PACK-GAP: 44/50 gaps are search, not missing packs
All 33 events
- 2026-10-18 05:02HONESTPROMPT verdict: GO for Detailed honest (D1), NO for Succinct-honest (S1)
- 2026-10-21 16:01A1-SUCCINCT17: Succinct as small-model default saves ~24 s p50 (projection)
- 2026-10-21 17:02G1-SWA5: Gemma multi-turn KV bug hypothesis killed on the phone
- 2026-10-23 00:11Night 10-23 program launched (kit for general questions, research 2x2 table)
- 2026-10-23 00:45Detector retrains (FT, FT2) rejected: lose health recall
- 2026-10-23 02:14W98 cooking/money veto on the health route closed NO (silences real health probes)
- 2026-10-23 09:13Engine fixes branch (CO card more vehicles, linear arithmetic, wider seed-phrase cue; CO hazard in PT/ES)
- 2026-10-23 09:19Facts table r11 (currency + Nobel builder root causes fixed)
- 2026-10-23 09:23False "Not found" banner traced to a fixed release line; bannerkept patch written
- 2026-10-23 11:00PARA135: answers in 1/3/5 paragraphs; support breaks at paragraph 2
- 2026-10-23 11:34UX decision page (25 feature rows) written
- 2026-10-23 11:41DEEPMODE: Deep Research multipass on 1.7B judged worse than Detailed and Succinct, 2.6-5.6x slower
- 2026-10-23 11:53Author closes research lines; CLOSING-TESTS plan
- 2026-10-23 11:58Author: close with experience tests, no phone; UXMATRIX launched
- 2026-10-23 12:19Author states the thesis ("right whenever the content is in a pack"); PACKPROOF launched
- 2026-10-23 12:20PLUGBATCH-SELF (route 85-89/110) → PLUGFIX (route 103/110, 1,645 tests)
- 2026-10-23 13:20UXMATRIX result
- 2026-10-23 13:27WARN-KEPT NO
- 2026-10-23 14:19PACKPROOF result (48 % vs 26 %; 65 % when delivered)
- 2026-10-23 14:24Closing tests declared complete; pages republished
- 2026-10-23 14:39PACK-GAP: 44/50 gaps are search, not missing packs
- 2026-10-23 14:58NOWIKI KEEP (Wikipedia packs stay)
- 2026-10-23 15:20QREWRITE NO-CEILING; "research closed" again
- 2026-10-23 15:39Author: CONTENT-FIRST, no model change; ARTCAP, D2Q, PACK-SHAPE launched
- 2026-10-23 17:07FACTBOOK-V2-CONFIRM (Opus) NOT-CONFIRMED by CE clause; v2 card drafted, upload waits
- 2026-10-23 17:44GOAL/HANDOFF written; session restart kills two agents; NIGHT-2026-10-24 program written
- 2026-10-23 18:21Validation criterion frozen (aa08650f); ENGINE-RC build
- 2026-10-23 18:50ENGINE-RC V0 identity PASS; V1 letter-FAIL (pre-existing); V3-L1, S1, S8 PASS; RETBENCH candidate closed by probes
- 2026-10-23 18:51S3 routes PASS on the engine-fix branch
- 2026-10-23 23:56VALIDATION-2026-10-24 written: 2,855 answers, 0 judge calls, judge deferred to author
- 2026-10-24 09:02Morning check; one machine unreachable, nothing lost
- 2026-10-24 14:02Author OKs judge runs (Haiku, length + follow-up)
- 2026-10-24 14:15Judge: Succinct PASS and stays; follow-ups 0.46 correct with or without carry
Related nodes on the map
This chapter on the map →Decisions
- D7: upload the research site only as v1.0 at…2026-10-18
- Close the research lines; no new arms after PLUGBATCH-SELF…2026-10-23
- Thesis to present with the app: right whenever the…2026-10-23
- CONTENT-FIRST: no model change; make the right chunk reach…2026-10-23
- Closed simple chat app for one way of use…2026-10-24
- Thesis restated: small models need external knowledge; a harness…2026-10-25