Decision ·
Thesis restated: small models need external knowledge; a harness built around a small model's real capabilities, tuned for the app user
State the thesis as: small models need external knowledge; we learned that and built a harness around a small model's real capabilities, tuned for the app user's experience; the conditional closing-test numbers are its evidence and honest limits
From the record
- Context
- decided by: author (publication decision 5, 2026-10-25)
- Why
- the closing tests support only the conditional form of the 2026-10-23 sentence; the restated thesis describes what was built and learned
- What followed
- replaces D-170 as the thesis statement; claims S40 and U41-U43 in content/claims-and-caveats.md
- Reversible
- yes (framing)
- Status
- kept
- Alternatives considered
- the conditional form as the thesis, the 2026-10-23 sentence quoted as the tested hypothesis
- the 2026-10-23 sentence as the main claim
Linked facts · 6
- The author restated the thesis on 2026-10-25: small models need external knowledge; the project learned that and built a harness around a small model's real capabilities, tuned for the app user's experience (health gate,…The author restated the thesis on 2026-10-25: small models need external knowledge; the project learned that and built a harness around a small model's real capabilities, tuned for the app user's experience (health gate, QuickMath, pack search and delivery, the Succinct persona, a fixed check-the-source line, refusal behaviour). It replaces the 2026-10-23 sentence "right whenever the content is in a pack" as the thesis statement; the conditional closing-test numbers are its evidence and its limitsthesis statement (author, 2026-10-25)Date2026-10-25Kinda record, not a measurement (set, status, judge, n, seeds: not recorded)Caveatwho: author; a framing statement, not a measurement. Evidence: with the matching pack the 1.7B is right on 48 % vs 26 % of pack-written self-tests; 65 % when the item's own article reaches the model vs 28 %. Limits: the article is delivered on 54 %; no pack reached the 0.75 bar; no gain on real development items. Never "the harness makes it half as good".Sourcenot in the public repositories yetOpen the data file
- PACKPROOF pooled correct-any with pack installed vs withoutPACKPROOF pooled correct-any with pack installed vs without0.48 vs 0.26, Δ +0.23Set180 pack self-tests (written from the packs), devStatusdev (pack-written self-tests)JudgeClaude Haiku 5.5-low labels; correct-anyn180Seeds4295 % CICI +0.16..+0.31; n 180; seed 42Date2026-10-23Caveatself-tests favour the pack; hosts differ by arm; 1 seed Metric: outcome-label shares of n items (Claude Haiku 5.5-low labels, UXMATRIX protocol); not the judge.v1 ratio.Sourcenot in the public repositories yetOpen the data file
- Correct-any when own article is kept vs notCorrect-any when own article is kept vs not0.65 (n 98) vs 0.28Set180 pack self-tests (written from the packs), devStatusdevJudgeClaude Haiku labelsn180Seedsnot recordedDate2026-10-23Caveatabout 2 in 3 when delivered'kept' = own article kept (article level), between-item split, n 98 kept.Sourcenot in the public repositories yetOpen the data file
- Own article delivered (kept)Own article delivered (kept)54 %; cited 28 %Set180 pack self-tests (written from the packs), devStatusdevJudge0-LLM funneln180Seedsnot recordedDate2026-10-23Caveatdelivery = bottleneck'kept' here is article-level (own article in the kept list), not the chunk-level delivery used for MAIN facts since 10-22; do not mix with the 42/92 chunk-level count.Sourcenot in the public repositories yetOpen the data file
- Packs reaching the 0.75 "proves" barPacks reaching the 0.75 "proves" bar0 of 9; 7 below 0.60 (not default-eligible by rule); best 0.60 (two Q&A packs), worst factbook 0.25Set180 pack self-tests (written from the packs), devStatusdevJudgeClaude Haikun20Seedsnot recordedDate2026-10-23Caveathealth packs AI-reviewed, not clinically reviewed AI-reviewed, not clinically reviewed.Sourcenot in the public repositories yetOpen the data file
- Pack self-test gain does not reproduce on real itemsPack self-test gain does not reproduce on real itemsself-test fact-added cell +0.139 (+0.075..+0.201) n 27 vs real −0.010 (−0.077..+0.050) n 10Setself-tests (pack-written) vs MAIN keyed (dev)Statusdev (pack-written self-tests)JudgeClaude Haikunnot recordedSeeds1Date2026-10-21Caveatpost hoc; small n on realSourcenot in the public repositories yetOpen the data file