RatLabA Boar Research
Closed recordas of what’s still open →

The research · Act IV, From the bar to a product · Chapter 13

Delivery, not coverage

With the 1.7B as main engine, the question became whether the one passage that holds the answer reaches the model. On questions written from the packs it often does, and answers jump; on real development questions the pack gain disappears, and almost every undelivered fact already sits in an installed pack. Search fixes delivered more facts; no answer-level gain was established, and one fix was withdrawn as unsafe.

2026-10-19 → 10-23← In the article: §5 What worked: the harness

The page reached the model

65right of every 100

It did not

28right of every 100

Search delivered the page54 %of the time

Same small model, same 180 questions written from its own library. Correct-any when own article is kept vs not0.65 (n 98) vs 0.28Set180 pack self-tests (written from the packs), devStatusdevJudgeClaude Haiku labelsn180Seeds1Date2026-10-23Caveatabout 2 in 3 when delivered'kept' = own article kept (article level), between-item split, n 98 kept.Sourcenot in the public repositories yetOpen the data file Own article delivered (kept)54 %; cited 28 %Set180 pack self-tests (written from the packs), devStatusdevJudge0-LLM funneln180Seedsnot recordedDate2026-10-23Caveatdelivery = bottleneck'kept' here is article-level (own article in the kept list), not the chunk-level delivery used for MAIN facts since 10-22; do not mix with the 42/92 chunk-level count.Sourcenot in the public repositories yetOpen the data file

Big gains on the library's own questions

On test questions written from a library's own pages, installing that library makes a big difference. On real practice questions, the same library made no measurable difference. Pack self-test gain does not reproduce on real itemsself-test fact-added cell +0.139 (+0.075..+0.201) n 27 vs real −0.010 (−0.077..+0.050) n 10Setself-tests (pack-written) vs MAIN keyed (dev)Statusdev (pack-written self-tests)JudgeClaude Haikunnot recordedSeeds1Date2026-10-21Caveatpost hoc; small n on realSourcenot in the public repositories yetOpen the data file

How you ask decides what it finds

Ask a yes/no question and the right page is found 83 % of the time; ask a bare follow-up like "and how long does it take?" and it is found 8 % of the time. Question form decides retrievalhit@kept base 50 %, keywords 62 %, yes/no 83 %, entity-free 10 %, situational 25 %, follow-up 8 %Setdev, agent-writtenStatusdevJudgereplaynnot recordedSeedsnot recordedDate2026-10-21Caveatfollow-up hole is a bench-engine artefactSourcenot in the public repositories yetOpen the data file

The book is on the shelf

Of 50 facts the app failed to bring to the model, 44 were already in a library on the phone. The problem was not missing knowledge; it was fetching it.

Better fetching, one fix withdrawn

Search fixes doubled how often the right passage arrived (21 → 42 of 92 facts). One more fix did better still, but it let unsafe health answers through on test prompts, so we withdrew it. Answers did not measurably improve.

AI-reviewed, not clinically reviewed.

Closed record · 1.0

Final · as of 2026-10-25 16:46 · 0 of 71 wins on the phone, U-total 0.165 (Claude) / 0.242 (gpt-5.6-sol), below the 0.50 bar

Delivery: getting the fact-bearing chunk to a 1.7B

Count delivery at chunk level; most misses are search, not missing content

We count a fact as delivered only when its quote sits inside one kept chunk; article-level counting had overstated delivery by 8–13 facts. On 92 keyed dev items, delivered facts went from 21 (no fix) to 28 with a global-IDF merge (word rarity computed across all installed packs, so small packs compete fairly; +6.2 points hit@kept, 0 lost), to 42 with three Stack Exchange search fixes (the search fixes), and to 47 with a fourth fix (F3, a health no-source fallback) that fired on 21 of 65 safety probes and raised unsafe answers from 9 to 13, so it was withdrawn. Answer-level gains stayed within noise: the search fixes +0.013 (−0.008…+0.035) on all 97 (Haiku, two seeds). Of the 50 facts still undelivered, 44 already sit in the default 13-pack kit. A compact per-pack index (W78) cut the data read per question from 125.09 to 21.26 MB and the cold first stage from 620 to 92 ms on a desktop replay.

Closed record · 1.0

Final · as of 2026-10-25 16:46 · 0 of 71 wins on the phone, U-total 0.165 (Claude) / 0.242 (gpt-5.6-sol), below the 0.50 bar

6.9 Self-test gains do not transfer; delivery does not separate answers

On 60 topical pack self-tests the pack gain concentrates on items where the pack adds the key fact (+0.139, +0.075…+0.201, n 27); on real development items the same cell is −0.010 (−0.077…+0.050, n 10), and the self-test gain is not a lexical-overlap artefact (+0.064 after reweighting to the real overlap distribution) Pack self-test gain does not reproduce on real itemsself-test fact-added cell +0.139 (+0.075..+0.201) n 27 vs real −0.010 (−0.077..+0.050) n 10Setself-tests (pack-written) vs MAIN keyed (dev)Statusdev (pack-written self-tests)JudgeClaude Haikunnot recordedSeeds1Date2026-10-21Caveatpost hoc; small n on realSourcenot in the public repositories yetOpen the data file. Retrieval depends on question form (hit@kept 83 % yes/no, 50 % base, 25 % situational, 10 % entity-free, 8 % bare follow-ups) Question form decides retrievalhit@kept base 50 %, keywords 62 %, yes/no 83 %, entity-free 10 %, situational 25 %, follow-up 8 %Setdev, agent-writtenStatusdevJudgereplaynnot recordedSeedsnot recordedDate2026-10-21Caveatfollow-up hole is a bench-engine artefactSourcenot in the public repositories yetOpen the data file. From 2026-10-22 delivery is counted at chunk level (article-level counts overstated it by 8–13). On MAIN 92 keyed items, delivered facts rose 21 → 28 (global-IDF merge) → 42 (SEFIX-124) → 47 (with F3, withdrawn after unsafe answers rose 9 → 13 on 65 probes) Chunk-level delivered facts on MAIN 92N0 21; gidf 28; SEFIX-124 42; SEFIX-ALL 47SetdevStatusdevJudgecountsnnot recordedSeedsnot recordedDate2026-10-22Caveatarticle-level counts overstated by 8-13Sourcenot in the public repositories yetOpen the data file; the safe set changes U by +0.013 (−0.008…+0.035; Haiku, two seeds) SEFIX-124: F1/F2/F4 without F3chunk-level 42/92, 0 lost; pooled +0.013SetMAIN 97, 2 seedsStatusdevJudgeClaude Haikunnot recordedSeeds295 % CI+0.013 (−0.008..+0.035)Date2026-10-22Caveatcriterion frozen: yes (07ea2b5e); arm verdict: not a lever-candidate; adopted for deliverySourcenot in the public repositories yetOpen the data file. Between items, delivered vs not scores 0.420 vs 0.419 (+0.001, −0.071…+0.069); within items +0.044 (+0.009…+0.082, n 42, Haiku), Opus on 28 delivered general items +0.013 On real items, delivering the fact does not separate answersdelivered 0.420 vs not 0.419 (+0.001, −0.071..+0.069); within-item +0.044 (+0.009..+0.082)SetdevStatusdevJudgeClaude Haiku; Claude Opus slice +0.013nnot recordedSeeds2Date2026-10-23Caveatpage must state plainlySourcenot in the public repositories yetOpen the data file. 44 of 50 undelivered facts already sit in the installed kit Undelivered facts are a search problem44 of 50 already in the K13 kitSetdevStatusdevJudge0 LLMnnot recordedSeedsnot recordedDate2026-10-23Sourcenot in the public repositories yetOpen the data file.

Badge rows the 8 public fact rows this depth cites; a number here without a chip comes from a row that is not public yet (see /data/)
Claim and valueSetStatusJudgenSeeds95 % CIDate
Pack self-test gain does not reproduce on real itemsself-test fact-added cell +0.139 (+0.075..+0.201) n 27 vs real −0.010 (−0.077..+0.050) n 10self-tests (pack-written) vs MAIN keyed (dev)dev (pack-written self-tests)Claude Haikunot recorded12026-10-21
Question form decides retrievalhit@kept base 50 %, keywords 62 %, yes/no 83 %, entity-free 10 %, situational 25 %, follow-up 8 %dev, agent-writtendevreplaynot recordednot recorded2026-10-21
Chunk-level delivered facts on MAIN 92N0 21; gidf 28; SEFIX-124 42; SEFIX-ALL 47devdevcountsnot recordednot recorded2026-10-22
SEFIX-124: F1/F2/F4 without F3chunk-level 42/92, 0 lost; pooled +0.013MAIN 97, 2 seedsdevClaude Haikunot recordednot recorded+0.013 (−0.008..+0.035)2026-10-22
On real items, delivering the fact does not separate answersdelivered 0.420 vs not 0.419 (+0.001, −0.071..+0.069); within-item +0.044 (+0.009..+0.082)devdevClaude Haiku; Claude Opus slice +0.013not recorded22026-10-23
Undelivered facts are a search problem44 of 50 already in the K13 kitdevdev0 LLMnot recordednot recorded2026-10-23
PK-3 OVERLAP: Is the self-test pack gain a lexical-overlap artefact?reweighted to real overlap +0.064 (+0.035..+0.095)cand80 vs 37 real goldsnot recordedreweightingnot recordednot recorded2026-10-21
F3 health fallback is unsafefires 21/65 probes; unsafe 9 → 13devdevblinded agent readingnot recordednot recorded2026-10-22

In one sentence

The phone usually has the book; the hard part is opening it at the right page.

Turning point

2026-10-22, the author's "packs are the strategy", followed the same day by the chunk-level rule that made the target measurable.

On the calendar

43 dated events · 19 Oct → 24 Oct

  1. 19 Oct · 17:51PACKREAD complete: 4B 0.650 > LFM2.5 0.554 > 1.7B 0.529 reading packs (D4 prompt)
  2. 21 Oct · 03:54FACTPRESENT: self-test pack gain does not reproduce on real dev items
  3. 21 Oct · 22:38QFORM: question form decides retrieval; reference kept 0.88 vs not kept 0.19
  4. 22 Oct · 16:05Author direction: packs are the strategy; device day stopped
  5. 22 Oct · 17:3xAuthor decision: small focused packs, not giant ones
All 43 events
  1. 2026-10-19 13:04FACTSAUDIT: facts r7 vs CIA Factbook, 5 table-wrong rows; r8 fixes + emergency numbers for 195 countries
  2. 2026-10-19 13:15PROMPT-LIT-2 verified (19 claims): small models misuse offered tools; LFM2.5 card says not for knowledge QA without retrieval
  3. 2026-10-19 13:30Author redirects to "tests with sources" (user has the matching pack): PACKREAD launched
  4. 2026-10-19 13:51SAFEPACK: emergency pack installed, engine is safe but unhelpful (SAFE-FAIL 26/30)
  5. 2026-10-19 17:51PACKREAD complete: 4B 0.650 > LFM2.5 0.554 > 1.7B 0.529 reading packs (D4 prompt)
  6. 2026-10-19 21:29RELGATE: 4B picks the right source 72 %; per-chunk relevance and sufficiency gates FAIL
  7. 2026-10-19 22:36PACKGAP plan: no new open corpus likely to move V-para (<= +0.01)
  8. 2026-10-19 23:50NIST-crypto pack and visa pack built; both inert on self-tests (+0.008, +0.007)
  9. 2026-10-20 01:51LITE17 NO: thinking, quote-then-answer, chunk filtering leave the 1.7B at ~0.50 vs 4B ~0.61
  10. 2026-10-20 13:04LITELAT on Pixel 8a: retrieval is 57 % of time to first token (~435 serial SQL calls)
  11. 2026-10-20 16:48BATCHRET: batched pack SQL (937 -> 48 calls, byte-identical retrieval 390/390); on the phone TTFT 9.12 -> 5.55 s
  12. 2026-10-20 18:28LITEPACKS NO CHANGE on MAIN; packs help only pack-matched self-tests
  13. 2026-10-21 03:54FACTPRESENT: self-test pack gain does not reproduce on real dev items
  14. 2026-10-21 15:01Simplewiki pack r2 (converter fix) published to the public pack dataset
  15. 2026-10-21 17:02E1-PAR on Pixel 8a (bench engine): Promise.all pack search pays (answer 10.32 → 7.39 s)
  16. 2026-10-21 18:20SHIP15B-PACKS no gain; STYLE15B Succinct non-inferior (Haiku)
  17. 2026-10-21 19:28NOREPACK97: --no-repack makes two hosts byte-identical 97/97
  18. 2026-10-21 22:38QFORM: question form decides retrieval; reference kept 0.88 vs not kept 0.19
  19. 2026-10-22KB snapshot + packs-in-catalog in both clean apps (Android, iOS): setup ~2 min vs ~280 s
  20. 2026-10-22T1 search-term extraction fails; author picks composer design "one button + chips for what changed"
  21. 2026-10-22 00:44RETRIEVE-SWEEP + chunk vectors: delivery up on pack questions, real-item recovery 3/25 (fail by rule)
  22. 2026-10-22 16:05Author direction: packs are the strategy; device day stopped
  23. 2026-10-22 17:3xAuthor decision: small focused packs, not giant ones
  24. 2026-10-22 17:58H4 global-IDF merge: first no-harm pack-composition gain (+6.2 pts kept)
  25. 2026-10-22 18:30COMPOSE: 10-13 small packs compose without crowding; answers flat
  26. 2026-10-22 21:45SEFIX-ALL (search fixes incl. the F3 health fallback): pooled two-seed gain about +0.022/+0.023 with CI low at about 0 (Claude Haiku); rests on F3, withdrawn for safety; the safe set is +0.013 (-0.008..+0.035); claim set (Claude Opus) neutral
  27. 2026-10-22 22:32Opus on covered domain: where the fact arrives U 0.16 → 0.33
  28. 2026-10-22 22:41F3 health fallback fails safety (unsafe 9 → 13); withdrawn
  29. 2026-10-22 22:56SEFIX-124 (safe set) +0.013: not a lever
  30. 2026-10-22 23:16F3b and detector retrain both fail
  31. 2026-10-23 00:40NIGHT B table not publishable: on real items the chunk-level "delivered" flag does not separate answer scores (0.420 vs 0.419)
  32. 2026-10-23 09:41W78 reading-cost stack (1-bit sidecar, IVF lists, lists stored in the pack): 125→21 MB/question, identity 96-97/97
  33. 2026-10-23 10:00KEEPPIN NO, FMT1 NO, W78B NO, PIN2 HOLD (thin screen)
  34. 2026-10-23 11:25Bench APK with W78 stack built, never run on a phone
  35. 2026-10-23 12:18STYLE-SEFIX: Detailed still loses to Succinct with search fixes on
  36. 2026-10-23 12:49U5 follow-ups with packs: source survives 1/19 on our engine
  37. 2026-10-23 16:28PACKSHAPE factbook LEVER-CANDIDATE, seed 43 repeats
  38. 2026-10-23 16:48ARTCAP NO, PACKSHAPE-FIRSTAID NO, D2Q NO
  39. 2026-10-23 17:23HEALTHKEEP-DIAG: health-pack misses mostly delivery
  40. 2026-10-23 17:59X2B-BUDGET NO
  41. 2026-10-23 18:21RETBENCH v1 (0-LLM retrieval bench) done; best config candidate
  42. 2026-10-24 19:13GROUNDED-DOSE rule frozen; SEARCHPORT re-gate (arm A) FAILS: ungrounded dose figures 9 vs 5
  43. 2026-10-24 19:19GROUNDED-DOSE arms B and C PASS on the bench engine; one grounded-yet-toxic misreading flagged

Related nodes on the map

This chapter on the map →

Concepts

Experiments

Decisions

Key findings

.md