RatLabA Boar Research
Closed recordas of what’s still open →

The research · 18 chapters · 5 acts

How a phone with no internet was measured, chapter by chapter

The 10-minute article tells the story once. Here every step has its own page: what was tried, what the numbers were, and what changed because of them. Each chapter reads at three depths, on the same page.

  • ACurious The story and the headline numbers, no jargon.
  • BTech How the system and the method work, with the numbers that drive each decision.
  • CAI pros Every number with set, status, judge, n, seeds and CI; power, noise and threats to validity.

Act I

Building the ruler

Before improving anything, decide how to measure "half as good".

  1. 012026-09-17 → 09-30

    The question

    The goal was simple to say and hard to measure, so the first job was building the ruler.

    ≤ 12 GBof phone memory, fully offlineThe bounty's hardware envelope.≤ 12 GB RAM, ≤ 50 GB total storage, offline, no Play Services, GrapheneOS-compatibleDate2026-10-02Kinda record, not a measurement (set, status, judge, n, seeds: not recorded)CaveatGrapheneOS not yet run on hardwareSourcenot in the public repositories yetOpen the data file

  2. 022026-10-01 → 10-03

    Building a ruler

    Every answer is graded twice, blind, against a frozen frontier answer, under rules written before the test.

    2 ordersevery answer pair is judged twice, swappedCombining the two orders: same winner keeps it, disagreement becomes a tie; rubric scores are averaged.ruleSetallKinda record, not a measurement (status, judge, n, seeds, date: not recorded)Sourcebenchmark scorers/vendor/eval-lib/judge-core.mjs:93-98Open the data file

  3. 032026-10-02 → 10-07

    The floor trap

    We grade on the scale where a blank answer scores zero, and still show the flattering scale beside it.

    0.31what a blank answer scores on the raw ratioA system that declines every question already scores about 0.31 on the raw ratio, because the rubric never goes below 1 of 5.always-decline raw ratio D = 0.306 (v0.2-weighted 0.307)Setv0.1 night runsStatusdevJudgeClaude Opus (judge.v1, both orders)n79 judged declines over 7 runsSeedsnot recordedDate2026-10-02CaveatThis is why the floor-adjusted ratio, not the raw ratio, is the headline.Sourcebenchmark bench/v0.3/plan/analysis-plan.md:24research record reports/BAR-TABLE.md:14research record README.mdOpen the data file

Act II

Pushing the engine

Seven builds, the walls, a bigger model that thinks, and a real phone.

  1. 042026-10-03 → 10-07

    Seven frozen builds

    Seven builds made the app safer and quicker, not smarter.

    +0.006held-out movement across the builds: noiseBefore the 2026-10-06 audit, a week of 4B improvements did not move the held-out score beyond noise.v0.2 raw: main 0.506 → cut 6 0.512, two seeds each, +0.006 (noise)Setv0.2Statusheld-out at run time (burnt → dev 2026-10-06)JudgeClaude Opus (judge.v1, both orders)n110Seeds2Date2026-10-06CaveatDo not compare 0.519 (one seed, s42) with the two-seed 0.506; the audit's "+0.007" and QUEUE:703 use other pairings.Sourcenot in the public repositories yetOpen the data file

  2. 052026-10-03 → 10-06

    The walls

    The small model reads well enough; what it lacks is knowledge the phone doesn't hold.

    0.507 → 0.606when the model is handed the perfect pageOracle ladder: closed-book 0.487 · engine 0.507 · question-chosen passages 0.567 · gold passages 0.606 · 1500-token gold 0.626 · Haiku no-web 0.653see valueSetv0.1 EN devStatusdevJudgeOpusnnot recordedSeedsnot recordedDate2026-10-03Caveatgold passages chosen with knowledge of the reference → upper boundSourcenot in the public repositories yetOpen the data file

  3. 062026-10-02, 10-06 → 10-10; dropped from delivery 10-21

    The deep tier

    A bigger model that thinks first got closest, 0.436, on a desktop, with no wins, and it was never shipped.

    0.436closest configuration, desktop only, 0 wins of 51Recommended deep configuration (35B mixture-of-experts + 512-token thinking budget) on the bounty-style set.raw 0.553 (0.524–0.581; with seed noise 0.513–0.592); floor-adjusted 0.436 (0.401–0.472; with seed noise 0.387–0.486); win 0/51; desktop p50 ≈ 79 sSetV-paraStatusdev-styleJudgeClaude Opus (judge.v1, both orders)n51Seeds2Date2026-10-09CaveatChosen as the best of ≥ 8 deep configurations on this same set: expect a winner's-curse discount of about 0.01–0.02. QUEUE prints raw 0.552; the two-seed aggregate is 0.553. Desktop only.Sourceresearch record reports/BAR-TABLE.md:39research record reports/BAR-TABLE.md:126research record README.md:28Open the data file

  4. 072026-10-02 → 10-22

    A phone on a desk, and in a hand

    The phone matches the desktop on quality; it is just slower, and the big model never left the desktop.

    0.529 = 0.529phone and desktop, same judged qualityOn a real 12 GB-class phone, in airplane mode, the frozen build gives the same judged quality as on the desktop.phone raw 0.529 (0.481–0.575) vs desktop 0.529; floor-adjusted 0.403 vs 0.402; paired −0.000 (−0.026…+0.028); 72/72 answered; fast-tier p50 30.7 s; app memory (PSS) peak 5.88 GB during model load, then ≤ 4.23 GBSetv0.1Statuspublic devJudgeClaude Haiku 5.5-low (L1 triage judge, N = 4, both orders)n57Seeds1 runDate2026-10-14CaveatHaiku-judged level; not comparable with Opus-judged ratios. The device has 10.7 GiB usable RAM.Sourceresearch record arms/device-x6/DEVICE-X6.md:15-18research record arms/device-x6/DEVICE-X6.md:27research record arms/device-x6/DEVICE-X6.md:27-30research record arms/device-x6/DEVICE-X6.md:27-28research record arms/device-x6/DEVICE-X6.md:13Open the data file

Act III

The lab and its instruments

An AI-agent lab, offline libraries, and a check on the judge itself.

  1. 082026-10-01 → 10-24

    The agent lab

    An AI-agent lab can run real research, if every rule is written down and every mistake becomes a new rule.

    54 of 101closed ideas died before any judge callHow arms died101 closed arms: 54 at 0 judge calls, 10 at Haiku L1, 12 at L1→Opus, 25 Opus directlyDate2026-10-14Kinda record, not a measurement (set, status, judge, n, seeds: not recorded)Caveatcitation fixSourcenot in the public repositories yetOpen the data file

  2. 092026-10-12 → 10-14

    Knowledge packs

    Pick the one library you need; more is not better.

    35 → 102of 180 prompts that hold the factThe one retrieval change that converted: append the retrieved article's neighbouring chunks plus a few vector-search candidates, under a 600-token budget (HEADPASS-X2b).Opus +0.066 (+0.043…+0.091); Haiku +0.056; fact chunk present in context 35 → 102 of 180Set180 pack self-testsStatusdev (designed on the same set)JudgeClaude Opus (judge.v1, both orders)n180Seeds1Date2026-10-14CaveatLEVER on pack-matched dev data only; inert on the claim kit (ceiling ≈ 0.011) and not a bounty-bar lever.Sourceresearch record arms/headpass-st/HEADPASS-ST.md:57research record arms/headpass-st/HEADPASS-ST.md:89research record reports/REPORT-PACKS-2026-10-13.md:21research record arms/headpass-st/HEADPASS-ST.md:57,89research record arms/headpass-stOpen the data file

  3. 102026-10-02 → 10-21

    Measuring the measurer

    A different judge changes the level, not the ranking, and not the wins.

    ≈ +0.10another judge family: higher level, same orderCross-family judge level offset on V-paraClaude stricter by ≈ +0.10; 4B floor-adj 0.361 (Codex) vs 0.261 (Claude); deep+think 512 floor-adj 0.577 (0.519-0.632) vs 0.436SetV-para (dev-style for deep; held-out for 4B)Statusheld-out (at run time)Judgecross-family Codex vs Claude Opusn51,Seeds2Date2026-10-16Caveatwin-rate 0.00-0.01 under both; ranking unchanged (Spearman 0.88-0.92)Sourcenot in the public repositories yetOpen the data file

Act IV

From the bar to a product

A smaller engine, a second ruler, delivery, closure, and one simple app.

  1. 112026-10-18 → 10-21

    A smaller engine for 8 GB phones

    For an 8 GB phone, speed and memory picked the small model; it reads worse, and we say so.

    13–16 sper answer on an 8 GB phone, vs 29–46 sPixel 8a speed, 1.7B vs 4B1.7B 13-16 tok/s, answers 13-16 s; 4B 3.6-6 tok/s, 29-46 s; 4B LMK kill at 4.4 GB RSSSetpublic dev questionsStatusdevJudgetimingnnot recordedSeedsnot recordedDate10-20Caveat4B 73/73 only after a rebootVERIFIER NOTE: Pre-BATCHRET, 9 public dev questions; not comparable with the 73-question p50 of 10-21.Sourcenot in the public repositories yetOpen the data file

  2. 122026-10-19 → 10-24

    A second ruler (bench v0.4)

    A fairer exam is written and sealed; the submitted app has sat it, and the grade waits for the author (as of 24 October 2026, 20:34).

    75new questions, sealed before any answerBench v0.4 frozen75 items (71 headline + 4 anchors); 329 checklist points supported on the open web (at least one family, none contradicting; the 32 balance-round points by both families); sha f9ad2f2fSetv0.4 (held-out ruler)Statusheld-out ruler (v0.4; exposed to two engines 2026-10-24)Date2026-10-20Kinda record, not a measurement (judge, n, seeds: not recorded)Caveatwritten by Claude + OpenAI gpt-5.6-sol (the judge families)Sourcenot in the public repositories yetOpen the data file

  3. 132026-10-19 → 10-23

    Delivery, not coverage

    The phone usually has the book; the hard part is opening it at the right page.

    44 of 50missed facts were already on the phoneUndelivered facts are a search problem44 of 50 already in the K13 kitSetdevStatusdevJudge0 LLMnnot recordedSeedsnot recordedDate2026-10-23Sourcenot in the public repositories yetOpen the data file

  4. 142026-10-23 → 10-24

    Closing the research: what a small model needs

    A small model needs a library: right about two times in three when the page arrives; the page arrives about half the time.

    48 % vs 26 %right with the matching library, vs withoutPACKPROOF pooled correct-any with pack installed vs without0.48 vs 0.26, Δ +0.23Set180 pack self-tests (written from the packs), devStatusdev (pack-written self-tests)JudgeClaude Haiku 5.5-low labels; correct-anyn180Seeds4295 % CICI +0.16..+0.31; n 180; seed 42Date2026-10-23Caveatself-tests favour the pack; hosts differ by arm; 1 seed Metric: outcome-label shares of n items (Claude Haiku 5.5-low labels, UXMATRIX protocol); not the judge.v1 ratio.Sourcenot in the public repositories yetOpen the data file

  5. 152026-10-15 → 10-25

    One simple app: LabRat

    The entry is a deliberately simple app, LabRat, that runs offline on a phone; as of 25 October 2026 its desktop dry run is below the bar and its final test on the phone, the one that counts, is below the bar too.

    0 bytessent or received, in airplane modeDevice smoke of BoarLite (APK 9d7fdc2e) on a Pixel 8a in airplane mode: first launch unpacks 18 files (3 GB) to the chat in 14 s; 0 bytes received or sent over the network; 0 crashes; PSS 3.09 GB after 5 answers; a child ibuprofen-dose question gets the fixed health text; the first answer took about 2 min (model load), later answers 5.9-7.4 s at 15.9-16.0 tok/sunpack 14 s; network rx 0 B / tx 0 B; PSS 3.09 GB; 0 crashesSetdevice smoke (hand questions, Pixel 8a, airplane mode)StatusdevJudgenone (hand check)n5 questionsSeedsnot recordedDate2026-10-24 18:45CaveatOne device, 5 hand questions. Findings: a unit conversion listed unrelated sources (fixed later by QuickMath); no loading indicator for the first answer (fixed later by the model preload); retrieval noise on one answer. Health content: AI-reviewed, not clinically reviewed. Nothing here is medical advice.Sourcenot in the public repositories yetOpen the data file

Act V

Where it stands

The live standing, what is still open, and how to check it yourself.

  1. 162026-10-05 → 10-25

    Where it stands (live)

    As of 2026-10-25 03:30 the research lines are closed and the submitted app (LabRat, formerly BoarLite) runs offline on a phone.

    0 of 71wins on the phone; U-total below the 0.50 bar under both judgesDEVICE-FINAL M1, the final ruler measured on the phone: LabRat on a Pixel 8a (8 GB, airplane mode) answered bench v0.4 (71 headline items + 4 anchors) on seeds 42 and 43. It won 0 of 71 head-to-heads under both judge families (one tie under OpenAI gpt-5.6-sol on seed 42). U-total, both seeds pooled: Claude Opus 0.165 (0.127–0.205), OpenAI gpt-5.6-sol 0.242 (0.194–0.296). Per seed: Claude 0.167 (0.129–0.209) and 0.162 (0.125–0.201); gpt-5.6-sol 0.244 (0.193–0.298) and 0.241 (0.190–0.297). The lower family, Claude at 0.165 (0.127–0.205), is below the 0.50 bar. 5 headline items were refusals; answer coverage 0.900 of 71 wins under both families; U-total 0.165 (Claude) / 0.242 (gpt-5.6-sol); below the 0.50 barSetbench v0.4 (exposed)Statusheld-out ruler (v0.4; exposed)JudgeClaude Opus + OpenAI gpt-5.6-sol via the Codex CLI (each vs its own references)n71 headline items + 4 anchorsSeeds42, 43Date2026-10-25 13:2xCaveatMetric: ruler v0.4 U-total (71 headline items, health weight 0.5); never on one axis with the judge.v1 ratio or outcome labels. v0.4 has been answered by our engines since 2026-10-24, so it is no longer unseen. One reference family per judge family; stage 1 not run; no errata. Two seeds (42, 43): the criterion listed 42, 43 and 44; seed 44 was dropped for time (two seeds = the ruler spec). Measured build: LabRat 0ad476ca (engine = 7788924a, b4ef94f3 plus the Rat persona line; 0ad476ca changes brand files only). The released LabRat 1.0.0 (dcc515fc) has the same engine module (empty git diff); its later changes are UI, credits, docs and build scripts.Sourcenot in the public repositories yetOpen the data file

  2. 172026-10-14 →

    Open fronts and hand-off

    The research is closed; what remains is the phone test of the simple app, hosting it, and posting the claim.

    2fronts still open, each with what would change

  3. 18—

    Reproduce it (appendix)

    Everything that can be rerun is public, and what cannot be rerun is named.

    139 + 411public files: benchmark and research recordPublic repos at releaseboar-bench 139 files / 2.26 MB; boar-research 411 files / 5.53 MBDate2026-10-15 01:18 / 10:07Kinda record, not a measurement (set, status, judge, n, seeds: not recorded)Caveatone commit each, author "BOAR R&D"Sourcenot in the public repositories yetOpen the data file