The research · Act I, Building the ruler · Chapter 01
The question
"Half as good as the internet plus a frontier AI" sounds like a goal; it is really a measurement problem, and the first finding was that nobody had a ruler for it.
A phone in airplane mode
Imagine you are on a mountain, on a long flight, or somewhere the network is down. You still have questions: what is this plant, how far is the next town, what does this error mean. BOAR keeps a small AI model and a few downloaded libraries on your phone and answers from them, with sources.
The bounty's rules
The challenge had strict limits: an Android phone with at most 12 GB of memory, at most 50 GB of downloaded files, fully offline, open source. And one target: "more than 50 % as good as internet search plus a frontier AI." Winning is decided by a public confirmation, not by a score.
Half as good as what, exactly?
That sentence hides a research problem. Half of what? Measured how, on which questions, on which phone? If you grade a student's essay against a professor's, "half as good" depends on who grades and what counts. Before improving anything, we had to build a way to measure that we could trust.
Where we started
When the author joined the project, the app ran a small 1.5-billion-parameter model over about 1,300 Wikipedia paragraphs. A review called it "solid engineering, weak research": good code, but nobody knew how good the answers were.
The problem as a spec
The spec: Android, ≤ 12 GB RAM, ≤ 50 GB, offline, half as good
The bounty fixes the hardware envelope and leaves the quality target vague. Hard limits: Android or GrapheneOS, at most 12 GB of RAM, at most 50 GB of assets, fully offline, no Play Services, public source. Target: "more than 50 % as good as internet search plus a frontier AI" on explanation, comparison, synthesis and reasoning questions, in English. We treated the envelope as a budget and the target as a measurement problem: before touching the engine we built a desktop runner that executes the app's real TypeScript answer engine, and a judge that compares every answer with a frozen frontier-plus-web reference.
Callout
"Pre-harness numbers (an older 61-question set, 0.26 → 0.48, a 1.5B model on an iPhone) used a different instrument and are not comparable to anything below."
Abstract
This is the record (v0.4-preprint, as of 2026-10-25; research lines closed 2026-10-23) of a five-week attempt to build an offline Android research assistant that is "more than 50 % as good as internet search plus a frontier AI", under a ≤ 12 GB RAM, ≤ 50 GB storage envelope (poidh bounty #31). Quality was first measured pairwise against frozen references written by Claude Opus with web search, judged by Claude Opus in both presentation orders on a 1–5 rubric, and reported as a raw ratio, a floor-adjusted ratio that removes the rubric minimum, and win/tie/loss; every deciding criterion was hashed before its run and held-out exposure was logged per touch. On a bounty-style set held out at run time (V-para, n = 51, two seeds) the frozen 4B build scores raw 0.413 (0.384–0.445), floor-adjusted 0.261 (0.224–0.300), and wins 0 of 51; a 35B mixture-of-experts with a 512-token thinking budget reaches floor-adjusted 0.436 (0.401–0.472) on development data, desktop only, 0 wins, and was not delivered. A cross-family judge raises all levels by ≈ 0.10 while preserving the ranking and a near-zero win-rate. Phone constraints then made a 1.7B model the main 8 GB engine (the 4B leads it by +0.109 under Opus on development data). A second ruler (75 items, half written by each judge family, frozen before any answer; U-total with a zero for non-answers; verdict = lower of two families) has been run on the desktop twin of the submitted app's pre-rename engine (one seed): 0 of 71 wins under both families and a lower-family U-total of 0.164 (0.123–0.206), below the 0.50 bar; we report it as history, because the headline is the same ruler run on the phone with the shipped app under a criterion frozen before generation, which landed on 2026-10-25. Our thesis is a design statement: small models need external knowledge, so we built a harness around a small model's real capabilities (health gate, code-computed arithmetic, pack search and delivery, a short answer style, a fixed check-the-source line, refusals). Its evidence and limits come from closing tests on pack-written self-tests: with the item's own article delivered, the 1.7B is right on 65 % of items vs 28 % otherwise (Claude Haiku outcome labels, one seed), delivery succeeds on 54 %, no pack reaches 0.75, and on real development items delivery does not separate answer quality. As of 2026-10-25, under the Claude judge, the 0.50 bar was not reached on the earlier held-out sets, and no unexposed held-out number exists. The submitted app, LabRat (formerly BoarLite), runs offline on an 8 GB phone; one search change in it was merged although its pre-registered safety criterion failed by the letter, under a trade-off its author accepted in advance, and we report both. We release the first benchmark and judge, a registry of negative results, openly licensed knowledge packs and a phone-parity study.
Key results
| V-para (51) | v0.2 (110) | |
|---|---|---|
| status at run time / now | held-out (4B runs) / dev-style | held-out / burnt → dev |
| 4B claim build, Opus, 2 seeds | raw 0.413 (0.384–0.445) · floor 0.261 (0.224–0.300) · 0/0/51 | raw 0.512 (0.478–0.549) · floor 0.388 (0.346–0.434) · 0/0/110, 0/1/109 |
| deep + think 512, Opus | raw 0.553 (0.524–0.581) · floor 0.436 (0.401–0.472) · 0/0/51 (2 seeds, dev-style) | raw 0.608 (0.575–0.641) · floor 0.508 (0.467–0.550) · 0/1/109 (1 seed, dev) |
| same, cross-family judge (post-hoc) | raw 0.667 (0.621–0.711) · floor 0.577 (0.519–0.632) · win 0.00–0.01 | raw 0.714 (0.666–0.759) · floor 0.639 (0.579–0.695) · win 6.4 % |
Key results
| quantity | value | badge |
|---|---|---|
| correct-any with vs without the matching pack | 0.48 vs 0.26 (+0.23, +0.16…+0.31) | 180 pack-written self-tests, 9 packs, 1.7B Succinct, seed 42, Claude Haiku labels, desktop twin, dev PACKPROOF pooled correct-any with pack installed vs without0.48 vs 0.26, Δ +0.23Set180 pack self-tests (written from the packs), devStatusdev (pack-written self-tests)JudgeClaude Haiku 5.5-low labels; correct-anyn180Seeds4295 % CICI +0.16..+0.31; n 180; seed 42Date2026-10-23Caveatself-tests favour the pack; hosts differ by arm; 1 seed Metric: outcome-label shares of n items (Claude Haiku 5.5-low labels, UXMATRIX protocol); not the judge.v1 ratio.Sourcenot in the public repositories yetOpen the data file |
| correct-any when the own article is kept vs not | 0.65 (n 98) vs 0.28; kept 54 % (article level) | same Correct-any when own article is kept vs not0.65 (n 98) vs 0.28Set180 pack self-tests (written from the packs), devStatusdevJudgeClaude Haiku labelsn180Seedsnot recordedDate2026-10-23Caveatabout 2 in 3 when delivered'kept' = own article kept (article level), between-item split, n 98 kept.Sourcenot in the public repositories yetOpen the data file |
| packs at the 0.75 bar | 0 of 9 (best 0.60, worst 0.25) | same, n 20 per pack Packs reaching the 0.75 "proves" bar0 of 9; 7 below 0.60 (not default-eligible by rule); best 0.60 (two Q&A packs), worst factbook 0.25Set180 pack self-tests (written from the packs), devStatusdevJudgeClaude Haikun20Seedsnot recordedDate2026-10-23Caveathealth packs AI-reviewed, not clinically reviewed AI-reviewed, not clinically reviewed.Sourcenot in the public repositories yetOpen the data file |
| undelivered facts already in the kit | 44 of 50 | MAIN 92 keyed, 0-LLM Undelivered facts are a search problem44 of 50 already in the K13 kitSetdevStatusdevJudge0 LLMnnot recordedSeedsnot recordedDate2026-10-23Sourcenot in the public repositories yetOpen the data file |
| shipped style on ENGINE-RC | correct-any 0.43, wrong-unwarned 0.12 (pack on) | UXMATRIX 60, seed 42, Haiku labels Succinct on ENGINE-RC (page numbers)ON CWS 0.30, correct-any 0.43, WU 0.12; OFF correct-any 0.22, WU 0.10SetUXMATRIX 60, devStatusdevJudgeClaude Haiku labelsn60Seeds42Date2026-10-24CaveatPASS vs frozen CWS ≥0.28, WU OFF ≤0.10 Metric: outcome-label shares of n items (Claude Haiku 5.5-low labels, UXMATRIX protocol); not the judge.v1 ratio.Sourcenot in the public repositories yetOpen the data file |
| bench v0.4 (U-total) | frozen 2026-10-20; references 75/75 per family; final run on BoarLite b4ef94f3 (desktop twin, one seed; history): win-rate 0 of 71 both families; U-total Claude 0.164 (0.123–0.208), gpt-5.6-sol 0.245 (0.194–0.300), minimum 0.164 (0.123–0.206), below the 0.50 bar; not a measurement of LabRat 7788924a (persona line added after it); on-phone DEVICE-FINAL running (the headline) | ,,, “Ruler v0.4 verdict run”, “DEVICE-FINAL: final on-phone matrix” |
1 Introduction: an underspecified target and a measurable one
The target phrase "more than 50 % as good as internet plus frontier AI" leaves the construct, the comparator, the question distribution and the judge unspecified. We operationalise it as a ratio of judged rubric scores between the system's answer and a frozen frontier-plus-web reference for the same question, on sets drawn to resemble the bounty's stated use (explanation, comparison, synthesis, reasoning). We make four contributions: (i) a public benchmark and pairwise judge with frozen references and a fail-closed pipeline; (ii) a pre-registered evaluation of a production-shaped offline system across seven frozen builds and a 35B deep tier; (iii) an analysis of the measuring instrument itself (rubric floor, seed versus judge noise, judge-family offset, reference length); (iv) a registry of negative results with power statements. We also document an unusual organisation: the work was coordinated and executed by AI agents under one human author, with independent AI reviewers in place of human sign-off.
2 Related work
Key paragraph (to be completed with citations by the author; no claims beyond positioning): LLM-as-judge evaluation with pairwise comparison and order swapping; position and self-preference bias; reference-based vs reference-free judging; retrieval-augmented generation on device; mixture-of-experts inference with expert offloading/streaming; test-time compute ("thinking") budgets; pre-registration and multiplicity in ML evaluation. We do not claim novelty for any single component; the contribution is the integrated, pre-registered record under a hard device envelope. Note to author: the private record contains a 10-12 literature pass (258 findings, 23 claims adversarially verified, 18 corrected). Pull citations from it rather than from memory.
Badge rows the 19 public fact rows this depth cites; a number here without a chip comes from a row that is not public yet (see /data/)
| Claim and value | Set | Status | Judge | n | Seeds | 95 % CI | Date |
|---|---|---|---|---|---|---|---|
| PACKPROOF pooled correct-any with pack installed vs without0.48 vs 0.26, Δ +0.23 | 180 pack self-tests (written from the packs), dev | dev (pack-written self-tests) | Claude Haiku 5.5-low labels; correct-any | 180 | 42 | CI +0.16..+0.31; n 180; seed 42 | 2026-10-23 |
| Correct-any when own article is kept vs not0.65 (n 98) vs 0.28 | 180 pack self-tests (written from the packs), dev | dev | Claude Haiku labels | 180 | not recorded | 2026-10-23 | |
| Packs reaching the 0.75 "proves" bar0 of 9; 7 below 0.60 (not default-eligible by rule); best 0.60 (two Q&A packs), worst factbook 0.25 | 180 pack self-tests (written from the packs), dev | dev | Claude Haiku | 20 | not recorded | 2026-10-23 | |
| Undelivered facts are a search problem44 of 50 already in the K13 kit | dev | dev | 0 LLM | not recorded | not recorded | 2026-10-23 | |
| Succinct on ENGINE-RC (page numbers)ON CWS 0.30, correct-any 0.43, WU 0.12; OFF correct-any 0.22, WU 0.10 | UXMATRIX 60, dev | dev | Claude Haiku labels | 60 | 42 | 2026-10-24 | |
| A judge from a different model family scores everything higher on V-para, but keeps the ranking and the near-zero win-rate.4B: raw 0.497 (0.449–0.547), floor-adjusted 0.361; deep + think 512: raw 0.667 (0.621–0.711), floor-adjusted 0.577 (0.519–0.632); deep − 4B +0.171 (+0.123…+0.219) vs +0.139 under Claude; per-item Spearman 0.88–0.92; win-rate 0.00–0.01 | V-para | dev-style | cross-family: OpenAI gpt-5.6-sol (via the Codex CLI) (post-hoc, not pre-registered) | 51 (408 judge calls) | 2 | 2026-10-16 | |
| On a real 12 GB-class phone, in airplane mode, the frozen build gives the same judged quality as on the desktop.phone raw 0.529 (0.481–0.575) vs desktop 0.529; floor-adjusted 0.403 vs 0.402; paired −0.000 (−0.026…+0.028); 72/72 answered; fast-tier p50 30.7 s; app memory (PSS) peak 5.88 GB during model load, then ≤ 4.23 GB | v0.1 | public dev | Claude Haiku 5.5-low (L1 triage judge, N = 4, both orders) | 57 | 1 run | 2026-10-14 | |
| The frozen 4B claim build on the bounty-style set (V-para) delivers about a quarter of the frontier reference's value above "nothing", and loses every head-to-head.raw 0.413 (0.384–0.445); floor-adjusted 0.261 (0.224–0.300); win/tie/loss 0/0/51 in both seeds | V-para | held-out for this run (cut 3 on 2026-10-05, cut 6 on 2026-10-06); dev-style for everything measured after 2026-10-06 | Claude Opus (judge.v1, both orders) | 51 | 2 (42, 43) | 2026-10-06 | |
| The same 4B claim build on the second set (v0.2, later burnt to dev).raw 0.512 (0.478–0.549); floor-adjusted 0.388 (0.346–0.434); win/tie/loss 0/0/110 and 0/1/109 (per seed) | v0.2 | held-out at run time; burnt → dev by ruling 2026-10-06 | Claude Opus (judge.v1, both orders) | 110 | 2 | 2026-10-06 | |
| Recommended deep configuration (35B mixture-of-experts + 512-token thinking budget) on the bounty-style set.raw 0.553 (0.524–0.581; with seed noise 0.513–0.592); floor-adjusted 0.436 (0.401–0.472; with seed noise 0.387–0.486); win 0/51; desktop p50 ≈ 79 s | V-para | dev-style | Claude Opus (judge.v1, both orders) | 51 | 2 | 2026-10-09 | |
| The cross-family judge on v0.2.4B: raw 0.617 (0.567–0.666), floor-adjusted 0.517 (2 seeds); deep + think 512: raw 0.714 (0.666–0.759), floor-adjusted 0.639 (0.579–0.695) (1 seed); deep − 4B +0.097 vs +0.096 under Claude; Spearman 0.90–0.97; win-rate 2.7 % / 6.4 % vs 0 % under Claude | v0.2 | dev (burnt) | cross-family: OpenAI gpt-5.6-sol (via the Codex CLI) (post-hoc, not pre-registered) | 110 (660 judge calls) | 4B 2, deep 1 | 2026-10-17 | |
| Own article delivered (kept)54 %; cited 28 % | 180 pack self-tests (written from the packs), dev | dev | 0-LLM funnel | 180 | not recorded | 2026-10-23 | |
| Bench v0.4 frozen75 items (71 headline + 4 anchors); 329 checklist points supported on the open web (at least one family, none contradicting; the 32 balance-round points by both families); sha f9ad2f2f | v0.4 (held-out ruler) | held-out ruler (v0.4; exposed to two engines 2026-10-24) | not recorded | not recorded | not recorded | 2026-10-20 | |
| REFS-V2, OpenAI gpt-5.6-sol half complete: 75 of 75 v0.4 references written with live web search under the frozen protocol, 0 hard failures, 97 calls (two items completed on a resumed run); references of both judge families now exist75 / 75; 0 hard failures | v0.4 (held-out ruler) | held-out ruler (exposed) | not recorded | 75 items (88 turns) | not recorded | 2026-10-24 20:05 | |
| Final ruler run (RULER-FINAL) on BoarLite's final engine b4ef94f3 exists as of 2026-10-24 20:34: desktop twin of the app path (QuickMath, search port ON_NO_HEALTH, health gate, history as the app sends it), 71 headline items + 4 anchors of bench v0.4, one seed, each judge family against its own references (Claude Opus vs the Claude references; OpenAI gpt-5.6-sol vs its references), 150 + 150 judge calls, addendum frozen before any generation; its numbers were held until the author's publication decision of 2026-10-25 (now public as; history, the headline is DEVICE-FINAL) | v0.4 (held-out ruler; exposed) | held-out ruler (v0.4; now exposed to three engines) | Claude Opus + OpenAI gpt-5.6-sol (each vs its own references) | 71 headline items + 4 anchors | 1 (42) | 2026-10-24 20:34 | |
| RULER-FINAL U-total of BoarLite b4ef94f3 on v0.4: Claude family 0.164 (0.123-0.208), OpenAI gpt-5.6-sol family 0.245 (0.194-0.300); minimum over families 0.164 (0.123-0.206); win-rate 0 of 71 (Claude) and 0 of 71 with 1 tie (gpt-5.6-sol); fixed-text refusals on 3 itemsClaude 0.164; gpt-5.6-sol 0.245; min 0.164; win-rate 0 | v0.4 (held-out ruler; exposed) | held-out ruler (v0.4; exposed) | Claude Opus + OpenAI gpt-5.6-sol (each vs its own references) | 71 headline items | 1 (42) | 2026-10-24 20:34 | |
| Engine change after the ruler run: in LabRat 7788924a the grounding prompt's identity line changed from "You are BOAR, an offline AI research app running on this phone…" to "You are Rat, the assistant in LabRat, an offline AI research app running on this phone…". The final ruler run (RULER-FINAL, 2026-10-24 20:34) was on b4ef94f3, before this change: its number is for b4ef94f3 (desktop twin), and DEVICE-FINAL measures LabRat 7788924a on the phonepersona line changed after RULER-FINAL | not recorded | not recorded | not recorded | not recorded | not recorded | 2026-10-25 02:20 | |
| DEVICE-FINAL criterion frozen (sha256 recorded) on 2026-10-25 03:3x, before any matrix generation: LabRat 7788924a answers on a Pixel 8a 8 GB in airplane mode on AC power. Matrix: M1 ruler v0.4 (75 items, 88 turns) on seeds 42, 43 and 44, judged as in RULER-FINAL (Claude Opus vs Claude references on all three seeds; OpenAI gpt-5.6-sol vs its references on seed 42, further seeds optional), reporting per family the U-total, CI, win-rate, refusals, the minimum over families, the three-seed mean and a phone-vs-desktop parity read against the b4ef94f3 desktop twin (not a verdict); M2 the 65 safety probes (0 LLM); M3 the 140 on-K13 pack self-tests and M4 the 60 UXMATRIX items (Claude Haiku labels); M5 latency, time to first token, decode speed, memory, battery temperature, three cold starts and the app's network counters over the night. Claim numbers = M1 (both families named, seeds stated, win-rate first) and M5 (speed, memory, offline); M2-M4 are product numbers. No tuning, no engine changematrix M1-M5; claim numbers = M1 + M5 | bench v0.4 (exposed); 65 safety probes; 140 on-K13 self-tests; UXMATRIX 60 | dev | Claude Opus + OpenAI gpt-5.6-sol (M1); Claude Haiku 5.5-low labels (M3, M4); none (M2, M5) | 75 items × 3 seeds + 265 product items | 42, 43, 44 (M1); 42 (M2-M4) | 2026-10-25 03:3x | |
| Bounty hard limits: Android/GrapheneOS, ≤ 12 GB RAM, ≤ 50 GB assets, fully offline, no Play Services, public repo | n/a | not recorded | n/a | not recorded | not recorded | 2026-10-02 |
In one sentence
The goal was simple to say and hard to measure, so the first job was building the ruler.
Turning point
09-29/30: the written diagnosis that quality and speed were measured on different machines with ad hoc scripts. Fixing the instrument comes before fixing the app.
On the calendar
16 dated events · 17 Sept → 4 Oct
- 17 SeptPublic post (by Vitalik Buterin) asks for an offline phone research tool ">50 % as good as internet + frontier AI"; bounty #31 is later opened on poidh, inspired by it
- 26 SeptAuthor does a read-only review of the app maintainer's existing bounty claim: "solid engineering, weak research"; default Qwen2.5-1.5B + ~1,300 Wikipedia paragraphs; biggest lever named = large streamed MoE + full Wikipedia
- 26 SeptAuthor joins the app maintainer as co-builder; thesis "quality per second + blind eval vs frontier"; official Android submission planned as a no-INTERNET variant
- 3 OctDirection change: single goal = win the bounty; lanes re-tasked
- 3 OctV-para blind set frozen (58 items, bounty-style paraphrases)
All 16 events
- 2026-09-17Public post (by Vitalik Buterin) asks for an offline phone research tool ">50 % as good as internet + frontier AI"; bounty #31 is later opened on poidh, inspired by it
- 2026-09-24Upstream app README benchmark on a hot Xiaomi (Dimensity 8300): LFM2.5-8B-A1B 14.8 tok/s, Qwen2.5-1.5B 11.4 (maintainer's own measurement, 17 questions)
- 2026-09-26Author does a read-only review of the app maintainer's existing bounty claim: "solid engineering, weak research"; default Qwen2.5-1.5B + ~1,300 Wikipedia paragraphs; biggest lever named = large streamed MoE + full Wikipedia
- 2026-09-26Author joins the app maintainer as co-builder; thesis "quality per second + blind eval vs frontier"; official Android submission planned as a no-INTERNET variant
- 2026-09-27First knowledge kit packs built: boar-crypto (36.1 MB), boar-wikivoyage-en (351 MB), boar-preparedness
- 2026-09-28Overnight agent-run UI and layout-performance review of the app; heavy work routed to the Mac mini queue
- 2026-09-29Research-pipeline kickoff: first written diagnosis (prefill dominates time-to-first-token; corpus coverage limits quality; 4B never run on a real Android)
- 2026-09-29Overnight on-device tuning on an older 61-question bounty-style set (v2): score 0.26 → 0.48 with the 1.5B + quick mode + packs; vegan-venue test 5 % → 85 %
- 2026-09-30First "in-RAM MoE" experiments: Ling-3.0-tiny and Maple-Preview 20B-A1B vs Qwen3-4B (desktop, app contexts)
- 2026-09-30Frozen frontier-plus-web reference answers for the public v0.1 set start being generated (09-30 → 10-02)
- 2026-09-30Author decides 10 open items after an overnight eval run (early metric, iPhone 13; not comparable to later ratios)
- 2026-09-30 22:28Frozen Opus+web reference answers generated for the 33 kept v0 items (prompt reference.v1, sha d5144343…)
- 2026-10-02 12:40Big-pack relevance gate fails replay → knowledge front closed for the bounty (kill rule K1)
- 2026-10-03Direction change: single goal = win the bounty; lanes re-tasked
- 2026-10-03V-para blind set frozen (58 items, bounty-style paraphrases)
- 2026-10-04Knowledge-pack sizing: web-only pack parked for the bounty