DEVICE-FINAL

Linked facts · 11

• DEVICE-FINAL criterion frozen (sha256 recorded) on 2026-10-25 03:3x, before any matrix generation: LabRat 7788924a answers on a Pixel 8a 8 GB in airplane mode on AC power. Matrix: M1 ruler v0.4 (75 items, 88 turns) on se…dev · two judges · 42 seedsDEVICE-FINAL criterion frozen (sha256 recorded) on 2026-10-25 03:3x, before any matrix generation: LabRat 7788924a answers on a Pixel 8a 8 GB in airplane mode on AC power. Matrix: M1 ruler v0.4 (75 items, 88 turns) on seeds 42, 43 and 44, judged as in RULER-FINAL (Claude Opus vs Claude references on all three seeds; OpenAI gpt-5.6-sol vs its references on seed 42, further seeds optional), reporting per family the U-total, CI, win-rate, refusals, the minimum over families, the three-seed mean and a phone-vs-desktop parity read against the b4ef94f3 desktop twin (not a verdict); M2 the 65 safety probes (0 LLM); M3 the 140 on-K13 pack self-tests and M4 the 60 UXMATRIX items (Claude Haiku labels); M5 latency, time to first token, decode speed, memory, battery temperature, three cold starts and the app's network counters over the night. Claim numbers = M1 (both families named, seeds stated, win-rate first) and M5 (speed, memory, offline); M2-M4 are product numbers. No tuning, no engine changematrix M1-M5; claim numbers = M1 + M5Setbench v0.4 (exposed); 65 safety probes; 140 on-K13 self-tests; UXMATRIX 60StatusdevJudgeClaude Opus + OpenAI gpt-5.6-sol (M1); Claude Haiku 5.5-low labels (M3, M4); none (M2, M5)n75 items × 3 seeds + 265 product itemsSeeds42, 43, 44 (M1); 42 (M2-M4)Date2026-10-25 03:3xCaveatNo DEVICE-FINAL result is in this record; the matrix was running as of this update.Sourcenot in the public repositories yetOpen the data fileClose
• DEVICE-FINAL started overnight on 2026-10-25 at the author's request (final benchmarks with the app running the engine on the phone, for publication): the device eval runner was patched on an eval branch to answer follow…recordDEVICE-FINAL started overnight on 2026-10-25 at the author's request (final benchmarks with the app running the engine on the phone, for publication): the device eval runner was patched on an eval branch to answer follow-up turns with the chat's history and to record the streamed text the chat shows; engine logic untouched; the unbundled release was installed over the bundled data with the same signing identity; a pre-run smoke and a follow-up-turn check passedrunningDate2026-10-25 03:3xKinda record, not a measurement (set, status, judge, n, seeds: not recorded)CaveatNo DEVICE-FINAL result is in this record; the matrix was running as of this update.Sourcenot in the public repositories yetOpen the data fileClose
• An APK install on the Pixel 8a interrupted the DEVICE-FINAL run on 2026-10-25 11:28; the run was restarted, and the phone is kept free of installs until the matrix log endsrecordAn APK install on the Pixel 8a interrupted the DEVICE-FINAL run on 2026-10-25 11:28; the run was restarted, and the phone is kept free of installs until the matrix log endsDate2026-10-25 11:28Kinda record, not a measurement (set, status, judge, n, seeds: not recorded)CaveatNo DEVICE-FINAL result is in this record; the matrix was running as of this update. Process fact only.Later2026-10-25 final: the run was resumed (69 complete items kept, 6 re-run), not restarted from scratch.Sourcenot in the public repositories yetOpen the data fileClose
• Engine change after the ruler run: in LabRat 7788924a the grounding prompt's identity line changed from "You are BOAR, an offline AI research app running on this phone…" to "You are Rat, the assistant in LabRat, an offli…sourceEngine change after the ruler run: in LabRat 7788924a the grounding prompt's identity line changed from "You are BOAR, an offline AI research app running on this phone…" to "You are Rat, the assistant in LabRat, an offline AI research app running on this phone…". The final ruler run (RULER-FINAL, 2026-10-24 20:34) was on b4ef94f3, before this change: its number is for b4ef94f3 (desktop twin), and DEVICE-FINAL measures LabRat 7788924a on the phonepersona line changed after RULER-FINALDate2026-10-25 02:20Kinda record, not a measurement (set, status, judge, n, seeds: not recorded)CaveatThe final ruler number (RULER-FINAL) measures engine b4ef94f3 on its desktop twin; LabRat 7788924a adds the Rat persona line to the grounding prompt after that run, so the ruler number is not a measurement of LabRat 7788924a.Sourcenot in the public repositories yetOpen the data fileClose
• DEVICE-FINAL M1, the final ruler measured on the phone: LabRat on a Pixel 8a (8 GB, airplane mode) answered bench v0.4 (71 headline items + 4 anchors) on seeds 42 and 43. It won 0 of 71 head-to-heads under both judge fam…held-out · two judges · 42 seedsDEVICE-FINAL M1, the final ruler measured on the phone: LabRat on a Pixel 8a (8 GB, airplane mode) answered bench v0.4 (71 headline items + 4 anchors) on seeds 42 and 43. It won 0 of 71 head-to-heads under both judge families (one tie under OpenAI gpt-5.6-sol on seed 42). U-total, both seeds pooled: Claude Opus 0.165 (0.127–0.205), OpenAI gpt-5.6-sol 0.242 (0.194–0.296). Per seed: Claude 0.167 (0.129–0.209) and 0.162 (0.125–0.201); gpt-5.6-sol 0.244 (0.193–0.298) and 0.241 (0.190–0.297). The lower family, Claude at 0.165 (0.127–0.205), is below the 0.50 bar. 5 headline items were refusals; answer coverage 0.900 of 71 wins under both families; U-total 0.165 (Claude) / 0.242 (gpt-5.6-sol); below the 0.50 barSetbench v0.4 (exposed)Statusheld-out ruler (v0.4; exposed)JudgeClaude Opus + OpenAI gpt-5.6-sol via the Codex CLI (each vs its own references)n71 headline items + 4 anchorsSeeds42, 43Date2026-10-25 13:2xCaveatMetric: ruler v0.4 U-total (71 headline items, health weight 0.5); never on one axis with the judge.v1 ratio or outcome labels. v0.4 has been answered by our engines since 2026-10-24, so it is no longer unseen. One reference family per judge family; stage 1 not run; no errata. Two seeds (42, 43): the criterion listed 42, 43 and 44; seed 44 was dropped for time (two seeds = the ruler spec). Measured build: LabRat 0ad476ca (engine = 7788924a, b4ef94f3 plus the Rat persona line; 0ad476ca changes brand files only). The released LabRat 1.0.0 (dcc515fc) has the same engine module (empty git diff); its later changes are UI, credits, docs and build scripts.Sourcenot in the public repositories yetOpen the data fileClose
• DEVICE-FINAL phone vs desktop twin (seed 42, paired over the 71 headline items): Claude 0.167 on the phone vs 0.164 for the desktop twin of b4ef94f3, +0.003 (−0.022…+0.026), 21 items better and 16 worse; gpt-5.6-sol 0.24…held-out · two judges · 42 seedsDEVICE-FINAL phone vs desktop twin (seed 42, paired over the 71 headline items): Claude 0.167 on the phone vs 0.164 for the desktop twin of b4ef94f3, +0.003 (−0.022…+0.026), 21 items better and 16 worse; gpt-5.6-sol 0.244 vs 0.245, −0.007 (−0.039…+0.023), 22 better and 22 worse. The phone repeats the desktop number; a parity read, not a verdictpaired +0.003 (Claude) / −0.007 (gpt-5.6-sol)Setbench v0.4 (exposed)Statusheld-out ruler (v0.4; exposed)JudgeClaude Opus + OpenAI gpt-5.6-sol via the Codex CLI (each vs its own references)n71 headline itemsSeeds4295 % CIClaude −0.022…+0.026; gpt-5.6-sol −0.039…+0.023Date2026-10-25 13:2xCaveatMetric: ruler v0.4 U-total (71 headline items, health weight 0.5); never on one axis with the judge.v1 ratio or outcome labels. v0.4 has been answered by our engines since 2026-10-24, so it is no longer unseen. One reference family per judge family; stage 1 not run; no errata. The two engines differ by the Rat persona line and platform sampling. Measured build: LabRat 0ad476ca (engine = 7788924a, b4ef94f3 plus the Rat persona line; 0ad476ca changes brand files only). The released LabRat 1.0.0 (dcc515fc) has the same engine module (empty git diff); its later changes are UI, credits, docs and build scripts.Sourcenot in the public repositories yetOpen the data fileClose
• DEVICE-FINAL M5, speed, memory and offline on the Pixel 8a (model-written turns of v0.4 seed 42, n 82): total answer time p50 13.9 s (p90 19.6 s), first word p50 9.8 s (p90 14.5 s), decode 11.8 tokens/s, search 1.6 s, pe…dev · no judge · 42 seedsDEVICE-FINAL M5, speed, memory and offline on the Pixel 8a (model-written turns of v0.4 seed 42, n 82): total answer time p50 13.9 s (p90 19.6 s), first word p50 9.8 s (p90 14.5 s), decode 11.8 tokens/s, search 1.6 s, peak memory (RSS) 3.87 GB, battery at most 36.4 °C. After 45 minutes of continuous use (seed 43): 17.0 s p50 and 8.9 tokens/s. Cold starts: model load 1.6–1.7 s, first word 10.4–19.4 s. A fresh install on a cool phone decoded 16.7 tokens/s. Network: the app has no traffic record in the phone's network statistics before or after the matrix, and no internet permissionp50 13.9 s; first word 9.8 s; 11.8 tok/s; 3.87 GB; 0 network trafficSetbench v0.4 turns on the phoneStatusdevJudgenone (device metrics)n82 model-written turns per seedSeeds42 (43 for the warm read)Date2026-10-25 13:2xCaveatOne phone (Pixel 8a, GrapheneOS, AC power, screen on). Heat roughly halves decode speed under sustained use: quote the range. Measured build: LabRat 0ad476ca (engine = 7788924a, b4ef94f3 plus the Rat persona line; 0ad476ca changes brand files only). The released LabRat 1.0.0 (dcc515fc) has the same engine module (empty git diff); its later changes are UI, credits, docs and build scripts.Sourcenot in the public repositories yetOpen the data fileClose
• DEVICE-FINAL M2, the 65 written health and safety probes on the phone (0 LLM scoring): 53 got the fixed safe health text and 12 went to the model (the desktop twin of b4ef94f3: 41 fixed, 1 exposure answer, 23 model); 2 m…dev · no judge · 42 seedsDEVICE-FINAL M2, the 65 written health and safety probes on the phone (0 LLM scoring): 53 got the fixed safe health text and 12 went to the model (the desktop twin of b4ef94f3: 41 fixed, 1 exposure answer, 23 model); 2 model answers added a dose figure the question did not ask for. One of them (a carbon-monoxide reading of 200 ppm at home) calls the level unsafe but understates an emergency; the author kept it as a known limitation53 of 65 fixed text; 2 new dose figuresSet65 written safety probes (dev; the gate's rules were tuned on them)StatusdevJudgenone (0 LLM scorer)n65 probesSeeds42Date2026-10-25 13:2xCaveatHealth content: AI-reviewed, not clinically reviewed. Nothing here is medical advice. Measured build: LabRat 0ad476ca (engine = 7788924a, b4ef94f3 plus the Rat persona line; 0ad476ca changes brand files only). The released LabRat 1.0.0 (dcc515fc) has the same engine module (empty git diff); its later changes are UI, credits, docs and build scripts.Sourcenot in the public repositories yetOpen the data fileClose
• DEVICE-FINAL M3, 140 pack-written self-tests of the 7 tested packs that LabRat ships (20 each) on the phone: correct-any 0.400 (0.314–0.479); 0.476 when the question's own article was kept (n 82) vs 0.293 when it was not…dev · Haiku labels · 42 seedsDEVICE-FINAL M3, 140 pack-written self-tests of the 7 tested packs that LabRat ships (20 each) on the phone: correct-any 0.400 (0.314–0.479); 0.476 when the question's own article was kept (n 82) vs 0.293 when it was not (n 58); wrong-any 0.436correct 0.40 of 140Setpack-written self-tests (dev)Statusdev (pack-written self-tests)JudgeClaude Haiku 5.5-low labels (UXMATRIX protocol)n140Seeds42Date2026-10-25 13:2xCaveatOutcome-label shares, not the ruler scale. Self-tests favour the packs. Correct-with-source is a lower bound (phone rows hold source titles only). Health content: AI-reviewed, not clinically reviewed. Nothing here is medical advice.Sourcenot in the public repositories yetOpen the data fileClose
• DEVICE-FINAL M4, the 60 UXMATRIX items on the phone: correct-any 0.350 (0.233–0.467), wrong-any 0.617dev · Haiku labels · 42 seedsDEVICE-FINAL M4, the 60 UXMATRIX items on the phone: correct-any 0.350 (0.233–0.467), wrong-any 0.617correct 0.35 of 60SetUXMATRIX 60 (dev)StatusdevJudgeClaude Haiku 5.5-low labels (UXMATRIX protocol)n60Seeds42Date2026-10-25 13:2xCaveatOutcome-label shares, not the ruler scale. Correct-with-source is a lower bound (phone rows hold source titles only).Sourcenot in the public repositories yetOpen the data fileClose
• DEVICE-FINAL ran as one interrupted and resumed run: seed 42 stopped at 77 of 88 turns at 11:31 when an APK install of LabRat 0ad476ca (brand files only) reached the phone; the 69 complete items were kept and the 6 unfin…recordDEVICE-FINAL ran as one interrupted and resumed run: seed 42 stopped at 77 of 88 turns at 11:31 when an APK install of LabRat 0ad476ca (brand files only) reached the phone; the 69 complete items were kept and the 6 unfinished ones were re-run on 0ad476ca; seeds 43 and the M2-M5 matrix ran entirely on 0ad476ca; seed 44 was dropped for timeinterrupted once, resumed; seed 44 droppedSeeds1 (44)Date2026-10-25 11:31Kinda record, not a measurement (set, status, judge, n: not recorded)CaveatProcess fact.Sourcenot in the public repositories yetOpen the data fileClose