The research · Act IV, From the bar to a product · Chapter 12
A second ruler (bench v0.4)
The first ruler scored a desktop engine against Claude-written references with a Claude judge on Claude-written questions.
Full thesis
For the submitted app the project wrote a second one: 75 new questions frozen on 2026-10-20 before any system answered them, written half by each judge family, scored as U-total with a zero for non-answers, two judge families and a split by whether the packs hold the answer. As of 2026-10-25 it has references from both families, a desktop-twin run of the submitted app's pre-rename engine that the author published as history (0 of 71 wins; lower family below the 0.50 bar), and an on-phone run that carries the headline.
Our first exam graded the wrong thing
The first ruler compared our desktop engine with answers written by one AI family, on questions written by that family, graded by that family. The bounty asks about an app on a phone. So we wrote a second ruler.
75 new questions, sealed before anyone answered
75 new questions, written half by Claude and half by an OpenAI model, fact-checked by both, and frozen on 20 October before any system had seen them. It is not written outside the Claude family; we say so. Bench v0.4 frozen75 items (71 headline + 4 anchors); 329 checklist points supported on the open web (at least one family, none contradicting; the 32 balance-round points by both families); sha f9ad2f2fSetv0.4 (held-out ruler)Statusheld-out ruler (v0.4; exposed to two engines 2026-10-24)Date2026-10-20Kinda record, not a measurement (judge, n, seeds: not recorded)Caveatwritten by Claude + OpenAI gpt-5.6-sol (the judge families)Sourcenot in the public repositories yetOpen the data file
A blank answer now scores zero
On the old scale a polite refusal still earned a few points. The new ruler first asks "did this even try to answer?"; if not, it scores zero. Then both AI families grade it, and the lower grade counts.
Graded, not yet published
By the evening of 24 October both AI families had written their answer keys, and the submitted app's engine had sat the exam on a desktop copy of the phone app, graded by both families. The grade exists; whether it is published is the author's call, and until then this page shows no number and no direction. A test on the phone itself ran on 25 October: 0 of 71 wins and a lower-family U-total of 0.165 on the phone, below the 0.50 bar.
The second ruler: bench v0.4 and the ruler spec
75 items frozen before any answer, U-total with a stage-1 zero, two judge families; on the desktop the submitted engine stayed below the bar, the phone run is the headline
Bench v0.4 has 75 items (71 headline + 4 anchors taken from the example questions in Vitalik Buterin's post that inspired the bounty, scored outside the headline), frozen on 2026-10-20 at 16:45 before any system answered; candidates came from Claude (36) and OpenAI gpt-5.6-sol (34) and were cross-fact-checked (329 checklist points supported on the open web). The score is U-total: Σ(app rubric mean − 1) / Σ(reference rubric mean − 1) over the 71 headline items, with health items weighted 0.5 (weight-1 beside), so the rubric floor maps to 0. Because judge.v1 still gave a bare refusal U 0.058, a stage-1 classifier (DECLINE2STAGE) zeroes detected non-answers before judge.v1 scores the rest. The verdict arm installs all six persona packs at once (A∪); the verdict statistic is the lower of the two judge families per bootstrap draw, and "meets the bar" requires CI low > 0.50 with every gate (RAM, storage, offline, speed, safety) passed. Spec r3 and r4 were frozen on 2026-10-21 before any app answer; r4 adds an information-available split: the packs hold the answer for 19 of 71 headline items, partly for 25, not at all for 27. As of 2026-10-24 20:34 both reference families are complete (75/75 each; the gpt-5.6-sol half at 20:05, 0 hard failures) REFS-V2, OpenAI gpt-5.6-sol half complete: 75 of 75 v0.4 references written with live web search under the frozen protocol, 0 hard failures, 97 calls (two items completed on a resumed run); references of both judge families now exist75 / 75; 0 hard failuresSetv0.4 (held-out ruler)Statusheld-out ruler (exposed)Judgenot recordedn75 items (88 turns)Seedsnot recordedDate2026-10-24 20:05CaveatProtocol frozen 2026-10-21 before any call (prompt 5e41177d, generator eb2fd7e2). Reference text not read by the coordinator.Sourcenot in the public repositories yetOpen the data file, and three engines have answered on desktop twins. The last of them is the final run on BoarLite’s earlier engine: one seed instead of the spec's two (the author's choice), each judge family against its own references, stage 1 not run, no errata, no phone run Final ruler run (RULER-FINAL) on BoarLite's final engine b4ef94f3 exists as of 2026-10-24 20:34: desktop twin of the app path (QuickMath, search port ON_NO_HEALTH, health gate, history as the app sends it), 71 headline items + 4 anchors of bench v0.4, one seed, each judge family against its own references (Claude Opus vs the Claude references; OpenAI gpt-5.6-sol vs its references), 150 + 150 judge calls, addendum frozen before any generation; its numbers were held until the author's publication decision of 2026-10-25 (now public as; history, the headline is DEVICE-FINAL)Setv0.4 (held-out ruler; exposed)Statusheld-out ruler (v0.4; now exposed to three engines)JudgeClaude Opus + OpenAI gpt-5.6-sol (each vs its own references)n71 headline items + 4 anchorsSeeds1 (42)Date2026-10-24 20:34CaveatDeviations from the spec: one seed (spec: two; the author's choice), one reference family per judge family, stage 1 (DECLINE2STAGE) not run, no errata, desktop twin not the APK on a phone (the spec labels desktop-twin numbers dev/parity evidence). v0.4 is the claim set: no tuning after this run.Sourcenot in the public repositories yetOpen the data file. Published by the author's decision of 2026-10-25 as history: 0 of 71 wins under both judge families (one gpt-5.6-sol tie); U-total (the desktop run of the earlier engine) Claude Opus 0.164 (0.123–0.208), OpenAI gpt-5.6-sol 0.245 (0.194–0.300); the lower family, 0.164 (0.123–0.206), is below the 0.50 bar; by information available (Claude family) 0.164 where the packs hold the answer (n 19), 0.174 partly (n 25), 0.155 not at all (n 27): with the information present, the 1.7B did not deliver it RULER-FINAL U-total of BoarLite b4ef94f3 on v0.4: Claude family 0.164 (0.123-0.208), OpenAI gpt-5.6-sol family 0.245 (0.194-0.300); minimum over families 0.164 (0.123-0.206); win-rate 0 of 71 (Claude) and 0 of 71 with 1 tie (gpt-5.6-sol); fixed-text refusals on 3 itemsClaude 0.164; gpt-5.6-sol 0.245; min 0.164; win-rate 0Setv0.4 (held-out ruler; exposed)Statusheld-out ruler (v0.4; exposed)JudgeClaude Opus + OpenAI gpt-5.6-sol (each vs its own references)n71 headline itemsSeeds1 (42)Date2026-10-24 20:34CaveatMetric: ruler v0.4 U-total (71 headline items, health weight 0.5); never on one axis with the judge.v1 ratio or outcome labels. Desktop twin, one seed. Published by the author's decision of 2026-10-25 (AD-15 (b)): BoarLite/LabRat is the author's own submission. History, not the headline: desktop twin of engine b4ef94f3, one seed, before the Rat persona line and the cleanup; not a measurement of LabRat 7788924a. The headline is the on-phone run DEVICE-FINAL (OF-8).Sourcenot in the public repositories yetOpen the data file. The paired delta against the native engine branch stays internal (it prints that branch's number). They are numbers for the earlier engine: the shipped LabRat changes the grounding prompt's identity line after this run Engine change after the ruler run: in LabRat 7788924a the grounding prompt's identity line changed from "You are BOAR, an offline AI research app running on this phone…" to "You are Rat, the assistant in LabRat, an offline AI research app running on this phone…". The final ruler run (RULER-FINAL, 2026-10-24 20:34) was on b4ef94f3, before this change: its number is for b4ef94f3 (desktop twin), and DEVICE-FINAL measures LabRat 7788924a on the phonepersona line changed after RULER-FINALDate2026-10-25 02:20Kinda record, not a measurement (set, status, judge, n, seeds: not recorded)CaveatThe final ruler number (RULER-FINAL) measures engine b4ef94f3 on its desktop twin; LabRat 7788924a adds the Rat persona line to the grounding prompt after that run, so the ruler number is not a measurement of LabRat 7788924a.Sourcenot in the public repositories yetOpen the data file, and the on-phone re-measure is the final on-phone run (“final on-phone matrix”, landed 2026-10-25; criterion frozen 2026-10-25) DEVICE-FINAL criterion frozen (sha256 recorded) on 2026-10-25 03:3x, before any matrix generation: LabRat 7788924a answers on a Pixel 8a 8 GB in airplane mode on AC power. Matrix: M1 ruler v0.4 (75 items, 88 turns) on seeds 42, 43 and 44, judged as in RULER-FINAL (Claude Opus vs Claude references on all three seeds; OpenAI gpt-5.6-sol vs its references on seed 42, further seeds optional), reporting per family the U-total, CI, win-rate, refusals, the minimum over families, the three-seed mean and a phone-vs-desktop parity read against the b4ef94f3 desktop twin (not a verdict); M2 the 65 safety probes (0 LLM); M3 the 140 on-K13 pack self-tests and M4 the 60 UXMATRIX items (Claude Haiku labels); M5 latency, time to first token, decode speed, memory, battery temperature, three cold starts and the app's network counters over the night. Claim numbers = M1 (both families named, seeds stated, win-rate first) and M5 (speed, memory, offline); M2-M4 are product numbers. No tuning, no engine changematrix M1-M5; claim numbers = M1 + M5Setbench v0.4 (exposed); 65 safety probes; 140 on-K13 self-tests; UXMATRIX 60StatusdevJudgeClaude Opus + OpenAI gpt-5.6-sol (M1); Claude Haiku 5.5-low labels (M3, M4); none (M2, M5)n75 items × 3 seeds + 265 product itemsSeeds42, 43, 44 (M1); 42 (M2-M4)Date2026-10-25 03:3xCaveatNo DEVICE-FINAL result is in this record; the matrix was running as of this update.Sourcenot in the public repositories yetOpen the data file (the ruler run predates the Rat persona line).
3.6 A second instrument for the submitted app
Because the first instrument scored a desktop engine against Claude-written references on Claude-written items, a second ruler was specified for the submitted app. Bench v0.4 has 75 items (71 headline; 4 anchors taken from the example questions in Vitalik Buterin's post that inspired the bounty, outside the headline), drafted by Claude (36 candidates) and OpenAI gpt-5.6-sol (34), cross-fact-checked (329 checklist points supported on the open web by at least one family with none contradicting), and frozen on 2026-10-20 before any system answered (sha f9ad2f2f) Bench v0.4 frozen75 items (71 headline + 4 anchors); 329 checklist points supported on the open web (at least one family, none contradicting; the 32 balance-round points by both families); sha f9ad2f2fSetv0.4 (held-out ruler)Statusheld-out ruler (v0.4; exposed to two engines 2026-10-24)Date2026-10-20Kinda record, not a measurement (judge, n, seeds: not recorded)Caveatwritten by Claude + OpenAI gpt-5.6-sol (the judge families)Sourcenot in the public repositories yetOpen the data file. The headline statistic U-total is a ratio of sums of rubric mass above the floor (mean − 1) against uncapped frontier-plus-web references from both families, with health items at weight 0.5 (weight-1 beside; decided after the items existed) Ruler headline weights the 11 health_safety items at 0.5, adds a safety gate, publishes weight-1 U beside itin spec v1 §2; reviewer F3 flags timing (decided after items existed)Date2026-10-20Kinda record, not a measurement (set, status, judge, n, seeds: not recorded)Caveatwho decided: author; why: bounty asks for casual lookup/research, never names health; alternatives: weight 1; reversible: yes AI-reviewed, not clinically reviewed.Sourcenot in the public repositories yetOpen the data file and a stage-1 classifier that zeroes detected non-answers (DECLINE2STAGE), because judge.v1 scores a bare refusal at U 0.058 (0.037–0.081) judge.v1 does not give declines a true zerobare refusal U 0.058 (0.037-0.081); hedge 0.053; fluent no-content 0.016SetdevStatusdevJudgeClaude Haikunnot recordedSeedsnot recordedDate2026-10-21Caveatleak in correctness dimensionSourcenot in the public repositories yetOpen the data file. A binary-checklist judge v2 failed its pre-set retest gate Judge v2 (binary checklist) test-retestCE test-retest 0.966 vs v1 0.924 (needed 0.974); SD 0.80x (needed 0.60x) -> NO-GOSetdevStatusdevDate10-20Kinda record, not a measurement (judge, n, seeds: not recorded)Sourcenot in the public repositories yetOpen the data file; keeping judge.v1 with the stage-1 zero (RULING R1) overrode a frozen FAIL and was accepted with conditions by the spec reviewer. Spec r3 (sha 0fac2286) and r4 (sha 82772e33, adding an information-available split, FACT-AVAIL, and ORACLE / ORACLE-0 rows) were frozen on 2026-10-21 before any app answer Ruler spec frozenr3 sha 0fac2286 (10-21 20:48); r4 sha 82772e33 (21:18)Date2026-10-21Kinda record, not a measurement (set, status, judge, n, seeds: not recorded)Caveatprivate pre-registrationSourcenot in the public repositories yetOpen the data file. The verdict arm installs all six persona packs (A∪); the verdict is the lower family per bootstrap draw with CI low > 0.50 and all gates. FACT-AVAIL labels: AVAILABLE 19 / PARTIAL 25 / NONE 27 of 71 (strict 15 / 23 / 33), lists quarantined Information-available labels on v0.4AVAILABLE 19 / PARTIAL 25 / NONE 27 (strict 15/23/33) of 71Setv0.4 (quarantined labels)Statusheld-out ruler (v0.4; exposed to two engines 2026-10-24)JudgeClaude Haiku + Claude Opusnnot recordedSeedsnot recordedDate2026-10-21Caveatcounts only; lists stay quarantinedSourcenot in the public repositories yetOpen the data file. As of 2026-10-24 20:34: references 75/75 for both families (Claude median 703 words; gpt-5.6-sol completed 20:05, 0 hard failures) REFS-V2, OpenAI gpt-5.6-sol half complete: 75 of 75 v0.4 references written with live web search under the frozen protocol, 0 hard failures, 97 calls (two items completed on a resumed run); references of both judge families now exist75 / 75; 0 hard failuresSetv0.4 (held-out ruler)Statusheld-out ruler (exposed)Judgenot recordedn75 items (88 turns)Seedsnot recordedDate2026-10-24 20:05CaveatProtocol frozen 2026-10-21 before any call (prompt 5e41177d, generator eb2fd7e2). Reference text not read by the coordinator.Sourcenot in the public repositories yetOpen the data file; the set counts as exposed from 2026-10-24 Bench v0.4 exposurefirst answers by two engines 10-24; any later engine change = developed after exposureSetv0.4 (held-out ruler, now exposed)Statusheld-out ruler (v0.4; exposed to two engines 2026-10-24)JudgeClaude Opusnnot recordedSeeds2Date2026-10-24Later2026-10-24 20:34: a third engine (BoarLite b4ef94f3) answered v0.4 in the final ruler run; its numbers are internal pending the author.Sourcenot in the public repositories yetOpen the data file and has now been answered by three engines on desktop twins. The final run (RULER-FINAL) Final ruler run (RULER-FINAL) on BoarLite's final engine b4ef94f3 exists as of 2026-10-24 20:34: desktop twin of the app path (QuickMath, search port ON_NO_HEALTH, health gate, history as the app sends it), 71 headline items + 4 anchors of bench v0.4, one seed, each judge family against its own references (Claude Opus vs the Claude references; OpenAI gpt-5.6-sol vs its references), 150 + 150 judge calls, addendum frozen before any generation; its numbers were held until the author's publication decision of 2026-10-25 (now public as; history, the headline is DEVICE-FINAL)Setv0.4 (held-out ruler; exposed)Statusheld-out ruler (v0.4; now exposed to three engines)JudgeClaude Opus + OpenAI gpt-5.6-sol (each vs its own references)n71 headline items + 4 anchorsSeeds1 (42)Date2026-10-24 20:34CaveatDeviations from the spec: one seed (spec: two; the author's choice), one reference family per judge family, stage 1 (DECLINE2STAGE) not run, no errata, desktop twin not the APK on a phone (the spec labels desktop-twin numbers dev/parity evidence). v0.4 is the claim set: no tuning after this run.Sourcenot in the public repositories yetOpen the data file scored BoarLite's engine b4ef94f3 with one seed (spec: two), each judge family against its own references (not the two-family mean), stage 1 not run, no errata; the spec labels desktop-twin numbers dev/parity evidence. Published by the author on 2026-10-25 as history: win-rate 0 of 71 under both families (one gpt-5.6-sol tie); U-total (desktop twin of engine b4ef94f3) 0.164 (0.123–0.208) Claude Opus, 0.245 (0.194–0.300) gpt-5.6-sol; minimum over families 0.164 (0.123–0.206), below the bar by the pre-registered rule; FACT-AVAIL (Claude) AVAILABLE 0.164 (n 19), PARTIAL 0.174 (n 25), NONE 0.155 (n 27), so the information's presence did not raise U-total RULER-FINAL U-total of BoarLite b4ef94f3 on v0.4: Claude family 0.164 (0.123-0.208), OpenAI gpt-5.6-sol family 0.245 (0.194-0.300); minimum over families 0.164 (0.123-0.206); win-rate 0 of 71 (Claude) and 0 of 71 with 1 tie (gpt-5.6-sol); fixed-text refusals on 3 itemsClaude 0.164; gpt-5.6-sol 0.245; min 0.164; win-rate 0Setv0.4 (held-out ruler; exposed)Statusheld-out ruler (v0.4; exposed)JudgeClaude Opus + OpenAI gpt-5.6-sol (each vs its own references)n71 headline itemsSeeds1 (42)Date2026-10-24 20:34CaveatMetric: ruler v0.4 U-total (71 headline items, health weight 0.5); never on one axis with the judge.v1 ratio or outcome labels. Desktop twin, one seed. Published by the author's decision of 2026-10-25 (AD-15 (b)): BoarLite/LabRat is the author's own submission. History, not the headline: desktop twin of engine b4ef94f3, one seed, before the Rat persona line and the cleanup; not a measurement of LabRat 7788924a. The headline is the on-phone run DEVICE-FINAL (OF-8).Sourcenot in the public repositories yetOpen the data file. The paired delta against the native engine branch is withheld (it carries that branch's absolute; the app maintainer's numbers stay internal). The engine it measured, b4ef94f3, precedes the rename to LabRat: 7788924a changes the grounding prompt's identity line ("You are Rat…"), so the RULER-FINAL number is not attributed to LabRat without that caveat Engine change after the ruler run: in LabRat 7788924a the grounding prompt's identity line changed from "You are BOAR, an offline AI research app running on this phone…" to "You are Rat, the assistant in LabRat, an offline AI research app running on this phone…". The final ruler run (RULER-FINAL, 2026-10-24 20:34) was on b4ef94f3, before this change: its number is for b4ef94f3 (desktop twin), and DEVICE-FINAL measures LabRat 7788924a on the phonepersona line changed after RULER-FINALDate2026-10-25 02:20Kinda record, not a measurement (set, status, judge, n, seeds: not recorded)CaveatThe final ruler number (RULER-FINAL) measures engine b4ef94f3 on its desktop twin; LabRat 7788924a adds the Rat persona line to the grounding prompt after that run, so the ruler number is not a measurement of LabRat 7788924a.Sourcenot in the public repositories yetOpen the data file (the ruler run predates the Rat persona line). DEVICE-FINAL (criterion frozen 2026-10-25 03:3x) DEVICE-FINAL criterion frozen (sha256 recorded) on 2026-10-25 03:3x, before any matrix generation: LabRat 7788924a answers on a Pixel 8a 8 GB in airplane mode on AC power. Matrix: M1 ruler v0.4 (75 items, 88 turns) on seeds 42, 43 and 44, judged as in RULER-FINAL (Claude Opus vs Claude references on all three seeds; OpenAI gpt-5.6-sol vs its references on seed 42, further seeds optional), reporting per family the U-total, CI, win-rate, refusals, the minimum over families, the three-seed mean and a phone-vs-desktop parity read against the b4ef94f3 desktop twin (not a verdict); M2 the 65 safety probes (0 LLM); M3 the 140 on-K13 pack self-tests and M4 the 60 UXMATRIX items (Claude Haiku labels); M5 latency, time to first token, decode speed, memory, battery temperature, three cold starts and the app's network counters over the night. Claim numbers = M1 (both families named, seeds stated, win-rate first) and M5 (speed, memory, offline); M2-M4 are product numbers. No tuning, no engine changematrix M1-M5; claim numbers = M1 + M5Setbench v0.4 (exposed); 65 safety probes; 140 on-K13 self-tests; UXMATRIX 60StatusdevJudgeClaude Opus + OpenAI gpt-5.6-sol (M1); Claude Haiku 5.5-low labels (M3, M4); none (M2, M5)n75 items × 3 seeds + 265 product itemsSeeds42, 43, 44 (M1); 42 (M2-M4)Date2026-10-25 03:3xCaveatNo DEVICE-FINAL result is in this record; the matrix was running as of this update.Sourcenot in the public repositories yetOpen the data file re-runs v0.4 on the phone with LabRat 7788924a, three seeds under Claude Opus and seed 42 under gpt-5.6-sol, with a phone-vs-desktop parity read against b4ef94f3 that is not a verdict; it landed on 2026-10-25 and is the headline.
Badge rows the 15 public fact rows this depth cites; a number here without a chip comes from a row that is not public yet (see /data/)
| Claim and value | Set | Status | Judge | n | Seeds | 95 % CI | Date |
|---|---|---|---|---|---|---|---|
| Bench v0.4 frozen75 items (71 headline + 4 anchors); 329 checklist points supported on the open web (at least one family, none contradicting; the 32 balance-round points by both families); sha f9ad2f2f | v0.4 (held-out ruler) | held-out ruler (v0.4; exposed to two engines 2026-10-24) | not recorded | not recorded | not recorded | 2026-10-20 | |
| Ruler headline weights the 11 health_safety items at 0.5, adds a safety gate, publishes weight-1 U beside itin spec v1 §2; reviewer F3 flags timing (decided after items existed) | not recorded | not recorded | not recorded | not recorded | not recorded | 2026-10-20 | |
| judge.v1 does not give declines a true zerobare refusal U 0.058 (0.037-0.081); hedge 0.053; fluent no-content 0.016 | dev | dev | Claude Haiku | not recorded | not recorded | 2026-10-21 | |
| Judge v2 (binary checklist) test-retestCE test-retest 0.966 vs v1 0.924 (needed 0.974); SD 0.80x (needed 0.60x) -> NO-GO | dev | dev | not recorded | not recorded | not recorded | 10-20 | |
| Ruler spec frozenr3 sha 0fac2286 (10-21 20:48); r4 sha 82772e33 (21:18) | not recorded | not recorded | not recorded | not recorded | not recorded | 2026-10-21 | |
| Information-available labels on v0.4AVAILABLE 19 / PARTIAL 25 / NONE 27 (strict 15/23/33) of 71 | v0.4 (quarantined labels) | held-out ruler (v0.4; exposed to two engines 2026-10-24) | Claude Haiku + Claude Opus | not recorded | not recorded | 2026-10-21 | |
| REFS-V2, OpenAI gpt-5.6-sol half complete: 75 of 75 v0.4 references written with live web search under the frozen protocol, 0 hard failures, 97 calls (two items completed on a resumed run); references of both judge families now exist75 / 75; 0 hard failures | v0.4 (held-out ruler) | held-out ruler (exposed) | not recorded | 75 items (88 turns) | not recorded | 2026-10-24 20:05 | |
| Bench v0.4 exposurefirst answers by two engines 10-24; any later engine change = developed after exposure | v0.4 (held-out ruler, now exposed) | held-out ruler (v0.4; exposed to two engines 2026-10-24) | Claude Opus | not recorded | 2 | 2026-10-24 | |
| Final ruler run (RULER-FINAL) on BoarLite's final engine b4ef94f3 exists as of 2026-10-24 20:34: desktop twin of the app path (QuickMath, search port ON_NO_HEALTH, health gate, history as the app sends it), 71 headline items + 4 anchors of bench v0.4, one seed, each judge family against its own references (Claude Opus vs the Claude references; OpenAI gpt-5.6-sol vs its references), 150 + 150 judge calls, addendum frozen before any generation; its numbers were held until the author's publication decision of 2026-10-25 (now public as; history, the headline is DEVICE-FINAL) | v0.4 (held-out ruler; exposed) | held-out ruler (v0.4; now exposed to three engines) | Claude Opus + OpenAI gpt-5.6-sol (each vs its own references) | 71 headline items + 4 anchors | 1 (42) | 2026-10-24 20:34 | |
| RULER-FINAL U-total of BoarLite b4ef94f3 on v0.4: Claude family 0.164 (0.123-0.208), OpenAI gpt-5.6-sol family 0.245 (0.194-0.300); minimum over families 0.164 (0.123-0.206); win-rate 0 of 71 (Claude) and 0 of 71 with 1 tie (gpt-5.6-sol); fixed-text refusals on 3 itemsClaude 0.164; gpt-5.6-sol 0.245; min 0.164; win-rate 0 | v0.4 (held-out ruler; exposed) | held-out ruler (v0.4; exposed) | Claude Opus + OpenAI gpt-5.6-sol (each vs its own references) | 71 headline items | 1 (42) | 2026-10-24 20:34 | |
| Engine change after the ruler run: in LabRat 7788924a the grounding prompt's identity line changed from "You are BOAR, an offline AI research app running on this phone…" to "You are Rat, the assistant in LabRat, an offline AI research app running on this phone…". The final ruler run (RULER-FINAL, 2026-10-24 20:34) was on b4ef94f3, before this change: its number is for b4ef94f3 (desktop twin), and DEVICE-FINAL measures LabRat 7788924a on the phonepersona line changed after RULER-FINAL | not recorded | not recorded | not recorded | not recorded | not recorded | 2026-10-25 02:20 | |
| DEVICE-FINAL criterion frozen (sha256 recorded) on 2026-10-25 03:3x, before any matrix generation: LabRat 7788924a answers on a Pixel 8a 8 GB in airplane mode on AC power. Matrix: M1 ruler v0.4 (75 items, 88 turns) on seeds 42, 43 and 44, judged as in RULER-FINAL (Claude Opus vs Claude references on all three seeds; OpenAI gpt-5.6-sol vs its references on seed 42, further seeds optional), reporting per family the U-total, CI, win-rate, refusals, the minimum over families, the three-seed mean and a phone-vs-desktop parity read against the b4ef94f3 desktop twin (not a verdict); M2 the 65 safety probes (0 LLM); M3 the 140 on-K13 pack self-tests and M4 the 60 UXMATRIX items (Claude Haiku labels); M5 latency, time to first token, decode speed, memory, battery temperature, three cold starts and the app's network counters over the night. Claim numbers = M1 (both families named, seeds stated, win-rate first) and M5 (speed, memory, offline); M2-M4 are product numbers. No tuning, no engine changematrix M1-M5; claim numbers = M1 + M5 | bench v0.4 (exposed); 65 safety probes; 140 on-K13 self-tests; UXMATRIX 60 | dev | Claude Opus + OpenAI gpt-5.6-sol (M1); Claude Haiku 5.5-low labels (M3, M4); none (M2, M5) | 75 items × 3 seeds + 265 product items | 42, 43, 44 (M1); 42 (M2-M4) | 2026-10-25 03:3x | |
| v0.4 candidate writersClaude 36, OpenAI gpt-5.6-sol 34; 302/307 points supported, 5 disputed | v0.4 drafts | frozen ruler set (held-out for the ruler; counts only) | not recorded | not recorded | not recorded | 10-20 | |
| RULER-FINAL information-available split: Claude family U-total 0.164 where the installed packs hold the answer (n 19), 0.174 partly (n 25), 0.155 not at all (n 27); gpt-5.6-sol family 0.272 / 0.238 / 0.233: with the information present, the 1.7B does not deliver itAVAILABLE 0.164 vs NONE 0.155 (Claude) | v0.4 (held-out ruler; exposed) | held-out ruler (v0.4; exposed) | Claude Opus + OpenAI gpt-5.6-sol | 19 / 25 / 27 headline items | 1 (42) | 2026-10-24 20:34 | |
| Current-city stratum weight7 items = 10.7 % of headline weight (7/65.5) | v0.4 tags | held-out ruler (v0.4; exposed to two engines 2026-10-24) | arithmetic | not recorded | not recorded | 2026-10-21 |
In one sentence
A fairer exam is written and sealed; the submitted app has sat it, and the grade waits for the author (as of 24 October 2026, 20:34).
Turning point
2026-10-20 16:45: the v0.4 item set is frozen before any system has seen it.
On the calendar
22 dated events · 18 Oct → 24 Oct
- 18 Oct · 16:07Author delegates D1-D7; adopted as recommended (no v0.3; submit with claim v4.2.4; site v1.0 only at closure)
- 19 Oct · 23:08BENCH-RULER synthesis: five misreadings of the bounty in the current bench; new public ruler design
- 20 Oct · 16:45v0.4 ruler item set FROZEN (75 items, sha f9ad2f2f...) before any system answered
- 20 Oct · 19:07Author "grill": 1.7B stays default, 4B optional tier for 10-12 GB phones, two-stage screens, health weight 0.5 in the ruler, safety-cards-first work archived
- 20 Oct · 20:24Author: ruler split into with-reference / no-reference parts; product refuses by default with "show anyway"; persona-pack arms
All 22 events
- 2026-10-18 16:07Author delegates D1-D7; adopted as recommended (no v0.3; submit with claim v4.2.4; site v1.0 only at closure)
- 2026-10-19 23:08BENCH-RULER synthesis: five misreadings of the bounty in the current bench; new public ruler design
- 2026-10-20 00:13v0.4 ruler candidate: 70 items (Claude 36, OpenAI gpt-5.6-sol 34), 5 disputed points for the author
- 2026-10-20 16:45v0.4 ruler item set FROZEN (75 items, sha f9ad2f2f...) before any system answered
- 2026-10-20 19:07Author "grill": 1.7B stays default, 4B optional tier for 10-12 GB phones, two-stage screens, health weight 0.5 in the ruler, safety-cards-first work archived
- 2026-10-20 20:24Author: ruler split into with-reference / no-reference parts; product refuses by default with "show anyway"; persona-pack arms
- 2026-10-20 20:39Ruler spec v1 r2 written + independent adversarial review (25 findings); verdict arm = union of persona packs
- 2026-10-20 21:45Persona map v1 drafted + independently approved (not frozen; waits author veto)
- 2026-10-21 03:51A-union (6 packs + gazetteer) on Pixel 8a with 1.7B / Gemma / 4B: no memory cost, +2.4 s serial pack search, 0 LMK
- 2026-10-21 04:01FLOOR-ANCHORS: decline arm INVALID under judge.v1 (U 0.058); true-zero judge variants NO-ADOPT
- 2026-10-21 04:39RV2 separability ladder (Haiku): ruler separates 4B from 1.7B/Gemma; 1.7B level with closed-book 1.5B
- 2026-10-21 05:28CV5 ladder under Opus: 1.7B vs closed-book 1.5B becomes separable (judge-dependent sentence)
- 2026-10-21 12:45RULING R1: judge = judge.v1 + DECLINE2STAGE zero per detected non-answer
- 2026-10-21 13:13GRILL session: honest 8 GB product + open ruler; no 35B delivery; research site; final score on the native APK
- 2026-10-21 20:28Author ruler decisions: verdict on the app with packs, OpenAI gpt-5.6-sol second judge, headline waits for new RC
- 2026-10-21 20:48RULER-SPEC v1 r3 frozen (private)
- 2026-10-21 20:58REFS-V2 Claude half frozen: 75/75 Opus+web references on v0.4 before any app answer
- 2026-10-21 21:18RULER-SPEC r4 frozen + FACT-AVAIL labels (19/71 AVAILABLE)
- 2026-10-24 16:08NATIVE-RC: the app's own engine vs ENGINE-RC (UXMATRIX, probes, factbook v2, ruler v0.4 Claude half)
- 2026-10-24 20:05References of the second judge family complete: 75 of 75 on v0.4
- 2026-10-24 20:34Final ruler run on BoarLite's engine (desktop twin, one seed, both judge families); numbers await the author's publication decision
- 2026-10-24 20:34RULER-FINAL: U-total Claude 0.164, gpt-5.6-sol 0.245, min 0.164, win-rate 0 (one seed)