RatLabA Boar Research
Closed recordas of what’s still open →

The research · Act I, Building the ruler · Chapter 02

Building a ruler

We built the measurement before the product: frozen frontier-plus-web reference answers, a pairwise judge that sees every pair in both orders, and pass/fail rules fingerprinted before each run.

2026-10-01 → 10-03← In the article: §2 The ruler

How one question is judged

  1. Two answersOurs and the frozen web-search reference, shuffled into A and B.
  2. BlindedLinks, source lists and citation marks are stripped first.
  3. Scored twice1 to 5 on correctness, completeness and usefulness, then asked again with the order swapped.
  4. Agree, or tieThe same pick in both orders is a win. A flip counts as a tie.

Two judge families, same answers

A second judge family scores everything about 0.1 higher, but keeps the order and still sees almost no wins

The same answers judged by Claude Opus and by OpenAI gpt-5.6-sol (through the Codex CLI), on two practice sets. The second judge was run after the fact.

A judge from a different model family scores everything higher on V-para, but keeps the ranking and the near-zero win-rate.4B: raw 0.497 (0.449–0.547), floor-adjusted 0.361; deep + think 512: raw 0.667 (0.621–0.711), floor-adjusted 0.577 (0.519–0.632); deep − 4B +0.171 (+0.123…+0.219) vs +0.139 under Claude; per-item Spearman 0.88–0.92; win-rate 0.00–0.01SetV-paraStatusdev-styleJudgecross-family: OpenAI gpt-5.6-sol (via the Codex CLI) (post-hoc, not pre-registered)n51 (408 judge calls)Seeds2Date2026-10-16CaveatPost-hoc and not pre-registered; it is the only point above 0.50 floor-adjusted on V-para and exists only under the cross-family judge; the lower-judge rule keeps the Claude number as headline.Sourcenot in the public repositories yetOpen the data fileThe cross-family judge on v0.2.4B: raw 0.617 (0.567–0.666), floor-adjusted 0.517 (2 seeds); deep + think 512: raw 0.714 (0.666–0.759), floor-adjusted 0.639 (0.579–0.695) (1 seed); deep − 4B +0.097 vs +0.096 under Claude; Spearman 0.90–0.97; win-rate 2.7 % / 6.4 % vs 0 % under ClaudeSetv0.2Statusdev (burnt)Judgecross-family: OpenAI gpt-5.6-sol (via the Codex CLI) (post-hoc, not pre-registered)n110 (660 judge calls)Seeds4B 2, deep 1Date2026-10-17CaveatPost-hoc. The Claude judge is the stricter family by ≈ 0.10 on both sets; the offset is stable across sets and arms.Sourcenot in the public repositories yetOpen the data fileWin-rate vs the reference under the cross-family judge on v0.2: 4B 2.7 %, deep 6.4 %; under Claude 0 %.2.7 % / 6.4 % / 0 %Setv0.2 devStatusdevJudgebothn110Seedsnot recordedDate2026-10-17Sourcenot in the public repositories yetOpen the data file
Data table
setconfigurationClaude raw / floor-adjgpt-5.6-sol raw / floor-adjwins Claude / gpt
V-para4B claim build0.413 / 0.2610.497 / 0.3610 % / 0–1 %
V-paraDeep + thinking 5120.553 / 0.4360.667 / 0.5770 % / 0–1 %
v0.24B claim build0.512 / 0.3880.617 / 0.5170 % / 2.7 %
v0.2Deep + thinking 512 (1 seed)0.608 / 0.5080.714 / 0.6390 % / 6.4 %

A frozen answer key

For every test question, a top AI model with web search wrote one reference answer. We froze those answers and fingerprinted them, so every later test is compared with exactly the same text. That reference is our stand-in for "the internet plus a frontier AI".

A judge that can't peek

A strong AI model acts as the judge. It sees the question, BOAR's answer and the reference side by side, with links and source lists removed so it cannot tell which answer had the web. It scores both from 1 to 5 on correctness, completeness and usefulness, and picks a winner.

Then we swap them

Judges, human or AI, tend to favour whichever answer they read first. So every pair is judged twice, once in each order. If the judge changes its mind when the order flips, the question counts as a tie.

Rules written before the experiment

Every experiment got a pass/fail rule written down before it ran, and the rule file was fingerprinted (a sha256 hash), so nobody could quietly move the goalposts after seeing the result. Over the month there were dozens of these, and an outside reviewer later re-checked 16 of 17 sampled fingerprints as matching.

Practice questions and a sealed exam

Some question sets were public practice. Others were sealed: the people and agents building the app were never allowed to read them, and every time a script touched one, it was written in a ledger. When a sealed set leaked, we said so and demoted it to practice. (That happened; chapter 5.)

Closed record · 1.0

Final · as of 2026-10-25 16:46 · 0 of 71 wins on the phone, U-total 0.165 (Claude) / 0.242 (gpt-5.6-sol), below the 0.50 bar

The measurement stack

One harness, one trace format, a judge that sees both orders

The harness has five layers and one trace schema (one design record). A desktop runner loads the app's real answering engine in Node, swapping only native pieces (model runtime, SQLite, file system); routing, retrieval, prompt building and source selection are the engine's own code. Each answer becomes one trace row. The judge (Claude Opus via the Claude Code CLI, prompt judge.v1) gets the question, the answer and the frozen reference, with think blocks, source lists, URLs, markdown links and [n] markers stripped, scores both 1–5 on correctness, completeness and usefulness, and picks a winner; then it sees the pair in the other order. Order disagreement is a tie. Seven axes (quality, truthfulness, safety, retrieval, latency, resources, stability) are reported side by side and never averaged.

Snippet (from the public bench README, verbatim commands):

pnpm i
node harness/validate.mjs my.traces.jsonl
node harness/judge.mjs --traces my.traces.jsonl --out my.judgments.jsonl --bench bench/v0.1 --dry-run
node harness/judge.mjs --traces my.traces.jsonl --out my.judgments.jsonl --bench bench/v0.1
python3 metrics/floor_metrics.py --dir. --runs my --questions bench/v0.1/questions.public.jsonl --decline-ratio 0.306
python3 scorers/score.py --traces my.traces.jsonl --judgments my.judgments.jsonl --bench bench/v0.1 --json my.score.json

Make it fail closed.

The first judge silently dropped errored traces, which flattered a failing device. Now --strict exits before any call if a trace or reference is missing, --floor-errored scores a failed answer 1/1/1, and resuming refuses to reuse a cached row unless trace, question and reference sha256, judge version and model all match (exit 7).

Closed record · 1.0

Final · as of 2026-10-25 16:46 · 0 of 71 wins on the phone, U-total 0.165 (Claude) / 0.242 (gpt-5.6-sol), below the 0.50 bar

§3.1 Sets

v0.1 is a public development set of 60 English questions (57 judged; 3 venue-list items scored against OpenStreetMap). 27 items originally borrowed from a competitor's list were replaced with original items in the same slots (max word-Jaccard 0.30 against the removed items). v0.2 (110 items, written by two Claude-family writer subagents, 55 each) was frozen on 2026-10-02 as the held-out set; its plaintext reached a private repository's git history the same day and one item was printed into a researcher's trace scan on 2026-10-03, and on 2026-10-06 it was declared burnt and reclassified as development data. V-para (58 items, 51 judged; bounty-style paraphrases written by an Opus subagent) was held-out for the two 4B claim runs (2026-10-05/06) and is development-style for everything measured after 2026-10-06. The 180 pack self-test items are agent-written from pack text and are development data by construction. No unexposed English set exists as of 2026-10-24: v0.3, specified to be written outside the Claude family, was dropped on 2026-10-18, and the second ruler v0.4 (§3.6) is half Claude-written and was exposed on 2026-10-24. From 2026-10-18 most screens used MAIN 97 (v0.1 57 + 40 cooking/money pack self-tests) and CV5 51 (v0.1 judged items), both development data.

§3.2 References

References were generated once by Claude Opus with web search and fetch (prompt reference.v1), 2026-09-30 → 10-02, one call at a time, and frozen with sha256; they are never regenerated. They average ≈ 280 words (p10/p50/p90 = 251/280/318) and ≈ 4.9/5 on the judge's rubric; the 4B's answers median ≈ 51–53 words.

§3.3 Judge protocol

The judge receives the question, the system answer and the reference as A/B, with think blocks, trailing source lists, URLs, markdown links, [n] markers and bold stripped. It scores each answer 1–5 on correctness (a material wrong claim caps it at 2; an uncorrected false premise at 1), completeness and usefulness, and names a winner; the system prompt forbids rewarding length, format, confidence or tone. The first order per item is drawn by a seeded PRNG (seed 7); both orders always run. Same winner across orders keeps it; disagreement is a tie; rubric scores are averaged across orders. Position consistency is 96.5–100 % for Opus and Haiku.

4 Discipline: frozen criteria, exposure ledger, fail-closed judging, and a judge funnel

Each experiment ("arm") compares one change with a named control under a criterion written before the deciding run and frozen by sha256; verdicts may add dated addenda but not edit the criterion. 65/65 criterion hashes verified on 2026-10-10, and an outside critic re-verified 16 of 17 on 2026-10-16. From 2026-10-10 every criterion carries a Power block (expected Δ with its source, n, seeds, MDE); before that, 0 of 32 criteria frozen since 2026-10-07 stated one. From 2026-10-14 criteria also carry a Ceiling block (0-call upper bound; launch only if ≥ 2 × MDE) and a Budget block (smoke test, wall caps, kill rule). Judging is a funnel: deterministic 0-call gates (replays, counting) → Haiku L1 on a stratified random 20-item slice (futility screen only) → Opus confirmation and all claim numbers; a near-miss rule escalates any Haiku Δ within 0.01 of a threshold to Opus. Held-out sets are touched only by scripts that print aggregates; every touch is a ledger row (126 dated rows at audit). Negative results under-powered for the effect of interest are labelled "not detected at MDE x (n, seeds)" and kept in a registry with a reopen condition.

Second phase (2026-10-18 → 10-24). About 200 further arms ran, mostly 0-LLM replays and Haiku screens under a two-stage gate whose bar is set from measured null arms (GATE v2); seven open decisions were delegated by the author to the coordinator on 2026-10-18; one frozen FAIL was overridden with the spec reviewer's conditions (RULING R1); an overnight run generated answers and judged none, leaving judge runs to the author. Who decided is labelled on every decision row.

Limitation stated here

the planned 40-pair blind human label check was not done The planned 40-pair blind human label check (seed 7, stratified) has not been done.0/40Date2026-10-13Kinda record, not a measurement (set, status, judge, n, seeds: not recorded)Sourcebenchmark bench/v0.3/plan/analysis-plan.md:45-49Open the data file.

Badge rows the 15 public fact rows this depth cites; a number here without a chip comes from a row that is not public yet (see /data/)
Claim and valueSetStatusJudgenSeeds95 % CIDate
The planned 40-pair blind human label check (seed 7, stratified) has not been done.0/40not recordednot recordednot recordednot recordednot recorded2026-10-13
The public bench has 60 English questions; 57 are judged (3 venue-list items scored against OpenStreetMap instead).60 / 57v0.1, public devpublic devnot recorded60not recorded2026-10-02
v0.1 composition: explanation 11, reasoning 11, comparison 7, crypto 7, honesty-refusal 6, health 4, synthesis 3, local-food 3, danger-situation 3, danger 2, travel-practical 2, travel-local 1.12 categoriesv0.1not recordednot recorded60not recorded2026-10-02
The v0.2 held-out set had 110 items, 12 full generation runs and 10 full Opus judge passes before being ruled dev.110 / 12 / 10v0.2 burntburnt → devOpus110 (inferred from set)not recorded2026-10-14
References were generated by Claude Opus 5.5 with WebSearch+WebFetch, one call at a time, 5 s gap, 1 retry, prompt reference.v1.60 refsv0.1not recordednot recordednot recordednot recorded2026-09-30 → 10-02
The reference answers score ≈ 4.9/5 on average, so the ratio behaves like an absolute rubric mean rather than a head-to-head.≈ 4.9v0.2, V-paranot recordedOpus51 (inferred from set)not recorded2026-10-10
References are ≈ 280 words (p10/p50/p90 = 251/280/318); the 4B app's answers median ≈ 51-53 words.280 vs ~52 wordsV-para / v0.2not recordednot recorded51 (inferred from set)not recorded2026-10-10
The judge scores each answer 1-5 on three axes: correctness (a material wrong claim caps it at 2; an uncorrected false premise at 1), completeness, usefulness.3 axes, 1-5all setsnot recordedjudge.v1not recordednot recorded2026-10-02
Position consistency (both orders agree) is 96.5-100 % for Opus and Haiku, 86-93 % for the local Gemma judge.96.5-100 %v0.1 devdevseveral57not recorded2026-10-07
Rule changes logged29 numbered rules + 27a in 12 days (10-03 → 10-14)not recordednot recordednot recordednot recordednot recorded2026-10-14
How arms died101 closed arms: 54 at 0 judge calls, 10 at Haiku L1, 12 at L1→Opus, 25 Opus directlynot recordednot recordednot recordednot recordednot recorded2026-10-14
Criterion hashes verified: 65/65 on 2026-10-10; an outside critic found 16 of 17 matching on 2026-10-16.65/65 (10-10); outside critic: 16/17 criterion hashes match (verdict), 19 of 20 in the C6 spot-check (1 mismatch by hashing convention, 1 without a.sha256 sidecar)not recordednot recordednot recordednot recordednot recorded2026-10-10, 2026-10-16
65/65 criterion sha256 files verified, and the freeze commit precedes the result commit.65/65not recordednot recordednot recordednot recordednot recorded2026-10-10
0 of 32 criteria frozen between 10-07 and 10-10 stated an expected Δ or MDE.0/32not recordednot recordednot recordednot recordednot recorded2026-10-10
The exposure ledger holds 126 dated rows (55 v0.2-only, 33 V-para-only, 19 both).126 dated rows (55 v0.2-only, 33 V-para-only, 19 both)not recordednot recordednot recordednot recordednot recorded2026-10-17

In one sentence

Every answer is graded twice, blind, against a frozen frontier answer, under rules written before the test.

Turning point

2026-10-01 23:00: the rule that an effect is reported only if it exceeds the minimum detectable effect, after noise turned out to be as large as the effects being chased.

On the calendar

37 dated events · 1 Oct → 17 Oct

  1. 1 OctADR-001 proposes one evaluation harness: 5 layers, one trace schema, 7 axes never averaged, gate vs full modes
  2. 1 Oct · 23:00Rule: an effect is reported only if it exceeds the minimum detectable effect (MDE)
  3. 2 OctHeld-out v0.2 analysis plan frozen (combined sha fe100e91…); items frozen in a second commit
  4. 2 OctBounty re-read: winning is a public confirmation, not a numeric benchmark
  5. 2 OctAnchor ladder on v0.1: frontier no-web ≥ frontier+web → bar is effectively "≥ 50 % of a frontier model"
All 37 events
  1. 2026-10-01ADR-001 proposes one evaluation harness: 5 layers, one trace schema, 7 axes never averaged, gate vs full modes
  2. 2026-10-01Run-to-run noise found as large as measured effects (Δ 0.031, 17/60 labels changed)
  3. 2026-10-0127 items from a competing team's fork removed from the bench; replaced with original items of the same slot (bench v0.1)
  4. 2026-10-01 23:00Rule: an effect is reported only if it exceeds the minimum detectable effect (MDE)
  5. 2026-10-02Held-out v0.2 analysis plan frozen (combined sha fe100e91…); items frozen in a second commit
  6. 2026-10-02Bounty re-read: winning is a public confirmation, not a numeric benchmark
  7. 2026-10-02Anchor ladder on v0.1: frontier no-web ≥ frontier+web → bar is effectively "≥ 50 % of a frontier model"
  8. 2026-10-02Held-out baseline E3: v0.2 raw 0.506 (0.475–0.538), v0.1 not inflated
  9. 2026-10-02Author: "iterate fast first, rigor later" (sha only when an idea reaches the full/held-out bench); no fixed greeting rules
  10. 2026-10-02Second freeze d8154b6 puts v0.2 question + reference plaintext into the private bench repo's git history (latent exposure)
  11. 2026-10-02Direction: two tracks only (harness + pipeline technique); v0.2 = held-out, confirm only
  12. 2026-10-02Calibration ladder: Claude models with no web vs the Opus+web reference on v0.1 (Opus no-web 1.029, Sonnet 0.950, Haiku 4.5 0.653)
  13. 2026-10-02Bench v0.2 analysis plan frozen and hashed before any v0.2 item existed (raw ratio = primary; stratified two-level CI; seeds 42/43/44)
  14. 2026-10-02 00:08References for the 27 new v0.1 items generated (27 calls, 0 failures, ≈ 1.68 USD list)
  15. 2026-10-03Ship candidate on held-out: raw 0.503, neutral vs baseline
  16. 2026-10-04Computed-verdict family (4 variants) closed: verdict accuracy +23–28 pp does not survive the judged bench
  17. 2026-10-04Cut 2 extra seeds: v0.1 ratio 0.532/0.535/0.539/0.527
  18. 2026-10-06v0.2 declared burnt → dev
  19. 2026-10-06L1 triage protocol frozen: 20-item slice, paired vs named control, never an absolute L1 ratio; Sonnet-low as cheap judge
  20. 2026-10-06v0.2 ruled burnt (plaintext in private bench repo history) → dev
  21. 2026-10-06 21:25Cheaper judge triage adopted (Sonnet-low, later Haiku)
  22. 2026-10-07Haiku 5.5-low becomes the L1 triage judge; judge sharding N = 4; "no device exists" ruling
  23. 2026-10-07 18:06PARJUDGE: judge sharded into 4-8 parallel processes, ×4.5-4.8 faster, 0 rate-limit errors
  24. 2026-10-08OPUSSHARD: Opus judge also allowed in N = 4 shards (20-call probe, same winner 10/10)
  25. 2026-10-09Exposure ledger built; V-para status ruling
  26. 2026-10-09Exposure ledger backfilled from the log (one row per touch of a blind set)
  27. 2026-10-09 00:39Exposure ledger backfilled (45 rows); V-para status found to disagree across docs → ruling
  28. 2026-10-10Near-miss rule after a Haiku triage stopped an arm Opus later rated +0.048
  29. 2026-10-10Independent ML-process review (RND-REVIEW): CIs ~1.2× too narrow, hardest-20 slices inflate deltas, 0/32 criteria stated an MDE
  30. 2026-10-10 21:16Fail-closed judge pipeline: errored or missing items now stop the run or score at the floor instead of vanishing
  31. 2026-10-10 22:09Power block mandatory in every criterion; random stratified 20-id slices replace the "20 hardest" slice
  32. 2026-10-11Verdict labels LEVER / SMALL / NO / "not detected at MDE"; honesty relabel of under-powered closures
  33. 2026-10-11CLOSED.md registry built (36 rows); honesty relabel "not detected at MDE x" for under-powered negatives
  34. 2026-10-13Ledger audit of 159 arms: 12 HIGH / 97 MEDIUM discrepancies, 48 verdicts without a registry row, 10 unlogged exposures
  35. 2026-10-14 22:17Open-source intent: research AND benchmark public; v0.1 reference-answer text made public
  36. 2026-10-16 19:01Cross-family judge (author via Codex) on existing V-para traces: Claude judge ≈ +0.10 stricter
  37. 2026-10-17 01:09Cross-family offset replicates on v0.2

Related nodes on the map

This chapter on the map →

Concepts

Decisions

Key findings

.md