RatLabA Boar Research
Closed recordas of what’s still open →

The research · Act III, The lab and its instruments · Chapter 10

Measuring the measurer

Claude wrote the questions, wrote the references and judged them, so the project measured its own instrument: judge noise is tiny, seed noise is not, and a judge from another company scores everything about 0.10 higher while keeping the same ranking and a near-zero win-rate.

2026-10-02 → 10-21← In the article: §2 The ruler

How one question is judged

  1. Two answersOurs and the frozen web-search reference, shuffled into A and B.
  2. BlindedLinks, source lists and citation marks are stripped first.
  3. Scored twice1 to 5 on correctness, completeness and usefulness, then asked again with the order swapped.
  4. Agree, or tieThe same pick in both orders is a win. A flip counts as a tie.

Two judge families, same answers

A second judge family scores everything about 0.1 higher, but keeps the order and still sees almost no wins

The same answers judged by Claude Opus and by OpenAI gpt-5.6-sol (through the Codex CLI), on two practice sets. The second judge was run after the fact.

A judge from a different model family scores everything higher on V-para, but keeps the ranking and the near-zero win-rate.4B: raw 0.497 (0.449–0.547), floor-adjusted 0.361; deep + think 512: raw 0.667 (0.621–0.711), floor-adjusted 0.577 (0.519–0.632); deep − 4B +0.171 (+0.123…+0.219) vs +0.139 under Claude; per-item Spearman 0.88–0.92; win-rate 0.00–0.01SetV-paraStatusdev-styleJudgecross-family: OpenAI gpt-5.6-sol (via the Codex CLI) (post-hoc, not pre-registered)n51 (408 judge calls)Seeds2Date2026-10-16CaveatPost-hoc and not pre-registered; it is the only point above 0.50 floor-adjusted on V-para and exists only under the cross-family judge; the lower-judge rule keeps the Claude number as headline.Sourcenot in the public repositories yetOpen the data fileThe cross-family judge on v0.2.4B: raw 0.617 (0.567–0.666), floor-adjusted 0.517 (2 seeds); deep + think 512: raw 0.714 (0.666–0.759), floor-adjusted 0.639 (0.579–0.695) (1 seed); deep − 4B +0.097 vs +0.096 under Claude; Spearman 0.90–0.97; win-rate 2.7 % / 6.4 % vs 0 % under ClaudeSetv0.2Statusdev (burnt)Judgecross-family: OpenAI gpt-5.6-sol (via the Codex CLI) (post-hoc, not pre-registered)n110 (660 judge calls)Seeds4B 2, deep 1Date2026-10-17CaveatPost-hoc. The Claude judge is the stricter family by ≈ 0.10 on both sets; the offset is stable across sets and arms.Sourcenot in the public repositories yetOpen the data fileWin-rate vs the reference under the cross-family judge on v0.2: 4B 2.7 %, deep 6.4 %; under Claude 0 %.2.7 % / 6.4 % / 0 %Setv0.2 devStatusdevJudgebothn110Seedsnot recordedDate2026-10-17Sourcenot in the public repositories yetOpen the data file
Data table
setconfigurationClaude raw / floor-adjgpt-5.6-sol raw / floor-adjwins Claude / gpt
V-para4B claim build0.413 / 0.2610.497 / 0.3610 % / 0–1 %
V-paraDeep + thinking 5120.553 / 0.4360.667 / 0.5770 % / 0–1 %
v0.24B claim build0.512 / 0.3880.617 / 0.5170 % / 2.7 %
v0.2Deep + thinking 512 (1 seed)0.608 / 0.5080.714 / 0.6390 % / 6.4 %

Claude graded Claude

One AI family wrote the test questions, wrote the reference answers and did the grading. That is a known risk: a judge may prefer answers that sound like itself.

A judge from another company

So we re-graded the same answers with a judge from another AI company, using the same rules. It scored everything about 0.10 higher. It kept the same order of who is better. And it still almost never preferred our answer over the frontier one. A judge from a different model family scores everything higher on V-para, but keeps the ranking and the near-zero win-rate.4B: raw 0.497 (0.449–0.547), floor-adjusted 0.361; deep + think 512: raw 0.667 (0.621–0.711), floor-adjusted 0.577 (0.519–0.632); deep − 4B +0.171 (+0.123…+0.219) vs +0.139 under Claude; per-item Spearman 0.88–0.92; win-rate 0.00–0.01SetV-paraStatusdev-styleJudgecross-family: OpenAI gpt-5.6-sol (via the Codex CLI) (post-hoc, not pre-registered)n51 (408 judge calls)Seeds2Date2026-10-16CaveatPost-hoc and not pre-registered; it is the only point above 0.50 floor-adjusted on V-para and exists only under the cross-family judge; the lower-judge rule keeps the Claude number as headline.Sourcenot in the public repositories yetOpen the data file

We keep the stricter one

Under the other judge, the big model with thinking crosses 0.50 on the honest scale. We do not headline that. Our rule, written before we had the number, says that when two judges disagree, the lower one is the headline.

Luck of the draw

Re-grading identical answers barely changes anything. Re-generating answers with a different random seed changes the score by about 0.02, sometimes 0.04. That is why important numbers use two runs.

Closed record · 1.0

Final · as of 2026-10-25 16:46 · 0 of 71 wins on the phone, U-total 0.165 (Claude) / 0.242 (gpt-5.6-sol), below the 0.50 bar

The numbers that drive this chapter

  • Second Claude judge (Fable 5.1): same winner 453/456 Second Claude judge (Fable 5.1) vs Opus: same winner 453/456 verdicts, Pearson r 0.939, κ (correctness ≥ 4) 0.894, ratio +0.004…+0.010.453/456Setv0.1 devStatusdevJudgeOpus vs Fablennot recordedSeedsnot recordedDate2026-10-02Caveatboth judges are Claude: tests noise, not self-preference; ≈ 23 USD listSourcebenchmark TRACE-FORMAT.md:78Open the data file.
  • Haiku 5.5-low vs Opus on v0.1: r 0.922, κ 0.651, ratio −0.069 (lead with these; "same winner 54/57" is near-uninformative because the reference wins ≈ 100 % of items) Haiku 5.5-low vs Opus on 57 v0.1 items: same winner 54/57, r 0.922, κ 0.651, ratio −0.069 lower, 3.52 s/call, 0 failures.54/57Setv0.1 devStatusdevJudgeHaiku vs Opusn57,Seedsnot recordedDate2026-10-07Caveat'same winner 54/57' carries almost no information because the reference wins ≈ 100 % of items (CRITIQUE-2026-10-16.md:32); quote the informative statistics instead: r 0.922, κ 0.651, level −0.069, Haiku Δ ≈ 0.70-0.85× the Opus Δ on pack armsCheckoverclaimSourceresearch record method/L1-PROTOCOL.md:11Open the data file.
  • Local open-weights judge (Gemma 4 26B): +0.11 lenient, κ 0.32/0.55, same ranking (-47) Local Gemma 4 26B judge: same winner as Opus 49/57 and 50/57, κ 0.32 / 0.55, ratio +0.110 / +0.112 above Opus, 41-43 s/call.+0.11Setv0.1 devStatusdevJudgeGemma vs Opusnnot recordedSeedsnot recordedDate2026-10-07Sourcenot in the public repositories yetOpen the data file.
  • Opus test-retest: σ ≈ 0.003 raw; per-item r 0.96; winner 51/51 Re-judging the exact same answers barely moves the score: judge noise is negligible next to seed noise.σ_judge ≈ 0.003 raw / 0.004 floor-adjusted; per-item r 0.96; same winner 51/51SetV-paraStatusdev-styleJudgeClaude Opus (judge.v1, both orders)n51 (102 Opus calls)Seeds1 (re-judge of s42)Date2026-10-11Sourceresearch record reports/BAR-TABLE.md:123research record reports/RULE-CHANGES.md:23Open the data file.
  • Seed noise: SD 0.020 raw / 0.025 floor on the deep tier; two seeds differ −0.039 (−0.072…−0.011) (-34) Three seeds of the recommended deep configuration show how much one run can move.raw mean 0.550, SD 0.020 (0.533 / 0.572 / 0.544); floor-adjusted mean 0.432, SD 0.025 (0.412 / 0.460 / 0.425); 0 wins each seedSetV-paraStatusdev-styleJudgeClaude Opus (judge.v1, both orders)n51Seeds3 (42–44)Date2026-10-10Sourceresearch record reports/BAR-TABLE.md:40research record reports/RULE-CHANGES.md:16Open the data file.
  • Reference length: shortening references to ≤ 80 words moves H floor-adj 0.412 → 0.514 (s42); ≈ 17 % of the gap is length, ≈ 83 % substance Shortening references to ≤ 80 words lifts H floor-adj 0.412 → 0.514 and K 0.261 → 0.322+0.102 (+0.068…+0.144) H; +0.060 KSetV-para dev, references alteredStatusdevJudgeOpusn51 (inferred from set)Seeds1Date2026-10-10Caveat≈ 17 % of H's gap is reference length, ≈ 83 % substance; not a bar readingSourceresearch record reports/BAR-TABLE.md:98Open the data file.
  • Correctness alone: H floor-adj 0.660 (0.608–0.712) on V-para; completeness 0.301, usefulness 0.341 On correctness alone H is above 0.50 floor-adj; the gap is in completeness and usefulnessV-para correctness floor-adj 0.660 (0.608–0.712) vs completeness 0.301 / usefulness 0.341SetV-para devStatusdevJudgeOpusn51 (inferred from set)Seeds2Date2026-10-10Caveat"correct but thin" also describes an always-decline system on those dims; not a bar readingSourceresearch record reports/BAR-TABLE.md:89,95Open the data file.
  • Cross-family judge (post-hoc, 408 + 660 calls, same rubric, orders and blinding): V-para 4B raw 0.497 / floor 0.361; H raw 0.667 / floor 0.577 (0.519–0.632); win-rate 0.00–0.01. v0.2: 4B 0.617 / 0.517 (2 seeds); H 0.714 / 0.639 (1 seed); deep − 4B gap +0.097 vs Claude +0.096; win-rate 2.7 % / 6.4 % vs 0 % A judge from a different model family scores everything higher on V-para, but keeps the ranking and the near-zero win-rate.4B: raw 0.497 (0.449–0.547), floor-adjusted 0.361; deep + think 512: raw 0.667 (0.621–0.711), floor-adjusted 0.577 (0.519–0.632); deep − 4B +0.171 (+0.123…+0.219) vs +0.139 under Claude; per-item Spearman 0.88–0.92; win-rate 0.00–0.01SetV-paraStatusdev-styleJudgecross-family: OpenAI gpt-5.6-sol (via the Codex CLI) (post-hoc, not pre-registered)n51 (408 judge calls)Seeds2Date2026-10-16CaveatPost-hoc and not pre-registered; it is the only point above 0.50 floor-adjusted on V-para and exists only under the cross-family judge; the lower-judge rule keeps the Claude number as headline.Sourcenot in the public repositories yetOpen the data file.
  • Outside critique: numbers real (15/17 re-derive), internal summaries overstated; 7 re-wordings applied Critique: most traced numbers re-derive15/17 numbers, 16/17 criterion hashes matchDate2026-10-16Kinda record, not a measurement (set, status, judge, n, seeds: not recorded)Sourcenot in the public repositories yetOpen the data file.
  • Later calibration (10-20/21, dev): Haiku and Opus agree on confident errors (κ 0.75) far better than on "correct" (κ 0.49; 0.22 on warned answers); Opus finds 98 correct answers Haiku misses vs 10 the other way, so Haiku-only correctness rates are low-biased Haiku vs Opus calibrationκ CE 0.75, κ correct 0.49 (warned 0.22); Opus finds 98 correct Haiku misses vs 10 the other way; U shift +0.022SetdevStatusdevJudgebothnnot recordedSeedsnot recordedDate2026-10-21Sourcenot in the public repositories yetOpen the data file. judge.v1 gives a bare refusal U 0.058 (0.037–0.081), not zero judge.v1 does not give declines a true zerobare refusal U 0.058 (0.037-0.081); hedge 0.053; fluent no-content 0.016SetdevStatusdevJudgeClaude Haikunnot recordedSeedsnot recordedDate2026-10-21Caveatleak in correctness dimensionSourcenot in the public repositories yetOpen the data file: the reason the second ruler adds a stage-1 zero (chapter 12).

Closed record · 1.0

Final · as of 2026-10-25 16:46 · 0 of 71 wins on the phone, U-total 0.165 (Claude) / 0.242 (gpt-5.6-sol), below the 0.50 bar

7.1 One family wrote, referenced and judged: what changes under another

Items (v0.2, V-para), references and the judge all come from the Claude family. We ran four checks. A second Claude judge (Fable 5.1) agrees with Opus on 453/456 verdicts. Haiku 5.5-low tracks Opus on v0.1 with r = 0.922, κ = 0.651 and a −0.069 level offset (agreement on the winner, 54/57, is near-uninformative because the reference wins almost every item) and compresses paired gaps to 0.70–0.95× of Opus's on pack arms and as low as 0.25× on one 4B-vs-deep pair. A local Gemma 4 26B judge is +0.11 lenient (κ 0.32/0.55) but preserves the deep − 4B ranking (+0.026 vs +0.029). Post hoc, the author re-judged existing traces with an OpenAI model using the same rubric, orders and blinding (408 calls on V-para, 660 on v0.2). It scores every configuration higher: on V-para the 4B rises from 0.413 / 0.261 to 0.497 / 0.361 and the 512 configuration from 0.553 / 0.436 to 0.667 / 0.577 (0.519–0.632); on v0.2 the 4B rises to 0.617 / 0.517 (two seeds) and the deep configuration to 0.714 / 0.639 (0.579–0.695) (one seed). The deep − 4B gap is preserved (V-para +0.171 vs +0.139; v0.2 +0.097 vs +0.096), per-item Spearman is 0.88–0.97, and win-rate stays near zero (V-para 0.00–0.01; v0.2 2.7 % and 6.4 % vs 0 %). Per the analysis plan, when two families differ by more than 0.05 raw the lower is the headline; we report the Claude numbers as headline and the cross-family numbers beside them.

§7.2 Noise budget

Re-judging unchanged answers with Opus gives σ_judge ≈ 0.003 raw (per-item r = 0.96; winner 51/51). Re-generating answers is the dominant noise: on the deep tier, seeds 42/43/44 give raw 0.533 / 0.572 / 0.544 (SD 0.020; floor-adjusted SD 0.025), and two seeds of the same configuration differ by −0.039 (−0.072…−0.011), a CI excluding zero. The 4B on V-para is seed-stable (identical ratios across seeds; per-item r = 0.81), and the desktop 4B path is byte-deterministic on a fixed host (20/20). A single 20-item Haiku pass carries ±0.03–0.04 of judge noise, about the size of the decision threshold, so slices screen for futility only; "hardest-20" slices inflated deltas 3–10× and were retired. The house reading rule is that a one-seed deep-tier Δ on V-para below 0.05 is seed noise (raised from 0.03 on 2026-10-16).

§7.3 Multiplicity and selection

Across ≈ 60 judged comparisons, ≈ 1.5 false "CI low > 0" results are expected at α = 0.05; the large levers survive a Bonferroni-style correction (X2b z ≈ 5.5, LITE5 z ≈ 6, thinking z ≈ 4.2). The larger risk is selection on the same development sets (winner's curse on the deep configuration; levers designed on the 180 self-tests), which is why development and held-out readings are separated throughout.

§7.4 Two judges on labels, refusals and correctness (2026-10-20/21)

On double-labelled development units Haiku and Opus agree on confident errors (κ 0.75) far better than on "correct" (κ 0.49; 0.22 on warned answers); Opus finds 98 correct answers that Haiku misses vs 10 the other way, a +0.02 U shift Haiku vs Opus calibrationκ CE 0.75, κ correct 0.49 (warned 0.22); Opus finds 98 correct Haiku misses vs 10 the other way; U shift +0.022SetdevStatusdevJudgebothnnot recordedSeedsnot recordedDate2026-10-21Sourcenot in the public repositories yetOpen the data file. A Haiku confident-error count moves by about ± 1.3 per 100 on retest of identical answers. judge.v1 gives non-answers a positive score (bare refusal 0.058, hedge 0.053, fluent no-content 0.016) judge.v1 does not give declines a true zerobare refusal U 0.058 (0.037-0.081); hedge 0.053; fluent no-content 0.016SetdevStatusdevJudgeClaude Haikunnot recordedSeedsnot recordedDate2026-10-21Caveatleak in correctness dimensionSourcenot in the public repositories yetOpen the data file, and attempts to fix this inside the rubric flipped 5–7 of 40 confident-error verdicts JUDGE-V1_1 / V1_1B: Can one rule line make judge.v1 give declines a true zero?declines U 0.000, but CE flips 7/40 and 5/40; retest SD 0.0265/0.0295 vs 0.0206Setdecline variants; 40 controlsJudgeClaude HaikuDate2026-10-21Kinda record, not a measurement (status, n, seeds: not recorded)Caveatcriterion frozen: yes (bf560ea9; 6edbacf6); arm verdict: NO-ADOPT bothSourcenot in the public repositories yetOpen the data file, hence the stage-1 zero of §3.6. Reference draws matter little: two uncapped Opus draws move U by ≤ 0.015 Reference draw noisetwo uncapped Opus draws move U ≤ 0.015SetdevStatusdevJudgeClaude Haikunnot recordedSeedsnot recordedDate2026-10-21Caveatbetween-family variance unmeasuredSourcenot in the public repositories yetOpen the data file. Same-family judge lift in the literature is 3.4–8.4 points for open-model families and untested for Claude and GPT Same-family judge lift in the literature3.4-8.4 pp (open-model families; Claude/GPT not tested)SetexternalStatusexternalDate2026-10-21Kinda record, not a measurement (judge, n, seeds: not recorded)Caveatexternal paper, abstract onlySourcenot in the public repositories yetOpen the data file.

Badge rows the 19 public fact rows this depth cites; a number here without a chip comes from a row that is not public yet (see /data/)
Claim and valueSetStatusJudgenSeeds95 % CIDate
Haiku vs Opus calibrationκ CE 0.75, κ correct 0.49 (warned 0.22); Opus finds 98 correct Haiku misses vs 10 the other way; U shift +0.022devdevbothnot recordednot recorded2026-10-21
judge.v1 does not give declines a true zerobare refusal U 0.058 (0.037-0.081); hedge 0.053; fluent no-content 0.016devdevClaude Haikunot recordednot recorded2026-10-21
JUDGE-V1_1 / V1_1B: Can one rule line make judge.v1 give declines a true zero?declines U 0.000, but CE flips 7/40 and 5/40; retest SD 0.0265/0.0295 vs 0.0206decline variants; 40 controlsnot recordedClaude Haikunot recordednot recorded2026-10-21
Reference draw noisetwo uncapped Opus draws move U ≤ 0.015devdevClaude Haikunot recordednot recorded2026-10-21
Same-family judge lift in the literature3.4-8.4 pp (open-model families; Claude/GPT not tested)externalexternalnot recordednot recordednot recorded2026-10-21
Second Claude judge (Fable 5.1) vs Opus: same winner 453/456 verdicts, Pearson r 0.939, κ (correctness ≥ 4) 0.894, ratio +0.004…+0.010.453/456v0.1 devdevOpus vs Fablenot recordednot recorded2026-10-02
Haiku 5.5-low vs Opus on 57 v0.1 items: same winner 54/57, r 0.922, κ 0.651, ratio −0.069 lower, 3.52 s/call, 0 failures.54/57v0.1 devdevHaiku vs Opus57,not recorded2026-10-07
Haiku compresses paired gaps: deep35 − 4B was +0.007 under Haiku vs +0.029 under Opus (×0.25).+0.007 vs +0.029v0.1 devdevboth57not recorded2026-10-07
Haiku L1 compresses paired gaps vs OpusHaiku Δ ≈ 0.70-0.95× the Opus Δ across six pack/retrieval arms (X1 0.70, pack 0.76, ONLY 0.83, X2b 0.85, ST2 0.88, ADD 0.95); per-item Δ r 0.75-0.84180 self-testsnot recordedHaiku vs Opus180 (inferred from set)not recorded2026-10-14/16
Local Gemma 4 26B judge: same winner as Opus 49/57 and 50/57, κ 0.32 / 0.55, ratio +0.110 / +0.112 above Opus, 41-43 s/call.+0.11v0.1 devdevGemma vs Opusnot recordednot recorded2026-10-07
The Gemma judge preserved the ranking: deep35 − 4B +0.026 vs Opus +0.029.+0.026 vs +0.029v0.1 devdevboth57not recorded2026-10-07
A judge from a different model family scores everything higher on V-para, but keeps the ranking and the near-zero win-rate.4B: raw 0.497 (0.449–0.547), floor-adjusted 0.361; deep + think 512: raw 0.667 (0.621–0.711), floor-adjusted 0.577 (0.519–0.632); deep − 4B +0.171 (+0.123…+0.219) vs +0.139 under Claude; per-item Spearman 0.88–0.92; win-rate 0.00–0.01V-paradev-stylecross-family: OpenAI gpt-5.6-sol (via the Codex CLI) (post-hoc, not pre-registered)51 (408 judge calls)22026-10-16
Re-judging the exact same answers barely moves the score: judge noise is negligible next to seed noise.σ_judge ≈ 0.003 raw / 0.004 floor-adjusted; per-item r 0.96; same winner 51/51V-paradev-styleClaude Opus (judge.v1, both orders)51 (102 Opus calls)1 (re-judge of s42)2026-10-11
Three seeds of the recommended deep configuration show how much one run can move.raw mean 0.550, SD 0.020 (0.533 / 0.572 / 0.544); floor-adjusted mean 0.432, SD 0.025 (0.412 / 0.460 / 0.425); 0 wins each seedV-paradev-styleClaude Opus (judge.v1, both orders)513 (42–44)2026-10-10
4B generation on desktop CPU is byte-deterministic20/20 identical across 5 pairingsnot recordednot recordednot recordednot recordednot recorded2026-10-13
On the 20 hardest items Haiku showed +0.029 where Opus gave +0.091; on the full set they agreed within 0.005 (+0.022 vs +0.026).×3 compression on sliceV-para devdevboth20not recorded2026-10-10
A single n=20 Haiku pass carries ±0.03–0.04 noiseper-item retest SD 0.22–0.28 rubric pointsdevdevHaiku18not recordedn 18–302026-10-16
Large levers survive multiplicity correctionX2b z ≈ 5.5, LITE5-180 z ≈ 6, DEEPTHINK z ≈ 4.2not recordednot recordednot recordednot recordednot recorded2026-10-16
Multiplicity check≈ 60 judged comparisons → ≈ 1.5 false "CI low > 0" expected; X2b z ≈ 5.5 survives Bonferroni over 100not recordednot recordednot recordednot recordednot recorded2026-10-16

In one sentence

A different judge changes the level, not the ranking, and not the wins.

Turning point

2026-10-16 19:01, the cross-family judge: the deep tier crosses 0.50 floor-adjusted only under the other family, and the win-rate does not move.

On the calendar

15 dated events · 2 Oct → 17 Oct

  1. 10 OctSEEDVAR: H over 3 seeds mean floor-adj 0.432 (SD 0.025); one-seed deep Δ < 0.03 = seed noise
  2. 10 Oct · 13:30R&D process review (10 Opus agents: 7 readers, 2 refuters, 1 critic; 76 findings): "measurement machine better than typical, pointed at the wrong target"
  3. 10 Oct · 22:31BAR-TABLE written (all configs vs bar); LENGTHSHARE: ≈ 17 % of H's gap is reference length, ≈ 83 % substance
  4. 16 OctAdversarial outside critique: numbers real, internal narrative overstated; 7 re-wordings applied 18:28
  5. 16 OctAdversarial critique: lite "= 4B" overstated (kit −0.111), lite gap is reading too (−0.070 at identical contexts); closed loop risk
All 15 events
  1. 2026-10-02Second Claude judge (Fable 5.1) agrees with Opus on 453/456 verdicts
  2. 2026-10-07JUDGE2: local open-weights judge (Gemma 4 26B) keeps the ranking but scores +0.11 above Opus
  3. 2026-10-07 17:58HAIKUJUDGE: Haiku 5.5-low adopted as L1 triage judge; JUDGE2 local Gemma judge = ranking only (+0.11 lenient)
  4. 2026-10-10SEEDVAR: H over 3 seeds mean floor-adj 0.432 (SD 0.025); one-seed deep Δ < 0.03 = seed noise
  5. 2026-10-10 13:30R&D process review (10 Opus agents: 7 readers, 2 refuters, 1 critic; 76 findings): "measurement machine better than typical, pointed at the wrong target"
  6. 2026-10-10 22:31BAR-TABLE written (all configs vs bar); LENGTHSHARE: ≈ 17 % of H's gap is reference length, ≈ 83 % substance
  7. 2026-10-11 03:00JUDGEVAR: re-judging unchanged answers gives σ_judge ≈ 0.003; common-mode CIs adopted for deep-tier rows
  8. 2026-10-16Adversarial outside critique: numbers real, internal narrative overstated; 7 re-wordings applied 18:28
  9. 2026-10-16Critique raises the one-seed noise rule from 0.03 to ≥ 0.05
  10. 2026-10-16Adversarial critique: lite "= 4B" overstated (kit −0.111), lite gap is reading too (−0.070 at identical contexts); closed loop risk
  11. 2026-10-16Lite tier (Qwen3-1.7B) evaluated; LITESAFETY PASS; critique corrects "1.7B = 4B"
  12. 2026-10-16 19:02XFAM: cross-family judge (OpenAI gpt-5.6-sol via the Codex CLI, 408 calls) on V-para traces: ≈ +0.10 vs Claude, same ranking, win-rate ≈ 0
  13. 2026-10-16 21:00Claim v4.2 / v4.2.1: judge family named on every ratio; native comparison kept internal
  14. 2026-10-17 01:09XFAM-V02: same judge on v0.2 (660 calls): offset holds (+0.10 raw / +0.13 floor-adj), deep − 4B gap identical
  15. 2026-10-17 20:52Claim v4.2.3: "the lower judge is the headline"; judge family named next to every ratio

Related nodes on the map

This chapter on the map →

Concepts

Experiments

Key findings

.md