The research · Act I, Building the ruler · Chapter 03
The floor trap
The obvious score, "our answer's points divided by the reference's", gives a system that refuses every question about 0.31 for free, so the project moved its headline to a floor-adjusted ratio and win-rate, and kept printing all three.
Try it · two scales, one answer
free points a blank answer gets the 0.50 line
The floor trap, measured
A blank answer already scores 0.306, so most of a raw 0.50 is free
The rubric never goes below 1 of 5. Remove that free point from both sides and the two builds land in different places.
bench/v0.3/plan/analysis-plan.md:24research record reports/BAR-TABLE.md:14research record README.mdOpen the data fileThe frozen 4B claim build on the bounty-style set (V-para) delivers about a quarter of the frontier reference's value above "nothing", and loses every head-to-head.raw 0.413 (0.384–0.445); floor-adjusted 0.261 (0.224–0.300); win/tie/loss 0/0/51 in both seedsSetV-paraStatusheld-out for this run (cut 3 on 2026-10-05, cut 6 on 2026-10-06); dev-style for everything measured after 2026-10-06JudgeClaude Opus (judge.v1, both orders)n51Seeds2 (42, 43)Date2026-10-06CaveatCut 7 = cut 6 on this set (the FACTS table fires 0 times on it). Public BAR-TABLE prints CI high 0.300; one claim draft prints 0.299 (rounding).Sourceresearch record reports/BAR-TABLE.md:35research record reports/SET-STATUS.md:15research record README.md:25research record reports/BAR-TABLE.mdOpen the data fileRecommended deep configuration (35B mixture-of-experts + 512-token thinking budget) on the bounty-style set.raw 0.553 (0.524–0.581; with seed noise 0.513–0.592); floor-adjusted 0.436 (0.401–0.472; with seed noise 0.387–0.486); win 0/51; desktop p50 ≈ 79 sSetV-paraStatusdev-styleJudgeClaude Opus (judge.v1, both orders)n51Seeds2Date2026-10-09CaveatChosen as the best of ≥ 8 deep configurations on this same set: expect a winner's-curse discount of about 0.01–0.02. QUEUE prints raw 0.552; the two-seed aggregate is 0.553. Desktop only.Sourceresearch record reports/BAR-TABLE.md:39research record reports/BAR-TABLE.md:126research record README.md:28Open the data fileData table
| what | raw (95 % CI) | floor-adjusted (95 % CI) |
|---|---|---|
| Always declines (blank-answer floor) | 0.306 | 0 by construction |
| 4B claim build | 0.413 (0.384–0.445) | 0.261 (0.224–0.3) |
| Deep + thinking 512 (desktop) | 0.553 (0.524–0.581) | 0.436 (0.401–0.472) |
A blank answer still gets points
The judge's scale runs from 1 to 5, not 0 to 5. So even an answer that says nothing earns a 1. If you divide BOAR's points by the reference's points, a version of BOAR that politely declines every single question already scores about 0.31.
So 0.50 was 61 % free
That means the bounty's 0.50 line was more than halfway reached by doing nothing. A number that rewards silence is not a fair ruler.
Subtract the free point
The fix: subtract the free point from both sides before dividing. On this "floor-adjusted" scale, silence scores 0 and matching the reference scores 1. A floor-adjusted 0.50 means "half of the frontier answer's value above nothing". It is a stricter scale, and we made it our headline. We still print the flattering one next to it, so you can see both.
And we count wins
A third, plainer number: on how many questions did the judge prefer BOAR's answer over the frontier answer, in both orders? We report it everywhere. You will see a lot of zeros.
Our real starting point
Here is the small model the phone runs, on the bounty-style questions it had never seen: 0.413 on the flattering scale, 0.261 on the honest one, and 0 wins out of 51. The frozen 4B claim build on the bounty-style set (V-para) delivers about a quarter of the frontier reference's value above "nothing", and loses every head-to-head.raw 0.413 (0.384–0.445); floor-adjusted 0.261 (0.224–0.300); win/tie/loss 0/0/51 in both seedsSetV-paraStatusheld-out for this run (cut 3 on 2026-10-05, cut 6 on 2026-10-06); dev-style for everything measured after 2026-10-06JudgeClaude Opus (judge.v1, both orders)n51Seeds2 (42, 43)Date2026-10-06CaveatCut 7 = cut 6 on this set (the FACTS table fires 0 times on it). Public BAR-TABLE prints CI high 0.300; one claim draft prints 0.299 (rounding).Sourceresearch record reports/BAR-TABLE.md:35research record reports/SET-STATUS.md:15research record README.md:25research record reports/BAR-TABLE.mdOpen the data file
Three scales
Raw, floor-adjusted, win/tie/loss: read the last two first
With S and R the mean rubric (mean of three axes, both orders averaged) of the app and of the reference: raw = ΣS / ΣR; floor-adjusted = Σ(S − 1) / Σ(R − 1); above-decline = (raw − D) / (1 − D) with D = 0.306, the measured raw score of a system that declines everything. Because the rubric's minimum is 1, raw 0.50 is about 61 % free. References score about 4.9/5, so the ratio behaves like an absolute rubric mean. Example on the 4B (v0.1, dev): raw 0.512 = floor-adjusted 0.387 = above-decline 0.297. We report all three, lead with floor-adjusted and win/tie/loss.
§3.4 Metrics
Let S_i and R_i be the order-averaged mean rubric of system and reference on item i. Raw = Σ S_i / Σ R_i. Floor-adjusted = Σ (S_i − 1) / Σ (R_i − 1). Above-decline = (raw − D)/(1 − D) with D = 0.306, the mean judged rubric of 79 declined answers over 7 runs divided by the mean reference rubric. Win/tie/loss is the order-combined winner per item. CIs: stratified (category) two-level bootstrap, items within stratum then runs within item, 2,000 resamples, seed 11, percentile; paired differences use the same procedure. For deep-tier rows we also report a common-mode CI that adds run-level seed variance: point ± 1.96·sqrt(SE_item² + (σ_run/√seeds)²).
§3.5 The bar and its history
The analysis plan, frozen and hashed on 2026-10-02 before any v0.2 item existed, set the primary metric to the raw ratio with a bar "CI low > 0.50" on both held-out sets and listed the floor-adjusted and above-decline ratios as pre-registered companion metrics, with win/tie/loss as a secondary. On 2026-10-02 the rubric floor was quantified (an always-decline system scores raw ≈ 0.31, ≈ 61 % of the bar). On 2026-10-06 an internal audit promoted the floor-adjusted ratio and win/tie/loss to the headline; the claim text adopted it on 2026-10-07. The primary research target since then is V-para floor-adjusted CI low ≥ 0.50 with two seeds; the formal raw rule is still printed. Some arm reports of 2026-10-08/09 read the bar with an additional "win-rate > 0" conjunct; from 2026-10-10 win-rate is "reported, never gated". We list this trail because the change of headline came after results; it went toward the stricter scale and all scales are printed for every row.
Badge rows the 4 public fact rows this depth cites; a number here without a chip comes from a row that is not public yet (see /data/)
| Claim and value | Set | Status | Judge | n | Seeds | 95 % CI | Date |
|---|---|---|---|---|---|---|---|
| Raw ratio = Σ S / Σ R, where S and R are the app's and reference's mean rubric (mean of 3 axes, both orders averaged).formula | all | not recorded | not recorded | not recorded | not recorded | not recorded | |
| CIs are a stratified (by category) two-level bootstrap: resample items within stratum, then runs within item; 2,000 resamples; seed 11; percentile 95 %.B 2000, seed 11 | all | not recorded | not recorded | not recorded | 1 (11) | 2026-10-02 | |
| Paired differences between two systems use the same stratified two-level bootstrap on the three scales.B 2000, seed 11 | all | not recorded | not recorded | not recorded | 1 (11) | not recorded | |
| On the 4B, raw 0.512 = 0.387 floor-removed = 0.297 above always-decline0.512 / 0.387 / 0.297 | v0.1 dev | dev | Opus | not recorded | 2 | 2 seeds; floor-removed CI 0.319–0.448 | 2026-10-02 |
In one sentence
We grade on the scale where a blank answer scores zero, and still show the flattering scale beside it.
Turning point
the 10-06 pipeline audit: "rigorous process, looping direction" (E-process timeline). The switch went toward the stricter scale; an outside critic later called it post-hoc but "not self-serving".
On the calendar
4 dated events · 2 Oct → 17 Oct
- 2 Oct · 11:15Author's morning decisions: bar on the raw ratio, but always publish raw + floor-removed + above-decline side by side
- 6 OctPipeline audit: rubric floor gives an always-decline system ≈ 0.31 "for free"; decision D1 = floor-adjusted headline; D5 = treat v0.2 as burnt
- 6 Oct · 22:40Honest-metric rewrite: headline becomes floor-adjusted ratio + win/tie/loss, raw second
- 17 OctClaim draft v4.2.4: Claude Opus stays the headline (lower judge), win-rate first