RatLabA Boar Research
Closed recordas of what’s still open →

The research · Act III, The lab and its instruments · Chapter 08

The agent lab

One independent author ran this as a small research organisation made of AI agents: a coordinator that freezes rules and logs everything, researchers in lanes that propose but never measure, independent reviewers that decide wording, and an unattended loop; most ideas died for free, and the rules got stricter after every mistake.

2026-10-01 → 10-24← In the article: §8 How it was made

The cemetery of ideas

Most ideas died here: 215 of 387 ended without a gain (187 closed, 28 too small to see), 57 adopted

Every experiment in the open registry, one mark each, placed on the day it was decided and filed by verdict. Point at, tap or focus a mark for its name and recorded result.

How arms died101 closed arms: 54 at 0 judge calls, 10 at Haiku L1, 12 at L1→Opus, 25 Opus directlyDate2026-10-14Kinda record, not a measurement (set, status, judge, n, seeds: not recorded)Caveatcitation fixSourcenot in the public repositories yetOpen the data file2 rows without a date are not plotted. Unverified results are marked "as recorded".
Data table
verdictmeaningcount
Adoptedshipped or recommended57
Paritysame quality, cheaper10
Measurementmeasured a property57
Toolingmethod or infrastructure32
Openparked or running16
Not detectedtoo small to see; not a no28
Closedthe idea failed its rule187
(undated, not plotted)2

A lab made of AI agents

There was no university and no team of humans. The author set the direction and the limits; AI agents did the rest in roles: a coordinator that wrote the rules and kept the log, researchers in separate lanes who proposed ideas but never graded their own, and independent reviewers who checked every public sentence.

Most ideas died for free

Of 101 ideas that were closed, 54 were killed by cheap checks before any AI judge looked at them.

Four days on autopilot

For four days the coordinator ran by itself, waking up on a timer: 133 wake-ups, about 10,500 judge calls, 62 machine-hours of answer generation. The result: one improvement, and only on practice questions.

Rules grew out of mistakes

A ten-hour run produced 45 empty answers out of 120 because nobody ran a ten-question test first. The same large model file got deleted twice by housekeeping. The loop sat idle for hours. Each mistake became a written rule: 29 of them in 12 days.

Closed record · 1.0

Final · as of 2026-10-25 16:46 · 0 of 71 wins on the phone, U-total 0.165 (Claude) / 0.242 (gpt-5.6-sol), below the 0.50 bar

How the work was run: an agent lab, with numbers

A coordinator, lanes, reviewers and a loop: 54 of 101 ideas died for free

The research ran as an AI-agent lab. A coordinator session owned the queue, wrote and hashed each pass/fail rule before a run, launched jobs on two Macs (a Mac mini M4 16 GB for long generations, a MacBook M1 Pro 16 GB for replays and small models) and logged every result. Research agents in lanes (retrieval, models, evaluation/safety, native performance) wrote technique cards but never ran or graded experiments. Independent reviewer agents decided wording. Jobs ran detached with sentinel files so they survived session restarts. Of 101 closed arms, 54 died at zero judge calls (deterministic replays, counting), 10 at the cheap Haiku slice, 12 after Opus confirmation and 25 with Opus directly. Over four unattended days the loop made 133 ticks and ≈ 10.5k judge calls and produced one dev-only lever; ≈ 31 machine-hours were avoidable, mostly long runs launched without a smoke test.

Closed record · 1.0

Final · as of 2026-10-25 16:46 · 0 of 71 wins on the phone, U-total 0.165 (Claude) / 0.242 (gpt-5.6-sol), below the 0.50 bar

4 Discipline: frozen criteria, exposure ledger, fail-closed judging, and a judge funnel

Each experiment ("arm") compares one change with a named control under a criterion written before the deciding run and frozen by sha256; verdicts may add dated addenda but not edit the criterion. 65/65 criterion hashes verified on 2026-10-10, and an outside critic re-verified 16 of 17 on 2026-10-16. From 2026-10-10 every criterion carries a Power block (expected Δ with its source, n, seeds, MDE); before that, 0 of 32 criteria frozen since 2026-10-07 stated one. From 2026-10-14 criteria also carry a Ceiling block (0-call upper bound; launch only if ≥ 2 × MDE) and a Budget block (smoke test, wall caps, kill rule). Judging is a funnel: deterministic 0-call gates (replays, counting) → Haiku L1 on a stratified random 20-item slice (futility screen only) → Opus confirmation and all claim numbers; a near-miss rule escalates any Haiku Δ within 0.01 of a threshold to Opus. Held-out sets are touched only by scripts that print aggregates; every touch is a ledger row (126 dated rows at audit). Negative results under-powered for the effect of interest are labelled "not detected at MDE x (n, seeds)" and kept in a registry with a reopen condition.

Second phase (2026-10-18 → 10-24). About 200 further arms ran, mostly 0-LLM replays and Haiku screens under a two-stage gate whose bar is set from measured null arms (GATE v2); seven open decisions were delegated by the author to the coordinator on 2026-10-18; one frozen FAIL was overridden with the spec reviewer's conditions (RULING R1); an overnight run generated answers and judged none, leaving judge runs to the author. Who decided is labelled on every decision row.

Limitation stated here

the planned 40-pair blind human label check was not done The planned 40-pair blind human label check (seed 7, stratified) has not been done.0/40Date2026-10-13Kinda record, not a measurement (set, status, judge, n, seeds: not recorded)Sourcebenchmark bench/v0.3/plan/analysis-plan.md:45-49Open the data file.

Badge rows the 7 public fact rows this depth cites; a number here without a chip comes from a row that is not public yet (see /data/)
Claim and valueSetStatusJudgenSeeds95 % CIDate
The planned 40-pair blind human label check (seed 7, stratified) has not been done.0/40not recordednot recordednot recordednot recordednot recorded2026-10-13
Rule changes logged29 numbered rules + 27a in 12 days (10-03 → 10-14)not recordednot recordednot recordednot recordednot recorded2026-10-14
How arms died101 closed arms: 54 at 0 judge calls, 10 at Haiku L1, 12 at L1→Opus, 25 Opus directlynot recordednot recordednot recordednot recordednot recorded2026-10-14
Criterion hashes verified: 65/65 on 2026-10-10; an outside critic found 16 of 17 matching on 2026-10-16.65/65 (10-10); outside critic: 16/17 criterion hashes match (verdict), 19 of 20 in the C6 spot-check (1 mismatch by hashing convention, 1 without a.sha256 sidecar)not recordednot recordednot recordednot recordednot recorded2026-10-10, 2026-10-16
65/65 criterion sha256 files verified, and the freeze commit precedes the result commit.65/65not recordednot recordednot recordednot recordednot recorded2026-10-10
0 of 32 criteria frozen between 10-07 and 10-10 stated an expected Δ or MDE.0/32not recordednot recordednot recordednot recordednot recorded2026-10-10
The exposure ledger holds 126 dated rows (55 v0.2-only, 33 V-para-only, 19 both).126 dated rows (55 v0.2-only, 33 V-para-only, 19 both)not recordednot recordednot recordednot recordednot recorded2026-10-17

In one sentence

An AI-agent lab can run real research, if every rule is written down and every mistake becomes a new rule.

Turning point

2026-10-10 13:30, the R&D process review (10 agents, 76 findings): "measurement machine better than typical, pointed at the wrong target"; deliverables unbuilt The 10-10 process review10 Opus agents, 76 findings; double-check: 32 claims → 10 verified, 20 partly, 2 falseDate2026-10-10Kinda record, not a measurement (set, status, judge, n, seeds: not recorded)Sourcenot in the public repositories yetOpen the data file.

On the calendar

45 dated events · 1 Oct → 16 Oct

  1. 1 Oct · 21:06Research lanes start: the models/latency research agent cycle 1 (21:06), the evaluation/safety research agent cycle 1 (21:10), the retrieval research agent cycle 1 (~21:45) under one written mandate
  2. 2 OctBench v0.2 written by two blind writer agents (55 items each); plan frozen (combined sha fe100e91)
  3. 3 OctA research agent's trace scan prints one v0.2 item; plaintext moved to blind/, copies quarantined
  4. 3 Oct"No human review": an independent Opus reviewer agent decides content; caveat "AI-reviewed, not clinically reviewed"
  5. 10 Oct · 21:16Judge pipeline made fail-closed (errored traces no longer silently dropped)
All 45 events
  1. 2026-10-01Research mandate: three researcher lanes (retrieval, models, evaluation/safety) write "technique cards"; a runner executes arms
  2. 2026-10-01 21:06Research lanes start: the models/latency research agent cycle 1 (21:06), the evaluation/safety research agent cycle 1 (21:10), the retrieval research agent cycle 1 (~21:45) under one written mandate
  3. 2026-10-01 22:45"Off-the-shelf scan" added to every lane (look for ready-made GitHub/HF artifacts first)
  4. 2026-10-02Bench v0.2 written by two blind writer agents (55 items each); plan frozen (combined sha fe100e91)
  5. 2026-10-02 04:45Blind-set rule widened after a filename-only grep leaked set composition
  6. 2026-10-02 06:10Strategy agent (the strategy agent) memo S1: three fronts, ~1 in 4 chance the CI lower bound clears 0.50; stop list
  7. 2026-10-03A research agent's trace scan prints one v0.2 item; plaintext moved to blind/, copies quarantined
  8. 2026-10-03"No human review": an independent Opus reviewer agent decides content; caveat "AI-reviewed, not clinically reviewed"
  9. 2026-10-03One v0.2 item printed into a research agent's trace scan; plaintext moved to an isolated folder
  10. 2026-10-07 08:02Overnight loop closed; 158 local commits, nothing pushed
  11. 2026-10-10 20:50Double-check of that review (5 Opus agents): of 32 claims, 10 verified, 20 partly, 2 false; one task withdrawn
  12. 2026-10-10 21:16Judge pipeline made fail-closed (missing ids/shards abort; tests 33/33)
  13. 2026-10-10 21:16Judge pipeline made fail-closed (errored traces no longer silently dropped)
  14. 2026-10-10 21:50STOP by the author ("para tudo"): all app/claim-package work halted
  15. 2026-10-10 22:09Criterion template with mandatory Power block; random-20 slices; bar table frozen
  16. 2026-10-11 03:4xLOOP-PROGRAM written: unattended self-paced loop, one experiment per tick
  17. 2026-10-12 11:27BAR-TABLE addendum: 0 floor-adjusted points adopted per 1,804 judge calls this cycle
  18. 2026-10-12 14:17RATIONALEPASS, NUMCHECK, EVALFIX, K2HALF, SEEDSPEC (0 Claude calls)
  19. 2026-10-12 20:41Loop stopped by the author, restarted 12 min later on a 30-min cron
  20. 2026-10-12 23:22Autonomy rule from the author: never ask for approval; paid API only when no free equivalent
  21. 2026-10-13 08:45Author challenges 9+ idle ticks; loop un-idled; rules 24 (never idle while 0-call work exists) and 25 (no Haiku cap)
  22. 2026-10-13 10:20Rule 25: no Haiku cap
  23. 2026-10-13 10:58Two session restarts in one day; nohup jobs survive, in-process agents die
  24. 2026-10-13 21:43Ledger audit workflow (54-58 Opus agents): 159 verified arm records; 48 verdicts with no closed-registry row, 10 exposures with no row
  25. 2026-10-13 23:14Judge resume provenance fix: resuming over another trace's rows now exits 7
  26. 2026-10-14Rule 28 (verdict + registry rows in the same commit) and Rule 29 (experiment economics: Ceiling, Budget, kill points, judge escalation); Rule 29 reviewed ACCEPT WITH EDITS
  27. 2026-10-14RULE 29 experiment economics: Ceiling + Budget + Transfer blocks; no Opus on dev sets
  28. 2026-10-14EXPERIMENT-ECONOMICS: of 101 closed arms 54 closed at 0 judge calls
  29. 2026-10-14 01:33Ledger fixes applied: closed-registry rows R50-R70
  30. 2026-10-14 08:04MICROD2Q (doc2query) fails; doc2query closed
  31. 2026-10-14 10:4xCoordinator session dies on an expired login token; mini idle 2.2 h
  32. 2026-10-14 16:35Author: wind-down to a decision point; no new arms
  33. 2026-10-14 19:56LOOP v2 retrospective: 133 ticks, ≈ 12.0k Claude calls (≈ 10.5k judge calls), 1 lever, ≈31 avoidable machine-hours
  34. 2026-10-15Research page + post reviewed by an independent agent: PUBLISHABLE AFTER FIXES; every number matched the release
  35. 2026-10-15 00:45Closing-sprint loop on in the live session
  36. 2026-10-15 10:07Public repos live (first push by the coordinator, explicitly authorised for those two repos only)
  37. 2026-10-15 10:48Illustrated research page + short group post ready
  38. 2026-10-15 18:58Maintainer package sent (140-question eval request, health-path port spec)
  39. 2026-10-15 19:28Session split: PR-watch session vs research session
  40. 2026-10-15 20:11LOOP v2 tooling T1–T7 built (queue runner, STATE line, smoke gate, log writers, judge preflight, lane locks, watchdog)
  41. 2026-10-15 23:38Coordinator's own hashing error: a valid freeze wrongly "corrected", then the correction corrected
  42. 2026-10-16 04:00Lanes idle by coordinator choice, then judge login dead (06:50); logged as a coordinator fault; "idle lane is a failure" restated
  43. 2026-10-16 06:50Judge authentication expired; no judge passes for ≈ 2.5 h; lanes idle ≈ 5 h (coordinator fault acknowledged)
  44. 2026-10-16 18:04SMALLROUTE (small model as router) opened and closed at S0
  45. 2026-10-16 18:28Independent adversarial CRITIQUE: 7 re-wordings applied

Related nodes on the map

This chapter on the map →

Concepts

Decisions

.md