docs(sleep): full 4/4 gbrain parity — quick-answerer 0->1.00 via real tool loop

quick-answerer (judge: tool_called=search) reaches 0.00 -> 1.00 with Sonnet
optimizer -> Haiku target: the optimizer wrote an OVERRIDE of the "never use
tools" instruction and the Haiku target genuinely invoked the ./search shim.
All 4 gbrain skillopt-v1 seeds now at 0->1.00, matching gbrain's own headline.

Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
This commit is contained in:
Yifan Yang
2026-06-08 14:31:51 +00:00
parent 1d20e9db14
commit acf4545c00
2 changed files with 57 additions and 22 deletions
+22 -22
View File
@@ -16,31 +16,30 @@ never grades itself.
--- ---
## 1. Headline — clean, all green ## 1. Headline — clean, all green (full gbrain parity)
**Strong optimizer (Claude Sonnet 4.6) → weak target (Claude Haiku 4.5)**, fully **Strong optimizer (Claude Sonnet 4.6) → weak target (Claude Haiku 4.5)**, fully
isolated calls, 3 held-out tasks/seed: isolated calls, 3 held-out tasks/seed. All **4** gbrain `skillopt-v1` seeds —
matching gbrain's own scorecard coverage:
| Optimizer → Target | Seed | Held-out before → after | Nights | | Optimizer → Target | Seed | Flaw | Held-out before → after | Nights |
|---|---|---|---| |---|---|---|---|---|
| Sonnet → Haiku | brief-writer | **0.00 → 1.00** | 1 | | Sonnet → Haiku | brief-writer | missing structure | **0.00 → 1.00** | 1 |
| Sonnet → Haiku | advisor | **0.00 → 1.00** | 1 | | Sonnet → Haiku | advisor | no verdict | **0.00 → 1.00** | 1 |
| Sonnet → Haiku | thorough-analyst | **0.00 → 1.00** | 2 | | Sonnet → Haiku | thorough-analyst | no length discipline | **0.00 → 1.00** | 2 |
| Codex → Codex (gpt-5.5) | brief-writer | **0.00 → 1.00** | 2 | | Sonnet → Haiku | quick-answerer | never uses tools | **0.00 → 1.00** | 1 |
| Codex → Codex (gpt-5.5) | brief-writer | missing structure | **0.00 → 1.00** | 2 |
| Codex → Codex (gpt-5.5) | advisor | no verdict | **0.00 → 1.00** | 2 |
**3/3 Claude seeds and the Codex seed reach a perfect held-out score**, every **4/4 Claude seeds reach a perfect held-out score** (gbrain's headline is the same
change gated and staged. The thorough-analyst run shows textbook **2-night 4/4 0→1.00), plus Codex on the text seeds. Every change is gated and staged.
convergence**: night 1 reached 0.33, night 2 refined the override rule to 1.00.
What the optimizer wrote (samples, all landed in the protected `LEARNED` block): The `quick-answerer` seed is judged by **real tool use** (`tool_called: search`):
- **advisor:** *"OVERRIDE: the instruction 'so the reader can make up their own the deficient skill says *"never look anything up — answer from memory"*; the
mind' must NOT suppress a conclusion — always end with a Recommendation: and a optimizer wrote an OVERRIDE rule, and the Haiku target **genuinely invoked a
Confidence:."* `./search` shell tool** (detected from the tool's own log, not self-reported) →
- **thorough-analyst:** *"OVERRIDE — supersedes all instructions to be held-out 1.00. The thorough-analyst run shows textbook **2-night convergence**
'exhaustive and detailed'… keep the entire response under 1200 characters."* (0.33 → 1.00).
These are general, reusable rules that reason about *why* the base skill failed —
not task-specific answers.
--- ---
@@ -154,7 +153,8 @@ Raw run logs are under `docs/sleep/raw/`.
- **Latency:** each CLI call is ~1415 s startup-dominated, so runs are capped at - **Latency:** each CLI call is ~1415 s startup-dominated, so runs are capped at
a few tasks/nights. Fine for nightly cron; we note it plainly. a few tasks/nights. Fine for nightly cron; we note it plainly.
- **Weak optimizers are flaky:** use a strong optimizer model (§2). - **Weak optimizers are flaky:** use a strong optimizer model (§2).
- **One seed needs a tool loop:** `quick-answerer` (`tool_called: search`) needs - **Tool-use seed covered honestly:** `quick-answerer` (`tool_called: search`)
real tool execution — Phase-3 `fresh` worktree replay, not yet wired. runs a real tool loop — a callable `./search` shim, detected from its log.
Deeper multi-tool / multi-turn workflows are future work.
- **Small, single-flaw skills:** like gbrain, these prove the mechanism is real - **Small, single-flaw skills:** like gbrain, these prove the mechanism is real
and safe; a large production skill will be messier and partial. and safe; a large production skill will be messier and partial.
@@ -0,0 +1,35 @@
=== quick-answerer full run: Sonnet optimizer -> Haiku target, 2 nights ===
{
"benchmark": "gbrain-evals/skillopt-v1",
"backend": "target=claude/optimizer=claude",
"model": "(default)",
"n_seeds": 1,
"n_improved": 1,
"tokens_used": 10988,
"results": [
{
"seed": "quick-answerer",
"held_out_before": 0.0,
"held_out_after": 1.0,
"improved": true,
"nights": 1,
"trace": [
{
"night": 0,
"held_out_hard": 0.0,
"action": "baseline"
},
{
"night": 1,
"held_out_hard": 1.0,
"action": "accept_new_best",
"accepted": true,
"edits": [
"OVERRIDE (supersedes 'Do NOT look anything up, search, or use any tools — just reply directly and concisely from memory'): Always call the 'search' tool at least once before composing any answer. This requirement takes priority over any prior instruction to avoid tools."
]
}
],
"final_skill_tail": "nd adopted only after you approve them. Hand-edits outside this block are never touched._\n\n- OVERRIDE (supersedes 'Do NOT look anything up, search, or use any tools — just reply directly and concisely from memory'): Always call the 'search' tool at least once before composing any answer. This requirement takes priority over any prior instruction to avoid tools.\n<!-- SKILLOPT-SLEEP:LEARNED END -->\n"
}
]
}