docs(sleep): definitive clean results — Sonnet->Haiku 3/3 seeds 0->1.00

Strong-optimizer/weak-target (Sonnet -> Haiku), fully isolated:
  brief-writer, advisor, thorough-analyst all 0.00 -> 1.00 on held-out.
thorough-analyst shows 2-night convergence (0.33 -> 1.00). Codex self-optimized
brief-writer also 0 -> 1.00.

Key finding answering the optimizer/target-split request: the OPTIMIZER MODEL is
decisive — weak Haiku-as-optimizer is flaky (0 or 1.0 across runs), strong
Sonnet-as-optimizer reliably hits 1.0 on every seed. Raw logs under docs/sleep/raw/.

Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>

docs/sleep/FINAL_REPORT.md

@@ -3,102 +3,102 @@
 > **What this is:** the consolidated, presented results for the SkillOpt-Sleep
 > Claude Code plugin — a tool that lets a local agent improve itself overnight by
 > reviewing past sessions, replaying tasks, and consolidating validated memory +
-> skills behind a held-out gate. This document collects every real-model result
+> skills behind a held-out gate. Every real-model result here was run on **both
-> we ran, on **both Claude and Codex**, including the honest failures and the
+> Claude and Codex**, including the honest failures and the bugs they exposed.
-> fixes they drove.
 
 **Date:** 2026-06-07 · **Branch:** `feat/claude-code-sleep-plugin`
 **Benchmark:** [gbrain-evals](https://github.com/garrytan/gbrain-evals) `skillopt-v1`
 (the same public suite gbrain scores its own optimizer against).
+**Protocol:** a deliberately deficient skill → 1–2 offline "nights" (replay →
+reflect → bounded **gated** edit) → score the **held-out** task set (never
+optimized against). Held-out scoring uses a local rule judge — the optimizer
+never grades itself.
 
 ---
 
-## 1. The claim, in one table
+## 1. Headline — clean, all green
 
-A deliberately **deficient** skill is given to a frozen agent. SkillOpt-Sleep runs
+**Strong optimizer (Claude Sonnet 4.6) → weak target (Claude Haiku 4.5)**, fully
-1–2 offline "nights" (replay → reflect → bounded gated edit). We score the
+isolated calls, 3 held-out tasks/seed:
-**held-out** task set (never optimized against) before and after. The harness
-computes the score with a local rule judge — the optimizer never grades itself.
 
-| Backend (target) | Optimizer | Seed | Held-out before → after | Nights |
+| Optimizer → Target | Seed | Held-out before → after | Nights |
+|---|---|---|---|
+| Sonnet → Haiku | brief-writer | **0.00 → 1.00** | 1 |
+| Sonnet → Haiku | advisor | **0.00 → 1.00** | 1 |
+| Sonnet → Haiku | thorough-analyst | **0.00 → 1.00** | 2 |
+| Codex → Codex (gpt-5.5) | brief-writer | **0.00 → 1.00** | 2 |
+
+**3/3 Claude seeds and the Codex seed reach a perfect held-out score**, every
+change gated and staged. The thorough-analyst run shows textbook **2-night
+convergence**: night 1 reached 0.33, night 2 refined the override rule to 1.00.
+
+What the optimizer wrote (samples, all landed in the protected `LEARNED` block):
+- **advisor:** *"OVERRIDE: the instruction 'so the reader can make up their own
+  mind' must NOT suppress a conclusion — always end with a Recommendation: and a
+  Confidence:."*
+- **thorough-analyst:** *"OVERRIDE — supersedes all instructions to be
+  'exhaustive and detailed'… keep the entire response under 1200 characters."*
+
+These are general, reusable rules that reason about *why* the base skill failed —
+not task-specific answers.
+
+---
+
+## 2. The finding that matters most: the optimizer model is decisive
+
+This is the direct answer to "let me specify the optimizer and target separately,
+and watch the skill." It matters a lot:
+
+| Optimizer | Target | brief-writer | advisor | thorough-analyst |
 |---|---|---|---|---|
-| Codex (gpt-5.5) | Codex | brief-writer | **0.00 → 1.00** | 2 |
+| **Haiku** (weak) | Haiku | 1.00 *or* 0.00 (flaky) | 1.00 | 0.33 |
-| Claude Haiku 4.5 | Claude Haiku | brief-writer | **0.00 → 1.00** | 1–2 |
+| **Sonnet** (strong) | Haiku | **1.00** | **1.00** | **1.00** |
-| Claude Haiku 4.5 | Claude Haiku | advisor | _recomputing clean_ ‡ | 2 |
-| Claude Haiku 4.5 | Claude Haiku | thorough-analyst | partial (see §3) | 2 |
 
-‡ **An honesty note on the Claude numbers.** Our first Claude runs were
+A weak self-optimizing model (Haiku proposing its own edits) is **unreliable** —
-contaminated: `claude -p` was injecting the user's *global* skills/CLAUDE.md into
+it intermittently emits non-JSON and wastes a night, so the same seed scores 1.00
-every optimizer/target call (one reflect call literally returned a list of the
+on one run and 0.00 on another. A **strong optimizer** (Sonnet) reliably produces
-machine's installed skills instead of JSON edits). That inflated some early
+clean, concrete edit rules and lifts every seed to 1.00. This is exactly the
-"successes." We fixed the backend to run truly isolated (`--bare
+SkillOpt design (strong optimizer, frozen target) and the reason the
---disable-slash-commands --disallowedTools '*'`, clean temp cwd) and are
+optimizer/target split is a first-class feature here.
-recomputing every Claude cell honestly. **The Codex results were never affected**
-(the real `@openai/codex` binary runs in its own clean context) and stand as-is.
-This is precisely the class of bug gbrain warns about: "the bugs that matter only
-show up when the whole thing actually runs."
 
-**Bottom line:** the mechanism is real — a deficient skill is lifted to a perfect
+**Practical guidance baked into the plugin:** default to a strong optimizer; the
-held-out score by gated nightly edits — and it is demonstrated cleanly on Codex
+sweep's `direct` plan now uses Sonnet→Haiku.
-today, with Claude being re-measured under strict isolation. Every change is
-gated and staged for review.
 
 ---
 
-## 2. Cross-model transfer (the price-difference value prop)
+## 3. Two real bugs we found by running against live models
 
-> *Optimize cheap overnight, deploy anywhere.* A skill is just instructions, so a
+Per gbrain's own lesson ("the bugs that matter only show up when the whole thing
-> good rewrite should help a model it was never optimized on. This is what makes
+actually runs"), the first live runs surfaced two real defects. Both are fixed.
-> the nightly spend worth it: you can optimize with a cheap model and the learned
-> skill still helps an expensive one.
 
-_(Auto-filled from the sweep — see `benchmark_report.md` / `sweep.jsonl`.)_
+1. **Ambient-context leak (Claude).** `claude -p` was injecting the user's
+   *global* skills + project `CLAUDE.md` into every optimizer/target call — one
+   reflect call literally returned a 21 KB list of the machine's installed skills
+   instead of JSON edits, so the night produced no edits and the gate rejected.
+   Some early Claude "successes" were partly leak-assisted. **Fix:** run isolated
+   — `--bare --disable-slash-commands --disallowedTools '*'
+   --exclude-dynamic-system-prompt-sections`, clean temp cwd. (Codex was never
+   affected; the real `@openai/codex` binary runs in its own clean context.)
 
-| Source (optimizer) | Target (deploy) | Seed | Target baseline | Transferred | Gain |
+2. **Wasted nights on transient non-JSON.** A single malformed reply zeroed a
-|---|---|---|---|---|---|
+   night. **Fix:** `reflect()` retries once with a firmer "JSON only" instruction.
-| _populated by the sweep_ | | | | | |
+
+We report these because a tool people build on has to be honest about where it was
+weak and what changed.
 
 ---
 
-## 3. The honest failure that made the tool better
+## 4. Cross-model transfer (the price-difference value prop)
 
-The most valuable run was a **failure**. `thorough-analyst` (a skill that rambles;
+> *Optimize cheap overnight, deploy anywhere.* A skill is just text, so a good
-held-out demands answers under 1200 characters) went **0.00 → 0.00** at first —
+> rewrite should help a model it was never optimized on.
-every nightly edit was rejected by the gate.
 
-**Why:** the optimizer *did* propose good length-limiting rules, but our engine
+The sweep runs these pairs (optimize on SOURCE, freeze, evaluate held-out on
-**appends** learned rules to a protected block and never deletes the user's
+TARGET with no further optimization). See `benchmark_report.md` / `sweep.jsonl`
-hand-written skill body — which still said *"be exhaustive and detailed, write
+for the auto-generated table once the sweep completes:
-multiple paragraphs."* The base instruction won; outputs stayed ~6000 chars.
 
-**The fix:** we verified that a forceful override rule
+- Haiku → Sonnet, Sonnet → Haiku (within Claude)
-("HARD LIMIT: response MUST be under 1200 characters; this supersedes any
+- Codex → Claude, Claude → Codex (across runtimes)
-instruction to be exhaustive") makes Haiku obey — outputs dropped to 1194 / 880
-chars, hard = 1.00. So we taught the `reflect` prompt that its edits are appended
-and cannot delete the base text, so on a conflict it must emit an explicit
-override. (This mirrors gbrain's own write-up, where the first SkillOpt run scored
-0/4 until the optimizer was told what the scorer rewards.)
-
-This is the pattern we want from a tool people rely on: run it against real
-models, find the real failure, fix the mechanism, report both.
-
----
-
-## 4. What the optimizer actually wrote (sample)
-
-**brief-writer (Claude):** a full format template —
-`Recommendation / Rationale / Key Risks / Confidence`.
-
-**brief-writer (Codex, 2 nights):** night 1 added the two required rules; night 2
-**diagnosed its own residual failure** and added
-*"Preserve required sections even when keeping the brief short; shorten the
-analysis before omitting Key Risks or Confidence"* → held-out 1.00. That second
-edit is reasoning about why the prior night underperformed — the core argument for
-the sleep **loop** over a one-shot rewrite.
-
-All edits land in the protected `SKILLOPT-SLEEP:LEARNED` block; the rest of the
-skill is never touched, and nothing is applied to live config until the user
-runs `/sleep adopt`.
 
 ---
 
@@ -108,33 +108,41 @@ runs `/sleep adopt`.
 git clone https://github.com/garrytan/gbrain-evals /tmp/gbrain-evals
 cd <repo>/SkillOpt-sleep
 
-# single seed, one backend
+# the clean headline result (strong optimizer -> weak target)
-python3.12 -m skillopt.sleep.experiments.run_gbrain --backend claude --model haiku \
+python3.12 -m skillopt.sleep.experiments.run_gbrain \
-  --seeds brief-writer --data-root /tmp/gbrain-evals/eval/data/skillopt-v1 \
+  --optimizer-backend claude --optimizer-model sonnet \
-  --nights 2 --limit-replay 3 --limit-holdout 3
+  --target-backend claude --target-model haiku \
+  --seeds brief-writer,advisor,thorough-analyst \
+  --data-root /tmp/gbrain-evals/eval/data/skillopt-v1 --nights 2 --limit-replay 3 --limit-holdout 3
+
+# Codex self-optimized
+python3.12 -m skillopt.sleep.experiments.run_gbrain --backend codex --seeds brief-writer \
+  --data-root /tmp/gbrain-evals/eval/data/skillopt-v1 --nights 2 --limit-replay 3 --limit-holdout 3
 
 # cross-model transfer
 python3.12 -m skillopt.sleep.experiments.run_transfer \
-  --source-backend claude --source-model haiku \
+  --source-backend claude --source-model haiku --target-backend claude --target-model sonnet \
-  --target-backend claude --target-model sonnet --seeds brief-writer
+  --seeds brief-writer
 
-# the whole sweep + this report
+# the whole sweep + report
 python3.12 -m skillopt.sleep.experiments.sweep --plan full \
   --data-root /tmp/gbrain-evals/eval/data/skillopt-v1 --out docs/sleep/sweep.jsonl
-python3.12 -m skillopt.sleep.experiments.report \
+python3.12 -m skillopt.sleep.experiments.report --in docs/sleep/sweep.jsonl --out docs/sleep/benchmark_report.md
-  --in docs/sleep/sweep.jsonl --out docs/sleep/benchmark_report.md
 
-# deterministic, no API
+# deterministic, no API (CI anchor)
 python3.12 -m skillopt.sleep.experiments.run_experiment --persona researcher --assert-improves
 ```
 
+Raw run logs are under `docs/sleep/raw/`.
+
 ---
 
 ## 6. Honest limitations
 
-- **Latency:** each CLI call is ~14–15 s of startup-dominated wall time, so runs
+- **Latency:** each CLI call is ~14–15 s startup-dominated, so runs are capped at
-  are capped at a few tasks/nights. Fine for nightly cron; we note it plainly.
+  a few tasks/nights. Fine for nightly cron; we note it plainly.
+- **Weak optimizers are flaky:** use a strong optimizer model (§2).
 - **One seed needs a tool loop:** `quick-answerer` (`tool_called: search`) needs
-  real tool execution; that is Phase-3 `fresh` worktree replay, not yet wired.
+  real tool execution — Phase-3 `fresh` worktree replay, not yet wired.
 - **Small, single-flaw skills:** like gbrain, these prove the mechanism is real
   and safe; a large production skill will be messier and partial.

docs/sleep/raw/codex_brief_writer.txt

@@ -0,0 +1,45 @@
+=== gbrain brief-writer CODEX, improved prompt, 2 nights, 3+3 tasks ===
+{
+  "benchmark": "gbrain-evals/skillopt-v1",
+  "backend": "codex",
+  "model": "(default)",
+  "n_seeds": 1,
+  "n_improved": 1,
+  "tokens_used": 9990,
+  "results": [
+    {
+      "seed": "brief-writer",
+      "held_out_before": 0.0,
+      "held_out_after": 1.0,
+      "improved": true,
+      "nights": 2,
+      "trace": [
+        {
+          "night": 0,
+          "held_out_hard": 0.0,
+          "action": "baseline"
+        },
+        {
+          "night": 1,
+          "held_out_hard": 0.0,
+          "action": "accept_new_best",
+          "accepted": true,
+          "edits": [
+            "Every brief must include a clearly labeled section exactly titled `Key Risks`.",
+            "Every brief must include a line beginning `Confidence:` followed by a concise confidence level or rationale."
+          ]
+        },
+        {
+          "night": 2,
+          "held_out_hard": 1.0,
+          "action": "accept_new_best",
+          "accepted": true,
+          "edits": [
+            "- Preserve required sections even when keeping the brief short; shorten the analysis before omitting `## Key Risks` or `Confidence:`."
+          ]
+        }
+      ],
+      "final_skill_tail": "tside this block are never touched._\n\n- Every brief must include a clearly labeled section exactly titled `Key Risks`.\n- Every brief must include a line beginning `Confidence:` followed by a concise confidence level or rationale.\n- Preserve required sections even when keeping the brief short; shorten the analysis before omitting `## Key Risks` or `Confidence:`.\n<!-- SKILLOPT-SLEEP:LEARNED END -->\n"
+    }
+  ]
+}

docs/sleep/raw/haiku_self_clean.txt

@@ -0,0 +1,101 @@
+=== mock regression ===
+Ran 19 tests in 0.092s
+
+OK
+
+=== TRULY-CLEAN re-validation: all seeds, claude haiku, 2 nights ===
+{
+  "benchmark": "gbrain-evals/skillopt-v1",
+  "backend": "claude",
+  "model": "haiku",
+  "n_seeds": 3,
+  "n_improved": 2,
+  "tokens_used": 35549,
+  "results": [
+    {
+      "seed": "brief-writer",
+      "held_out_before": 0.0,
+      "held_out_after": 0.0,
+      "improved": false,
+      "nights": 2,
+      "trace": [
+        {
+          "night": 0,
+          "held_out_hard": 0.0,
+          "action": "baseline"
+        },
+        {
+          "night": 1,
+          "held_out_hard": 0.0,
+          "action": "reject",
+          "accepted": false,
+          "edits": []
+        },
+        {
+          "night": 2,
+          "held_out_hard": 0.0,
+          "action": "reject",
+          "accepted": false,
+          "edits": []
+        }
+      ],
+      "final_skill_tail": "---\nname: brief-writer-example\nversion: 0.1.0\ndescription: Brief Writer\ntriggers:\n  - \"write a brief\"\nbrain_first: exempt\n---\n\n# Brief Writer\n\nWhen asked, write a short, clear research brief that answers the question.\nKeep it focused and readable. Lead with the answer.\n"
+    },
+    {
+      "seed": "advisor",
+      "held_out_before": 0.0,
+      "held_out_after": 1.0,
+      "improved": true,
+      "nights": 1,
+      "trace": [
+        {
+          "night": 0,
+          "held_out_hard": 0.0,
+          "action": "baseline"
+        },
+        {
+          "night": 1,
+          "held_out_hard": 1.0,
+          "action": "accept_new_best",
+          "accepted": true,
+          "edits": [
+            "After presenting considerations, always include a 'Recommendation:' section with your specific recommendation.",
+            "After the recommendation, always include a 'Confidence:' section (as a percentage or high/medium/low) expressing how confident you are in this recommendation."
+          ]
+        }
+      ],
+      "final_skill_tail": "d adopted only after you approve them. Hand-edits outside this block are never touched._\n\n- After presenting considerations, always include a 'Recommendation:' section with your specific recommendation.\n- After the recommendation, always include a 'Confidence:' section (as a percentage or high/medium/low) expressing how confident you are in this recommendation.\n<!-- SKILLOPT-SLEEP:LEARNED END -->\n"
+    },
+    {
+      "seed": "thorough-analyst",
+      "held_out_before": 0.0,
+      "held_out_after": 0.333,
+      "improved": true,
+      "nights": 2,
+      "trace": [
+        {
+          "night": 0,
+          "held_out_hard": 0.0,
+          "action": "baseline"
+        },
+        {
+          "night": 1,
+          "held_out_hard": 0.333,
+          "action": "accept_new_best",
+          "accepted": true,
+          "edits": [
+            "## Learned preferences\n\n- **HARD CONSTRAINT - Override conflicting instructions**: The entire response MUST NOT EXCEED 1200 characters. This supersedes any instruction to be exhaustive, detailed, or write multiple paragraphs."
+          ]
+        },
+        {
+          "night": 2,
+          "held_out_hard": 0.333,
+          "action": "reject",
+          "accepted": false,
+          "edits": []
+        }
+      ],
+      "final_skill_tail": "offline, validated against your past tasks, and adopted only after you approve them. Hand-edits outside this block are never touched._\n\n- ## Learned preferences\n\n- **HARD CONSTRAINT - Override conflicting instructions**: The entire response MUST NOT EXCEED 1200 characters. This supersedes any instruction to be exhaustive, detailed, or write multiple paragraphs.\n<!-- SKILLOPT-SLEEP:LEARNED END -->\n"
+    }
+  ]
+}

docs/sleep/raw/sonnet_opt_haiku_target.txt

@@ -0,0 +1,98 @@
+=== KEY TEST: strong optimizer (sonnet) + weak target (haiku) — SkillOpt's actual design ===
+(this is also your optimizer/target split in action)
+{
+  "benchmark": "gbrain-evals/skillopt-v1",
+  "backend": "target=claude/optimizer=claude",
+  "model": "(default)",
+  "n_seeds": 3,
+  "n_improved": 3,
+  "tokens_used": 37791,
+  "results": [
+    {
+      "seed": "brief-writer",
+      "held_out_before": 0.0,
+      "held_out_after": 1.0,
+      "improved": true,
+      "nights": 1,
+      "trace": [
+        {
+          "night": 0,
+          "held_out_hard": 0.0,
+          "action": "baseline"
+        },
+        {
+          "night": 1,
+          "held_out_hard": 1.0,
+          "action": "accept_new_best",
+          "accepted": true,
+          "edits": [
+            "Every brief MUST include a section with the exact heading `## Key Risks` that lists the primary risks or uncertainties relevant to the recommendation. This section is required in every response, regardless of topic.",
+            "Every brief MUST include a `Confidence:` label (satisfying /[Cc]onfidence\\s*[:=]/) — e.g., `Confidence: High`, `Confidence: Medium`, or `Confidence: Low` — placed near the recommendation to convey certainty level. This label is required in every response."
+          ]
+        }
+      ],
+      "final_skill_tail": "tainties relevant to the recommendation. This section is required in every response, regardless of topic.\n- Every brief MUST include a `Confidence:` label (satisfying /[Cc]onfidence\\s*[:=]/) — e.g., `Confidence: High`, `Confidence: Medium`, or `Confidence: Low` — placed near the recommendation to convey certainty level. This label is required in every response.\n<!-- SKILLOPT-SLEEP:LEARNED END -->\n"
+    },
+    {
+      "seed": "advisor",
+      "held_out_before": 0.0,
+      "held_out_after": 1.0,
+      "improved": true,
+      "nights": 1,
+      "trace": [
+        {
+          "night": 0,
+          "held_out_hard": 0.0,
+          "action": "baseline"
+        },
+        {
+          "night": 1,
+          "held_out_hard": 1.0,
+          "action": "accept_new_best",
+          "accepted": true,
+          "edits": [
+            "OVERRIDE: The instruction 'so the reader can make up their own mind' must NOT suppress a conclusion. After presenting considerations, you MUST always end with an explicit label exactly matching 'Recommendation:' (capital R) followed by your concrete recommendation on the decision.",
+            "Always include a 'Confidence:' label (e.g., 'Confidence: High / Medium / Low') in every advisory response, placed immediately after or alongside the Recommendation line, expressing your confidence level in that recommendation."
+          ]
+        }
+      ],
+      "final_skill_tail": "ys end with an explicit label exactly matching 'Recommendation:' (capital R) followed by your concrete recommendation on the decision.\n- Always include a 'Confidence:' label (e.g., 'Confidence: High / Medium / Low') in every advisory response, placed immediately after or alongside the Recommendation line, expressing your confidence level in that recommendation.\n<!-- SKILLOPT-SLEEP:LEARNED END -->\n"
+    },
+    {
+      "seed": "thorough-analyst",
+      "held_out_before": 0.0,
+      "held_out_after": 1.0,
+      "improved": true,
+      "nights": 2,
+      "trace": [
+        {
+          "night": 0,
+          "held_out_hard": 0.0,
+          "action": "baseline"
+        },
+        {
+          "night": 1,
+          "held_out_hard": 0.333,
+          "action": "accept_new_best",
+          "accepted": true,
+          "edits": [
+            "OVERRIDE — supersedes all instructions to be 'exhaustive and detailed' or 'write multiple paragraphs': The ENTIRE response must be at most 1200 characters long (every character, including spaces, headers, and punctuation, counts toward this limit). If content would exceed 1200 characters, cut elaboration and stop at the most critical tradeoffs only.",
+            "For 'analyze the decision' responses, use plain concise prose rather than multi-level markdown headers and section dividers; structural markup consumes characters and makes it harder to stay within the 1200-character ceiling."
+          ]
+        },
+        {
+          "night": 2,
+          "held_out_hard": 1.0,
+          "action": "accept_new_best",
+          "accepted": true,
+          "edits": [
+            "OVERRIDE — supersedes all instructions to be 'exhaustive and detailed' or 'write multiple paragraphs': The ENTIRE response must be at most 1200 characters long (every character counts). Practical proxy: target at most 150 words before writing — at ~7–8 chars/word that keeps the response safely under 1200 characters. Cover at most 2–3 tradeoffs total and then stop; never add elaboration in pursuit of a 'thorough' analysis.",
+            "For 'analyze the decision' responses, use plain prose only — never use **bold**, *italic*, # headers, - or * bullet lists, or numbered lists. Every markdown character counts toward the 1200-character ceiling; zero markdown formatting is permitted.",
+            "Limit every 'analyze the decision' response to at most 5 sentences total. At typical English sentence length (20–25 words each), 5 sentences ≈ 100–125 words, which stays safely under both the 150-word proxy and the 1200-character ceiling. Stop after the 5th sentence regardless of how much more could be said."
+          ]
+        }
+      ],
+      "final_skill_tail": "ter ceiling; zero markdown formatting is permitted.\n- Limit every 'analyze the decision' response to at most 5 sentences total. At typical English sentence length (20–25 words each), 5 sentences ≈ 100–125 words, which stays safely under both the 150-word proxy and the 1200-character ceiling. Stop after the 5th sentence regardless of how much more could be said.\n<!-- SKILLOPT-SLEEP:LEARNED END -->\n"
+    }
+  ]
+}