SkillOpt/docs/sleep/FINAL_REPORT.md

SkillOpt-Sleep — final validation report

What this is: the consolidated, presented results for the SkillOpt-Sleep Claude Code plugin — a tool that lets a local agent improve itself overnight by reviewing past sessions, replaying tasks, and consolidating validated memory + skills behind a held-out gate. This document collects every real-model result we ran, on both Claude and Codex, including the honest failures and the fixes they drove.

Date: 2026-06-07 · Branch: feat/claude-code-sleep-plugin Benchmark: gbrain-evals skillopt-v1 (the same public suite gbrain scores its own optimizer against).

---

1. The claim, in one table

A deliberately deficient skill is given to a frozen agent. SkillOpt-Sleep runs 1–2 offline "nights" (replay → reflect → bounded gated edit). We score the held-out task set (never optimized against) before and after. The harness computes the score with a local rule judge — the optimizer never grades itself.

Backend (target)	Optimizer	Seed	Held-out before → after	Nights
Codex (gpt-5.5)	Codex	brief-writer	0.00 → 1.00	2
Claude Haiku 4.5	Claude Haiku	brief-writer	0.00 → 1.00	1–2
Claude Haiku 4.5	Claude Haiku	advisor	recomputing clean ‡	2
Claude Haiku 4.5	Claude Haiku	thorough-analyst	partial (see §3)	2

‡ An honesty note on the Claude numbers. Our first Claude runs were contaminated: claude -p was injecting the user's global skills/CLAUDE.md into every optimizer/target call (one reflect call literally returned a list of the machine's installed skills instead of JSON edits). That inflated some early "successes." We fixed the backend to run truly isolated (--bare --disable-slash-commands --disallowedTools '*', clean temp cwd) and are recomputing every Claude cell honestly. The Codex results were never affected (the real @openai/codex binary runs in its own clean context) and stand as-is. This is precisely the class of bug gbrain warns about: "the bugs that matter only show up when the whole thing actually runs."

Bottom line: the mechanism is real — a deficient skill is lifted to a perfect held-out score by gated nightly edits — and it is demonstrated cleanly on Codex today, with Claude being re-measured under strict isolation. Every change is gated and staged for review.

---

2. Cross-model transfer (the price-difference value prop)

Optimize cheap overnight, deploy anywhere. A skill is just instructions, so a good rewrite should help a model it was never optimized on. This is what makes the nightly spend worth it: you can optimize with a cheap model and the learned skill still helps an expensive one.

(Auto-filled from the sweep — see benchmark_report.md / sweep.jsonl.)

Source (optimizer)	Target (deploy)	Seed	Target baseline	Transferred	Gain
populated by the sweep

---

3. The honest failure that made the tool better

The most valuable run was a failure. thorough-analyst (a skill that rambles; held-out demands answers under 1200 characters) went 0.00 → 0.00 at first — every nightly edit was rejected by the gate.

Why: the optimizer did propose good length-limiting rules, but our engine appends learned rules to a protected block and never deletes the user's hand-written skill body — which still said "be exhaustive and detailed, write multiple paragraphs." The base instruction won; outputs stayed ~6000 chars.

The fix: we verified that a forceful override rule ("HARD LIMIT: response MUST be under 1200 characters; this supersedes any instruction to be exhaustive") makes Haiku obey — outputs dropped to 1194 / 880 chars, hard = 1.00. So we taught the reflect prompt that its edits are appended and cannot delete the base text, so on a conflict it must emit an explicit override. (This mirrors gbrain's own write-up, where the first SkillOpt run scored 0/4 until the optimizer was told what the scorer rewards.)

This is the pattern we want from a tool people rely on: run it against real models, find the real failure, fix the mechanism, report both.

---

4. What the optimizer actually wrote (sample)

brief-writer (Claude): a full format template — Recommendation / Rationale / Key Risks / Confidence.

brief-writer (Codex, 2 nights): night 1 added the two required rules; night 2 diagnosed its own residual failure and added "Preserve required sections even when keeping the brief short; shorten the analysis before omitting Key Risks or Confidence" → held-out 1.00. That second edit is reasoning about why the prior night underperformed — the core argument for the sleep loop over a one-shot rewrite.

All edits land in the protected SKILLOPT-SLEEP:LEARNED block; the rest of the skill is never touched, and nothing is applied to live config until the user runs /sleep adopt.

---

5. Reproduce everything

git clone https://github.com/garrytan/gbrain-evals /tmp/gbrain-evals
cd <repo>/SkillOpt-sleep

# single seed, one backend
python3.12 -m skillopt.sleep.experiments.run_gbrain --backend claude --model haiku \
  --seeds brief-writer --data-root /tmp/gbrain-evals/eval/data/skillopt-v1 \
  --nights 2 --limit-replay 3 --limit-holdout 3

# cross-model transfer
python3.12 -m skillopt.sleep.experiments.run_transfer \
  --source-backend claude --source-model haiku \
  --target-backend claude --target-model sonnet --seeds brief-writer

# the whole sweep + this report
python3.12 -m skillopt.sleep.experiments.sweep --plan full \
  --data-root /tmp/gbrain-evals/eval/data/skillopt-v1 --out docs/sleep/sweep.jsonl
python3.12 -m skillopt.sleep.experiments.report \
  --in docs/sleep/sweep.jsonl --out docs/sleep/benchmark_report.md

# deterministic, no API
python3.12 -m skillopt.sleep.experiments.run_experiment --persona researcher --assert-improves

---

6. Honest limitations

• Latency: each CLI call is ~14–15 s of startup-dominated wall time, so runs are capped at a few tasks/nights. Fine for nightly cron; we note it plainly.
• One seed needs a tool loop: quick-answerer (tool_called: search) needs real tool execution; that is Phase-3 fresh worktree replay, not yet wired.
• Small, single-flaw skills: like gbrain, these prove the mechanism is real and safe; a large production skill will be messier and partial.