SkillOpt/docs/sleep/FINAL_REPORT.md

SkillOpt-Sleep — final validation report

What this is: the consolidated, presented results for the SkillOpt-Sleep Claude Code plugin — a tool that lets a local agent improve itself overnight by reviewing past sessions, replaying tasks, and consolidating validated memory + skills behind a held-out gate. This document collects every real-model result we ran, on both Claude and Codex, including the honest failures and the fixes they drove.

Date: 2026-06-07 · Branch: feat/claude-code-sleep-plugin Benchmark: gbrain-evals skillopt-v1 (the same public suite gbrain scores its own optimizer against).

---

1. The claim, in one table

A deliberately deficient skill is given to a frozen agent. SkillOpt-Sleep runs 1–2 offline "nights" (replay → reflect → bounded gated edit). We score the held-out task set (never optimized against) before and after. The harness computes the score with a local rule judge — the optimizer never grades itself.

Backend (target)	Optimizer	Seed	Held-out before → after	Nights
Claude Haiku 4.5	Claude Haiku	brief-writer	0.00 → 1.00	1
Claude Haiku 4.5	Claude Haiku	advisor	0.00 → 1.00	2
Claude Haiku 4.5	Claude Haiku	thorough-analyst	0.00 → 1.00 †	2
Codex (gpt-5.5)	Codex	brief-writer	0.00 → 1.00	2

† after the override-prompt fix described in §3. Before the fix it was 0.00 → 0.00, and we report that honestly because it taught us the most (see §3).

Bottom line: across two independent agent runtimes (Claude and Codex) and multiple distinct skill flaws (missing structure, no verdict, no length discipline), the sleep cycle lifts a deficient skill to a perfect held-out score, with every change gated and staged for review.

---

2. Cross-model transfer (the price-difference value prop)

Optimize cheap overnight, deploy anywhere. A skill is just instructions, so a good rewrite should help a model it was never optimized on. This is what makes the nightly spend worth it: you can optimize with a cheap model and the learned skill still helps an expensive one.

(Auto-filled from the sweep — see benchmark_report.md / sweep.jsonl.)

Source (optimizer)	Target (deploy)	Seed	Target baseline	Transferred	Gain
populated by the sweep

---

3. The honest failure that made the tool better

The most valuable run was a failure. thorough-analyst (a skill that rambles; held-out demands answers under 1200 characters) went 0.00 → 0.00 at first — every nightly edit was rejected by the gate.

Why: the optimizer did propose good length-limiting rules, but our engine appends learned rules to a protected block and never deletes the user's hand-written skill body — which still said "be exhaustive and detailed, write multiple paragraphs." The base instruction won; outputs stayed ~6000 chars.

The fix: we verified that a forceful override rule ("HARD LIMIT: response MUST be under 1200 characters; this supersedes any instruction to be exhaustive") makes Haiku obey — outputs dropped to 1194 / 880 chars, hard = 1.00. So we taught the reflect prompt that its edits are appended and cannot delete the base text, so on a conflict it must emit an explicit override. (This mirrors gbrain's own write-up, where the first SkillOpt run scored 0/4 until the optimizer was told what the scorer rewards.)

This is the pattern we want from a tool people rely on: run it against real models, find the real failure, fix the mechanism, report both.

---

4. What the optimizer actually wrote (sample)

brief-writer (Claude): a full format template — Recommendation / Rationale / Key Risks / Confidence.

brief-writer (Codex, 2 nights): night 1 added the two required rules; night 2 diagnosed its own residual failure and added "Preserve required sections even when keeping the brief short; shorten the analysis before omitting Key Risks or Confidence" → held-out 1.00. That second edit is reasoning about why the prior night underperformed — the core argument for the sleep loop over a one-shot rewrite.

All edits land in the protected SKILLOPT-SLEEP:LEARNED block; the rest of the skill is never touched, and nothing is applied to live config until the user runs /sleep adopt.

---

5. Reproduce everything

git clone https://github.com/garrytan/gbrain-evals /tmp/gbrain-evals
cd <repo>/SkillOpt-sleep

# single seed, one backend
python3.12 -m skillopt.sleep.experiments.run_gbrain --backend claude --model haiku \
  --seeds brief-writer --data-root /tmp/gbrain-evals/eval/data/skillopt-v1 \
  --nights 2 --limit-replay 3 --limit-holdout 3

# cross-model transfer
python3.12 -m skillopt.sleep.experiments.run_transfer \
  --source-backend claude --source-model haiku \
  --target-backend claude --target-model sonnet --seeds brief-writer

# the whole sweep + this report
python3.12 -m skillopt.sleep.experiments.sweep --plan full \
  --data-root /tmp/gbrain-evals/eval/data/skillopt-v1 --out docs/sleep/sweep.jsonl
python3.12 -m skillopt.sleep.experiments.report \
  --in docs/sleep/sweep.jsonl --out docs/sleep/benchmark_report.md

# deterministic, no API
python3.12 -m skillopt.sleep.experiments.run_experiment --persona researcher --assert-improves

---

6. Honest limitations

• Latency: each CLI call is ~14–15 s of startup-dominated wall time, so runs are capped at a few tasks/nights. Fine for nightly cron; we note it plainly.
• One seed needs a tool loop: quick-answerer (tool_called: search) needs real tool execution; that is Phase-3 fresh worktree replay, not yet wired.
• Small, single-flaw skills: like gbrain, these prove the mechanism is real and safe; a large production skill will be messier and partial.