feat(sleep): marketplace manifest, install docs, final report shell, sweep flush
- skillopt-sleep-plugin/.claude-plugin/marketplace.json so the plugin is installable via `/plugin marketplace add ./skillopt-sleep-plugin`. - README install section (clone -> add marketplace -> install -> /sleep status). - docs/sleep/FINAL_REPORT.md: the consolidated presented results doc (real Claude+Codex, transfer, and the honest thorough-analyst failure + fix). - sweep.py flushes stdout for live monitoring. Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
This commit is contained in:
Yifan Yang
@@ -0,0 +1,132 @@
|
||||
# SkillOpt-Sleep — final validation report
|
||||
|
||||
> **What this is:** the consolidated, presented results for the SkillOpt-Sleep
|
||||
> Claude Code plugin — a tool that lets a local agent improve itself overnight by
|
||||
> reviewing past sessions, replaying tasks, and consolidating validated memory +
|
||||
> skills behind a held-out gate. This document collects every real-model result
|
||||
> we ran, on **both Claude and Codex**, including the honest failures and the
|
||||
> fixes they drove.
|
||||
|
||||
**Date:** 2026-06-07 · **Branch:** `feat/claude-code-sleep-plugin`
|
||||
**Benchmark:** [gbrain-evals](https://github.com/garrytan/gbrain-evals) `skillopt-v1`
|
||||
(the same public suite gbrain scores its own optimizer against).
|
||||
|
||||
---
|
||||
|
||||
## 1. The claim, in one table
|
||||
|
||||
A deliberately **deficient** skill is given to a frozen agent. SkillOpt-Sleep runs
|
||||
1–2 offline "nights" (replay → reflect → bounded gated edit). We score the
|
||||
**held-out** task set (never optimized against) before and after. The harness
|
||||
computes the score with a local rule judge — the optimizer never grades itself.
|
||||
|
||||
| Backend (target) | Optimizer | Seed | Held-out before → after | Nights |
|
||||
|---|---|---|---|---|
|
||||
| Claude Haiku 4.5 | Claude Haiku | brief-writer | **0.00 → 1.00** | 1 |
|
||||
| Claude Haiku 4.5 | Claude Haiku | advisor | **0.00 → 1.00** | 2 |
|
||||
| Claude Haiku 4.5 | Claude Haiku | thorough-analyst | **0.00 → 1.00** † | 2 |
|
||||
| Codex (gpt-5.5) | Codex | brief-writer | **0.00 → 1.00** | 2 |
|
||||
|
||||
† after the override-prompt fix described in §3. Before the fix it was 0.00 → 0.00,
|
||||
and we report that honestly because it taught us the most (see §3).
|
||||
|
||||
**Bottom line:** across two independent agent runtimes (Claude and Codex) and
|
||||
multiple distinct skill flaws (missing structure, no verdict, no length
|
||||
discipline), the sleep cycle lifts a deficient skill to a perfect held-out score,
|
||||
with every change gated and staged for review.
|
||||
|
||||
---
|
||||
|
||||
## 2. Cross-model transfer (the price-difference value prop)
|
||||
|
||||
> *Optimize cheap overnight, deploy anywhere.* A skill is just instructions, so a
|
||||
> good rewrite should help a model it was never optimized on. This is what makes
|
||||
> the nightly spend worth it: you can optimize with a cheap model and the learned
|
||||
> skill still helps an expensive one.
|
||||
|
||||
_(Auto-filled from the sweep — see `benchmark_report.md` / `sweep.jsonl`.)_
|
||||
|
||||
| Source (optimizer) | Target (deploy) | Seed | Target baseline | Transferred | Gain |
|
||||
|---|---|---|---|---|---|
|
||||
| _populated by the sweep_ | | | | | |
|
||||
|
||||
---
|
||||
|
||||
## 3. The honest failure that made the tool better
|
||||
|
||||
The most valuable run was a **failure**. `thorough-analyst` (a skill that rambles;
|
||||
held-out demands answers under 1200 characters) went **0.00 → 0.00** at first —
|
||||
every nightly edit was rejected by the gate.
|
||||
|
||||
**Why:** the optimizer *did* propose good length-limiting rules, but our engine
|
||||
**appends** learned rules to a protected block and never deletes the user's
|
||||
hand-written skill body — which still said *"be exhaustive and detailed, write
|
||||
multiple paragraphs."* The base instruction won; outputs stayed ~6000 chars.
|
||||
|
||||
**The fix:** we verified that a forceful override rule
|
||||
("HARD LIMIT: response MUST be under 1200 characters; this supersedes any
|
||||
instruction to be exhaustive") makes Haiku obey — outputs dropped to 1194 / 880
|
||||
chars, hard = 1.00. So we taught the `reflect` prompt that its edits are appended
|
||||
and cannot delete the base text, so on a conflict it must emit an explicit
|
||||
override. (This mirrors gbrain's own write-up, where the first SkillOpt run scored
|
||||
0/4 until the optimizer was told what the scorer rewards.)
|
||||
|
||||
This is the pattern we want from a tool people rely on: run it against real
|
||||
models, find the real failure, fix the mechanism, report both.
|
||||
|
||||
---
|
||||
|
||||
## 4. What the optimizer actually wrote (sample)
|
||||
|
||||
**brief-writer (Claude):** a full format template —
|
||||
`Recommendation / Rationale / Key Risks / Confidence`.
|
||||
|
||||
**brief-writer (Codex, 2 nights):** night 1 added the two required rules; night 2
|
||||
**diagnosed its own residual failure** and added
|
||||
*"Preserve required sections even when keeping the brief short; shorten the
|
||||
analysis before omitting Key Risks or Confidence"* → held-out 1.00. That second
|
||||
edit is reasoning about why the prior night underperformed — the core argument for
|
||||
the sleep **loop** over a one-shot rewrite.
|
||||
|
||||
All edits land in the protected `SKILLOPT-SLEEP:LEARNED` block; the rest of the
|
||||
skill is never touched, and nothing is applied to live config until the user
|
||||
runs `/sleep adopt`.
|
||||
|
||||
---
|
||||
|
||||
## 5. Reproduce everything
|
||||
|
||||
```bash
|
||||
git clone https://github.com/garrytan/gbrain-evals /tmp/gbrain-evals
|
||||
cd <repo>/SkillOpt-sleep
|
||||
|
||||
# single seed, one backend
|
||||
python3.12 -m skillopt.sleep.experiments.run_gbrain --backend claude --model haiku \
|
||||
--seeds brief-writer --data-root /tmp/gbrain-evals/eval/data/skillopt-v1 \
|
||||
--nights 2 --limit-replay 3 --limit-holdout 3
|
||||
|
||||
# cross-model transfer
|
||||
python3.12 -m skillopt.sleep.experiments.run_transfer \
|
||||
--source-backend claude --source-model haiku \
|
||||
--target-backend claude --target-model sonnet --seeds brief-writer
|
||||
|
||||
# the whole sweep + this report
|
||||
python3.12 -m skillopt.sleep.experiments.sweep --plan full \
|
||||
--data-root /tmp/gbrain-evals/eval/data/skillopt-v1 --out docs/sleep/sweep.jsonl
|
||||
python3.12 -m skillopt.sleep.experiments.report \
|
||||
--in docs/sleep/sweep.jsonl --out docs/sleep/benchmark_report.md
|
||||
|
||||
# deterministic, no API
|
||||
python3.12 -m skillopt.sleep.experiments.run_experiment --persona researcher --assert-improves
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. Honest limitations
|
||||
|
||||
- **Latency:** each CLI call is ~14–15 s of startup-dominated wall time, so runs
|
||||
are capped at a few tasks/nights. Fine for nightly cron; we note it plainly.
|
||||
- **One seed needs a tool loop:** `quick-answerer` (`tool_called: search`) needs
|
||||
real tool execution; that is Phase-3 `fresh` worktree replay, not yet wired.
|
||||
- **Small, single-flaw skills:** like gbrain, these prove the mechanism is real
|
||||
and safe; a large production skill will be messier and partial.
|
||||
Reference in New Issue
Block a user