Yifan Yang
4e7add899d
Add skillopt/sleep — a deployment-time companion to SkillOpt that gives a
local Claude agent a nightly "sleep cycle":
harvest ~/.claude transcripts -> mine recurring tasks -> replay offline
-> consolidate (reflect -> bounded edit -> held-out GATE) -> stage -> adopt
Synthesizes SkillOpt (validation-gated bounded text optimization, reusing
skillopt.evaluation.gate verbatim), Claude Dreams (offline consolidation;
input never mutated; review-then-adopt), and the agent-sleep paper
(short-term experience -> long-term competence).
Engine (skillopt/sleep/, import-light, py>=3.10):
- harvest.py read-only parse of session JSONL + history.jsonl
- mine.py sessions -> TaskRecords (heuristic miner + LLM hook)
- backend.py MockBackend (deterministic, no API) + AnthropicBackend
- replay.py offline re-run -> (hard, soft) scores
- consolidate.py one SkillOpt epoch behind a held-out gate
- memory.py protected-region edits to SKILL.md / CLAUDE.md
- staging.py stage proposals; adopt with backup (Dreams safety contract)
- cycle.py + __main__.py orchestrator + CLI (run/dry-run/status/adopt/harvest)
Plugin (skillopt-sleep-plugin/): plugin.json, /sleep command, skillopt-sleep
skill, SessionEnd hook, bundled runner + cron generator.
Validation (deterministic, no API): persona experiment proves held-out lift
(researcher 0.33->1.0, programmer 0.32->1.0) AND that the gate rejects an
injected harmful edit. 13 stdlib-unittest tests pass, incl. full cycle +
adopt-with-backup and parsing of real on-disk transcripts.
Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
2.4 KiB
2.4 KiB
SkillOpt-Sleep — validation experiment results
Generated: 2026-06-07 (autonomous offline session) Backend: mock (deterministic, no API). Reproducible via the commands below.
$ python3.12 -m skillopt.sleep.experiments.run_experiment --persona researcher --nights 4 --json
{
"persona": "researcher",
"backend": "mock",
"nights_run": 1,
"baseline_holdout": 0.3333,
"after_holdout": 1.0,
"lift": 0.6667,
"improved": true,
"gate_blocks_harmful": true,
"final_skill_excerpt": "T -->\n## Learned preferences & procedures\n\n_This block is maintained by SkillOpt-Sleep. Edits here are proposed offline, validated against your past tasks, and adopted only after you approve them. Hand-edits outside this block are never touched._\n\n- Always wrap the final answer in <answer>...</answer> tags.\n- Report arXiv ids in the exact form arXiv:XXXX.XXXXX.\n<!-- SKILLOPT-SLEEP:LEARNED END -->\n",
"trace": [
{
"night": 0,
"holdout_score": 0.3333,
"action": "baseline",
"n_edits": 0
},
{
"night": 1,
"holdout_score": 1.0,
"action": "accept_new_best",
"accepted": true,
"n_edits": 2,
"edits": [
"Always wrap the final answer in <answer>...</answer> tags.",
"Report arXiv ids in the exact form arXiv:XXXX.XXXXX."
],
"n_rejected": 0
}
]
}
$ python3.12 -m skillopt.sleep.experiments.run_experiment --persona programmer --nights 4 --json
{
"persona": "programmer",
"backend": "mock",
"nights_run": 1,
"baseline_holdout": 0.3194,
"after_holdout": 1.0,
"lift": 0.6806,
"improved": true,
"gate_blocks_harmful": true,
"final_skill_excerpt": "laude Code sessions.\n\n<!-- SKILLOPT-SLEEP:LEARNED START -->\n## Learned preferences & procedures\n\n_This block is maintained by SkillOpt-Sleep. Edits here are proposed offline, validated against your past tasks, and adopted only after you approve them. Hand-edits outside this block are never touched._\n\n- Write git commit subjects in imperative mood, max 50 chars.\n<!-- SKILLOPT-SLEEP:LEARNED END -->\n",
"trace": [
{
"night": 0,
"holdout_score": 0.3194,
"action": "baseline",
"n_edits": 0
},
{
"night": 1,
"holdout_score": 1.0,
"action": "accept_new_best",
"accepted": true,
"n_edits": 1,
"edits": [
"Write git commit subjects in imperative mood, max 50 chars."
],
"n_rejected": 0
}
]
}