url/SkillOpt
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: sync documentation with post-v0.2 changes

Browse Source
This commit is contained in:
Yif-Yang
2026-07-14 17:11:40 +00:00
parent efb30b4bcc
commit f31bf8c06b
44 changed files with 2285 additions and 2025 deletions
Download Patch File Download Diff File Expand all files Collapse all files
plugins/README.md
+133 -217
View File
@@ -1,262 +1,178 @@
# SkillOpt-Sleep — plugins for Claude Code, Codex, Copilot, and Devin
# SkillOpt-Sleep integrations
**Your coding agent forgets everything between sessions. SkillOpt-Sleep fixes
that.** While you sleep, it reviews what you did today, notices the rules you
keep repeating ("always add a LIMIT", "answers in `\boxed{}`", "cite the
source"), and writes them into your agent's long-term memory and skills — but
only the rules that actually make it score better on *your own* past tasks. You
wake up to an agent that's better at *your* work, and you approve every change
before it sticks.
**SkillOpt-Sleep** reviews recent agent sessions, mines recurring tasks, replays
them, and proposes bounded updates to memory and skills. A held-out validation
gate decides whether a proposal is worth staging, and nothing live changes until
the user explicitly adopts it.
One engine, four thin shells. It synthesizes **SkillOpt** (validation-gated
bounded text optimization — the research in this repo), **Claude Dreams**
(offline consolidation; input never mutated; review-then-adopt), and the **agent
sleep** idea (short-term experience → long-term competence).
The shared engine lives in [`skillopt_sleep/`](../skillopt_sleep) and has no
runtime dependency on the paper's `skillopt/` experiment package.
> **Open-source tool, decoupled from the research.** The engine lives in the
> top-level [`skillopt_sleep/`](../skillopt_sleep) package with **zero
> dependency** on the paper's `skillopt/` experiment code (the validation gate is
> vendored). Use it without the research stack.
## Available integrations
---
Four integrations wrap the shared `skillopt_sleep` CLI. OpenClaw is a separate
reference adaptation with its own backend and setup assumptions.
| Platform | Folder | Mechanism | Status |
|---|---|---|---|
| **Claude Code** | [`claude-code/`](claude-code) | `.claude-plugin` + `/skillopt-sleep` + `/skillopt-sleep-handoff` commands + skill + hooks | full, installable |
| **Codex** | [`codex/`](codex) | user-level `skillopt-sleep` skill + shared runner | full |
| **Copilot** | [`copilot/`](copilot) | MCP server (`sleep_*` tools) + `copilot-instructions` | full (MCP) |
| **Devin** | [`devin/`](devin) | MCP server (`sleep_*` tools) + Devin ATIF-v1.7 harvest + `.devin/rules` | full (MCP) |
| **Claude Code** | [`claude-code/`](claude-code) | marketplace plugin, commands, skill, and hooks | installable shared-engine integration |
| **Codex** | [`codex/`](codex) | user-level skill and shared runner | installable shared-engine integration |
| **GitHub Copilot** | [`copilot/`](copilot) | MCP server exposing seven `sleep_*` tools | shared-engine MCP integration |
| **Devin** | [`devin/`](devin) | MCP server plus Devin transcript conversion | shared-engine MCP integration |
| **OpenClaw** | [`openclaw/`](openclaw) | custom DeepSeek/Ollama wrapper | independent reference adaptation; review and adapt before use |
## Install (pick your agent)
## Install
Clone the repository first unless an installed `skillopt-sleep` CLI is sufficient
for your workflow.
| Platform | Install | Then |
|---|---|---|
| **Claude Code** | `/plugin marketplace add microsoft/SkillOpt` → `/plugin install skillopt-sleep` | `/skillopt-sleep status` |
| **Codex** | `git clone` → `bash plugins/codex/install.sh` | `/skillopt-sleep status` |
| **Copilot** | `git clone` → register `plugins/copilot/mcp_server.py` as an MCP server | ask "run the sleep cycle" |
| **Devin** | `git clone` → `devin mcp add skillopt-sleep -- python3 plugins/devin/mcp_server.py` | ask "run the sleep cycle" |
| **Claude Code** | from the repository root, `/plugin marketplace add ./plugins/claude-code`, then `/plugin install skillopt-sleep@skillopt-sleep` | `/skillopt-sleep status` |
| **Codex** | `bash plugins/codex/install.sh` | ask Codex to use the `skillopt-sleep` skill |
| **Copilot** | register `plugins/copilot/mcp_server.py` using its example MCP config | ask Copilot to run `sleep_status` |
| **Devin** | register `plugins/devin/mcp_server.py` using its example MCP config | ask Devin to run `sleep_status` |
| **OpenClaw** | follow and adapt [`openclaw/README.md`](openclaw/README.md) | validate paths, credentials, and tasks locally |
Requirements: Python ≥ 3.10 and the agent's CLI on PATH. All three call the same
[`run-sleep.sh`](run-sleep.sh) → `python -m skillopt_sleep`, so behaviour is
identical everywhere. Default backend is `mock` (no API spend); `--backend
claude|codex|copilot` uses your own budget.
Python 3.10 or newer is required. Real CLI backends also require the selected
agent CLI to be installed and authenticated.
---
The shared [`run-sleep.sh`](run-sleep.sh) supports both source checkouts and
installed packages. If it cannot find the repository, it tries the
`skillopt-sleep` executable on `PATH` (including `uv tool`/`pipx` installs), then
an importable `skillopt_sleep` module. Install with `uv tool install skillopt` or
`pip install skillopt` when using that fallback.
## How it works: one "night", in plain terms
> **Version note.** This integration reference tracks `main`. PyPI 0.2.0
> supports the base Sleep CLI, while handoff, Sleep support for non-Azure
> OpenAI-compatible endpoints, and `--preferences` require a source checkout
> from `main` until the next release.
```
harvest your past sessions → mine the tasks you keep doing → replay them offline
→ reflect on failures → propose a few rule edits → KEEP only edits that raise
your held-out score → stage a proposal → (you) review & adopt
## One sleep cycle
```text
harvest supported local sessions → mine recurring tasks → replay tasks
→ reflect and propose bounded edits → validate on held-out real tasks
→ stage proposal → (you) review and adopt
```
Nothing live changes until you `adopt`; every adopt backs up the prior file.
The default backend is `mock`: it makes no provider calls and is useful for
checking plumbing. A real backend is required for model-driven mining and genuine
optimization.
### The split that keeps it honest: dream-train / real-val / real-test
## Data boundary
This is the heart of the design, borrowed from the SkillOpt paper's
train/selection/test protocol:
- Harvesting is local and read-only. The `mock` backend has no model-provider
data path and no API spend.
- A real backend sends truncated transcript excerpts and derived task content to
the provider selected for mining, replay, judging, and reflection.
- Outbound prompts are not currently guaranteed to be free of secrets. Do not
use a third-party provider on sensitive transcripts without reviewing the data
source and the provider's retention policy.
- For a reviewable workflow, export tasks first, inspect and redact the JSON, set
its top-level `"reviewed"` field to `true`, and then use the task file with a
real backend:
| Split | Where it comes from | What it's for |
```bash
python -m skillopt_sleep harvest --project "$(pwd)" --output reviewed-tasks.json
python -m skillopt_sleep dry-run --project "$(pwd)" --backend codex \
--tasks-file reviewed-tasks.json --progress
```
Real backends reject task files that are still marked unreviewed.
For the separate API-key and Azure managed-identity transport boundaries, see
[OpenAI-compatible endpoints](../docs/sleep/openai-compatible-endpoints.md).
## Supported CLI surface
Actions:
| Action | Behavior |
|---|---|
| `status` | show state and the latest staged proposal |
| `dry-run` | harvest, mine, replay, and report; stage nothing |
| `run` | run the full cycle and stage a proposal |
| `adopt` | apply the latest staged proposal, with backups |
| `harvest` | inspect or export mined tasks |
| `schedule` / `unschedule` | install or remove the managed nightly cron entry |
Common implemented flags include:
| Flag | Default | Purpose |
|---|---|---|
| **train** | your real tasks **+ optional "dreamed" variants** | what the optimizer *learns from*. Over-dreaming here is fine — it's imagination. |
| **val** (selection) | **your real tasks only**, held out | the **gate**: an edit is kept only if it raises this score. Stops overfitting. |
| **test** | **your real tasks only**, held out, never seen during optimization | the **final score** we report. Kept as close to your real usage as possible. |
| `--backend mock\|claude\|codex\|copilot\|handoff\|azure_openai` | `mock` | select who performs model calls |
| `--model NAME` | backend default | select a backend-specific model |
| `--source claude\|codex\|auto` | `claude` | select the transcript source |
| `--project PATH` | current directory | select the project and invoked harvest scope |
| `--scope invoked\|all` | `invoked` | limit transcript harvesting |
| `--target-skill-path PATH` | managed skill | select a specific `SKILL.md` to stage/adopt |
| `--tasks-file PATH` | none | replay a reviewed task file instead of harvesting |
| `--max-sessions N` / `--max-tasks N` | unset → `3 × tasks` / `40` tasks | bound harvested work; these are not hard token or wall-clock budgets |
| `--edit-budget N` | `4` | cap bounded edits per cycle |
| `--preferences "..."` | empty | add house rules to the reflection prior |
| `--progress` | off | print phase progress to stderr |
| `--auto-adopt` | off | adopt an accepted proposal without a separate command |
| `--json` | off | emit machine-readable output where supported |
So you can **dream up extra training examples** to learn a rule robustly, while
the rule is still **judged on real, unseen tasks**. A `dream` task can *never*
land in val or test — that invariant is unit-tested.
The nightly CLI does **not** currently expose `--gate`, `--rollouts-k`,
`--optimizer-model`, `--target-model`, `--budget-tokens`, or `--budget-minutes`.
Do not pass experiment-harness flags to the main CLI.
---
### Preferences
## What each feature does **for you** (with examples)
Every control below works on all three platforms (pass it after the action,
e.g. `/skillopt-sleep run --rollouts-k 3`).
### `--preferences "..."` — tell it your house rules
The single most useful knob. Free text that steers what the optimizer writes,
as a prior. Use it to encode the conventions you're tired of repeating.
`--preferences` is the main user-facing steering knob:
```bash
# A backend engineer:
/skillopt-sleep run --preferences "Always use async/await, never callbacks. \
Prefer pytest over unittest. Commit subjects in imperative mood under 50 chars."
# A data analyst:
/skillopt-sleep run --preferences "Every SQL query must end with LIMIT 1000 unless \
I say otherwise. Money in USD with 2 decimals. Prefer CTEs over nested subqueries."
# A researcher:
/skillopt-sleep run --preferences "Cite sources as [Author, Year]. Math answers in \
\\boxed{}. Keep explanations under 150 words unless I ask for depth."
python -m skillopt_sleep run --backend codex --project "$(pwd)" \
--preferences "Prefer pytest. Keep commit subjects imperative and concise."
```
*What it does for you:* the next morning your agent already follows these
without you re-typing them, and the rules are validated against your real tasks
(if a "preference" actually hurts your held-out score, the gate drops it).
### `--gate on|off` — strict vs. greedy
Preferences guide reflection but remain subject to the validation gate.
- `on` (default): an edit is kept **only if it raises your held-out score**.
Safe — blocks plausible-but-wrong rules and reward-hacking.
- `off`: greedy — keep edits without the strict check (still reports whether
quality moved).
### Advanced config
*What it does for you:* leave it `on` for trust. Flip it `off` when you're
exploring and want to see everything the optimizer proposes.
The JSON/YAML config under `~/.skillopt-sleep/` supports additional engine keys,
including `gate_mode`, `gate_metric`, `dream_rollouts`, `dream_factor`, `recall_k`,
`evolve_memory`, and `evolve_skill`. These are config keys, not aliases for the
unsupported CLI flags listed above. Shipping defaults are conservative:
`gate_mode="on"`, `dream_rollouts=1`, `dream_factor=0`, and `recall_k=0`.
### `--rollouts-k K` — learn from contrast, not just failure
### Handoff backend
Re-runs each task `K` times and learns from the difference between the **good**
and **bad** attempts, not just a single failure.
```bash
/skillopt-sleep run --rollouts-k 3
```
*What it does for you:* a much stronger signal. If your agent gets a task right 1
time in 3, the optimizer figures out *what the winning attempt did* and makes it
reliable.
### `--optimizer-model` / `--target-model` — optimize cheap, deploy anywhere
Use a strong model to *write* the rules and a cheap model to *run* your tasks.
The learned skill then helps the cheap model — or any model.
```bash
/skillopt-sleep run --optimizer-model sonnet --target-model haiku
```
*What it does for you:* spend a little on a smart optimizer overnight; your
everyday cheap/fast agent inherits the upgrade. (Verified: a skill optimized on
one model lifts a different one — cross-model and even cross-runtime
Codex↔Claude.)
### `--budget-tokens N` / `--budget-minutes M` — cap the spend
You decide how much the nightly "dreaming" costs; it auto-plans how many nights
× how many rollouts fit.
```bash
/skillopt-sleep run --backend claude --budget-tokens 60000
```
*What it does for you:* predictable cost. It stops cleanly when the budget is hit
and tells you what it skipped.
### multi-objective (accuracy ↑, tokens ↓, latency ↓)
The reward can weight not just correctness but **cost and speed**, so a skill can
learn to be cheaper and faster, not only more accurate. *What it does for you:*
"answer directly instead of opening five files" becomes a learned habit.
### `--backend handoff` — session-executed calls (no API subprocess)
For subscription seats and environments where the engine shouldn't spawn
`claude -p` / API calls itself. The engine still runs every deterministic
stage (harvest → mine → replay scoring → gate → stage), but each model call
(attempt / judge / reflect) is written to a prompt file that **your own agent
session answers between engine runs**:
`--backend handoff` keeps model subprocesses out of the engine. It writes pending
model calls to `.skillopt-sleep-handoff/PROMPTS.md` and `pending.json`, exits with
code 3, and resumes after answers are placed in `answers/<id>.md`:
```bash
python -m skillopt_sleep run --backend handoff --project "$(pwd)"
# exit 3 => .skillopt-sleep-handoff/PROMPTS.md + pending.json were written
# answer each prompt (each in a FRESH context) into answers/<id>.md
# re-run the same command => it resumes from the answers and either
# finishes (exit 0) or stages the next prompt batch (exit 3)
# answer each prompt in a fresh context, then run the same command again
```
A typical night converges in 3–6 rounds: baseline attempts → reflect →
candidate re-scoring per accepted edit. Resume is stateless — replay is
deterministic and answers are cached by prompt hash, so re-running skips
everything already answered. Mined tasks are pinned to
`.skillopt-sleep-handoff/tasks.json` on the first round, so the sessions that
answer the prompts can't shift the task set and invalidate earlier answers.
On a completed real run the handoff directory is archived to
`.skillopt-sleep-handoff.night<N>.done`.
Answering held-out prompts from a context that has already seen their references
contaminates the validation gate. Claude Code's `/skillopt-sleep-handoff` command
automates the loop with isolated fresh-context subagents.
On Claude Code, `/skillopt-sleep-handoff run` drives the whole loop for you,
answering each prompt in an isolated fresh-context subagent.
## Validation
**Integrity rule:** answer every prompt in a fresh context (a subagent with no
conversation history). Answering from a session that has already seen the
mined tasks and their references contaminates the held-out gate and fakes the
improvement score.
*What it does for you:* the sleep cycle runs entirely on your interactive
session's subscription budget — no API key, no headless subprocess — while the
gate, splits, and staging discipline stay in the engine.
Limitations: `--rollouts-k > 1` gives no contrastive spread (identical prompt
→ identical answer file), and tool-loop tasks fall back to the single-shot
`TOOL_CALL:` marker convention.
### `schedule` / `unschedule` — set it and forget it
Built-in nightly scheduling (no manual cron):
The deterministic no-provider check exercises consolidation and the gate:
```bash
/skillopt-sleep schedule --hour 3 --minute 17 # runs every night for this project
/skillopt-sleep unschedule # stop it
python -m skillopt_sleep.experiments.run_experiment \
--persona researcher --assert-improves
```
*What it does for you:* it just gets better while you sleep. The nightly run only
*stages* a proposal — adopting is still your call (or add `--auto-adopt` when you
schedule, if you trust it).
---
Real-model benchmark results and their limitations are documented in
[`docs/sleep/RESULTS.md`](../docs/sleep/RESULTS.md). The benchmark recipes are not
the shipping CLI defaults.
## Full action / flag reference
## Safety summary
| Action | Does |
|---|---|
| `status` | nights so far + the latest staged proposal (read-only) |
| `dry-run` | harvest→mine→replay→report; **stages nothing** |
| `run` | full cycle; **stages** a proposal; nothing live changes |
| `adopt` | apply the staged proposal to `CLAUDE.md`/`SKILL.md` (backs up first) |
| `harvest` | debug: print the recurring tasks it mined |
| `schedule` / `unschedule` | install/remove the nightly cron entry |
| Flag | Default | Meaning |
|---|---|---|
| `--backend mock\|claude\|codex\|copilot\|handoff` | `mock` | who runs/optimizes (mock = free; handoff = your own session answers) |
| `--preferences "..."` | – | your house rules, as a prior |
| `--gate on\|off` | `on` | strict held-out gate vs. greedy |
| `--rollouts-k K` | `1` | multi-rollout contrastive reflection |
| `--optimizer-model` / `--target-model` | – | split the optimizer from the target |
| `--budget-tokens` / `--budget-minutes` | – | cap the nightly spend |
| `--scope invoked\|all` | `invoked` | this project only, or all projects |
| `--auto-adopt` | off | apply without manual review (power users) |
Deep dive: [the SkillOpt-Sleep guide section](https://microsoft.github.io/SkillOpt/docs/guideline.html#sleep).
---
## Does it actually work?
Yes — measured with **real models on both Claude and Codex**, scored on held-out
tasks the optimizer never trained on:
- **gbrain-evals `skillopt-v1`** (the public suite gbrain scores SkillOpt on):
deficient skills go **0.00 → 1.00** on all 4 seeds, including a real tool-use
loop; cross-model transfer is positive; the gate blocks regressions.
→ [the SkillOpt-Sleep guide section](https://microsoft.github.io/SkillOpt/docs/guideline.html#sleep)
- **Academic daily-cases** (math / spreadsheet / search-QA, the paper's 4:1:5
split with dream-augmented train): see
[the SkillOpt-Sleep guide section](https://microsoft.github.io/SkillOpt/docs/guideline.html#sleep).
- **Fresh load-test** (a "SQL must always include LIMIT" analyst, built from
scratch): held-out **0.00 → 1.00** on both backends.
→ [the SkillOpt-Sleep guide section](https://microsoft.github.io/SkillOpt/docs/guideline.html#sleep)
Try the deterministic proof yourself (no API key, no spend):
```bash
python -m skillopt_sleep.experiments.run_experiment --persona researcher --assert-improves
```
It prints the held-out score rising to 1.0 as the gate accepts the right rules,
and confirms the gate **rejects** an injected harmful edit.
---
## Safety
- **Read-only** harvest of your sessions. `mock` replay has no side effects.
- Proposals are **staged**, never auto-applied (unless you opt in with `--auto-adopt`).
- Every adopt writes a backup. Per-night token/time budget caps. Secrets redacted.
- Session harvesting is read-only.
- `mock` replay makes no provider calls.
- `run` stages proposals; `adopt` is the normal live-change boundary.
- Adoption backs up existing target files.
- `--max-sessions` and `--max-tasks` bound work, but the main CLI does not yet
enforce a hard token or elapsed-time budget.
- Treat real-backend transcript excerpts as data shared with the selected
provider.
plugins/claude-code/README.md
+45 -19
View File
@@ -10,7 +10,7 @@ SkillOpt-Sleep is the **deployment-time** companion to
[SkillOpt](https://github.com/microsoft/SkillOpt). SkillOpt trains a skill
offline on a benchmark; SkillOpt-Sleep applies the same discipline to *your own
daily usage*: bounded text edits, accepted only through a held-out validation
gate, with rejected edits kept as negative feedback.
gate, with rejected candidates recorded in the cycle report for review.
It synthesizes three ideas:
@@ -32,7 +32,8 @@ then adopt or discard" contract). Every adopt backs up the prior file first.
## Install
**Requirements:** Python ≥ 3.10, and the `claude` CLI (and/or `codex` CLI) on PATH.
**Requirements:** Python ≥ 3.10. A real CLI backend additionally requires its
corresponding `claude` or `codex` executable on `PATH` and authenticated.
```bash
# 1) get the code (the plugin ships inside the SkillOpt repo)
@@ -40,7 +41,7 @@ git clone https://github.com/microsoft/SkillOpt.git
cd SkillOpt
# 2) add the plugin to Claude Code as a local marketplace
/plugin marketplace add ./skillopt-sleep-plugin
/plugin marketplace add ./plugins/claude-code
/plugin install skillopt-sleep@skillopt-sleep
# 3) verify
@@ -48,15 +49,21 @@ cd SkillOpt
```
The plugin's bundled runner (`scripts/sleep.sh`) auto-selects a Python ≥ 3.10
interpreter and calls the `skillopt_sleep` engine in the repo. No `pip install`
is required for the default `mock` backend or for `claude`/`codex` backends —
they shell out to the CLIs you already have.
interpreter and calls the `skillopt_sleep` engine. A source checkout needs no
`pip install`. If the marketplace cache does not contain a usable source tree,
the shared runner falls back first to a `skillopt-sleep` executable on `PATH`
(including `uv tool`/`pipx` installs), then to an importable Python module. Use
`uv tool install skillopt` or `pip install skillopt` for that fallback.
> **Version note.** This page tracks `main`. PyPI 0.2.0 provides the base Sleep
> CLI, but handoff mode and `--preferences` require a source checkout from
> `main` until the next release.
## Quick start
```bash
# from inside any project you use with Claude Code:
/skillopt-sleep dry-run # safe preview: what it would learn, no changes staged
/skillopt-sleep dry-run # preview what it would learn; no changes staged
/skillopt-sleep run # full cycle: stages a reviewed proposal (still no live edits)
/skillopt-sleep status # see history + the latest staged proposal
/skillopt-sleep adopt # apply the staged proposal to CLAUDE.md / SKILL.md (with backup)
@@ -74,8 +81,19 @@ python -m skillopt_sleep run --project "$(pwd)" --backend codex # real lift v
```
Default backend is **`mock`** — deterministic, no API spend — so you can try the
plumbing for free. Switch to `--backend claude` or `--backend codex` for genuine
improvement on your own budget.
plumbing for free. Switch to `--backend claude` or `--backend codex` for
model-driven mining and optimization on your own budget; an accepted gain is
task- and model-dependent, not guaranteed.
### Data boundary for real backends
Harvesting `~/.claude` is local and read-only, and the `mock` backend makes no
provider calls. A real backend sends truncated transcript excerpts and derived
tasks to the selected provider for mining, replay, judging, and reflection.
Outbound prompts are not currently guaranteed to be secret-free. Review your
session data and provider policy before using a real backend on a sensitive
project; the [shared integration guide](../README.md#data-boundary) describes a
reviewed task-file workflow.
### Handoff mode (session answers the model calls)
@@ -95,7 +113,7 @@ python -m skillopt_sleep run --backend handoff --project "$(pwd)" # resume
Answer every prompt in a **fresh context** — a session that has already seen
the mined tasks and their references would contaminate the held-out gate.
Details: [the plugins README](../README.md#--backend-handoff--session-executed-calls-no-api-subprocess).
Details: [the plugins README](../README.md#handoff-backend).
## Does it actually improve? (real models, public benchmark)
@@ -114,8 +132,8 @@ Both took a brief-writer with no risks section / no confidence level and, within
1–2 nights, proposed gated edits that lifted the held-out score to perfect —
into the protected `LEARNED` block, nothing else touched. The Codex 2-night
trace even shows the optimizer **diagnosing its own residual failure** and
adding a meta-rule to fix it. Full writeup + reproduction:
[the SkillOpt-Sleep guide section](https://microsoft.github.io/SkillOpt/docs/guideline.html#sleep).
adding a meta-rule to fix it. See the recorded results and limitations in
[`docs/sleep/RESULTS.md`](../../docs/sleep/RESULTS.md).
Reproduce:
@@ -138,24 +156,32 @@ python -m skillopt_sleep.experiments.run_experiment --persona programmer --asse
Each prints the held-out score rising from baseline toward 1.0 as the gate
accepts the general rules your tasks need, and confirms the gate **rejects** an
injected harmful edit. Recorded output: [the SkillOpt-Sleep guide section](https://microsoft.github.io/SkillOpt/docs/guideline.html#sleep).
injected harmful edit. Context for the measured experiments is in
[`docs/sleep/RESULTS.md`](../../docs/sleep/RESULTS.md).
## Schedule it nightly
```bash
"${CLAUDE_PLUGIN_ROOT}/scripts/install-cron.sh" "$(pwd)" # prints a crontab line; installs nothing
/skillopt-sleep schedule --hour 3 --minute 17
/skillopt-sleep unschedule
```
The built-in scheduler creates a managed cron entry and logs under the project.
The scheduled run stages proposals unless `--auto-adopt` is explicitly selected.
## Safety
- **Read-only** harvest of `~/.claude`. `mock` replay has no side effects.
- Proposals are **staged**, never auto-applied (unless you opt in with `--auto-adopt`).
- Every adopt writes a backup under the staging dir's `backup/`.
- Per-night **token/task budget caps**; secrets redacted from prompts.
- `fresh` replay (Phase 3) runs only in throwaway git worktrees.
- `--max-sessions` and `--max-tasks` bound work, but the main CLI does not enforce
a hard token or wall-clock budget.
- Real backends share truncated session/task content with the selected provider;
do not assume outbound prompts have been fully redacted.
## Status
Phase 1 (engine + deterministic experiment + plugin surface) is complete.
Phase 3 adds the real-API miner/judge and `fresh` worktree replay. See
[`docs/superpowers/specs/2026-06-07-skillopt-sleep-claude-code-plugin-design.md`](../docs/superpowers/specs/2026-06-07-skillopt-sleep-claude-code-plugin-design.md).
The engine, deterministic experiment, Claude/Codex CLI backends, handoff mode,
and staged adoption flow are implemented. Advanced experiment-harness flags are
not automatically available on the nightly CLI; see the
[shared integration reference](../README.md#supported-cli-surface).
plugins/claude-code/commands/skillopt-sleep-handoff.md
+13 -6
View File
@@ -21,10 +21,11 @@ isolated context so the validation gate stays honest.
Repeat until the engine exits 0 (done) — at most 8 rounds:
1. **Run the engine** via the bundled runner:
1. **Run the engine** via the bundled runner. Split `$ARGUMENTS` into the action
and remaining options, and preserve those options on every resumed round:
```bash
"${CLAUDE_PLUGIN_ROOT}/scripts/sleep.sh" <action> --backend handoff --project "$(pwd)" --scope invoked
"${CLAUDE_PLUGIN_ROOT}/scripts/sleep.sh" <action> --backend handoff --project "$(pwd)" --scope invoked <remaining options>
```
- exit 0 → the night is complete; go to "Finish" below.
@@ -49,10 +50,13 @@ Repeat until the engine exits 0 (done) — at most 8 rounds:
## Finish
- `Read` the `report.md` in the staging dir the engine printed and show
the user: held-out baseline → candidate score, the gate decision, the
proposed edits, and where the proposal is staged.
- Tell the user nothing live changed; offer `/skillopt-sleep adopt`.
- For `run`, if the engine prints a staging directory, `Read` its `report.md`
and show the user: held-out baseline → candidate score, the gate decision,
the proposed edits, and where the proposal is staged. If an accepted proposal
was staged, tell the user nothing live changed and offer
`/skillopt-sleep adopt`.
- For `dry-run`, no staging directory or `report.md` is created; summarize the
final stdout instead.
- The engine archives `.skillopt-sleep-handoff/` on a completed real run;
do not delete it yourself.
@@ -65,3 +69,6 @@ Repeat until the engine exits 0 (done) — at most 8 rounds:
set. Do not edit that file.
- If a batch looks like it contains secrets or content the user would not
want re-processed, stop and ask before answering.
- Handoff files apply pattern-based secret redaction, but that is not a
guarantee that prompts are free of sensitive data. Treat the pending batch as
private user data and do not copy it into chat, logs, or commits.
plugins/claude-code/commands/skillopt-sleep.md
+40 -24
View File
@@ -1,5 +1,5 @@
---
description: Run or manage the SkillOpt-Sleep self-evolution cycle (review past sessions, replay tasks offline, consolidate validated memory + skills; can also schedule nightly runs)
description: Run or manage the SkillOpt-Sleep self-evolution cycle (review past sessions, replay tasks through a selected backend, consolidate validated memory + skills, or schedule nightly runs)
argument-hint: "[run | dry-run | status | adopt | harvest | schedule | unschedule] (default: status)"
allowed-tools: Bash, Read
---
@@ -7,10 +7,11 @@ allowed-tools: Bash, Read
# /skillopt-sleep — SkillOpt-Sleep nightly self-evolution
You are driving **SkillOpt-Sleep**: a tool that lets this user's Claude agent
improve offline by reviewing past sessions, replaying recurring tasks, and
improve from past usage by reviewing sessions, replaying recurring tasks, and
consolidating what it learns into **validated** memory (`CLAUDE.md`) and skills
(`SKILL.md`). It is gated like SkillOpt: a change is kept only if it improves a
held-out replay score, and nothing live is modified until the user adopts it.
(`SKILL.md`). With the default gate enabled, a change is kept only if it improves
a held-out replay score. Nothing live is modified until adoption unless the
user explicitly requests `--auto-adopt`.
## Requested action: $ARGUMENTS
@@ -18,11 +19,14 @@ held-out replay score, and nothing live is modified until the user adopts it.
## How to run it
The engine is the `skillopt_sleep` Python package in this repo. Use the
**plugin's bundled runner** so the right interpreter and repo are on the path:
The engine is the `skillopt_sleep` Python package in this repo. Split
`$ARGUMENTS` into the first action token and its remaining options, then use the
**plugin's bundled runner** so the right interpreter and repo are on the path.
Preserve the user's remaining options (for example `--preferences`, `--backend`,
or `--target-skill-path`) instead of silently dropping them:
```bash
"${CLAUDE_PLUGIN_ROOT}/scripts/sleep.sh" <action> --project "$(pwd)" --scope invoked
"${CLAUDE_PLUGIN_ROOT}/scripts/sleep.sh" <action> --project "$(pwd)" --scope invoked <remaining options>
```
`<action>` is one of:
@@ -30,37 +34,49 @@ The engine is the `skillopt_sleep` Python package in this repo. Use the
| action | what it does |
|--------------|--------------|
| `status` | show how many nights have run + the latest staged proposal (READ-ONLY) |
| `dry-run` | harvest → mine → replay → report, but **stage nothing** (safe preview) |
| `run` | full cycle: also **stage** a reviewed proposal (still does NOT touch live files) |
| `dry-run` | harvest → mine → replay → report, but **stage nothing** (no-staging preview) |
| `run` | full cycle: **stage** a validation report and any accepted proposal; only explicit `--auto-adopt` may also update live files |
| `adopt` | apply the latest staged proposal to live `CLAUDE.md` / `SKILL.md` (backs up first) |
| `harvest` | debug: print the recurring tasks mined from recent sessions |
| `schedule` | install a nightly cron entry for this project (`--hour --minute`, off-:00 by default) |
| `unschedule` | remove the nightly cron entry (`--all` to remove every managed entry) |
Default backend is `mock` (deterministic, no API spend). To use real budget for
genuine improvement, add `--backend claude` or `--backend codex`. To steer what
the optimizer writes, add `--preferences "<your house rules>"`.
model-driven optimization, add `--backend claude` or `--backend codex`. An
accepted gain is evidence on this run's held-out tasks, not a guarantee of
general improvement; results depend on the tasks, model, and checks. To steer
what the optimizer writes, add `--preferences "<your house rules>"`.
## Steps to follow
1. **Run the requested action** via the bundled runner above. Capture stdout.
2. **For `run` / `dry-run`:** after it completes, `Read` the generated
`report.md` in the staging dir it prints, and show the user:
- held-out score: baseline → candidate (the proof it helped)
1. **Run the requested action** via the bundled runner above. Capture stdout and
stderr.
2. **For `run`:** if it prints a staging directory, `Read` its `report.md` and
show the user:
- held-out score: baseline → candidate (evidence on this run's held-out tasks)
- the gate decision (accept/reject) and the exact edits it proposes
- where the proposal is staged
3. **For `run` that produced an accepted proposal:** tell the user the diff is
staged and that **nothing live changed yet**. Offer to run `/skillopt-sleep adopt`.
4. **For `adopt`:** confirm which live files were updated and that backups were
3. **For `dry-run`:** no staging directory or `report.md` is created. Summarize
the score, gate decision, and edits from stdout (or request `--json` when
machine-readable output is useful).
4. **For `run` that produced an accepted proposal:** inspect whether stdout says
it was auto-adopted. If not, tell the user nothing live changed and offer
`/skillopt-sleep adopt`; if it was, report the updated paths explicitly.
5. **For `adopt`:** confirm which live files were updated and that backups were
written under the staging dir's `backup/`.
5. **Never** edit `CLAUDE.md` or `SKILL.md` yourself — only the `adopt` action
does that, with a backup. Respect the review gate.
6. **Never** edit `CLAUDE.md` or `SKILL.md` yourself — let the engine's explicit
`adopt` or user-requested `--auto-adopt` path apply its manifest and backup
behavior. Respect the review gate.
## Safety reminders
- Harvest is **read-only** over `~/.claude`. Replay in `mock` mode runs no
shell side effects.
- The cycle stages proposals; the user is in control of adoption.
- If the user asks to schedule this nightly, point them at
`${CLAUDE_PLUGIN_ROOT}/scripts/install-cron.sh` (prints a crontab line; does
not install anything without confirmation).
- The cycle stages proposals by default; auto-adoption requires explicit opt-in.
- A real backend sends truncated transcript excerpts and derived tasks to its
provider for mining, replay, judging, and reflection. Pattern-based redaction
is not a guarantee that outbound prompts are secret-free. For sensitive data,
use `mock` or first run `harvest --output <file>`, review/redact the file, set
`"reviewed": true`, and then pass it with `--tasks-file`.
- `schedule` manages a cron entry when `crontab` is available; otherwise it
prints a line for manual installation.
plugins/claude-code/scripts/install-cron.sh
+2 -1
View File
@@ -24,6 +24,7 @@ cat <<EOF
${MIN} ${HOUR} * * * "${RUNNER}" run --project "${PROJECT}" --scope invoked --backend ${BACKEND} >> "${PROJECT}/.skillopt-sleep/cron.log" 2>&1
#
# For fully-autonomous adoption (power users), append: --auto-adopt
# To spend real API budget for genuine lift, set BACKEND=anthropic above.
# To use the authenticated Claude CLI for model-driven optimization, set
# BACKEND=claude above.
# ────────────────────────────────────────────────────────────────────────────
EOF
plugins/claude-code/skills/skillopt-sleep/SKILL.md
+60 -31
View File
@@ -1,25 +1,29 @@
---
name: skillopt-sleep
description: "Use when the user wants their Claude agent to self-improve from past usage, asks about a nightly/offline 'sleep' or 'dream' cycle, memory/skill consolidation, or says things like 'make my agent better the more I use it', 'review my past sessions', 'learn my preferences', 'consolidate what you learned', 'run the sleep cycle', or wants to schedule offline self-optimization. Drives the skillopt_sleep engine: harvest past sessions -> mine recurring tasks -> replay offline -> consolidate validated CLAUDE.md/SKILL.md behind a held-out gate."
description: "Use when the user wants their Claude agent to self-improve from past usage, asks about a nightly/offline 'sleep' or 'dream' cycle, memory/skill consolidation, or says things like 'make my agent better the more I use it', 'review my past sessions', 'learn my preferences', 'consolidate what you learned', 'run the sleep cycle', or wants to schedule background self-optimization. Drives the skillopt_sleep engine: harvest past sessions -> mine recurring tasks -> replay through a selected backend -> consolidate validated CLAUDE.md/SKILL.md behind a held-out gate."
---
# SkillOpt-Sleep: offline self-evolution for a local Claude agent
# SkillOpt-Sleep: usage-driven self-evolution for a local Claude agent
SkillOpt-Sleep gives the user's agent a **sleep cycle**. While the user is
offline (e.g. nightly), it reviews their real past Claude Code sessions,
re-runs recurring tasks on their own API budget, and consolidates what it
learns into **memory** (`CLAUDE.md`) and **skills** (`SKILL.md`) — but only
keeps changes that pass a held-out validation gate, and only after the user
adopts them. The agent gets measurably better at *this* user's recurring work,
SkillOpt-Sleep gives the user's agent a **sleep cycle**. On demand or on a
nightly schedule, it reviews real past Claude Code sessions, re-runs recurring
tasks through the selected backend, and consolidates what it
learns into **memory** (`CLAUDE.md`) and **skills** (`SKILL.md`). With the
default validation gate enabled, it keeps only changes that improve a held-out
score. Live files change only through explicit adoption or a user-requested
`--auto-adopt`. It aims to improve this user's recurring work, while making
each accepted proposal measurable on the run's held-out tasks,
with no model-weight training. It is the deployment-time analogue of training:
short-term experience → long-term competence.
It synthesizes three ideas:
- **SkillOpt** — the skill/memory doc is trainable text; bounded add/delete/replace
edits; accepted only through a held-out gate; rejected edits become negative feedback.
- **Claude Dreams** — offline consolidation that reads past sessions and rebuilds
memory (dedup/merge/resolve); the input is never mutated; output is reviewed then adopted.
- **Agent sleep** — periodic offline replay turns episodes into durable skill.
edits; accepted only through a held-out gate; rejected edits are recorded in
the run report for review.
- **Claude Dreams** — consolidation that reads past sessions and proposes changes
inside protected learned blocks; the input is never mutated, and output is
reviewed before adoption.
- **Agent sleep** — periodic background replay turns episodes into durable skill.
## When to use this skill
@@ -34,9 +38,14 @@ Trigger when the user wants any of:
1. **Harvest** — read `~/.claude/projects/*/<session>.jsonl` + `~/.claude/history.jsonl` (READ-ONLY) → session digests.
2. **Mine** — digests → `TaskRecord`s (recurring intents + outcome labels + checkable refs where possible).
3. **Replay** — re-run tasks offline under the *current* skill+memory → (hard, soft) scores.
4. **Consolidate** — reflect on failures → propose bounded edits → **gate** on a held-out slice; accept only if it strictly improves.
5. **Stage** — write `proposed_CLAUDE.md`, `proposed_SKILL.md`, a diff, and `report.md` into `<project>/.skillopt-sleep/staging/<date>/`. **Nothing live changes.**
3. **Replay** — re-run tasks through the selected backend under the *current*
skill+memory → (hard, soft) scores.
4. **Consolidate** — reflect on failures → propose bounded edits → **gate** on a held-out slice; with the default gate enabled, accept only if it strictly improves.
5. **Stage** — write the accepted `proposed_CLAUDE.md` and/or
`proposed_SKILL.md`, plus `report.md`, `report.json`, `manifest.json`, and
`diagnostics.json` into `<project>/.skillopt-sleep/staging/<timestamp>/`.
**Nothing live changes.** A rejected run still has a report but no proposed
live-file replacement.
6. **Adopt** — explicit (or opt-in auto): copy staged files over live ones, backing up first.
## How to drive it
@@ -45,14 +54,20 @@ Prefer the `/skillopt-sleep` command. Under the hood it calls the bundled runner
```bash
"${CLAUDE_PLUGIN_ROOT}/scripts/sleep.sh" status # what's happened
"${CLAUDE_PLUGIN_ROOT}/scripts/sleep.sh" dry-run --project "$(pwd)" # safe preview
"${CLAUDE_PLUGIN_ROOT}/scripts/sleep.sh" dry-run --project "$(pwd)" # no-staging preview
"${CLAUDE_PLUGIN_ROOT}/scripts/sleep.sh" run --project "$(pwd)" # full cycle, stages a proposal
"${CLAUDE_PLUGIN_ROOT}/scripts/sleep.sh" adopt --project "$(pwd)" # apply staged proposal (with backup)
```
- Default backend is `mock` (deterministic, **no API spend**) — good for trying the plumbing.
- Add `--backend claude` or `--backend codex` to spend the user's real budget for genuine improvement.
- Scope defaults to the invoked project; `--scope all` harvests every project.
- Add `--backend claude` or `--backend codex` to spend the user's real budget
for model-driven optimization. A held-out gain is run-specific evidence, not
a guarantee of broader improvement; results depend on the tasks, model, and
checks.
- Scope defaults to the invoked project; `--scope all` harvests every Claude
project into the current run's configured targets.
- A real backend sends truncated transcript/task content to its provider. See
the data-boundary rules below before using one with sensitive sessions.
### Scheduling
@@ -63,25 +78,30 @@ Prefer the `/skillopt-sleep` command. Under the hood it calls the bundled runner
Installs a nightly cron entry. `unschedule --all` removes every managed entry.
## All CLI flags
## Common CLI flags
| Flag | Default | Description |
|------|---------|-------------|
| `--project PATH` | cwd | Project directory to evolve |
| `--scope all\|invoked` | invoked | Harvest scope |
| `--backend mock\|claude\|codex\|copilot` | mock | Replay backend (mock = no API spend) |
| `--backend mock\|claude\|codex\|copilot\|handoff\|azure_openai` | mock | Backend (mock = no provider calls) |
| `--model NAME` | backend default | Override the model used for replay |
| `--source claude\|codex\|auto` | claude | Transcript source |
| `--lookback-hours N` | 72 | Harvest window |
| `--max-sessions N` | unlimited | Cap harvested sessions |
| `--max-sessions N` | derived | Cap harvested sessions; defaults to 3 × max tasks (120 with current defaults) |
| `--max-tasks N` | 40 | Cap mined tasks |
| `--target-skill-path PATH` | auto | Explicit SKILL.md to evolve |
| `--target-skill-path PATH` | `~/.claude/skills/skillopt-sleep-learned/SKILL.md` | Explicit SKILL.md to evolve |
| `--tasks-file PATH` | — | Reviewed TaskRecord JSON (skip harvest) |
| `--progress` | off | Print phase progress to stderr |
| `--auto-adopt` | off | Auto-adopt if gate passes |
| `--edit-budget N` | 4 | Max bounded edits per night |
| `--preferences TEXT` | empty | Add house rules to the optimizer's reflection prior |
| `--json` | off | Machine-readable JSON output |
The CLI also has source/runtime path overrides (`--claude-home`, `--codex-home`,
and `--codex-path`) and action-specific flags. Use
`python -m skillopt_sleep <action> --help` as the authoritative surface.
## Config keys (`~/.skillopt-sleep/config.json`)
Beyond the CLI flags, advanced behavior is controlled via config:
@@ -99,28 +119,37 @@ The sleep cycle can consolidate both:
- **SKILL.md** — the managed skill file (bounded edits: add/delete/replace)
- **CLAUDE.md** — the project memory (same bounded edits)
Both are gated by the same held-out validation score. Set `evolve_memory: false` to consolidate only skills, or `evolve_skill: false` for only memory.
With the default gate enabled, both are evaluated by the same held-out score.
Set `evolve_memory: false` to consolidate only skills, or `evolve_skill: false`
for only memory.
## Hard rules
- **Never** hand-edit the user's `CLAUDE.md` / `SKILL.md` as part of this skill.
Only the `adopt` action changes live files, and it backs them up first.
Let the engine's explicit `adopt` or user-requested `--auto-adopt` path apply
the staging manifest and back up existing live files first.
- Harvest is read-only. `mock` replay has no side effects.
- Real backends send truncated transcript excerpts and derived tasks to the
selected provider for mining, replay, judging, and reflection. The Claude
transcript path is not guaranteed to remove every secret before those calls.
Review provider policy and session contents first. For sensitive data, use
`mock` or run `harvest --output <file>`, inspect/redact the JSON, set
`"reviewed": true`, and replay it with `--tasks-file`; real backends refuse an
unreviewed task file.
- Always show the user the **held-out baseline → candidate** score and the
exact proposed edits before suggesting adoption. Evidence before adoption.
- If asked whether it really helps, run
- If asked to demonstrate the mechanism without provider calls, run
`python -m skillopt_sleep.experiments.run_experiment --persona researcher --json`
— a deterministic demo that proves held-out lift and that the gate blocks
harmful edits.
— a deterministic synthetic demo of held-out lift and gate rejection. It
validates the mechanism, not effectiveness on the user's own tasks.
## Validate / demo
```bash
# deterministic proof (no API): held-out score rises, gate blocks regressions
# deterministic synthetic demo (no API): score rises and the gate blocks a regression
python -m skillopt_sleep.experiments.run_experiment --persona researcher --assert-improves
python -m skillopt_sleep.experiments.run_experiment --persona programmer --assert-improves
```
See the SkillOpt-Sleep guide section for recorded output and
`docs/superpowers/specs/2026-06-07-skillopt-sleep-claude-code-plugin-design.md`
for the full design.
See the [SkillOpt-Sleep documentation](https://github.com/microsoft/SkillOpt/tree/main/docs/sleep)
for recorded results, limitations, and the supported integration surface.
plugins/codex/README.md
+18 -8
View File
@@ -9,7 +9,8 @@ as the Claude Code plugin (`skillopt_sleep`), wrapped for Codex.
> [gbrain-evals](https://github.com/garrytan/gbrain-evals) `skillopt-v1`
> benchmark, a deliberately deficient skill goes **0.00 → 1.00** on a held-out
> set with the Codex backend (incl. the tool-use seed via a real tool loop).
> See [the SkillOpt-Sleep guide section](https://microsoft.github.io/SkillOpt/docs/guideline.html#sleep).
> See the recorded results and limitations in
> [`docs/sleep/RESULTS.md`](../../docs/sleep/RESULTS.md).
## What Codex supports (and what we use)
@@ -22,8 +23,8 @@ rules. The shared runner remains a plain shell entrypoint that the skill calls.
## Install
```bash
git clone <repo-url> SkillOpt-Sleep
cd SkillOpt-Sleep
git clone https://github.com/microsoft/SkillOpt.git
cd SkillOpt
bash plugins/codex/install.sh # installs the skill
export SKILLOPT_SLEEP_REPO="$(pwd)" # so the runner is found from anywhere
```
@@ -60,15 +61,19 @@ python -m skillopt_sleep run --project "$(pwd)" --source codex --backend codex \
`~/.codex/archived_sessions`. Use `--codex-home /path/to/.codex` to point at a
different Codex home, or `--source auto` to try Codex archives first and fall
back to Claude Code transcripts. Default backend is `mock` (no API spend).
`--backend codex` uses your Codex budget for real improvement. Bound live runs
`--backend codex` uses your Codex budget for model-driven optimization; an
accepted gain is task-dependent, not guaranteed. Bound live runs
with `--max-sessions` and `--max-tasks`; add `--progress` because Codex-backed
mining, replay, and reflection can be slow and otherwise quiet. Use
`--target-skill-path` to stage/adopt into a repo-scoped Codex skill such as
`.agents/skills/<name>/SKILL.md`; target runs over-sample mined tasks and
prefer tasks that match the target skill's path, headings, and content. All the
controllable knobs (`--gate on|off`, `--rollouts-k`, `--budget-tokens`,
`--preferences`, optimizer/target split) work identically — see
[the SkillOpt-Sleep guide section](https://microsoft.github.io/SkillOpt/docs/guideline.html#sleep).
prefer tasks that match the target skill's path, headings, and content. The
implemented main-CLI flags work the same across the shared integrations, and
`--preferences "..."` is available for house rules. Advanced keys such as
`gate_mode`, `dream_rollouts`, and `recall_k` belong in the Sleep config; the
nightly CLI does not expose `--gate`, `--rollouts-k`, token/time-budget, or
optimizer/target-split flags. See the
[shared CLI reference](../README.md#supported-cli-surface).
For privacy-sensitive projects, split the run into reviewable steps:
@@ -86,6 +91,11 @@ Inspect/redact the JSON and set `"reviewed": true` before using a real backend.
`--tasks-file` skips archive harvest/mining and replays only the reviewed JSON
tasks; real backends refuse task files still marked `"reviewed": false`.
This review step matters even though the Codex transcript converter removes
known secret-shaped strings: pattern-based redaction is not a guarantee. A real
backend sends truncated transcript/task content to the selected provider, while
`--backend mock` makes no provider calls.
## Notes / status
- Codex's `exec` runs shell, so the real-tool-loop replay (e.g. the
plugins/codex/skills/skillopt-sleep/SKILL.md
+71 -30
View File
@@ -1,16 +1,23 @@
---
name: skillopt-sleep
description: "Use when the user wants Codex to self-improve from past usage, asks about a nightly/offline 'sleep' or 'dream' cycle, wants Codex to review past sessions, learn preferences, consolidate memory/skills, run dry-run/run/adopt/status for SkillOpt-Sleep, or schedule offline self-optimization. Drives the skillopt_sleep engine: harvest past sessions -> mine recurring tasks -> replay offline -> consolidate validated memory + skills behind a held-out gate."
description: "Use when the user wants Codex to self-improve from past usage, asks about a nightly/offline 'sleep' or 'dream' cycle, wants Codex to review past sessions, learn preferences, consolidate memory/skills, run dry-run/run/adopt/status for SkillOpt-Sleep, or schedule background self-optimization. Drives the skillopt_sleep engine: harvest past sessions -> mine recurring tasks -> replay through a selected backend -> consolidate validated memory + skills behind a held-out gate."
---
# SkillOpt-Sleep: offline self-evolution for a local Codex agent
# SkillOpt-Sleep: usage-driven self-evolution for a local Codex agent
SkillOpt-Sleep gives the user's Codex agent a sleep cycle. While the user is
offline or on demand, it reviews past local sessions, re-runs recurring tasks
on the user's own budget, and consolidates what it learns into memory and
skills. It keeps only changes that pass a held-out validation gate, and live
files change only after the user explicitly adopts a staged proposal. There is
no model-weight training.
SkillOpt-Sleep gives the user's Codex agent a sleep cycle. On demand or on a
nightly schedule, it reviews past local sessions, re-runs recurring tasks
through the selected backend, and proposes changes to a configured skill and to
the project's `CLAUDE.md`. With the default validation gate enabled, it keeps
only changes that improve a held-out score. Live files change only through
explicit adoption or a user-requested `--auto-adopt`. There is no model-weight
training.
The current shared engine does **not** write `AGENTS.md`. For a Codex-visible
result, always select a Codex skill explicitly with `--target-skill-path` (for
example `.agents/skills/<name>/SKILL.md`). If project `CLAUDE.md` is not a
desired secondary target, set `"evolve_memory": false` in
`~/.skillopt-sleep/config.json` before running.
## When to use
@@ -28,13 +35,15 @@ Trigger when the user wants any of:
configuration and normalize them into session digests.
2. **Mine** - turn digests into recurring `TaskRecord`s with outcomes and
checkable references where possible.
3. **Replay** - re-run mined tasks offline under the current skill and memory.
3. **Replay** - re-run mined tasks through the selected backend under the
current skill and memory.
4. **Consolidate** - reflect on failures and propose bounded edits.
5. **Gate** - accept edits only when the held-out validation score improves.
5. **Gate** - with the default gate enabled, accept edits only when the held-out
validation score improves.
6. **Stage** - write the proposal under
`<project>/.skillopt-sleep/staging/<date>/`; nothing live changes.
7. **Adopt** - only after explicit user approval, copy staged files over live
files with backups.
7. **Adopt** - explicitly, or through user-requested auto-adopt, copy staged
files over live files with backups for existing targets.
## How to drive it
@@ -43,32 +52,47 @@ finds the engine and a Python >= 3.10 automatically.
```bash
# point at the repo if it isn't auto-detected from CWD:
export SKILLOPT_SLEEP_REPO=/path/to/SkillOpt-Sleep
export SKILLOPT_SLEEP_REPO=/path/to/SkillOpt
TARGET_SKILL=.agents/skills/example/SKILL.md
bash "$SKILLOPT_SLEEP_REPO/plugins/run-sleep.sh" status --project "$(pwd)"
bash "$SKILLOPT_SLEEP_REPO/plugins/run-sleep.sh" harvest --project "$(pwd)"
bash "$SKILLOPT_SLEEP_REPO/plugins/run-sleep.sh" dry-run --project "$(pwd)" --backend mock
bash "$SKILLOPT_SLEEP_REPO/plugins/run-sleep.sh" run --project "$(pwd)" --backend codex
bash "$SKILLOPT_SLEEP_REPO/plugins/run-sleep.sh" run --project "$(pwd)" --source codex # harvest from Codex Desktop
bash "$SKILLOPT_SLEEP_REPO/plugins/run-sleep.sh" harvest --project "$(pwd)" \
--source codex --target-skill-path "$TARGET_SKILL"
bash "$SKILLOPT_SLEEP_REPO/plugins/run-sleep.sh" dry-run --project "$(pwd)" \
--source codex --target-skill-path "$TARGET_SKILL" --backend mock
bash "$SKILLOPT_SLEEP_REPO/plugins/run-sleep.sh" run --project "$(pwd)" \
--source codex --target-skill-path "$TARGET_SKILL" --backend codex \
--max-sessions 5 --max-tasks 3 --progress
bash "$SKILLOPT_SLEEP_REPO/plugins/run-sleep.sh" adopt --project "$(pwd)"
```
Actions are `status`, `harvest`, `dry-run`, `run`, `adopt`, `schedule`, and `unschedule`.
- Default backend is `mock`, which is deterministic and spends no API budget.
- `--backend codex` uses the user's Codex budget for real improvement.
- `--backend codex` uses the user's Codex budget for model-driven optimization.
An accepted held-out gain is run-specific evidence, not a guarantee of
broader improvement; results depend on the tasks, model, and checks.
- `--source codex` reads Codex Desktop archived sessions from `~/.codex/archived_sessions`;
use `--codex-home /path/to/.codex` if the archive lives elsewhere.
- `--target-skill-path` is required for a Codex skill target. Without it, the
shared default is a Claude-managed skill under `~/.claude/skills/`, not an
`.agents` skill.
- Keep `dry-run --backend mock` as the first smoke check unless the user
explicitly asked for a real optimization run.
### Scheduling
```bash
bash "$SKILLOPT_SLEEP_REPO/plugins/run-sleep.sh" schedule --project "$(pwd)" --hour 3 --minute 17
bash "$SKILLOPT_SLEEP_REPO/plugins/run-sleep.sh" schedule --project "$(pwd)" \
--backend codex --hour 3 --minute 17
bash "$SKILLOPT_SLEEP_REPO/plugins/run-sleep.sh" unschedule --project "$(pwd)"
```
Installs a nightly cron entry. `unschedule --all` removes every managed entry.
The scheduler persists the project, backend, time, and optional auto-adopt flag;
it does not persist `--source` or `--target-skill-path` from this command. Before
scheduling a Codex-targeted run, set `"transcript_source": "codex"` and an
absolute `"target_skill_path"` in `~/.skillopt-sleep/config.json`. On systems
without `crontab`, `schedule` prints a line for manual installation.
`unschedule --all` removes every managed entry.
### All backends
@@ -76,6 +100,8 @@ Installs a nightly cron entry. `unschedule --all` removes every managed entry.
- `--backend claude` — uses the Claude CLI
- `--backend codex` — uses the Codex CLI
- `--backend copilot` — uses the GitHub Copilot CLI
- `--backend handoff` — emits prompt/answer files for an interactive session
- `--backend azure_openai` — uses the configured Azure OpenAI endpoint
### Additional flags
@@ -96,7 +122,10 @@ Installs a nightly cron entry. `unschedule --all` removes every managed entry.
### Memory consolidation
The sleep cycle consolidates both **memory** (AGENTS.md / CLAUDE.md) and **skills** (SKILL.md) by default. Each is independently toggleable via `evolve_memory` / `evolve_skill` config keys. Both are gated by the same held-out validation score.
The shared sleep cycle consolidates project **memory** (`CLAUDE.md`) and the
selected **skill** (`SKILL.md`) by default. It does not update `AGENTS.md`.
Each target is independently toggleable through `evolve_memory` /
`evolve_skill`, and both are gated by the same held-out validation score.
## Steps
@@ -104,16 +133,23 @@ The sleep cycle consolidates both **memory** (AGENTS.md / CLAUDE.md) and **skill
2. For `dry-run` and `run`, report the held-out baseline -> candidate score,
gate action, task count, session count, and exact proposed edits.
3. If a staging directory is printed, read `report.md` before summarizing.
4. `run` only stages a proposal; nothing live changes until `adopt`.
5. Offer adoption only after the user has reviewed the staged proposal.
6. Never hand-edit the user's `AGENTS.md`, memory, or skills as a substitute
for `adopt`; adoption is the safety boundary and writes backups first.
4. `run` stages by default; if `--auto-adopt` was explicitly supplied, report
the paths it updated instead of claiming nothing changed.
5. Offer adoption only after the user has reviewed a still-staged proposal.
6. Never hand-edit the configured `CLAUDE.md` or target skill as a substitute
for the engine's adopt path; adoption is the safety boundary and backs up
existing targets first.
## Hard rules
- Harvest is read-only. Do not edit archived sessions or raw transcripts.
- Keep raw secrets, credentials, private user data, and unsanitized transcript
contents out of messages, logs, generated artifacts, and commits.
- Codex transcript harvesting removes known secret-shaped strings, developer
instructions, and raw tool payloads, but pattern-based redaction is not a
guarantee. A real backend still sends truncated transcript/task content to
its provider. Review sensitive sessions and provider policy first; prefer a
reviewed `--tasks-file` workflow when the data boundary matters.
- Keep raw secrets, credentials, private user data, and transcript contents out
of messages, logs, generated artifacts, and commits.
- Show validation evidence before recommending adoption.
- Treat generated edits as proposals, not as source of truth.
- Do not rely on deprecated custom prompts or `/sleep` slash commands for this
@@ -122,11 +158,16 @@ The sleep cycle consolidates both **memory** (AGENTS.md / CLAUDE.md) and **skill
## Validate
```bash
python -m skillopt_sleep dry-run --project "$(pwd)" --backend mock --json
python -m skillopt_sleep dry-run --project "$(pwd)" --source codex \
--target-skill-path .agents/skills/example/SKILL.md --backend mock --json
python -m skillopt_sleep.experiments.run_gbrain --backend codex \
--seeds brief-writer --data-root /path/to/gbrain-evals/eval/data/skillopt-v1 \
--nights 2 --limit-replay 3 --limit-holdout 3
```
A deficient skill goes 0.00 -> 1.00 on a held-out set; the optimizer's edits
are gated on real-task performance.
In the recorded `brief-writer` gbrain run, the deliberately deficient fixture
went 0.00 -> 1.00 on that run's held-out set. Treat this as reproducible
benchmark evidence for that configuration, not a guarantee for other skills,
tasks, or models; see the
[recorded results](https://github.com/microsoft/SkillOpt/blob/main/docs/sleep/RESULTS.md)
for context and limitations.
plugins/copilot/README.md
+17 -9
View File
@@ -27,8 +27,8 @@ Requires Python ≥ 3.10. No third-party packages — the server is pure stdlib.
"mcpServers": {
"skillopt-sleep": {
"command": "python3",
"args": ["/abs/path/SkillOpt-Sleep/plugins/copilot/mcp_server.py"],
"env": { "SKILLOPT_SLEEP_REPO": "/abs/path/SkillOpt-Sleep" }
"args": ["/abs/path/SkillOpt/plugins/copilot/mcp_server.py"],
"env": { "SKILLOPT_SLEEP_REPO": "/abs/path/SkillOpt" }
}
}
}
@@ -42,14 +42,21 @@ Requires Python ≥ 3.10. No third-party packages — the server is pure stdlib.
## Use
Ask Copilot things like *"run the sleep cycle"*, *"what did the last sleep
propose?"*, *"adopt the staged sleep proposal"*. Copilot calls the MCP tools:
`sleep_status`, `sleep_dry_run`, `sleep_run`, `sleep_adopt`, `sleep_harvest`.
propose?"*, *"adopt the staged sleep proposal"*. The server exposes seven MCP
tools: `sleep_status`, `sleep_dry_run`, `sleep_run`, `sleep_adopt`,
`sleep_harvest`, `sleep_schedule`, and `sleep_unschedule`.
Each tool takes optional `project`, `backend` (`mock`/`claude`/`codex`/`copilot`), and
`scope` arguments. Default backend is `mock` (no API spend). The `copilot`
backend drives the GitHub Copilot CLI (`copilot -p ... --output-format json`)
and requires the `copilot` CLI to be installed and authenticated.
Harvesting is local and read-only, and the default `mock` backend makes no
provider calls. A real backend sends truncated transcript excerpts and derived
tasks to the selected provider. Outbound prompts are not currently guaranteed
to be secret-free; review sensitive data and provider policy first. See the
[shared data-boundary guidance](../README.md#data-boundary).
For speed, the `copilot` backend runs each call against an isolated
`COPILOT_HOME` with built-in MCP servers and custom instructions disabled, so
your user MCP servers (including this project's own) are not spawned per call
@@ -65,12 +72,13 @@ printf '%s\n' \
'{"jsonrpc":"2.0","id":2,"method":"tools/list"}' \
| SKILLOPT_SLEEP_REPO="$(pwd)" python3 plugins/copilot/mcp_server.py
```
You should see the server info and the five `sleep_*` tools.
You should see the server info and all seven `sleep_*` tools.
## Notes / status
- MCP is the stable, official Copilot extension surface, so this is the most
portable of the three integrations (one server → CLI + IDE).
- The engine and all its controls (gate on/off, multi-rollout, budget,
preferences, optimizer/target split) are identical across platforms — see
[the SkillOpt-Sleep guide section](https://microsoft.github.io/SkillOpt/docs/guideline.html#sleep).
portable shared-engine integration (one server → CLI + IDE).
- The MCP schema exposes the main CLI's implemented controls, including task and
session caps, target-skill selection, scheduling, and staged adoption. It does
not add experiment-only gate, rollout, token/time-budget, or optimizer/target
split flags. See the [shared CLI reference](../README.md#supported-cli-surface).
plugins/copilot/copilot-instructions.snippet.md
+22 -9
View File
@@ -8,26 +8,29 @@ automatically as ambient guidance.)
This project has SkillOpt-Sleep available via an MCP server (`skillopt-sleep`).
It gives the agent a nightly "sleep cycle": it reviews past sessions, replays
recurring tasks offline, and consolidates validated memory + skills behind a
held-out gate.
recurring tasks through a selected backend, and stages validation-gated changes
to project `CLAUDE.md` and a configured `SKILL.md`.
When the user asks to "run the sleep cycle", "review my past sessions", "learn
my preferences", or "make the agent improve from past usage", use the MCP tools:
- `sleep_status` — what's happened + the latest staged proposal
- `sleep_dry_run` — safe preview, stages nothing
- `sleep_run` — full cycle, stages a reviewed proposal (nothing live changes)
- `sleep_adopt` — apply the staged proposal (backs up first)
- `sleep_dry_run` — no-staging preview; a real backend still makes provider calls
- `sleep_run` — full cycle, stages a validation-gated proposal by default;
explicit `auto_adopt` may update live files
- `sleep_adopt` — apply the staged proposal (backs up an existing live file first)
- `sleep_harvest` — list mined recurring tasks
- `sleep_schedule` — install a nightly cron entry (set `hour`/`minute`)
- `sleep_unschedule` — remove the nightly cron entry
### Key parameters (pass as MCP tool arguments)
- `backend` — `mock` (default, free), `claude`, `codex`, or `copilot`
- `backend` — `mock` (default, no provider calls), `claude`, `codex`, or `copilot`
- `source` — `claude`, `codex`, or `auto` (where to read transcripts)
- `target_skill_path` — explicit SKILL.md to evolve
- `tasks_file` — pre-built TaskRecord JSON (skip harvest)
- `target_skill_path` — explicit SKILL.md to evolve; use this for a skill that
the current agent actually loads
- `tasks_file` — reviewed TaskRecord JSON (skip harvest); real backends require
its metadata to contain `"reviewed": true`
- `max_tasks` / `max_sessions` — cap workload
- `auto_adopt` — auto-adopt if the gate passes
- `json` — machine-readable output for programmatic use
@@ -40,4 +43,14 @@ my preferences", or "make the agent improve from past usage", use the MCP tools:
Always show the user the held-out baseline → candidate score and the proposed
edits before suggesting `sleep_adopt`. Never hand-edit the user's memory/skill
files; only `sleep_adopt` does that, with a backup.
files; use `sleep_adopt` (or an explicitly requested `auto_adopt`) so the engine
applies its staging manifest and backup behavior.
Harvesting is local and read-only, and `backend: "mock"` makes no provider
calls. A real backend sends truncated transcript excerpts and derived tasks to
the selected provider; outbound prompts are not guaranteed to be secret-free.
Review sensitive data and provider policy before selecting a real backend.
`sleep_schedule` persists only the project, backend, time, and optional
auto-adopt setting. Put a non-default transcript source or target skill in
`~/.skillopt-sleep/config.json` before scheduling it.
plugins/copilot/skillopt/README.md
+14 -5
View File
@@ -66,11 +66,20 @@ skill"*, or *"evaluate this skill on the dataset"*. Copilot calls the MCP tools:
| `skillopt_train` | `config` | Runs a reflective optimization loop. Long-running; spends budget. |
| `skillopt_eval` | `config`, `skill` | Evaluates one skill markdown file; no training. |
Common optional args (both train and eval): `env`, `backend`,
`optimizer_model`, `target_model`, `out_root`, `cfg_options` (space-separated
`KEY=VALUE` YAML overrides), and `extra_args` (raw passthrough flags for the
underlying script). `skillopt_train` also accepts `num_epochs`, `batch_size`,
`seed`, and `use_gate`.
Common optional args (both train and eval): `env`, `backend`,
`optimizer_model`, `target_model`, `out_root`, `cfg_options` (space-separated
`section.key=value` YAML overrides), and `extra_args` (raw passthrough flags
for the underlying script). `skillopt_train` also accepts `num_epochs`,
`batch_size`, `seed`, and `use_gate`. `use_gate` defaults to `true`; setting it
to `false` still records validation scores but force-accepts every candidate,
which changes the optimization semantics.
The MCP schema's `backend` argument follows the underlying script's
`--backend` choices. Role-specific and generic OpenAI-compatible backends can
be selected through config or `extra_args` (for example,
`--optimizer_backend openai_compatible --target_backend openai_compatible`);
see the repository's [backend guide](../../../docs/guide/new-backend.md) for
the required environment variables.
Runs can be very long. The server's subprocess timeout defaults to 6 hours;
override it with the `SKILLOPT_RUN_TIMEOUT` environment variable (seconds).
plugins/copilot/skillopt/copilot-instructions.snippet.md
+34 -33
View File
@@ -1,33 +1,34 @@
<!--
Copy this block into your repo's .github/copilot-instructions.md so Copilot
knows the SkillOpt research-engine tools exist. (Copilot reads
copilot-instructions.md automatically as ambient guidance.)
-->
## SkillOpt (research skill-optimization engine)
This repo exposes the core **SkillOpt** training/eval engine via an MCP server
(`skillopt`). SkillOpt is validation-gated, text-space skill optimization: it
reflects on rollouts, makes bounded edits to a skill, and keeps a change only
if it improves a held-out validation set.
When the user asks to "optimize a skill", "train on <benchmark>", "run
SkillOpt", "evaluate this skill", or "what configs can I run", use the MCP
tools:
- `skillopt_list_configs` — list the benchmark YAML configs you can pass as `config`
- `skillopt_train` — run a reflective skill-optimization loop on a config (long-running; spends API/compute budget)
- `skillopt_eval` — evaluate a single skill markdown file on a dataset (no training)
Guidance:
- Always run `skillopt_list_configs` first if you don't already know a valid `config` path.
- `skillopt_train` and `skillopt_eval` are long-running and consume the user's
model backend/budget — confirm the `config`, `backend`, and model choices
with the user before launching, and surface the held-out gate result when the
run finishes.
- For one-off YAML overrides use `cfg_options` (e.g. `seed=123 batch_size=40`);
for any other underlying flag use `extra_args`.
This is distinct from the **SkillOpt-Sleep** MCP server (`skillopt-sleep`,
`sleep_*` tools), which evolves a local coding agent from past sessions rather
than running the research benchmarks.
<!--
Copy this block into your repo's .github/copilot-instructions.md so Copilot
knows the SkillOpt research-engine tools exist. (Copilot reads
copilot-instructions.md automatically as ambient guidance.)
-->
## SkillOpt (research skill-optimization engine)
This repo exposes the core **SkillOpt** training/eval engine via an MCP server
(`skillopt`). SkillOpt is validation-gated, text-space skill optimization: it
reflects on rollouts, makes bounded edits to a skill, and keeps a change only
if it improves a held-out validation set.
When the user asks to "optimize a skill", "train on <benchmark>", "run
SkillOpt", "evaluate this skill", or "what configs can I run", use the MCP
tools:
- `skillopt_list_configs` — list the benchmark YAML configs you can pass as `config`
- `skillopt_train` — run a reflective skill-optimization loop on a config (long-running; spends API/compute budget)
- `skillopt_eval` — evaluate a single skill markdown file on a dataset (no training)
Guidance:
- Always run `skillopt_list_configs` first if you don't already know a valid `config` path.
- `skillopt_train` and `skillopt_eval` are long-running and consume the user's
model backend/budget — confirm the `config`, `backend`, and model choices
with the user before launching. Surface the held-out gate result for training,
or the evaluation score and output directory for eval-only runs.
- For one-off YAML overrides use dotted `cfg_options` for structured configs
(e.g. `train.seed=123 train.batch_size=40`);
for any other underlying flag use `extra_args`.
This is distinct from the **SkillOpt-Sleep** MCP server (`skillopt-sleep`,
`sleep_*` tools), which evolves a local coding agent from past sessions rather
than running the research benchmarks.
plugins/devin/README.md
+16 -4
View File
@@ -54,13 +54,25 @@ Requires Python ≥ 3.10. No third-party packages — the server is pure stdlib.
| Tool | What it does |
|---|---|
| `sleep_status` | nights run so far + latest staged proposal |
| `sleep_dry_run` | preview cycle — no staging, no changes |
| `sleep_dry_run` | preview cycle — no staging; a real backend still makes provider calls |
| `sleep_run` | full cycle; stages a proposal for review |
| `sleep_adopt` | apply the staged proposal; syncs skill to the workspace |
| `sleep_harvest` | debug: list the recurring tasks mined |
| `sleep_schedule` | install a nightly cron entry (`--hour` / `--minute`) |
| `sleep_unschedule` | remove the nightly cron entry |
Default backend is `mock` (no API spend); `--backend claude|codex` uses your own
budget. Same engine and `sleep_*` interface as the other plugins — all call
`python -m skillopt_sleep`.
Default backend is `mock` (no API spend); the `claude`, `codex`, and `copilot`
backends use the corresponding authenticated CLI and budget. The seven tools
call the same `python -m skillopt_sleep` actions as the other shared-engine
integrations.
## Data boundary
The Devin harvester reads local ATIF transcripts, agentmemory, and skill files
and converts them into the engine's session format. The `mock` backend keeps
that workflow local. A real backend sends truncated excerpts and derived tasks
to the selected provider for mining, replay, judging, and reflection. The
conversion step is not a guarantee that outbound prompts contain no secrets;
review sensitive sources and provider policy before enabling a real backend.
See the [shared data-boundary guidance](../README.md#data-boundary) and
[implemented CLI reference](../README.md#supported-cli-surface).
plugins/devin/devin-rules.snippet.md
+28 -9
View File
@@ -3,17 +3,36 @@
You have access to a nightly self-evolution cycle via the `skillopt-sleep` MCP
server. Use these tools to improve your long-term skills over time:
- **`sleep_status`** — how many nights have run + the latest staged proposal
- **`sleep_dry_run`** — preview a cycle without changing anything
- **`sleep_run`** — run a full cycle; stages a proposal for review
- **`sleep_adopt`** — apply the staged proposal to `.devin/skills/skillopt-sleep-learned/SKILL.md`
- **`sleep_status`** — refresh the converted local cache, then show how many
nights have run and the latest staged proposal
- **`sleep_dry_run`** — refresh the converted local cache and preview a cycle
without engine staging/adoption; a real backend still makes provider calls
- **`sleep_run`** — run a full cycle; stages a proposal by default, while an
explicit `auto_adopt` may also update live files
- **`sleep_adopt`** — apply the staged proposal, then sync the managed skill to
`.devin/skills/skillopt-sleep-learned/SKILL.md` when `project` is the Devin
workspace and that workspace already contains a `.devin/` directory
- **`sleep_harvest`** — debug: list the recurring tasks mined from recent sessions
- **`sleep_schedule`** / **`sleep_unschedule`** — install/remove a nightly cron run
- **`sleep_schedule`** / **`sleep_unschedule`** — low-level shared-engine cron
controls; the current scheduled command does not run Devin's conversion step,
so do not use it as an unattended Devin-harvest workflow
When a user asks about the sleep cycle, skill evolution, or improving your
long-term memory, prefer calling these tools over explaining the concept.
When a user asks about the sleep cycle or skill evolution, prefer calling these
tools over explaining the concept.
Default backend is `mock` (no API spend). Pass `backend: "claude"` or
`backend: "codex"` with your own API key for real LLM-driven optimization.
Always pass the absolute Devin workspace as `project`, especially for
`sleep_adopt`. Default backend is `mock` (no provider calls). The `claude`,
`codex`, and `copilot` backend values use the corresponding installed and
authenticated CLI; they do not require this plugin to implement a separate
API-key flow.
The Devin conversion and mock workflow stay local. A real backend sends
truncated transcript excerpts and derived tasks to the selected provider for
mining, replay, judging, and reflection; conversion is not a guarantee that
outbound prompts contain no secrets. Review local sources and provider policy
before selecting a real backend.
For a reviewed task file, pass `tasks_file`; before using it with a real backend,
inspect/redact it and ensure its metadata contains `"reviewed": true`.
Place this file at `.devin/rules/skillopt-sleep.md` in your workspace.
plugins/openclaw/README.md
+86 -83
View File
@@ -1,112 +1,115 @@
# OpenClaw Plugin for SkillOpt-Sleep
# OpenClaw reference adaptation for SkillOpt-Sleep
Thin shell for running [SkillOpt-Sleep](https://github.com/microsoft/SkillOpt) on [OpenClaw](https://github.com/openclaw/openclaw).
This directory is a contributed reference for connecting
[SkillOpt-Sleep](https://github.com/microsoft/SkillOpt) to
[OpenClaw](https://github.com/openclaw/openclaw) with a custom DeepSeek/Ollama
backend.
## What it does
> **Reference status.** This is not one of the shared, plug-and-play
> `skillopt_sleep` wrappers. Several scripts and the sample config contain
> environment-specific absolute paths and assumptions from the original setup,
> and the contributed wrapper has unresolved Python 3.10 syntax and backend
> factory-signature gaps. The current checkout is not directly runnable; treat
> it as porting source material, not an installation.
Adds a nightly "sleep cycle" to any OpenClaw agent. The cycle:
## Included components
1. **Harvests** recent session transcripts from `~/.openclaw/agents/<name>/sessions/*.jsonl`
2. **Mines** recurring task patterns using the optimizer LLM
3. **Replays** each pattern with the current `SKILL.md` (baseline) and a candidate `SKILL.md` (with proposed edits)
4. **Gates** the candidate against the held-out score (rejects regressions)
5. **Stages** the accepted proposal in `~/.skillopt-sleep/staging/<night>/`
6. Leaves adoption to the operator (Ethan)
| File | Purpose |
|---|---|
| `run_sleep.py` | custom cycle entry point |
| `skillopt_sleep_openclaw.py` | DeepSeek Chat Completions backend plus local Ollama embeddings |
| `run_sleep_cron.sh` | category-oriented cron wrapper |
| `slash_sleep.py` | experimental `/sleep` command helper |
| `config.json` | example engine configuration |
| `SKILL.md` | OpenClaw skill manifest |
| `tests/*.json` | example task sets for research, DevOps, and wiki workflows |
Nothing live changes until you adopt. Every adopt backs up first.
The adaptation imports the shared engine but registers its own backend and
maintains its own wrapper behavior. Changes to the shared CLI documentation do
not automatically make every option available through these custom scripts.
## Install
## Intended cycle
The plugin is a thin wrapper around the engine at `~/.openclaw/workspace/SkillOpt/skillopt_sleep/`:
```bash
# 1. Clone the engine (one-time)
cd ~/.openclaw/workspace
git clone https://github.com/microsoft/SkillOpt.git
# 2. Install the OpenClaw skill (this folder)
ln -s /path/to/openclaw ~/.openclaw/workspace/skills/skillopt-sleep
# 3. Configure
cp ~/.openclaw/workspace/skills/skillopt-sleep/config.json ~/.skillopt-sleep/config.json
$EDITOR ~/.skillopt-sleep/config.json
# Set backend = "openclaw-deepseek"
# Set model = "deepseek-v4-pro" (or "deepseek-v4-flash" for budget)
# 4. Set API key
echo 'export DEEPSEEK_API_KEY="sk-..."' >> ~/.openclaw/.env
# 5. Add the nightly cron
(crontab -l 2>/dev/null; echo "0 3 * * * cd ~/.openclaw/workspace/skills/skillopt-sleep && bash run_sleep_cron.sh >> ~/.skillopt-sleep/nightly.log 2>&1") | crontab -
```text
harvest supported session data or load a task file
→ replay with the current skill
→ propose bounded edits
→ validate the candidate on held-out tasks
→ stage a proposal for operator review
```
## Use
The intended safety boundary is manual adoption: review the generated report and
staged files before changing a live skill.
### Manual trigger
## Adapt before use
```bash
# Run one cycle now
python3 ~/.openclaw/workspace/skills/skillopt-sleep/run_sleep.py
1. Clone SkillOpt into a location you control:
# Dry run (report only)
python3 ~/.openclaw/workspace/skills/skillopt-sleep/run_sleep.py --dry-run
```bash
git clone https://github.com/microsoft/SkillOpt.git
cd SkillOpt/plugins/openclaw
```
# One category only
python3 ~/.openclaw/workspace/skills/skillopt-sleep/run_sleep.py --tasks tests/research-cron-tasks.json
```
2. Inspect and replace the sample absolute paths in `run_sleep.py`,
`slash_sleep.py`, `run_sleep_cron.sh`, and `config.json`. Confirm the engine
checkout, OpenClaw workspace, state directory, skill directory, and task-file
paths all point to isolated test locations.
### Slash command
3. Review `config.json`. In particular, do not assume that values such as
`max_tokens_per_night` or `replay_mode` are enforced by this custom wrapper
merely because they appear in the example config.
```bash
# In any OpenClaw session
/sleep status
/sleep run
/sleep run research-cron
/sleep dry-run
/sleep adopt # adopt most recent accepted proposal
/sleep reject # discard most recent
/sleep cost
```
4. Supply credentials through your normal secret-management mechanism. Do not
commit a DeepSeek key or place it in a world-readable file.
## Architecture
5. Resolve every known porting gap listed in [`SKILL.md`](SKILL.md), add
isolated tests for your adapted backend, and verify that `--help` imports
cleanly on Python 3.10+. Only then start with a dry run and one reviewed
task file. The target command should be shaped like:
```
plugins/openclaw/
├── README.md # this file
├── run_sleep_cron.sh # wrapper for cron invocation
├── run_sleep.py # main entry point
├── slash_sleep.py # /sleep command implementation
├── skillopt_sleep_openclaw.py # DeepSeek + Ollama backend
├── config.json # engine config
├── SKILL.md # OpenClaw skill manifest
└── tests/ # held-out test sets
├── research-cron-tasks.json
├── devops-tasks.json
└── wiki-tasks.json
```
```bash
cd /path/to/SkillOpt/plugins/openclaw
python3 run_sleep.py --config /path/to/reviewed-config.json \
--tasks tests/research-cron-tasks.json --dry-run
```
The OpenClaw shell is one engine (skillopt_sleep/) + one backend (DeepSeek/Ollama) + four thin wrappers (cron, slash, skill, tests).
6. Inspect the report, paths, network destinations, and proposed edits before
considering a non-dry run or scheduling.
## Why this matters for OpenClaw
## Data boundary
OpenClaw currently has no built-in "self-evolving skills" mechanism. The community has:
The custom `openclaw-deepseek` backend sends task, skill, response, rubric, and
reflection content to the configured DeepSeek endpoint. Its embedding helper can
send truncated text to the configured local Ollama service. Do not assume these
outbound prompts have been fully redacted; inspect transcript/task inputs and the
provider's retention policy before using real data.
- **Manual skills** — Ethan writes them
- **LLM-generated skills** — one-shot, no validation
- **Self-revision** — unbounded, no quality bar
Use HTTPS for a remote DeepSeek-compatible endpoint. Keep any plaintext Ollama
endpoint on a trusted loopback interface. For a network-free engine smoke test,
use the shared SkillOpt-Sleep CLI with `--backend mock` rather than assuming this
custom wrapper is isolated.
SkillOpt-Sleep adds a 4th option: **validated self-evolution**. The skill is the training target, the engine is the optimizer, the gate is the quality bar, the operator is the human-in-the-loop.
## Scheduling
## Validation
`run_sleep_cron.sh` and the scheduling helpers are examples, not portable
installers. Adapt their paths, create log directories, verify their environment,
and run the exact command manually before adding a cron entry. Scheduled runs
must preserve the same manual-adoption and credential boundaries as interactive
runs.
Validated on the public [gbrain-evals](https://github.com/garrytan/gbrain-evals) `skillopt-v1` benchmark with real Claude and Codex (deficient skills 0.00 → 1.00 on held-out, all 4 seeds).
## Validation scope
End-to-end test on our own 14-task held-out set: pipeline runs, gate correctly rejects non-improvements, staging artifacts land in `~/.skillopt-sleep/staging/<night>/`.
The bundled JSON files are example held-out task sets, not a universal OpenClaw
benchmark. Provider cost and quality depend on the selected model, task content,
number of calls, and pricing at run time; this reference does not promise a fixed
nightly cost. Validate the adapted workflow in an isolated workspace before
using it on live skills.
## Cost
Measured: ~$0.02/night with `deepseek-v4-pro` at 12 tasks/night. ~$0.59/month, $7.18/year.
For the supported shared-engine CLI and its current flags, see the
[integration reference](../README.md#supported-cli-surface). For measured
SkillOpt-Sleep results and limitations, see
[`docs/sleep/RESULTS.md`](../../docs/sleep/RESULTS.md).
## License
MIT (same as SkillOpt core).
MIT, consistent with SkillOpt core.
plugins/openclaw/SKILL.md
+81 -107
View File
@@ -1,129 +1,103 @@
---
name: skillopt-sleep
description: Validate and refine agent skills through nightly sleep cycles with held-out gates. Wraps Microsoft's SkillOpt-Sleep engine for the OpenClaw/DeepSeek stack.
description: Reference-only OpenClaw adaptation of SkillOpt-Sleep. Use it to study or port the contributed DeepSeek wrapper, not as a ready-to-run installation.
---
# skillopt-sleep — OpenClaw Adaptation of Microsoft SkillOpt-Sleep
# SkillOpt-Sleep OpenClaw reference adaptation
A nightly self-improvement loop that reads our session transcripts, mines recurring workflow patterns, replays them with proposed skill edits, and gates the proposals against a held-out test set. Only improvements that beat baseline are staged for human adoption.
This directory is a contributed **reference**, not a supported, plug-and-play
OpenClaw integration. It illustrates one way to connect the shared
`skillopt_sleep` cycle to a custom DeepSeek Chat Completions backend and a set of
environment-specific task fixtures.
## When To Use
Do not run or schedule the files unchanged. Several scripts and the sample
configuration preserve assumptions from the contributor's original machine,
and parts of the wrapper have not yet been ported to the current shared-engine
interfaces. Start with the directory's [README.md](README.md), which is the
authoritative status and adaptation guide.
- After Hermes's Weekly Skill Review (or as its replacement)
- When a skill is being used 10+ times/week and could be tighter
- Before promoting a new skill from `skill-proposals/` to `skills/`
- When a skill regresses in observed quality
## What is included
## What It Does (One Cycle)
- `skillopt_sleep_openclaw.py` — a contributed DeepSeek backend prototype. It
also contains an Ollama embedding helper, but that helper is not wired into
the current shared sleep cycle.
- `run_sleep.py` — a custom cycle wrapper with environment-specific paths and a
backend-registration shim.
- `slash_sleep.py` — an experimental command helper written for an older
staging-manifest shape.
- `run_sleep_cron.sh` — a machine-specific category runner, not a portable cron
installer.
- `config.json` — a sample configuration, not a set of guaranteed or enforced
runtime limits.
- `tests/*.json` — example task fixtures from one environment, not a universal
OpenClaw benchmark.
```
harvest session transcripts -> mine recurring task patterns
-> replay each pattern (current skill vs proposed)
-> GATE: must improve held-out score
-> stage proposal
-> Ethan adopts (manual)
```
## Known porting gaps
Nothing live changes until Ethan adopts. Every adopt backs up first.
Before treating this as an integration, a maintainer must at least:
## Architecture
1. Replace every absolute workspace, repository, state, skill, log, and task
path with explicit user configuration.
2. Update the custom backend factory to the current `get_backend` call contract,
including the project directory, and update its backend methods and edit
records to the current protocol.
3. Replace the experimental adoption logic with the current staging manifest
and `skillopt_sleep.staging.adopt` behavior. Current staging artifacts use
`proposed_SKILL.md` / `proposed_CLAUDE.md`, `manifest.json`, and report files;
they do not expose the old `manifest.proposed_skill` field.
4. Decide how real OpenClaw transcripts are converted into a supported session
format. Pointing `claude_home` at an arbitrary agent directory does not by
itself make its files Claude Code-compatible JSONL.
5. Build scheduling around the adapted wrapper. The shared scheduler launches
the shared CLI; it does not automatically preserve this custom backend or
its category task-file flow.
6. Add isolated end-to-end tests for dry-run, accepted/rejected gates, staging,
adoption and backup, credential failure, and scheduled execution.
```
skills/skillopt-sleep/
├── SKILL.md # this file
├── config.json # engine config (backend, budgets, etc.)
├── run_sleep.py # entry point
└── skillopt_sleep_openclaw.py # DeepSeek/Ollama backend
```
Until those gaps are resolved, use the supported shared
`python -m skillopt_sleep` CLI with `--backend mock` to test SkillOpt-Sleep itself,
and treat this directory only as source material for a future OpenClaw port.
The engine itself is at `~/.openclaw/workspace/SkillOpt/skillopt_sleep/` (cloned from microsoft/SkillOpt).
## Shared-engine features are not wrapper features
## Usage
At this revision the supported shared CLI backends are `mock`, `claude`,
`codex`, `copilot`, `handoff`, and `azure_openai`; the
[plugin integration reference](../README.md#supported-cli-surface) is the
authoritative list. The shared engine can consolidate a selected skill and
project `CLAUDE.md` memory (controlled by `evolve_skill` and `evolve_memory`),
and its `schedule` / `unschedule` actions manage shared-engine cron entries.
Those capabilities do **not** make the custom OpenClaw wrapper portable: the
shared scheduler will not invoke the prototype backend or its category
fixtures. Use the shared documentation for those features, not this reference
SKILL.
```bash
# Run one cycle with current config
cd ~/.openclaw/workspace/skills/skillopt-sleep
python3 run_sleep.py
## Data and credential boundary
# Dry run (report only, no staging)
python3 run_sleep.py --dry-run
The prototype DeepSeek backend sends task, skill, memory, response, rubric, and
reflection content to its configured Chat Completions endpoint. Its source also
contains a helper that can send text to an Ollama service if a future port wires
that helper into the cycle. Neither path should be assumed to remove every
secret or private detail.
# Use a pre-built task set (recommended for testing)
python3 run_sleep.py --tasks tests/research-cron-tasks.json
```
Before any port is tested with real data:
## Scheduling
- use isolated, synthetic or explicitly reviewed task files;
- replace sample business names, personal references, URLs, and machine paths;
- load credentials through the operator's secret-management mechanism;
- verify TLS and retention policy for every remote endpoint; and
- inspect all staged artifacts before adoption.
```bash
python3 slash_sleep.py schedule --hour 3 --minute 17
python3 slash_sleep.py unschedule
python3 slash_sleep.py unschedule --all
```
The bundled fixtures are examples only. Their scores and any old cost estimates
do not establish effectiveness, safety, or a stable nightly price for another
OpenClaw deployment.
Installs a nightly cron entry using the shared SkillOpt-Sleep scheduler. This is an alternative to the external `run_sleep_cron.sh` script.
## Further information
## Alternative backends
- [OpenClaw README](README.md) — current reference status and adaptation checklist
- [plugin integration reference](../README.md) — supported shared-engine CLI
surface and data boundary
- [SkillOpt-Sleep documentation](../../docs/sleep/README.md) — concepts,
results, and limitations
While OpenClaw defaults to `openclaw-deepseek` (DeepSeek V4 Pro + Ollama), the shared engine also supports:
- `--backend mock` — deterministic, no API spend (for testing)
- `--backend claude` — uses the Claude CLI
- `--backend codex` — uses the Codex CLI
- `--backend copilot` — uses the GitHub Copilot CLI
These can be used via the engine directly (`python -m skillopt_sleep`).
## Shared-engine flags
When invoking the engine directly, all standard flags are available:
- `--source codex` / `--source auto` — harvest from Codex Desktop sessions
- `--tasks-file PATH` — use a pre-built task set
- `--target-skill-path PATH` — explicit SKILL.md target
- `--max-tasks N` / `--max-sessions N` — cap workload
- `--progress` — print phase progress
- `--json` — machine-readable output
- `--auto-adopt` — auto-adopt if gate passes
Config keys: `preferences`, `gate_mode`, `gate_metric`, `dream_rollouts`, `recall_k`, `evolve_memory`, `evolve_skill`.
## Config (config.json)
Key knobs:
- `backend: "openclaw-deepseek"` — our custom backend
- `model: "deepseek-v4-pro"` — optimizer model
- `edit_budget: 3` — max bounded edits per night
- `gate_mode: "on"` — validation-gated (rejects regressions)
- `auto_adopt: false` — require Ethan to adopt manually
- `max_tasks_per_night: 12` — cap to control cost
## Cost Estimate
Per night: 12 tasks × (1 attempt + 1 judge + 1 reflect) × ~$0.005/1K tokens × ~3K tokens/call ≈ **$0.50-2.00/night**.
## Outputs
- Report: `~/.skillopt-sleep/state.json` (running totals)
- Staging: `~/.skillopt-sleep/staging/<night>/`
- `report.md` — readable summary
- `best_skill.md` — proposed skill
- `edits.json` — bounded edit list
- `before.md` / `after.md` — diffs
## Held-Out Test Sets (Phase 2)
Located at `tests/<category>-tasks.json`. Each task has:
- `prompt` — the recurring task
- `reference` — exact-match gold answer
- `rubric` — soft score rubric (0-1)
- `domain` — research/devops/wiki/etc.
Currently building for 3 categories:
- research-cron-output
- devops-infrastructure-check
- wiki-canonical-guide
## When NOT To Use
- For a one-off workflow (not a recurring pattern)
- During a crisis/incident (humans must lead)
- When session transcripts are < 24h old (not enough signal)
- For skills < 300 tokens (over-optimization risk)
Contributions that turn this reference into a portable integration should add
tests and update all three documents together.