url/SkillOpt
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: sync documentation with post-v0.2 changes

Browse Source
This commit is contained in:
Yif-Yang
2026-07-14 17:11:40 +00:00
parent efb30b4bcc
commit f31bf8c06b
44 changed files with 2285 additions and 2025 deletions
Download Patch File Download Diff File Expand all files Collapse all files
plugins/codex/README.md
+18 -8
View File
@@ -9,7 +9,8 @@ as the Claude Code plugin (`skillopt_sleep`), wrapped for Codex.
> [gbrain-evals](https://github.com/garrytan/gbrain-evals) `skillopt-v1`
> benchmark, a deliberately deficient skill goes **0.00 → 1.00** on a held-out
> set with the Codex backend (incl. the tool-use seed via a real tool loop).
> See [the SkillOpt-Sleep guide section](https://microsoft.github.io/SkillOpt/docs/guideline.html#sleep).
> See the recorded results and limitations in
> [`docs/sleep/RESULTS.md`](../../docs/sleep/RESULTS.md).
## What Codex supports (and what we use)
@@ -22,8 +23,8 @@ rules. The shared runner remains a plain shell entrypoint that the skill calls.
## Install
```bash
git clone <repo-url> SkillOpt-Sleep
cd SkillOpt-Sleep
git clone https://github.com/microsoft/SkillOpt.git
cd SkillOpt
bash plugins/codex/install.sh # installs the skill
export SKILLOPT_SLEEP_REPO="$(pwd)" # so the runner is found from anywhere
```
@@ -60,15 +61,19 @@ python -m skillopt_sleep run --project "$(pwd)" --source codex --backend codex \
`~/.codex/archived_sessions`. Use `--codex-home /path/to/.codex` to point at a
different Codex home, or `--source auto` to try Codex archives first and fall
back to Claude Code transcripts. Default backend is `mock` (no API spend).
`--backend codex` uses your Codex budget for real improvement. Bound live runs
`--backend codex` uses your Codex budget for model-driven optimization; an
accepted gain is task-dependent, not guaranteed. Bound live runs
with `--max-sessions` and `--max-tasks`; add `--progress` because Codex-backed
mining, replay, and reflection can be slow and otherwise quiet. Use
`--target-skill-path` to stage/adopt into a repo-scoped Codex skill such as
`.agents/skills/<name>/SKILL.md`; target runs over-sample mined tasks and
prefer tasks that match the target skill's path, headings, and content. All the
controllable knobs (`--gate on|off`, `--rollouts-k`, `--budget-tokens`,
`--preferences`, optimizer/target split) work identically — see
[the SkillOpt-Sleep guide section](https://microsoft.github.io/SkillOpt/docs/guideline.html#sleep).
prefer tasks that match the target skill's path, headings, and content. The
implemented main-CLI flags work the same across the shared integrations, and
`--preferences "..."` is available for house rules. Advanced keys such as
`gate_mode`, `dream_rollouts`, and `recall_k` belong in the Sleep config; the
nightly CLI does not expose `--gate`, `--rollouts-k`, token/time-budget, or
optimizer/target-split flags. See the
[shared CLI reference](../README.md#supported-cli-surface).
For privacy-sensitive projects, split the run into reviewable steps:
@@ -86,6 +91,11 @@ Inspect/redact the JSON and set `"reviewed": true` before using a real backend.
`--tasks-file` skips archive harvest/mining and replays only the reviewed JSON
tasks; real backends refuse task files still marked `"reviewed": false`.
This review step matters even though the Codex transcript converter removes
known secret-shaped strings: pattern-based redaction is not a guarantee. A real
backend sends truncated transcript/task content to the selected provider, while
`--backend mock` makes no provider calls.
## Notes / status
- Codex's `exec` runs shell, so the real-tool-loop replay (e.g. the
plugins/codex/skills/skillopt-sleep/SKILL.md
+71 -30
View File
@@ -1,16 +1,23 @@
---
name: skillopt-sleep
description: "Use when the user wants Codex to self-improve from past usage, asks about a nightly/offline 'sleep' or 'dream' cycle, wants Codex to review past sessions, learn preferences, consolidate memory/skills, run dry-run/run/adopt/status for SkillOpt-Sleep, or schedule offline self-optimization. Drives the skillopt_sleep engine: harvest past sessions -> mine recurring tasks -> replay offline -> consolidate validated memory + skills behind a held-out gate."
description: "Use when the user wants Codex to self-improve from past usage, asks about a nightly/offline 'sleep' or 'dream' cycle, wants Codex to review past sessions, learn preferences, consolidate memory/skills, run dry-run/run/adopt/status for SkillOpt-Sleep, or schedule background self-optimization. Drives the skillopt_sleep engine: harvest past sessions -> mine recurring tasks -> replay through a selected backend -> consolidate validated memory + skills behind a held-out gate."
---
# SkillOpt-Sleep: offline self-evolution for a local Codex agent
# SkillOpt-Sleep: usage-driven self-evolution for a local Codex agent
SkillOpt-Sleep gives the user's Codex agent a sleep cycle. While the user is
offline or on demand, it reviews past local sessions, re-runs recurring tasks
on the user's own budget, and consolidates what it learns into memory and
skills. It keeps only changes that pass a held-out validation gate, and live
files change only after the user explicitly adopts a staged proposal. There is
no model-weight training.
SkillOpt-Sleep gives the user's Codex agent a sleep cycle. On demand or on a
nightly schedule, it reviews past local sessions, re-runs recurring tasks
through the selected backend, and proposes changes to a configured skill and to
the project's `CLAUDE.md`. With the default validation gate enabled, it keeps
only changes that improve a held-out score. Live files change only through
explicit adoption or a user-requested `--auto-adopt`. There is no model-weight
training.
The current shared engine does **not** write `AGENTS.md`. For a Codex-visible
result, always select a Codex skill explicitly with `--target-skill-path` (for
example `.agents/skills/<name>/SKILL.md`). If project `CLAUDE.md` is not a
desired secondary target, set `"evolve_memory": false` in
`~/.skillopt-sleep/config.json` before running.
## When to use
@@ -28,13 +35,15 @@ Trigger when the user wants any of:
configuration and normalize them into session digests.
2. **Mine** - turn digests into recurring `TaskRecord`s with outcomes and
checkable references where possible.
3. **Replay** - re-run mined tasks offline under the current skill and memory.
3. **Replay** - re-run mined tasks through the selected backend under the
current skill and memory.
4. **Consolidate** - reflect on failures and propose bounded edits.
5. **Gate** - accept edits only when the held-out validation score improves.
5. **Gate** - with the default gate enabled, accept edits only when the held-out
validation score improves.
6. **Stage** - write the proposal under
`<project>/.skillopt-sleep/staging/<date>/`; nothing live changes.
7. **Adopt** - only after explicit user approval, copy staged files over live
files with backups.
7. **Adopt** - explicitly, or through user-requested auto-adopt, copy staged
files over live files with backups for existing targets.
## How to drive it
@@ -43,32 +52,47 @@ finds the engine and a Python >= 3.10 automatically.
```bash
# point at the repo if it isn't auto-detected from CWD:
export SKILLOPT_SLEEP_REPO=/path/to/SkillOpt-Sleep
export SKILLOPT_SLEEP_REPO=/path/to/SkillOpt
TARGET_SKILL=.agents/skills/example/SKILL.md
bash "$SKILLOPT_SLEEP_REPO/plugins/run-sleep.sh" status --project "$(pwd)"
bash "$SKILLOPT_SLEEP_REPO/plugins/run-sleep.sh" harvest --project "$(pwd)"
bash "$SKILLOPT_SLEEP_REPO/plugins/run-sleep.sh" dry-run --project "$(pwd)" --backend mock
bash "$SKILLOPT_SLEEP_REPO/plugins/run-sleep.sh" run --project "$(pwd)" --backend codex
bash "$SKILLOPT_SLEEP_REPO/plugins/run-sleep.sh" run --project "$(pwd)" --source codex # harvest from Codex Desktop
bash "$SKILLOPT_SLEEP_REPO/plugins/run-sleep.sh" harvest --project "$(pwd)" \
--source codex --target-skill-path "$TARGET_SKILL"
bash "$SKILLOPT_SLEEP_REPO/plugins/run-sleep.sh" dry-run --project "$(pwd)" \
--source codex --target-skill-path "$TARGET_SKILL" --backend mock
bash "$SKILLOPT_SLEEP_REPO/plugins/run-sleep.sh" run --project "$(pwd)" \
--source codex --target-skill-path "$TARGET_SKILL" --backend codex \
--max-sessions 5 --max-tasks 3 --progress
bash "$SKILLOPT_SLEEP_REPO/plugins/run-sleep.sh" adopt --project "$(pwd)"
```
Actions are `status`, `harvest`, `dry-run`, `run`, `adopt`, `schedule`, and `unschedule`.
- Default backend is `mock`, which is deterministic and spends no API budget.
- `--backend codex` uses the user's Codex budget for real improvement.
- `--backend codex` uses the user's Codex budget for model-driven optimization.
An accepted held-out gain is run-specific evidence, not a guarantee of
broader improvement; results depend on the tasks, model, and checks.
- `--source codex` reads Codex Desktop archived sessions from `~/.codex/archived_sessions`;
use `--codex-home /path/to/.codex` if the archive lives elsewhere.
- `--target-skill-path` is required for a Codex skill target. Without it, the
shared default is a Claude-managed skill under `~/.claude/skills/`, not an
`.agents` skill.
- Keep `dry-run --backend mock` as the first smoke check unless the user
explicitly asked for a real optimization run.
### Scheduling
```bash
bash "$SKILLOPT_SLEEP_REPO/plugins/run-sleep.sh" schedule --project "$(pwd)" --hour 3 --minute 17
bash "$SKILLOPT_SLEEP_REPO/plugins/run-sleep.sh" schedule --project "$(pwd)" \
--backend codex --hour 3 --minute 17
bash "$SKILLOPT_SLEEP_REPO/plugins/run-sleep.sh" unschedule --project "$(pwd)"
```
Installs a nightly cron entry. `unschedule --all` removes every managed entry.
The scheduler persists the project, backend, time, and optional auto-adopt flag;
it does not persist `--source` or `--target-skill-path` from this command. Before
scheduling a Codex-targeted run, set `"transcript_source": "codex"` and an
absolute `"target_skill_path"` in `~/.skillopt-sleep/config.json`. On systems
without `crontab`, `schedule` prints a line for manual installation.
`unschedule --all` removes every managed entry.
### All backends
@@ -76,6 +100,8 @@ Installs a nightly cron entry. `unschedule --all` removes every managed entry.
- `--backend claude` — uses the Claude CLI
- `--backend codex` — uses the Codex CLI
- `--backend copilot` — uses the GitHub Copilot CLI
- `--backend handoff` — emits prompt/answer files for an interactive session
- `--backend azure_openai` — uses the configured Azure OpenAI endpoint
### Additional flags
@@ -96,7 +122,10 @@ Installs a nightly cron entry. `unschedule --all` removes every managed entry.
### Memory consolidation
The sleep cycle consolidates both **memory** (AGENTS.md / CLAUDE.md) and **skills** (SKILL.md) by default. Each is independently toggleable via `evolve_memory` / `evolve_skill` config keys. Both are gated by the same held-out validation score.
The shared sleep cycle consolidates project **memory** (`CLAUDE.md`) and the
selected **skill** (`SKILL.md`) by default. It does not update `AGENTS.md`.
Each target is independently toggleable through `evolve_memory` /
`evolve_skill`, and both are gated by the same held-out validation score.
## Steps
@@ -104,16 +133,23 @@ The sleep cycle consolidates both **memory** (AGENTS.md / CLAUDE.md) and **skill
2. For `dry-run` and `run`, report the held-out baseline -> candidate score,
gate action, task count, session count, and exact proposed edits.
3. If a staging directory is printed, read `report.md` before summarizing.
4. `run` only stages a proposal; nothing live changes until `adopt`.
5. Offer adoption only after the user has reviewed the staged proposal.
6. Never hand-edit the user's `AGENTS.md`, memory, or skills as a substitute
for `adopt`; adoption is the safety boundary and writes backups first.
4. `run` stages by default; if `--auto-adopt` was explicitly supplied, report
the paths it updated instead of claiming nothing changed.
5. Offer adoption only after the user has reviewed a still-staged proposal.
6. Never hand-edit the configured `CLAUDE.md` or target skill as a substitute
for the engine's adopt path; adoption is the safety boundary and backs up
existing targets first.
## Hard rules
- Harvest is read-only. Do not edit archived sessions or raw transcripts.
- Keep raw secrets, credentials, private user data, and unsanitized transcript
contents out of messages, logs, generated artifacts, and commits.
- Codex transcript harvesting removes known secret-shaped strings, developer
instructions, and raw tool payloads, but pattern-based redaction is not a
guarantee. A real backend still sends truncated transcript/task content to
its provider. Review sensitive sessions and provider policy first; prefer a
reviewed `--tasks-file` workflow when the data boundary matters.
- Keep raw secrets, credentials, private user data, and transcript contents out
of messages, logs, generated artifacts, and commits.
- Show validation evidence before recommending adoption.
- Treat generated edits as proposals, not as source of truth.
- Do not rely on deprecated custom prompts or `/sleep` slash commands for this
@@ -122,11 +158,16 @@ The sleep cycle consolidates both **memory** (AGENTS.md / CLAUDE.md) and **skill
## Validate
```bash
python -m skillopt_sleep dry-run --project "$(pwd)" --backend mock --json
python -m skillopt_sleep dry-run --project "$(pwd)" --source codex \
--target-skill-path .agents/skills/example/SKILL.md --backend mock --json
python -m skillopt_sleep.experiments.run_gbrain --backend codex \
--seeds brief-writer --data-root /path/to/gbrain-evals/eval/data/skillopt-v1 \
--nights 2 --limit-replay 3 --limit-holdout 3
```
A deficient skill goes 0.00 -> 1.00 on a held-out set; the optimizer's edits
are gated on real-task performance.
In the recorded `brief-writer` gbrain run, the deliberately deficient fixture
went 0.00 -> 1.00 on that run's held-out set. Treat this as reproducible
benchmark evidence for that configuration, not a guarantee for other skills,
tasks, or models; see the
[recorded results](https://github.com/microsoft/SkillOpt/blob/main/docs/sleep/RESULTS.md)
for context and limitations.