url/SkillOpt
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

merge into: url/SkillOpt:main
Branches Tags
url/SkillOpt:main url/SkillOpt:fix/minimax-optimizer-deployment url/SkillOpt:fix/handoff-notask-and-corrupt-snapshot url/SkillOpt:fix/claude-backend-clear-last-error url/SkillOpt:refactor/reflect-default-base url/SkillOpt:feat/skill-aware-reflection url/SkillOpt:copilot/update-env-adapter-documentation url/SkillOpt:docs/guideline url/SkillOpt:users/GitHubPolicyService/a41a3ce1-e5a1-4e18-810b-cfb8d2d21c29 url/SkillOpt:users/GitHubPolicyService/51be9fd7-b70b-4938-a8ee-3622f17d8046 url/SkillOpt:users/GitHubPolicyService/eb4071d7-d09f-411c-a688-a6f89d362a6a url/SkillOpt:users/GitHubPolicyService/c90b1514-6d1f-4da6-9662-3b96a06a9fb2 url/SkillOpt:users/GitHubPolicyService/8ded453c-4b9b-473f-8865-2117f99a82fd
url/SkillOpt:v0.2.0 url/SkillOpt:v0.1.0
..
pull from: url/SkillOpt:fix/claude-backend-clear-last-error
Branches Tags
url/SkillOpt:main url/SkillOpt:fix/minimax-optimizer-deployment url/SkillOpt:fix/handoff-notask-and-corrupt-snapshot url/SkillOpt:fix/claude-backend-clear-last-error url/SkillOpt:refactor/reflect-default-base url/SkillOpt:feat/skill-aware-reflection url/SkillOpt:copilot/update-env-adapter-documentation url/SkillOpt:docs/guideline url/SkillOpt:users/GitHubPolicyService/a41a3ce1-e5a1-4e18-810b-cfb8d2d21c29 url/SkillOpt:users/GitHubPolicyService/51be9fd7-b70b-4938-a8ee-3622f17d8046 url/SkillOpt:users/GitHubPolicyService/eb4071d7-d09f-411c-a688-a6f89d362a6a url/SkillOpt:users/GitHubPolicyService/c90b1514-6d1f-4da6-9662-3b96a06a9fb2 url/SkillOpt:users/GitHubPolicyService/8ded453c-4b9b-473f-8865-2117f99a82fd
url/SkillOpt:v0.2.0 url/SkillOpt:v0.1.0

1 Commits
main .. fix/claude-backend-clear-last-error

Author SHA1 Message Date
Yif Yang fad8fdb01f fix(sleep): reset ClaudeCliBackend.last_call_error at each call start 
Follow-up to #126. ClaudeCliBackend recorded last_call_error on failure but
never reset it, so a failure followed by a successful call still reported the
stale error in persisted diagnostics — making a healthy run look failed. Both
_call() and attempt_with_tools() now clear it at the start, matching
CodexCliBackend.

Note: the "TimeoutExpired leaks the prompt into logs" concern raised in review
does NOT apply on current main — the prompt is passed via stdin (input=), not
argv (fixed by #98), so TimeoutExpired's str() only contains the argv (no
prompt). Verified empirically.

Adds a regression test asserting a prior failure is cleared on the next
success.

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-07-12 16:33:18 +00:00
88 changed files with 2140 additions and 7980 deletions
Expand all files Collapse all files
.env.example
+4 -30
View File
@@ -8,7 +8,6 @@ export AZURE_OPENAI_API_VERSION=2024-12-01-preview
# Authentication: choose one method
# Option 1: API Key
export AZURE_OPENAI_API_KEY=
export AZURE_OPENAI_AUTH_MODE=api_key
# Option 2: Azure CLI (no API key needed, recommended on Azure VMs)
# export AZURE_OPENAI_AUTH_MODE=azure_cli
# Option 3: Managed Identity
@@ -16,45 +15,20 @@ export AZURE_OPENAI_AUTH_MODE=api_key
# export AZURE_OPENAI_MANAGED_IDENTITY_CLIENT_ID=your-client-id
# ── OpenAI-compatible endpoints ──────────────────────────────────────
# Path 1: generic research backend. Select openai_compatible explicitly as
# model.optimizer_backend and/or model.target_backend.
# export OPENAI_COMPATIBLE_BASE_URL=https://api.deepseek.com/v1
# export OPENAI_COMPATIBLE_API_KEY=sk-...
# export OPENAI_COMPATIBLE_MODEL=deepseek-chat
# Per-role overrides use OPTIMIZER_OPENAI_COMPATIBLE_* and
# TARGET_OPENAI_COMPATIBLE_* (BASE_URL, API_KEY, MODEL, TEMPERATURE,
# MAX_TOKENS, TIMEOUT_SECONDS).
# For scripts/train.py and scripts/eval_only.py, also set model.optimizer and
# model.target in YAML (or via --cfg-options). Those role model values are
# applied after backend initialization and override *_MODEL environment values.
# Path 2: research openai_chat compatibility mode. This reuses the Azure-family
# variables but creates a plain OpenAI client (no Azure auth or api-version).
# Set AUTH_MODE to openai_compatible and reuse AZURE_OPENAI_ENDPOINT / _API_KEY.
# The plain OpenAI client is used; no Azure auth, no api-version header.
# export AZURE_OPENAI_ENDPOINT=https://api.openai.com/v1
# export AZURE_OPENAI_API_KEY=sk-...
# export AZURE_OPENAI_AUTH_MODE=openai_compatible
# Path 3: SkillOpt-Sleep. `skillopt-sleep run --backend azure_openai` uses the
# same three AZURE_* variables from path 2. Optional Sleep-only controls:
# export SKILLOPT_SLEEP_COMPAT_MAX_TOKENS=8192
# export SKILLOPT_SLEEP_CHAT_EXTRA_BODY='{"provider_option": true}'
# ── Claude Code CLI (for claude_chat backend) ─────────────────────────
# Install and authenticate the `claude` CLI before use. For a non-default path:
# export CLAUDE_CLI_BIN=/path/to/claude
# ANTHROPIC_API_KEY is one authentication option understood by the CLI; SkillOpt
# does not create a direct Anthropic API client for this backend.
# ── Anthropic / Claude (for claude_chat backend) ─────────────────────
# export ANTHROPIC_API_KEY=sk-ant-...
# ── Qwen Local Model (for qwen_chat backend) ────────────────────────
# export QWEN_CHAT_BASE_URL=http://localhost:8000/v1
# export QWEN_CHAT_MODEL=Qwen/Qwen3.5-4B
# The train/eval entry points likewise override this model with
# model.optimizer/model.target for the selected Qwen roles.
# ── MiniMax (for minimax_chat backend) ──────────────────────────────
# export MINIMAX_BASE_URL=https://api.minimax.io/v1
# export MINIMAX_API_KEY=...
# When MiniMax is the target, set model.minimax_model in YAML. The current
# adapter shares one deployment across MiniMax roles; mixed-backend runs cannot
# independently select a MiniMax optimizer model and a different target model.
# export MINIMAX_MODEL=MiniMax-M2.7
CHANGELOG.md
+1 -59
View File
@@ -15,65 +15,7 @@ All notable changes to SkillOpt are documented here. This project adheres to
per night). Mined tasks are pinned per night so answering sessions cannot
shift the task set. Ships a `/skillopt-sleep-handoff` Claude Code command
that automates the loop with fresh-context subagents to protect the
held-out gate (thanks @dimitarvdenev, #125).
- **Generic OpenAI-compatible research backend** for optimizer and target
calls, with configurable base URL, API key, model, and timeout (thanks
@nankingjing, #115).
- **OpenAI-compatible SkillOpt-Sleep endpoint support** for providers such as
DeepSeek and self-hosted vLLM servers (thanks @Alphaxalchemy, #129; hardened
in #138).
- End-to-end wiring for the documented reflection `--preferences` option
(thanks @AKhozya, #131).
### Changed
- Claude Code's Sleep plugin can now use a `pip`/`uv`-installed
`skillopt-sleep` when no repository checkout is present (thanks
@ichoosetoaccept, #107).
- Qwen reasoning-model requests now use `max_completion_tokens` and omit
unsupported temperature parameters (thanks @chirag127, #128).
- Configuration files are read explicitly as UTF-8 (thanks @nankingjing,
#124).
### Fixed
- Preserve fractional rollout hard scores instead of coercing them to binary
values (thanks @zixuanguo786-ctrl, #104).
- Reject duplicate and overlapping IDs while materializing SearchQA manifests
(thanks @zixuanguo786-ctrl, #105).
- Make JSON-array extraction robust to unmatched braces and keep malformed
scans linear-time (thanks @zixuanguo786-ctrl, #103; follow-up #136).
- Package Markdown prompt assets in wheels and tolerate Windows temporary-file
cleanup failures (thanks @nankingjing, #135; follow-up #137).
- Exclude sub-agent transcripts and plugin-generated sessions from Sleep task
mining (thanks @codeL1985, #99).
- Normalize validation-gate density against the proposed edits and handle
zero-edit candidates safely (thanks @SparshGarg999, #102).
- Route optimizer-role MiniMax calls through the MiniMax backend (thanks
@jcforever1, #116).
- Surface Claude CLI spawn failures instead of silently turning them into zero
scores (thanks @Phoenix0531-sudo, #126).
- Improve Claude CLI behavior on Windows, including `.cmd` resolution and
long-prompt handling (thanks @codeL1985, #98).
- Preserve the scheduler's established annealing contract while expanding its
endpoint and sequence coverage (thanks @nankingjing, #123; follow-up #133).
### Security
- Prevent managed-identity credentials from being sent to non-Azure or
non-HTTPS endpoints, and isolate compatible-provider request extensions
from native Azure mode in SkillOpt-Sleep (#138, following
@Alphaxalchemy's #129).
### Tests
- Strengthen SkillOpt-Sleep verifier-discipline assertions, including recorded
scores and gate actions (thanks @Tanmay9223, #96).
- Add focused coverage for the validation-gate decision core and edit-budget
schedulers (thanks @nankingjing, #122, #123).
### Acknowledgements 🙏
Thank you to the contributors behind this unreleased work:
@AKhozya, @Alphaxalchemy, @Phoenix0531-sudo, @SparshGarg999,
@Tanmay9223, @chirag127, @codeL1985, @dimitarvdenev,
@ichoosetoaccept, @jcforever1, @nankingjing, and
@zixuanguo786-ctrl.
held-out gate.
## [0.2.0] — 2026-07-02
CONTRIBUTING.md
+7 -12
View File
@@ -7,7 +7,7 @@ Thank you for your interest in contributing! SkillOpt welcomes contributions of
```bash
git clone https://github.com/microsoft/SkillOpt.git
cd SkillOpt
python -m pip install -e ".[dev,docs]"
pip install -e ".[dev]"
```
## How to Contribute
@@ -16,28 +16,23 @@ python -m pip install -e ".[dev,docs]"
Open a GitHub issue with reproduction steps, expected/actual behavior, and your config file (remove API keys).
### 🔧 Add a Benchmark
See the [guide](docs/guide/new-benchmark.md) and use the scaffold at
`skillopt/envs/_template/`. Register the adapter lazily in both
`scripts/train.py` and `scripts/eval_only.py`, and add focused tests.
See the [guide](docs/guide/new-benchmark.md) and use the scaffold at `skillopt/envs/_template/`.
### 🤖 Add a Model Backend
First check whether the built-in `openai_compatible` backend covers the
provider. Otherwise follow the function-based backend contract in the
[backend guide](docs/guide/new-backend.md), including routing, configuration,
token accounting, and no-network tests.
See the [guide](docs/guide/new-backend.md).
### 📝 Improve Documentation
```bash
python -m mkdocs serve # Preview at http://localhost:8000
pip install -e ".[docs]"
mkdocs serve # Preview at http://localhost:8000
```
## Pull Request Process
1. Fork the repo and create a feature branch
2. Make changes and run focused tests plus `python -m pytest -q`
2. Make changes and test with an existing benchmark
3. Submit a PR with a clear description
4. For documentation changes, run `python -m mkdocs build --strict`
5. Ensure CI passes
4. Ensure CI passes
## Code Style
- Follow existing patterns in the codebase
README.md
+8 -14
View File
@@ -9,12 +9,12 @@
<a href="https://trendshift.io/repositories/38498?utm_source=trendshift-badge&utm_medium=badge&utm_campaign=badge-trendshift-38498" target="_blank" rel="noopener noreferrer"><img src="https://trendshift.io/api/badge/trendshift/repositories/38498/weekly?language=Python" alt="microsoft%2FSkillOpt | Trendshift" width="250" height="55"/></a>
</p>
> 📖 **For installation, data preparation, training/eval commands, configuration, and framework internals, start with the versioned [SkillOpt documentation](https://github.com/microsoft/SkillOpt/blob/main/docs/index.md). A concise rendered overview is available in the [Documentation & Reproduction Guide](https://microsoft.github.io/SkillOpt/docs/guideline.html), and longer-form engineering analysis appears on the [Technical Blog](https://microsoft.github.io/SkillOpt/blog/). We also maintain a [Changelog](CHANGELOG.md) for released and unreleased changes.**
> 📖 **For installation, data preparation, training/eval commands, the full configuration reference, and framework internals, see the [Documentation & Reproduction Guide](https://microsoft.github.io/SkillOpt/docs/guideline.html)** (rendered on GitHub Pages).
---
## News 🔥🔥🔥
- **[2026-07-02]** 🚀 **SkillOpt [v0.2.0](https://github.com/microsoft/SkillOpt/releases/tag/v0.2.0) is out on [PyPI](https://pypi.org/project/skillopt/)!** Headline feature: **SkillOpt-Sleep**, a nightly offline self-evolution engine (harvest → mine → replay → consolidate behind a held-out validation gate), now shipped as the `skillopt-sleep` CLI. It also includes experimental multi-objective, replay, and dream-rollout controls; the main CLI keeps conservative defaults and does not expose every experiment-harness control as a flag. The release source adds integration shells for **Claude Code, Codex, Copilot, and Devin**, plus an **OpenClaw reference adaptation**; these plugin/MCP files live in the repository rather than the PyPI wheel. It also adds SearchQA split materialization, Windows robustness, and hardened JSON parsing. See the [release notes](https://github.com/microsoft/SkillOpt/releases/tag/v0.2.0) for full release details and contributor acknowledgements.
- **[2026-07-02]** 🚀 **SkillOpt [v0.2.0](https://github.com/microsoft/SkillOpt/releases/tag/v0.2.0) is out on [PyPI](https://pypi.org/project/skillopt/)!** Headline feature: **SkillOpt-Sleep**, a nightly offline self-evolution engine (harvest → mine → replay → consolidate, all behind a held-out validation gate) with multi-objective reward, experience replay + dream rollouts, and long-term memory — now shipped as the `skillopt-sleep` CLI. This release also adds cross-tool backends and plugin shells for **Claude, Codex, Copilot, Devin, and OpenClaw**, SearchQA split materialization, Windows robustness, and hardened JSON parsing. See the [release notes](https://github.com/microsoft/SkillOpt/releases/tag/v0.2.0) for the full changelog and contributor acknowledgements.
- **[2026-06-15]** 😴 **SkillOpt-Sleep (preview)** — a nightly offline self-evolution companion for local coding agents (Claude Code / Codex / Copilot): review past sessions, replay recurring tasks, and consolidate validated skills behind a held-out gate. See **[`docs/sleep/README.md`](docs/sleep/README.md)** for what it is, how to use it, and results.
- **[2026-06-03]** 🎉 **[gbrain](https://github.com/garrytan/gbrain), [gbrain-evals](https://github.com/garrytan/gbrain-evals/blob/main/docs/benchmarks/2026-06-03-skillopt.md), and [darwin-skill](https://github.com/alchaincyf/darwin-skill) have all integrated SkillOpt.**
- **[2026-06-02]** 🎉 **SkillOpt [v0.1.0](https://github.com/microsoft/SkillOpt/releases/tag/v0.1.0) is now available on [PyPI](https://pypi.org/project/skillopt/)!** Install with `pip install skillopt`. This initial release includes the full training loop (rollout → reflect → aggregate → select → update → evaluate), multi-backend support (OpenAI / Azure / Claude / Qwen / MiniMax), six built-in benchmarks, and WebUI dashboard.
@@ -31,9 +31,9 @@ which reliably improves over its starting point under feedback.
**SkillOpt treats the skill document as the trainable state of a frozen
agent**, and trains it with the discipline that makes weight-space
optimization reproducible. A separate optimizer model turns scored rollouts
into bounded add / delete / replace edits on a single skill document; in the
default paper-style path, a candidate edit is accepted only when it strictly
improves a held-out validation score. A textual learning-rate budget, a rejected-edit buffer,
into bounded add / delete / replace edits on a single skill document; a
candidate edit is accepted only when it strictly improves a held-out
validation score. A textual learning-rate budget, a rejected-edit buffer,
and an epoch-wise slow / meta update make skill training stable while
adding **zero inference-time model calls** at deployment.
@@ -64,9 +64,7 @@ https://github.com/user-attachments/assets/eb12d3bc-371c-467f-904d-91b61f339ed7
### Adding a new backend
A backend = a chat / exec target (e.g. `openai_chat`, `claude_chat`,
`qwen_chat`, `minimax_chat`, `openai_compatible`, `codex_exec`,
`claude_code_exec`). If a provider implements the OpenAI Chat Completions
protocol, try the built-in `openai_compatible` backend before adding code. See
`qwen_chat`, `minimax_chat`, `codex_exec`, `claude_code_exec`). See
[`docs/guide/new-backend.md`](docs/guide/new-backend.md) for the full
contract; in short you add a `skillopt/model/<name>_backend.py` module,
register it in `skillopt/model/common.py` + `backend_config.py`, and wire
@@ -75,9 +73,8 @@ and `minimax_backend.py` are good templates.
### Adding a new benchmark
A benchmark = a `skillopt/envs/<name>/` package with an adapter, a data loader,
a scored rollout helper, a YAML config, and optionally an initial seed skill.
See
A benchmark = a `skillopt/envs/<name>/` package with a `dataloader.py`, a
`rollout.py`, and an `initial.md` seed skill. See
[`docs/guide/new-benchmark.md`](docs/guide/new-benchmark.md) for the full
contract; the simplest reference is `skillopt/envs/searchqa/`.
@@ -96,9 +93,6 @@ python -m skillopt_webui.app
| `--host` | `0.0.0.0` | Bind address |
| `--share` | off | Create a public Gradio share link |
The default host listens on every network interface. Use
`--host 127.0.0.1` for local-only access.
---
## Citation
blog/gating-reflection-safe-updates/index.html
-1808
View File
File diff suppressed because it is too large Load Diff
blog/index.html
-149
View File
@@ -1,149 +0,0 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>SkillOpt Technical Blog</title>
<meta name="description" content="Technical notes from the SkillOpt team on skill optimization, evaluation, safety, and agent learning systems.">
<link rel="canonical" href="https://microsoft.github.io/SkillOpt/blog/">
<meta property="og:type" content="website">
<meta property="og:site_name" content="SkillOpt">
<meta property="og:title" content="SkillOpt Technical Blog">
<meta property="og:description" content="Technical notes on skill optimization, evaluation, safety, and agent learning systems.">
<meta property="og:url" content="https://microsoft.github.io/SkillOpt/blog/">
<meta property="og:image" content="https://microsoft.github.io/SkillOpt/skillopt-assets/teaser-1.png">
<meta name="twitter:card" content="summary_large_image">
<meta name="twitter:title" content="SkillOpt Technical Blog">
<meta name="twitter:description" content="Technical notes on skill optimization, evaluation, safety, and agent learning systems.">
<meta name="twitter:image" content="https://microsoft.github.io/SkillOpt/skillopt-assets/teaser-1.png">
<link rel="icon" href="data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 100 100'%3E%3Crect width='100' height='100' rx='16' fill='%23245fc7'/%3E%3Ctext x='50' y='68' text-anchor='middle' font-size='58' font-family='Arial' font-weight='700' fill='white'%3ES%3C/text%3E%3C/svg%3E">
<style>
:root {
--ink: #172033;
--muted: #596579;
--line: #d9deea;
--paper: #fff;
--wash: #f7f9fc;
--blue: #245fc7;
--blue-soft: #eaf1ff;
--shadow: 0 14px 44px rgba(23, 32, 51, .08);
}
* { box-sizing: border-box; }
body {
margin: 0;
color: var(--ink);
background: var(--wash);
font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif;
line-height: 1.6;
}
a { color: var(--blue); }
.skip-link {
position: fixed;
left: 16px;
top: -80px;
z-index: 100;
padding: 10px 14px;
border-radius: 8px;
color: #fff;
background: var(--blue);
}
.skip-link:focus { top: 12px; }
header {
border-bottom: 1px solid var(--line);
background: rgba(255, 255, 255, .95);
}
.header-inner {
display: flex;
align-items: center;
justify-content: space-between;
gap: 24px;
max-width: 1040px;
margin: 0 auto;
padding: 16px 24px;
}
.brand {
color: var(--ink);
font-weight: 750;
text-decoration: none;
}
nav { display: flex; flex-wrap: wrap; gap: 16px; font-size: 14px; }
nav a { color: var(--muted); text-decoration: none; }
nav a:hover, nav a:focus-visible { color: var(--blue); text-decoration: underline; }
main { max-width: 1040px; margin: 0 auto; padding: 72px 24px 96px; }
.eyebrow {
color: var(--blue);
font-size: 13px;
font-weight: 750;
letter-spacing: .08em;
text-transform: uppercase;
}
h1 { margin: 10px 0 16px; font-size: clamp(40px, 7vw, 68px); line-height: 1.02; }
.intro { max-width: 760px; color: var(--muted); font-size: 20px; }
.post-list { margin-top: 52px; }
.post-card {
display: grid;
grid-template-columns: minmax(0, 1fr) auto;
gap: 30px;
padding: 34px;
border: 1px solid var(--line);
border-radius: 16px;
background: var(--paper);
box-shadow: var(--shadow);
}
.post-card h2 { margin: 8px 0 12px; font-size: clamp(25px, 4vw, 36px); line-height: 1.15; }
.post-card h2 a { color: var(--ink); text-decoration: none; }
.post-card h2 a:hover, .post-card h2 a:focus-visible { color: var(--blue); text-decoration: underline; }
.meta { color: var(--muted); font-size: 14px; }
.summary { max-width: 720px; margin: 16px 0 0; color: #344054; }
.read-link { align-self: center; white-space: nowrap; font-weight: 700; }
footer {
max-width: 1040px;
margin: 0 auto;
padding: 26px 24px 42px;
border-top: 1px solid var(--line);
color: var(--muted);
font-size: 14px;
}
@media (max-width: 720px) {
.header-inner { align-items: flex-start; flex-direction: column; }
main { padding-top: 48px; }
.post-card { grid-template-columns: 1fr; padding: 24px; }
.read-link { justify-self: start; }
}
</style>
</head>
<body>
<a class="skip-link" href="#main-content">Skip to content</a>
<header>
<div class="header-inner">
<a class="brand" href="../">SkillOpt</a>
<nav aria-label="Site navigation">
<a href="../">Project</a>
<a href="../docs/guideline.html">Documentation</a>
<a href="https://arxiv.org/abs/2605.23904">Paper</a>
<a href="https://github.com/microsoft/SkillOpt">Code</a>
</nav>
</div>
</header>
<main id="main-content">
<span class="eyebrow">Microsoft Research · SkillOpt</span>
<h1>Technical Blog</h1>
<p class="intro">Scoped experiments and engineering notes on optimizing agent skills, evaluating text-space updates, and deploying self-improving systems responsibly.</p>
<section class="post-list" aria-labelledby="latest-posts">
<h2 id="latest-posts">Latest posts</h2>
<article class="post-card">
<div>
<div class="meta"><time datetime="2026-07-14">July 14, 2026</time> · Ziwei Zhou, Ziyang Gong, and Yifan Yang</div>
<h2><a href="gating-reflection-safe-updates/">Expanded SkillOpt Ablations, Skill-Aware Reflection, and SkillOpt-Sleep</a></h2>
<p class="summary">A three-part report with expanded SkillOpt ablations, a skill-aware reflection design with memory consolidation, and a controlled study of the SkillOpt-Sleep plugin.</p>
</div>
<a class="read-link" href="gating-reflection-safe-updates/" aria-label="Read Expanded SkillOpt Ablations, Skill-Aware Reflection, and SkillOpt-Sleep">Read article →</a>
</article>
</section>
</main>
<footer>SkillOpt Technical Blog · <a href="https://github.com/microsoft/SkillOpt">github.com/microsoft/SkillOpt</a></footer>
</body>
</html>
ckpt/README.md
+14 -22
View File
@@ -10,10 +10,9 @@ provided skills on a given split without re-running the training loop.
> skills as portable artifacts. If you want to *train* your own skill,
> use `scripts/train.py` per the top-level README.
>
> This is the first optimized-skill artifact batch. We plan to continue
> uploading remaining paper artifacts as they are cleaned and verified. All
> six lightweight ID/path split manifests are already checked in under
> `data/`; most still require materializing their upstream benchmark payload.
> This is the first artifact batch. We plan to continue uploading the
> remaining optimized skills and benchmark split manifests as they are
> cleaned and verified.
## What's here
@@ -36,17 +35,12 @@ longitudinal guidance — that's expected, not a formatting issue.
invoking the optimizer. Example for SearchQA against the test split:
```bash
# The checked-in SearchQA split is ID-only; materialize full examples first.
python -m pip install -e ".[searchqa]"
python scripts/materialize_searchqa.py
python scripts/eval_only.py \
--config configs/searchqa/default.yaml \
--skill ckpt/searchqa/gpt5.5_skill.md \
--split valid_unseen \
--split_dir data/searchqa_split \
--split_dir data/searchqa_id_split \
--azure_openai_endpoint https://your-resource.openai.azure.com/ \
--azure_openai_auth_mode api_key \
--target_model gpt-5.5
```
@@ -58,16 +52,13 @@ is the selection / validation split, `train` is the training split, and
## On comparing to the paper numbers
To compare against the paper-reported cells, use the same dataset split and
scorer. SearchQA's ID manifest is checked in at `data/searchqa_id_split/` (400
train / 200 selection / 1400 test); the materializer writes the runnable
payload to `data/searchqa_split/`. All six lightweight split manifests are
checked in under `data/`. ALFWorld's manifest records game-file paths; the
other ID manifests still require you to materialize the corresponding
upstream benchmark payload into the documented `split_dir`. See
[`data/README.md`](../data/README.md) for the exact status of each benchmark.
When using `split_mode: ratio` instead, the loader is deterministic from
`split_seed` (default `42`) + `split_ratio` (default `2:1:7`), so a given
`data_path` + seed reproduces across machines.
scorer. SearchQA's split is checked in at `data/searchqa_id_split/` (400
train / 200 selection / 1400 test). For the other benchmarks, point
`--split_dir` at your own materialized split; the loader is deterministic
from `split_seed` (default `42`) + `split_ratio` (default `2:1:7`) when
`split_mode: ratio` is used, so a given `data_path` + seed reproduces
across machines. Explicit per-benchmark split manifests are being prepared
for upload — see issues #14 and #21.
## Why force-accept vs. gated slow-update matters
@@ -83,5 +74,6 @@ Current `main` defaults to `false` (force-accept mode), a newer
post-submission behavior where the slow-update guidance is written into
`current_skill` and `best_skill` unconditionally at the epoch boundary. If
you re-train with the current default, you may produce a *different*
`best_skill.md` than the one checked in here. Both modes are supported; see
the [configuration reference](../docs/reference/config.md).
`best_skill.md` than the one checked in here. Both modes are supported;
see the top-level README's "Configuration -> Slow-update acceptance mode"
section.
docs/contributing.md
+10 -19
View File
@@ -15,7 +15,6 @@ pip install -e ".[dev]"
### 🐛 Bug Reports
Open an issue with:
- Steps to reproduce
- Expected vs actual behavior
- Config file used (sanitize API keys)
@@ -26,29 +25,21 @@ Open an issue with:
See [Add a New Benchmark](guide/new-benchmark.md) for the implementation guide.
**Checklist:**
- [ ] Data loader in `skillopt/envs/<benchmark>/dataloader.py`
- [ ] Scored rollout implementation in `skillopt/envs/<benchmark>/rollout.py`
- [ ] Per-item `predictions/<id>/conversation.json` artifacts for shared reflection
- [ ] Environment adapter in `skillopt/envs/<benchmark>/adapter.py`
- [ ] Data loader in `skillopt/envs/<benchmark>/loader.py`
- [ ] Environment adapter in `skillopt/envs/<benchmark>/env.py`
- [ ] Config file in `configs/<benchmark>/default.yaml`
- [ ] Lazy registration in `scripts/train.py` and `scripts/eval_only.py`
- [ ] Focused tests and an optional seed skill referenced by `env.skill_init`
- [ ] Documentation update
- [ ] Registration in `skillopt/envs/__init__.py`
- [ ] Documentation page in `docs/`
### 🤖 New Model Backend
See [Add a New Model Backend](guide/new-backend.md) for the implementation guide.
**Checklist:**
- [ ] Function-based backend module in `skillopt/model/<name>_backend.py`
- [ ] Alias and default model in `skillopt/model/common.py`
- [ ] Optimizer/target whitelist entries in `skillopt/model/backend_config.py`
- [ ] Dispatch, token tracking, and setter forwarding in `skillopt/model/__init__.py`
- [ ] YAML/CLI wiring when the backend exposes structured config fields
- [ ] Focused routing, configuration, tool-call, and token-accounting tests
- [ ] `.env.example` and backend/configuration reference updates
- [ ] Backend in `skillopt/model/<backend>.py`
- [ ] Registration in `skillopt/model/__init__.py`
- [ ] API key entry in `.env.example`
- [ ] Documentation update
### 📝 Documentation
@@ -68,9 +59,9 @@ mkdocs serve # Preview at http://localhost:8000
## Pull Request Process
1. Fork the repository
2. Create a feature branch: `git switch -c feature/my-benchmark`
2. Create a feature branch: `git checkout -b feature/my-benchmark`
3. Make your changes
4. Run focused tests, the full test suite, and `mkdocs build --strict` when docs change
4. Test with an existing benchmark config
5. Submit a PR with a clear description
## License
docs/guide/configuration.md
+9 -82
View File
@@ -20,53 +20,15 @@ Benchmark configs inherit from `_base_/default.yaml` and override specific value
## Key Parameters
### Model Backends
`optimizer_backend` controls reflection and skill editing;
`target_backend` controls task rollout. The legacy `backend` field remains for
backward compatibility, but explicit role fields are the clearest configuration.
### Model
```yaml
model:
backend: azure_openai # High-level compatibility label
optimizer_backend: openai_chat
target_backend: openai_chat
optimizer: gpt-5.5 # Optimizer deployment/model
target: gpt-5.5 # Target deployment/model
azure_openai_auth_mode: api_key
backend: azure_openai # azure_openai | openai_chat | claude_code_exec | qwen
optimizer: gpt-5.5 # Optimizer model (for reflection)
target: gpt-5.5 # Target model (for rollout)
```
| Backend | Optimizer | Target | Configuration |
|---|:---:|:---:|---|
| `openai_chat` | ✓ | ✓ | Azure OpenAI, or its explicit compatibility auth mode |
| `openai_compatible` | ✓ | ✓ | Generic OpenAI Chat Completions endpoint |
| `claude_chat` | ✓ | ✓ | Claude Code CLI (`claude -p`) |
| `qwen_chat` | ✓ | ✓ | Qwen served through an OpenAI-compatible local endpoint |
| `minimax_chat` | ✓ | ✓ | MiniMax API |
| `codex_exec` | — | ✓ | Codex CLI execution harness |
| `claude_code_exec` | — | ✓ | Claude Code CLI execution harness |
The current MiniMax adapter has one shared deployment. Set
`model.minimax_model` when MiniMax is the target; a mixed-backend run cannot
independently select a MiniMax optimizer model and a different target model.
For a generic compatible provider, select the role backends explicitly rather
than relying on a high-level shorthand:
```yaml
model:
optimizer_backend: openai_compatible
target_backend: openai_compatible
optimizer: deepseek-chat
target: deepseek-chat
```
The train/eval entry points apply `model.optimizer` and `model.target` after
backend initialization. For the selected roles, these YAML values override
`OPENAI_COMPATIBLE_MODEL`, `QWEN_CHAT_MODEL`, and their per-role environment
forms. The environment model variables mainly seed direct library use; always
set the role models in a training or evaluation config.
### Training
```yaml
@@ -134,16 +96,9 @@ Notes:
```yaml
evaluation:
use_gate: true # Validation gating (accept/reject updates)
gate_metric: hard # hard | soft | mixed
gate_mixed_weight: 0.5 # Soft-score weight when metric=mixed
use_semantic_density: false # Optional instruction-density bonus
eval_test: true # Run test evaluation after training
```
The default and paper-style setting is `use_gate: true`. Setting it to `false`
still records selection scores but force-accepts every candidate, so it changes
the optimization semantics and should be reported explicitly.
### Environment (Data)
```yaml
@@ -162,7 +117,6 @@ Override any config value from the command line:
```bash
python scripts/train.py \
--config configs/searchqa/default.yaml \
--cfg-options \
optimizer.learning_rate=16 \
optimizer.lr_scheduler=linear \
gradient.analyst_workers=8
@@ -174,38 +128,11 @@ Model credentials are loaded from environment variables:
| Variable | Backend | Description |
|---|---|---|
| `AZURE_OPENAI_ENDPOINT` | `openai_chat` | Azure resource URL, or compatibility-mode base URL |
| `AZURE_OPENAI_API_VERSION` | `openai_chat` | Azure API version |
| `AZURE_OPENAI_AUTH_MODE` | `openai_chat` | `api_key`, `azure_cli`, `managed_identity`, or `openai_compatible` |
| `AZURE_OPENAI_API_KEY` | `openai_chat` | Required when auth mode is `api_key` or `openai_compatible` |
| `OPENAI_COMPATIBLE_BASE_URL` | `openai_compatible` | Generic Chat Completions base URL |
| `OPENAI_COMPATIBLE_API_KEY` | `openai_compatible` | Provider API key; optional for local servers |
| `OPENAI_COMPATIBLE_MODEL` | `openai_compatible` | Shared provider model ID for direct library use; train/eval YAML role models take precedence |
| `CLAUDE_CLI_BIN` | `claude_chat` | Optional path to the `claude` executable; defaults to `claude` |
| `ANTHROPIC_API_KEY` | `claude_chat` | Optional authentication method understood by the Claude CLI, not a direct SkillOpt API client |
| `QWEN_CHAT_BASE_URL` | `qwen_chat` | Local Qwen/vLLM endpoint |
| `QWEN_CHAT_MODEL` | `qwen_chat` | Served model name for direct library use; train/eval YAML role models take precedence |
| `MINIMAX_BASE_URL` | `minimax_chat` | MiniMax-compatible base URL |
| `MINIMAX_API_KEY` | `minimax_chat` | MiniMax API key |
`OPTIMIZER_` and `TARGET_` prefixes provide per-role overrides for the
Azure, OpenAI-compatible, and Qwen variable families. See the
[Configuration Reference](../reference/config.md) for exact names.
`claude_chat` launches the installed Claude Code CLI with `claude -p`; install
and authenticate that CLI before use. Setting `ANTHROPIC_API_KEY` is one way
the CLI may authenticate, but SkillOpt does not call the Anthropic API
directly through this backend.
### Three OpenAI-compatible paths
- Research, generic provider: select `openai_compatible` and use
`OPENAI_COMPATIBLE_*`.
- Research, Azure-family compatibility mode: keep `openai_chat`, set
`AZURE_OPENAI_AUTH_MODE=openai_compatible`, and use `AZURE_OPENAI_*`.
- SkillOpt-Sleep: run with `--backend azure_openai` and use the same
compatibility-mode `AZURE_OPENAI_*` variables. Sleep does not read the
research backend's role-specific variables.
| `AZURE_OPENAI_ENDPOINT` | azure_openai | Azure resource endpoint |
| `AZURE_OPENAI_API_KEY` | azure_openai | Azure API key |
| `OPENAI_API_KEY` | openai | OpenAI API key |
| `ANTHROPIC_API_KEY` | claude | Anthropic API key |
| `QWEN_API_BASE` | qwen | Local Qwen vLLM endpoint |
## Full Reference
docs/guide/dl-analogy.md
+3 -5
View File
@@ -14,9 +14,10 @@ SkillOpt is designed around a core insight: **optimizing natural-language prompt
| **Gradient aggregation** | Patch aggregation | Merge similar edits |
| **Gradient clipping** | Edit selection | Cap max edits per step |
| **Learning rate** | `learning_rate` | Max number of edits applied per step |
| **LR scheduler** | `lr_scheduler` | Edit-budget schedule: cosine, linear, constant, or autonomous |
| **LR scheduler** | `lr_scheduler` | Decay schedule: cosine, linear, constant |
| **SGD step** | Skill update | Apply selected patches to document |
| **Validation set** | Selection split | Gate checks improvement before accepting |
| **Early stopping** | Gate patience | Reject updates that don't improve |
| **Training step** | Step | One rollout → reflect → update cycle |
| **Epoch** | Epoch | Full pass with slow update + meta memory |
| **Momentum** | Slow update | Longitudinal comparison at epoch boundary |
@@ -33,10 +34,7 @@ SkillOpt is designed around a core insight: **optimizing natural-language prompt
1. **Familiar mental model**: ML practitioners immediately understand how to tune SkillOpt
2. **Principled hyperparameter search**: Grid search over `learning_rate` × `lr_scheduler` works just like in DL
3. **Reusable mechanisms**: Gating provides validation-based model selection, while slow update plays a momentum-like role across epochs
The gate is a per-candidate accept/reject decision. SkillOpt does not implement
a gate-patience counter or stop training after a run of rejected candidates.
3. **Proven mechanisms**: Gating ≈ validation-based selection, patience ≈ early stopping, slow update ≈ momentum — all with strong theoretical motivation
## Hyperparameter Transfer Rules
docs/guide/first-experiment.md
+40 -67
View File
@@ -4,43 +4,17 @@ This guide walks through running a complete SkillOpt training on SearchQA.
## 1. Choose a Benchmark
SkillOpt includes ready-to-use configs for several benchmarks. End-to-end
runtime depends on the chosen models, provider latency, worker limits, and
dataset size, so the project does not promise fixed wall-clock estimates.
SkillOpt includes ready-to-use configs for several benchmarks:
| Benchmark | Modality | Additional setup |
| Benchmark | Difficulty | Typical Runtime |
|---|---|---|
| SearchQA | Text QA | Materialize the released ID manifest |
| DocVQA | Document/image QA | Obtain and materialize images and examples |
| ALFWorld | Embodied agent | Install ALFWorld and download its assets |
| SearchQA | ⭐ Easy | ~30 min |
| DocVQA | ⭐⭐ Medium | ~2 hours |
| ALFWorld | ⭐⭐⭐ Hard | ~3 hours |
We'll use **SearchQA** because it is the simplest text-only walkthrough.
We'll use **SearchQA** as it's the fastest to complete.
## 2. Install and Materialize SearchQA
The repository contains a stable SearchQA ID manifest, not the full runnable
examples. From a source checkout, install the data extra and materialize the
split once:
```bash
python -m pip install -e ".[searchqa]"
python scripts/materialize_searchqa.py
```
By default, the materializer reads `data/searchqa_id_split/` and writes the
train/validation/test payloads expected by the config to
`data/searchqa_split/`; both paths have command-line overrides.
## 3. Configure
Configure and export one model backend as described in
[Installation](installation.md#environment-variables). For example:
```bash
cp .env.example .env
# Edit .env, choose one authentication mode, then export it:
set -a; source .env; set +a
```
## 2. Configure
Review the config file:
@@ -68,55 +42,56 @@ evaluation:
use_gate: true # (validation gating)
```
## 4. Train
## 3. Train
```bash
python scripts/train.py \
--config configs/searchqa/default.yaml \
--out_root outputs/searchqa_first_run
python scripts/train.py --config configs/searchqa/default.yaml
```
The command prints the resolved backend/data configuration, per-step rollout
and gate progress, and the generated output directory.
## 5. Monitor
The explicit `--out_root` above creates this run directory:
You'll see output like:
```
outputs/searchqa_first_run/
├── config.json
├── runtime_state.json
├── history.json
├── best_skill.md
├── skills/
│ └── skill_vXXXX.md
[Step 1/8] Rollout: 20 items, 4 workers...
[Step 1/8] Score: 0.65 → Reflect...
[Step 1/8] 6 edit patches generated
[Step 1/8] Selected 4 edits (lr=8, cosine → 7.7)
[Step 1/8] Gate: val score 0.68 > 0.65 ✓ ACCEPT
[Step 2/8] ...
```
## 4. Monitor
Training outputs are saved to `outputs/<benchmark>/<run_id>/`:
```
outputs/searchqa/2024-01-15_10-30-00/
├── steps/
│ └── step_XXXX/
│ ├── candidate_skill.md
│ ├── step_record.json
│ └── trajectory_digest.json
│ ├── step_0001/
│ │ ├── candidate_skill.md
│ │ ├── step_record.json
│ │ └── trajectory_digest.json
│ └── step_0002/
├── slow_update/
│ └── epoch_XX/
└── meta_skill/
└── epoch_XX/
│ └── epoch_02/
├── meta_skill/
│ └── epoch_02/
├── skills/
│ └── step_0001.md
├── best_skill.md
├── history.json
└── config.yaml
```
## 6. Evaluate
## 5. Evaluate
Evaluate the best skill on the test split:
```bash
python scripts/eval_only.py \
--config configs/searchqa/default.yaml \
--skill outputs/searchqa_first_run/best_skill.md \
--split valid_unseen
--skill outputs/searchqa/<run_id>/skills/best_skill.md
```
The `--skill` path above is the training artifact. Evaluation writes
`eval_summary.json` to its own timestamped `outputs/eval_.../` directory unless
you pass an explicit `--out_root`; it does not overwrite the training run.
## WebUI
Prefer a graphical interface? Launch the WebUI:
@@ -126,9 +101,7 @@ pip install -e ".[webui]"
python -m skillopt_webui.app
```
Then open `http://localhost:7860` in your browser to configure parameters and
launch training. The default host is `0.0.0.0`; pass `--host 127.0.0.1` for a
local-only dashboard.
Then open `http://localhost:7860` in your browser to configure parameters and launch training.
## Next Steps
docs/guide/installation.md
+20 -95
View File
@@ -3,44 +3,16 @@
## Requirements
- Python ≥ 3.10
- For research training/evaluation, access to at least one configured model
backend (hosted API, local server, or an installed execution CLI)
- The SkillOpt-Sleep `mock` backend needs no credentials
- At least one model API key (Azure OpenAI, OpenAI, Anthropic, or local Qwen)
## Choose an Install
### PyPI
Use PyPI for the Python packages and installed commands:
```bash
python -m pip install skillopt
skillopt-sleep --help
```
This installs `skillopt-train`, `skillopt-eval`, and `skillopt-sleep`. The wheel
does not include the repository's benchmark configs, data materializers,
agent-integration shells/MCP servers, or development tests; use a source
checkout for those files.
!!! important "PyPI versus `main`"
These docs track the latest `main`. The current PyPI release is `0.2.0`.
The generic research `openai_compatible` backend, SkillOpt-Sleep handoff,
Sleep support for non-Azure OpenAI-compatible endpoints, and the Sleep
`--preferences` flag landed after that release and require a source install
from `main` until the next release.
### Source checkout
## Quick Install
```bash
git clone https://github.com/microsoft/SkillOpt.git
cd SkillOpt
python -m pip install -e .
pip install -e .
```
Use the source checkout for paper reproduction, built-in benchmark configs,
and contributions.
## Optional Dependencies
Install extras for specific benchmarks or backends:
@@ -48,115 +20,68 @@ Install extras for specific benchmarks or backends:
=== "ALFWorld"
```bash
python -m pip install -e ".[alfworld]"
pip install -e ".[alfworld]"
```
=== "Claude agent SDK (optional)"
=== "Claude Backend"
```bash
python -m pip install -e ".[claude]"
pip install -e ".[claude]"
```
This extra does not install the `claude` executable. The research
`claude_chat` backend launches `claude -p`, so install and authenticate the
Claude Code CLI separately. The SDK extra is only needed when selecting an
SDK-backed Claude Code exec path.
=== "Qwen (Local)"
```bash
python -m pip install -e ".[qwen]"
```
=== "SearchQA data"
```bash
python -m pip install -e ".[searchqa]"
pip install -e ".[qwen]"
```
=== "WebUI"
```bash
python -m pip install -e ".[webui]"
pip install -e ".[webui]"
```
=== "Development"
```bash
python -m pip install -e ".[dev]"
pip install -e ".[dev]"
```
=== "All"
```bash
python -m pip install -e ".[alfworld,claude,qwen,searchqa,webui,docs,dev]"
pip install -e ".[alfworld,claude,qwen,webui,dev]"
```
## Environment Variables
From a source checkout, copy the template and fill in only the backend you
will use:
Copy the example `.env` file and fill in your credentials:
```bash
cp .env.example .env
```
SkillOpt does not automatically load `.env`; export it into the current shell
before running commands:
```bash
set -a
source .env
set +a
```
For Azure OpenAI with API-key authentication, the minimum settings are:
Edit `.env` with your API keys:
```ini
# Azure OpenAI (default backend)
AZURE_OPENAI_ENDPOINT=https://your-resource.openai.azure.com/
AZURE_OPENAI_API_VERSION=2024-12-01-preview
AZURE_OPENAI_API_KEY=your-key
AZURE_OPENAI_AUTH_MODE=api_key
# Or use OpenAI directly
OPENAI_API_KEY=sk-...
# Or Anthropic Claude
ANTHROPIC_API_KEY=sk-ant-...
```
Use `AZURE_OPENAI_AUTH_MODE=azure_cli` for Azure CLI credentials, or
`managed_identity` with an optional
`AZURE_OPENAI_MANAGED_IDENTITY_CLIENT_ID`.
The research `claude_chat` backend is a Claude Code CLI adapter, not a direct
Anthropic API client. Install and authenticate `claude`, and set
`CLAUDE_CLI_BIN` only if the executable is not available as `claude` on
`PATH`. `ANTHROPIC_API_KEY` is one authentication option the CLI may consume.
OpenAI-compatible servers have three distinct entry points:
1. The research engine's generic `openai_compatible` backend uses
`OPENAI_COMPATIBLE_BASE_URL`, `OPENAI_COMPATIBLE_API_KEY`, and
`OPENAI_COMPATIBLE_MODEL`.
2. The research `openai_chat` backend can use
`AZURE_OPENAI_AUTH_MODE=openai_compatible` with
`AZURE_OPENAI_ENDPOINT` and `AZURE_OPENAI_API_KEY`.
3. SkillOpt-Sleep uses the same Azure-family variables as item 2 with
`skillopt-sleep run --backend azure_openai`.
For research train/eval commands, `model.optimizer` and `model.target` in the
YAML config are applied after backend initialization. They override model-name
environment variables such as `OPENAI_COMPATIBLE_MODEL` and
`QWEN_CHAT_MODEL`; set both role models explicitly when selecting those
backends.
!!! tip
You only need to configure the backend you plan to use. See
[Configuration](configuration.md#model-backends) for exact backend names
and role-specific overrides.
You only need credentials for the backend you plan to use. Azure OpenAI is the default.
## Verify Installation
```bash
python -c "import skillopt; print('SkillOpt ready!')"
skillopt-train --help
skillopt-eval --help
skillopt-sleep --help
```
## Next Steps
docs/guide/local-env-smoke.md
+9 -15
View File
@@ -35,14 +35,9 @@ Use the split names your adapter maps to SkillOpt phases:
- `val` or `valid_seen` for selection/gating
- `test` or `valid_unseen` for final evaluation
## 2. Support a genuinely offline mock mode
## 2. Support an offline mock mode
Add a configuration flag such as `mock: true` to your adapter. In mock mode,
`rollout()` should return deterministic responses without calling external
model APIs. The inherited `EnvAdapter.reflect()` does call the configured
optimizer backend, so a no-credential smoke test must also override
`reflect()` in mock mode to return a small, schema-valid deterministic patch
(and delegate to `super().reflect(...)` otherwise).
Add a configuration flag such as `mock: true` to your adapter. In mock mode, `rollout()` should return deterministic responses without calling external model APIs.
This lets you verify the SkillOpt loop with a fast command such as:
@@ -51,14 +46,13 @@ python scripts/train.py \
--config configs/myenv/tiny_mock.yaml
```
Mock mode should still exercise the trainer's normal artifact paths, including:
Mock mode should still write the same artifacts as a real run, for example:
- `config.json`, `runtime_state.json`, and `history.json`
- `skills/skill_vXXXX.md`
- `steps/step_XXXX/ranked_edits.json`
- `steps/step_XXXX/candidate_skill.md`
- `steps/step_XXXX/step_record.json`
- the final `summary.json`
- `responses.json`
- `rollout_results.json`
- `ranked_edits.json`
- `candidate_skill.md`
- `summary.json`
## 3. Keep the smoke config tiny
@@ -133,7 +127,7 @@ For the real tiny run, verify that:
- the run completes
- `summary.json` is written
- the step directory's `ranked_edits.json` contains the expected ranking metadata
- `ranked_edits.json` contains the expected ranking metadata
- any optimizer bridge log marks the response schema as valid
- no generated files are written outside `out_root`
docs/guide/new-backend.md
+100 -170
View File
@@ -1,200 +1,130 @@
# Add a New Model Backend
SkillOpt's model layer is function-based: each chat backend is a Python module
that exposes the call, token-tracking, and deployment-setting functions used by
`skillopt.model`. There is no backend base class or registry object to subclass.
SkillOpt supports multiple LLM backends. This guide shows how to add your own.
## Built-in: the generic OpenAI-compatible backend
## Backend Architecture
!!! note "Version requirement"
This backend landed after v0.2.0. Install from the latest `main` until it is
included in the next release.
```
skillopt/model/
├── base.py # Abstract base class
├── azure_openai.py # Azure OpenAI backend
├── openai_model.py # Direct OpenAI backend
├── claude.py # Anthropic Claude backend
├── qwen.py # Local Qwen (vLLM) backend
└── your_backend.py # Your new backend
```
Before writing a new backend, check whether your provider already speaks the
OpenAI Chat Completions protocol. Most do, in which case you can use the
built-in **`openai_compatible`** backend
(`skillopt/model/openai_compatible_backend.py`) with no code changes.
## Step 1: Create the Backend
A single `base_url` + `api_key` pair lets you point SkillOpt at, for example:
| Provider | `base_url` | Example model |
|---|---|---|
| DeepSeek | `https://api.deepseek.com/v1` | `deepseek-chat` |
| Groq | `https://api.groq.com/openai/v1` | `llama-3.3-70b-versatile` |
| Together AI | `https://api.together.xyz/v1` | `meta-llama/Llama-3.3-70B-Instruct-Turbo` |
| Ollama (local) | `http://localhost:11434/v1` | `qwen2.5:7b` |
| vLLM / SGLang / TGI | `http://localhost:8000/v1` | your served model |
| LiteLLM proxy | `http://localhost:4000` | any proxied model |
| OpenRouter / Fireworks / xAI / … | provider base URL | provider model id |
### Python API
Select and configure the backend directly when embedding SkillOpt as a Python
library:
Create `skillopt/model/your_backend.py`:
```python
import skillopt.model as model
from skillopt.model.base import ModelBackend, ModelResponse
# Use the generic backend for both optimizer and target calls.
model.set_backend("openai_compatible")
model.configure_openai_compatible(
base_url="https://api.deepseek.com/v1",
api_key="sk-...",
model="deepseek-chat",
class YourBackend(ModelBackend):
"""Your custom model backend."""
def __init__(self, cfg: dict):
super().__init__(cfg)
self.model_name = cfg.get('model_name', 'your-default-model')
self.api_key = os.environ.get('YOUR_API_KEY', '')
self.client = self._init_client()
def _init_client(self):
"""Initialize API client."""
# TODO: Set up your API client
pass
async def generate(
self,
messages: list[dict],
temperature: float = 0.7,
max_tokens: int = 4096,
**kwargs
) -> ModelResponse:
"""
Generate a completion.
Args:
messages: Chat messages [{"role": "...", "content": "..."}]
temperature: Sampling temperature
max_tokens: Maximum tokens in response
Returns:
ModelResponse with content, usage, and metadata
"""
response = await self.client.chat(
model=self.model_name,
messages=messages,
temperature=temperature,
max_tokens=max_tokens,
)
return ModelResponse(
content=response.text,
usage={
'prompt_tokens': response.usage.input,
'completion_tokens': response.usage.output,
},
model=self.model_name,
)
async def generate_with_tools(
self,
messages: list[dict],
tools: list[dict],
**kwargs
) -> ModelResponse:
"""Generate with tool/function calling support."""
# Optional: implement if your model supports tool use
raise NotImplementedError("Tool use not supported")
```
`configure_openai_compatible()` also accepts `optimizer_*` and `target_*`
arguments when the two roles use different endpoints or models.
## Step 2: Register the Backend
### Environment variables
Add to `skillopt/model/__init__.py`:
The shared variables below configure both roles. Role-specific
`OPTIMIZER_OPENAI_COMPATIBLE_*` and `TARGET_OPENAI_COMPATIBLE_*` variables take
precedence:
```python
from .your_backend import YourBackend
```bash
export OPENAI_COMPATIBLE_BASE_URL="https://api.groq.com/openai/v1"
export OPENAI_COMPATIBLE_API_KEY="gsk_..."
export OPENAI_COMPATIBLE_MODEL="llama-3.3-70b-versatile"
# Optional: OPENAI_COMPATIBLE_TEMPERATURE, _MAX_TOKENS, _TIMEOUT_SECONDS
BACKEND_REGISTRY = {
# ... existing backends ...
'your_backend': YourBackend,
}
```
For direct library use, `OPTIMIZER_BACKEND=openai_compatible` and/or
`TARGET_BACKEND=openai_compatible` select the role. The training and evaluation
scripts resolve backend selection from their config, so set the split fields
explicitly there:
## Step 3: Configure
Use your backend in any config:
```yaml
model:
optimizer_backend: openai_compatible
target_backend: openai_compatible
optimizer: llama-3.3-70b-versatile
target: llama-3.3-70b-versatile
backend: your_backend
model_name: your-model-id
temperature: 0.7
max_tokens: 4096
```
Equivalently, override those fields on the command line:
Set credentials via environment variable:
```bash
python scripts/train.py --config configs/searchqa/default.yaml \
--cfg-options \
model.optimizer_backend=openai_compatible \
model.target_backend=openai_compatible \
model.optimizer=llama-3.3-70b-versatile \
model.target=llama-3.3-70b-versatile
export YOUR_API_KEY="your-key"
```
Do not rely on the legacy high-level `model.backend` label to replace the two
role-specific fields in a structured config.
## Required Interface
The generic backend uses the official `openai` SDK and the Chat Completions
API. It records token usage through the shared tracker, supports provider tool
calling through `chat_*_messages(..., tools=...)`, and exposes `count_tokens()`
(tiktoken when available, with a character-based fallback). Provider-specific
Responses API features are outside this backend's contract.
Your backend must implement these methods:
Only write a new backend when the provider is not compatible with this surface
or requires behavior that cannot be expressed by its configuration.
| Method | Required | Description |
|---|---|---|
| `generate()` | ✅ | Basic text generation |
| `generate_with_tools()` | Optional | Tool/function calling |
| `count_tokens()` | Optional | Token counting for context management |
## Backend architecture
## Tips
The active split optimizer/target dispatcher is the public
`skillopt/model/__init__.py` module:
```text
skillopt/model/
├── common.py # aliases, default models, token/response helpers
├── backend_config.py # optimizer/target whitelists and runtime selection
├── __init__.py # public API and split-role dispatch
├── openai_compatible_backend.py # generic Chat Completions example
├── qwen_backend.py # raw-HTTP chat example with per-role config
├── minimax_backend.py # compact raw-HTTP chat example
├── codex_harness.py # target-only exec harnesses
└── router.py # legacy single-backend compatibility surface
```
`router.py` is not the dispatcher used by the current training loop. Update it
only if the new backend must also be exposed through that legacy single-backend
API.
## Step 1: implement the module contract
Create a module such as `skillopt/model/your_backend.py`. Copy the signatures
from `openai_compatible_backend.py` or `qwen_backend.py`; model calls in the
current framework are synchronous.
For a chat backend that supports both roles, the public module surface is:
| Function | Purpose |
|---|---|
| `chat_optimizer(...)` | Optimizer system/user call; returns `(text, usage)` |
| `chat_target(...)` | Target system/user call; returns `(text, usage)` |
| `chat_optimizer_messages(...)` | Optimizer message-list call, including optional tools |
| `chat_target_messages(...)` | Target message-list call, including optional tools |
| `get_token_summary()` | Return per-stage counters plus `_total` |
| `reset_token_tracker()` | Clear this backend's counters |
| `set_optimizer_deployment(name)` | Change the optimizer model at runtime |
| `set_target_deployment(name)` | Change the target model at runtime |
| `set_reasoning_effort(effort)` | Apply or safely ignore the shared reasoning setting |
Every call returns a usage dict with `prompt_tokens`, `completion_tokens`, and
`total_tokens`. Use `TokenTracker` from `skillopt.model.common` and record each
call exactly once. Message-list calls that accept tools should return the
compatibility message objects from `common.py` when `return_message=True`.
Provider-specific configuration helpers and `count_tokens()` are optional, but
their state must be safe to update while calls may run concurrently. Keep
credentials out of logs and persisted artifacts.
Exec-style targets do not implement this chat contract. They are target-only
and are integrated through `codex_harness.py` plus environment-specific rollout
code.
## Step 2: register and route the backend
A new backend normally requires all of the following:
1. Add its canonical name, aliases, and default model to
`skillopt/model/common.py`.
2. Add the canonical name to the appropriate optimizer and/or target whitelist
in `skillopt/model/backend_config.py`. Do not advertise a role the module
cannot execute.
3. Import the module in `skillopt/model/__init__.py` and add dispatch branches
for every supported call surface.
4. Include its counters in `get_token_summary()` / `reset_token_tracker()` and
forward the shared deployment/reasoning setters where applicable.
5. If it has YAML settings, add structured-to-flat mappings in
`skillopt/config.py`, wire them through `scripts/train.py` and
`scripts/eval_only.py`, and document their precedence over environment
variables.
6. Update `router.py` only when legacy single-backend compatibility is part of
the intended feature.
Backend selection in `scripts/train.py` must use
`model.optimizer_backend` and `model.target_backend`. A high-level
`model.backend` alias alone is not a substitute for this explicit split.
## Step 3: test the integration
Add focused tests under `tests/` that do not call a live provider. At minimum,
cover:
- optimizer and target whitelist validation;
- routing for text and message-list calls;
- role-specific configuration precedence;
- tool-call compatibility, if supported;
- deployment/reasoning setters;
- token accounting, including a single correct `_total`;
- actionable errors for missing credentials or invalid responses.
Then run the focused test, the full suite, and the documentation build:
```bash
python -m pytest tests/test_your_backend.py -q
python -m pytest tests/ -q
mkdocs build --strict
```
Also update `.env.example`, the configuration reference, and the backend table
in the API reference. Add an optional dependency extra only when the backend
requires a package that is not already a core dependency.
!!! tip
- Test your backend with `python -c "from skillopt.model.your_backend import YourBackend"` first
- Use `async` methods for all API calls — SkillOpt uses asyncio throughout
- Implement retry logic with exponential backoff for production use
- Add your API key to `.env.example` when submitting a PR
docs/guide/new-benchmark.md
+23 -60
View File
@@ -14,18 +14,16 @@ To add a benchmark you implement four things:
1. **A `SplitDataLoader` subclass** — knows how to load train / val / test
item dicts from disk.
2. **A rollout helper** — runs the target model on a batch of items, scores
each prediction, and persists the per-item conversation consumed by the
shared reflection stage.
2. **A rollout helper** — runs the target model on a batch of items
under the current skill and scores each prediction.
3. **An `EnvAdapter` subclass** — wires the loader + rollout helper into
SkillOpt's lifecycle (`build_*_env`, `rollout`, and `get_task_types`).
The shared `reflect()` implementation is inherited unless the benchmark
needs custom reflection logic.
SkillOpt's lifecycle (`build_*_env`, `rollout`, `reflect`,
`get_task_types`).
4. **A YAML config** — references your env name plus the standard
train / optimizer / gradient knobs.
Then lazy registration in the training and evaluation scripts makes it
discoverable without importing optional dependencies at startup.
Then one line in `scripts/train.py`'s `_register_builtins()` makes it
discoverable.
---
@@ -101,8 +99,8 @@ def _score(prediction: str, ground_truth: str) -> tuple[int, float]:
return hard, soft
def _rollout_one(item: dict, skill_content: str, *, prediction_dir: Path,
max_completion_tokens: int) -> dict:
def _rollout_one(item: dict, skill_content: str,
*, max_completion_tokens: int) -> dict:
system = skill_content
user = (
f"Question: {item['question']}\n\n"
@@ -115,33 +113,14 @@ def _rollout_one(item: dict, skill_content: str, *, prediction_dir: Path,
max_completion_tokens=max_completion_tokens,
)
hard, soft = _score(prediction, item.get("ground_truth", ""))
# EnvAdapter.reflect() reads this exact trajectory path. Keep item IDs
# unique and filesystem-safe.
task_dir = prediction_dir / str(item["id"])
task_dir.mkdir(parents=True, exist_ok=True)
conversation = [
{"role": "system", "content": system},
{"role": "user", "content": user},
{"role": "assistant", "content": prediction},
]
(task_dir / "conversation.json").write_text(
json.dumps(conversation, ensure_ascii=False, indent=2),
encoding="utf-8",
)
return {
"id": str(item["id"]),
"hard": hard,
"soft": soft,
"predicted_answer": prediction,
"task_description": item.get("question", ""),
"question": item.get("question", ""),
"reference_text": item.get("reference_text", ""),
"task_type": item.get("task_type", "docfaithful"),
"target_system_prompt": system,
"target_user_prompt": user,
"n_turns": 1,
}
@@ -149,18 +128,15 @@ def run_batch(*, items: list[dict], skill_content: str, out_root: str,
workers: int = 4, max_completion_tokens: int = 4096) -> list[dict]:
"""Run a batch of episodes sequentially or with a thread pool."""
os.makedirs(out_root, exist_ok=True)
prediction_dir = Path(out_root, "predictions")
# For brevity we go sequentially — swap in concurrent.futures.ThreadPoolExecutor
# when network / model latency dominates.
results = [
_rollout_one(item, skill_content,
prediction_dir=prediction_dir,
max_completion_tokens=max_completion_tokens)
for item in items
]
Path(out_root, "rollouts.json").write_text(
json.dumps(results, ensure_ascii=False, indent=2),
encoding="utf-8",
json.dumps(results, ensure_ascii=False, indent=2)
)
return results
```
@@ -174,17 +150,9 @@ Two design points worth flagging:
- **Use `skillopt.model.chat_target`**, not raw OpenAI/Claude calls.
That routes through whichever **chat** target backend the user
configured (`openai_chat` / `claude_chat` / `qwen_chat` /
`minimax_chat` / `openai_compatible`) without your adapter caring.
Exec-style backends (`codex_exec`, `claude_code_exec`) need
environment-specific rollout code —
see `skillopt/model/codex_harness.py` together with the rollout modules in
`skillopt/envs/searchqa/`, `skillopt/envs/docvqa/`, or
`skillopt/envs/officeqa/` for working examples.
- **Persist a conversation for reflection.** The shared `EnvAdapter.reflect()`
looks under `<rollout_dir>/predictions/<result-id>/conversation.json` and
skips results whose trajectory is absent or empty. Returning `hard`/`soft`
scores alone is sufficient for evaluation, but it cannot produce learning
patches.
`minimax_chat`) without your adapter caring. Exec-style backends
(`codex_exec`, `claude_code_exec`) need env-specific rollout code —
see `skillopt/envs/swebench/` for an example.
## Step 4 — Implement the environment adapter
@@ -309,11 +277,9 @@ the answer against `item["ground_truth"]`, and returns a list of dicts:
]
```
The trainer requires `id`, `hard`, and `soft` for scoring. The remaining fields
are preserved on `RolloutResult.extras` (see `skillopt/types.py`). The shared
reflection implementation combines those fields with each persisted
`predictions/<id>/conversation.json`; without that file the result is omitted
from reflection.
The trainer only requires `id`, `hard`, `soft`. The rest is preserved on
`RolloutResult.extras` (see `skillopt/types.py`) and is what your
`reflect()` consumes via `run_minibatch_reflect`.
## Step 5 — Register the adapter
@@ -328,11 +294,9 @@ and add to `_register_builtins()`:
pass # docfaithful deps not installed — skip
```
Mirror the same lazy registration in
[`scripts/eval_only.py`](https://github.com/microsoft/SkillOpt/blob/main/scripts/eval_only.py)
so standalone evaluation can resolve the environment too. There is **no
`BENCHMARK_REGISTRY` dict in `skillopt/envs/__init__.py`**; both entry points
keep a small lazy registry so optional dependencies do not break `--help`.
There is **no `BENCHMARK_REGISTRY` dict in `skillopt/envs/__init__.py`** —
the registry lives in `scripts/train.py` and is populated lazily so that
optional deps don't break `--help`.
## Step 6 — Create the YAML config
@@ -358,7 +322,9 @@ optimizer:
env:
name: docfaithful
# Point to an existing Markdown file. Use an empty file to start blank.
# Optional: a seed skill document. Create this file (or any markdown
# file) yourself before the first run, or omit the key to let SkillOpt
# start from an empty skill.
skill_init: skillopt/envs/docfaithful/skills/initial.md
split_mode: split_dir
split_dir: data/docfaithful_split
@@ -375,7 +341,7 @@ env:
## Step 7 — Run
```bash
# Create the file referenced by env.skill_init before the first run:
# If you set skill_init above, create the seed skill first:
# mkdir -p skillopt/envs/docfaithful/skills
# echo "# DocFaithful initial skill" > skillopt/envs/docfaithful/skills/initial.md
@@ -398,11 +364,8 @@ you forgot to implement one of the four abstract methods on `EnvAdapter`:
`hard` / `soft`.
- Noisy scoring kills the optimizer. Spend time on `run_batch`'s scoring
before you spend time on prompts.
- If training repeatedly reports `skip_no_patches`, first verify that every
rollout result has a non-empty
`rollout/predictions/<id>/conversation.json` using the same `id` string.
- If your benchmark needs heavy optional deps (selenium, vllm, ...),
wrap both registration blocks with `try / except ImportError` (Step 5)
wrap the registration block with `try / except ImportError` (Step 5)
so people without those deps can still `--help`.
- Copy `skillopt/envs/_template/` as a starting skeleton — it now
implements the real abstract methods.
docs/guide/skill-document.md
+7 -30
View File
@@ -38,45 +38,23 @@ During training, the skill document is modified by **edit patches**:
2. **Modifications**: Refining existing rules that are partially correct
3. **Deletions**: Removing rules that consistently lead to errors
Selected edits are applied together to produce a candidate skill. With the
validation gate enabled, that candidate replaces the current skill only when
its score on the selection split strictly improves.
SkillOpt may maintain two protected, machine-managed regions:
```markdown
<!-- SLOW_UPDATE_START -->
... epoch-level longitudinal guidance ...
<!-- SLOW_UPDATE_END -->
<!-- APPENDIX_START -->
... skill-aware execution reminders ...
<!-- APPENDIX_END -->
```
Normal edit patches cannot modify either region. Slow update owns the first;
optional skill-aware reflection owns the second. Preserve these markers when
copying or manually inspecting a trained skill.
Each edit is validated through the **gate** mechanism before being permanently accepted.
## Initial Skill
You can start training with:
- **Empty skill**: Point `env.skill_init` to an empty Markdown file
- **Empty skill**: The system learns everything from scratch
- **Seed skill**: Provide initial instructions to bootstrap training
- **Pre-trained skill**: Transfer a skill from a related benchmark
Configure the initial skill in your YAML:
```yaml
env:
skill_init: path/to/initial_skill.md
train:
init_skill: "path/to/initial_skill.md" # or omit for empty
```
To start from scratch, create an empty Markdown file and use its path. A missing
path currently also starts blank, so using an explicit file avoids silently
treating a typo as an empty skill.
## Skill Quality Metrics
Track your skill's evolution through:
@@ -84,16 +62,15 @@ Track your skill's evolution through:
- **Validation score**: Primary metric on the selection split
- **Test score**: Final metric on held-out test data
- **Skill length**: Total tokens in the document
- **Candidate acceptance rate**: Fraction of candidate skill updates that pass
gating; multiple proposed edits can be combined into one candidate
- **Edit acceptance rate**: Fraction of proposed edits that pass gating
## Best Practices
!!! tip "Tips for better skills"
1. **Start with a seed skill** (`env.skill_init`) if you have domain knowledge — it converges faster
2. **Use cosine LR schedule** — aggressive early exploration + careful late refinement
3. **Enable slow update** (`optimizer.use_slow_update: true`) to counter forgetting across epochs
4. **Enable meta skill** (`optimizer.use_meta_skill: true`) so the optimizer accumulates strategy memory
3. **Enable slow update** (`use_slow_update: true`) to prevent forgetting across epochs
4. **Enable meta skill** (`use_meta_skill: true`) so the optimizer accumulates strategy memory
## Next Steps
docs/guide/training-loop.md
+9 -22
View File
@@ -37,11 +37,12 @@ scores = evaluate(predictions, ground_truth)
### 2. Reflect (Backward Pass)
The **optimizer** model analyzes trajectory minibatches and produces **edit
patches** — structured suggestions for improving the skill document. Failure
minibatches are always eligible for analysis; successful trajectories are also
analyzed unless `gradient.failure_only` is enabled. Independent minibatches can
run concurrently according to `gradient.analyst_workers`.
The **optimizer** model analyzes failed trajectories and produces **edit patches** — structured suggestions for improving the skill document.
Two modes:
- **Shallow**: Analyze each trajectory independently
- **Deep**: Cross-reference multiple failures to find systemic issues
```python
# Analogy: computing gradients
@@ -73,31 +74,17 @@ Selected edits are applied to the skill document, producing a new version.
### 6. Gate (Validation)
The updated skill is evaluated on a **selection split** (analogous to a
validation set). With the gate enabled, the candidate is accepted only when its
configured gate score (`hard`, `soft`, or `mixed`) is strictly higher than the
current skill's score. With `evaluation.use_gate: false`, validation is still
recorded but candidates are force-accepted.
The updated skill is evaluated on a **selection split** (analogous to a validation set). The update is only accepted if performance improves.
## Epoch Boundary Mechanisms
### Slow Update
At the end of each epoch (starting from epoch 2), the system performs a
**longitudinal comparison**: it rolls out both the previous epoch's skill and
the current skill on the same samples, categorizes items as
improved/regressed/persistent-fail/stable-success, then generates high-level
**guidance** for the skill document. Depending on
`optimizer.slow_update_gate_with_selection`, that guidance is either checked on
the selection split or applied unconditionally. Its purpose is to counter
cross-epoch forgetting.
At the end of each epoch (starting from epoch 2), the system performs a **longitudinal comparison**: it rolls out both the previous epoch's skill and the current skill on the same samples, categorizes items as improved/regressed/persistent_fail/stable_success, then generates high-level **guidance** that is injected into the skill document. This prevents catastrophic forgetting of earlier improvements.
### Meta Skill
A **meta-skill memory** accumulates high-level strategy notes across the training
run. Starting at the end of epoch 2, the optimizer compares the previous and
current epoch, writes a compact memory, and provides the prior epoch's memory as
additional context during later reflection and update stages.
A **meta-skill memory** accumulates high-level strategy notes across the entire training run. At the end of each epoch, the optimizer reflects on what changed between epochs and produces a compact memory that is provided as additional context during future reflection steps.
## Next Steps
docs/guideline.html
+988 -496
View File
File diff suppressed because it is too large Load Diff
docs/index.md
+11 -56
View File
@@ -18,19 +18,6 @@ hide:
---
## Two Complementary Workflows
| Workflow | Package / command | Use it for |
|---|---|---|
| **Research engine** | `skillopt`, `skillopt-train`, `skillopt-eval` | Train and evaluate skill documents on explicit benchmark splits. |
| **SkillOpt-Sleep (preview)** | `skillopt_sleep`, `skillopt-sleep` | Review supported coding-agent sessions and stage proposed memory/skill updates for human adoption. |
They share the idea of bounded text updates and validation, but they are
separate entry points with different configs and safety boundaries. Start with
the [SkillOpt-Sleep overview](sleep/README.md) before using real session data.
---
## How It Works
<div class="pipeline-container" markdown>
@@ -119,52 +106,29 @@ SkillOpt brings the familiar deep-learning training paradigm to agentic prompt o
| **ALFWorld** | Embodied AI | `configs/alfworld/` |
| **OfficeQA** | Enterprise QA | `configs/officeqa/` |
| **SearchQA** | Open-domain QA | `configs/searchqa/` |
| **LiveMathematicianBench** | Math reasoning | `configs/livemathematicianbench/` |
| **SpreadsheetBench** | Spreadsheet editing | `configs/spreadsheetbench/` |
---
## Model Backends
Optimizer and target roles are configured separately. Chat backends include
Azure OpenAI (`openai_chat`), the provider-neutral
`openai_compatible` backend, the Claude Code CLI (`claude_chat`), Qwen, and
MiniMax. Codex and Claude Code exec harnesses are target-only and require
adapter support. Despite its name, `claude_chat` launches `claude -p`; it is
not a direct Anthropic API client.
If a provider implements OpenAI Chat Completions, begin with the
[built-in compatible backend](guide/new-backend.md#built-in-the-generic-openai-compatible-backend)
instead of adding a new integration. See [Configuration](guide/configuration.md)
for authentication and per-role overrides.
| **LiveMathBench** | Math reasoning | `configs/livemathematicianbench/` |
| **SWEBench** | Software Engineering | `configs/swebench/` |
| + 5 more | Various | See [docs](guide/first-experiment.md) |
---
## Quick Example
```bash
# Clone and install the research checkout plus the SearchQA data extra
git clone https://github.com/microsoft/SkillOpt.git
cd SkillOpt
python -m pip install -e ".[searchqa]"
# Install
pip install -e .
# Configure credentials (choose one auth mode in .env)
cp .env.example .env
set -a; source .env; set +a
# Configure credentials
export AZURE_OPENAI_ENDPOINT="https://your-resource.openai.azure.com/"
export AZURE_OPENAI_API_KEY="your-key"
# Materialize the runnable split from the checked-in ID manifest
python scripts/materialize_searchqa.py
# Train on SearchQA into a predictable output directory
python scripts/train.py \
--config configs/searchqa/default.yaml \
--out_root outputs/searchqa_quickstart
# Train on SearchQA
python scripts/train.py --config configs/searchqa/default.yaml
# Evaluate best skill
python scripts/eval_only.py \
--config configs/searchqa/default.yaml \
--skill outputs/searchqa_quickstart/best_skill.md \
--split valid_unseen
--skill outputs/best_skill.md
```
---
@@ -203,13 +167,4 @@ python scripts/eval_only.py \
[:octicons-arrow-right-24: WebUI Guide](guide/first-experiment.md#webui)
- :material-weather-night:{ .lg .middle } **SkillOpt-Sleep**
---
Test the deployment companion with the no-provider mock path, then review
its data boundary before selecting a real backend.
[:octicons-arrow-right-24: Sleep Overview](sleep/README.md)
</div>
docs/reference/api.md
+18 -21
View File
@@ -17,9 +17,7 @@ browse [`skillopt/envs/`](https://github.com/microsoft/SkillOpt/tree/main/skillo
`skillopt/envs/base.py` — abstract adapter that connects the SkillOpt
trainer to an environment (benchmark, simulator, REST API, ...).
Subclasses **must** implement the four abstract methods below. Reflection has a
shared default implementation and only needs to be overridden for
environment-specific behavior.
Subclasses **must** implement the five abstract methods below.
```python
from abc import ABC, abstractmethod
@@ -32,10 +30,6 @@ class EnvAdapter(ABC):
def setup(self, cfg: dict) -> None: ...
def get_dataloader(self) -> BaseDataLoader | None: ...
def requires_ray(self) -> bool: ... # default False
def reflect(self, results: list[dict], skill_content: str,
out_dir: str, **kwargs) -> list[dict | None]:
"""Delegate to the shared minibatch reflection pipeline."""
...
# ── Abstract methods (subclasses MUST implement) ────────────────────
@@ -59,21 +53,26 @@ class EnvAdapter(ABC):
It MAY contain env-specific extra keys (parsed into RolloutResult.extras).
"""
@abstractmethod
def reflect(self, results: list[dict], skill_content: str,
out_dir: str, **kwargs) -> list[dict | None]:
"""Turn rollout results into a list of raw patch dicts.
Each dict (or None to drop the slot) MUST contain:
- "patch": {"edits": [...]} a Patch.to_dict() payload
- "source_type": "failure" | "success"
"""
@abstractmethod
def get_task_types(self) -> list[str]:
"""Distinct task-type strings used for stratified sampling."""
```
The default `reflect()` delegates to `run_minibatch_reflect` and returns raw
patch dicts with a `patch` payload plus a `failure` or `success` source type.
It expects each rollout to persist a non-empty trajectory at
`<rollout_dir>/predictions/<result-id>/conversation.json`; results without that
file can be scored but are skipped during reflection.
The trainer also calls several default-implemented helpers on every adapter:
The trainer also calls a few default-implemented helpers on every adapter:
`build_reference_text`, `get_reference_metadata`, `attach_reference_context`,
`select_representative_items`, and `build_env_from_batch`. Read the docstrings
in `skillopt/envs/base.py` if you need to override any of these — most
benchmarks do not.
benchmarks don't.
### `BaseDataLoader` / `SplitDataLoader`
@@ -162,17 +161,16 @@ into `RolloutResult.extras`.
### `GateResult` / `GateAction`
`skillopt/evaluation/gate.py` — the validation-gate decision types
returned for each candidate optimization step, and optionally for a separate
epoch-end slow-update candidate.
returned each epoch.
---
## Registering an environment
Environments are not registered via decorators or a `BENCHMARK_REGISTRY`
dict. The training and standalone-evaluation entry points each keep a lazy
`_ENV_REGISTRY`, populated by `_register_builtins()` in `scripts/train.py` and
`scripts/eval_only.py`. Add the environment to both entry points. See
dict. The trainer keeps a lazy registry inside `scripts/train.py` —
`_ENV_REGISTRY` — populated by `_register_builtins()`. To add a new env
you append a `try / except ImportError` block there. See
[Add a New Benchmark](../guide/new-benchmark.md) for the full step-by-step.
---
@@ -189,8 +187,7 @@ not via a base class subclass. Supported values (as of this writing):
| `claude_chat` | ✓ | ✓ |
| `qwen_chat` | ✓ | ✓ |
| `minimax_chat` | ✓ | ✓ |
| `openai_compatible` | ✓ | ✓ |
| `codex_exec` | ✓ | ✓ |
| `codex_exec` | — | ✓ |
| `claude_code_exec` | — | ✓ |
See `skillopt/model/backend_config.py` for the live whitelist and
docs/reference/cli.md
+6 -72
View File
@@ -1,17 +1,9 @@
# CLI Reference
> **Version note.** This reference tracks `main`. PyPI 0.2.0 does not yet
> include the generic research `openai_compatible` backend, Sleep handoff,
> Sleep support for non-Azure OpenAI-compatible endpoints, or the Sleep
> `--preferences` flag; use a source install from `main` for those features
> until the next release.
## Training
```bash
python scripts/train.py --config <config.yaml> [overrides...]
# Installed equivalent:
skillopt-train --config <config.yaml> [overrides...]
```
### Arguments
@@ -19,15 +11,13 @@ skillopt-train --config <config.yaml> [overrides...]
| Argument | Description |
|---|---|
| `--config` | Path to YAML config file (required) |
| `--cfg-options key=value [...]` | Override structured config parameters |
| `key=value` | Override any config parameter |
### Examples
```bash
# Basic training
python scripts/train.py \
--config configs/searchqa/default.yaml \
--out_root outputs/searchqa_run
python scripts/train.py --config configs/searchqa/default.yaml
# With overrides
python scripts/train.py \
@@ -44,8 +34,6 @@ python scripts/train.py \
```bash
python scripts/eval_only.py --config <config.yaml> --skill <skill.md>
# Installed equivalent:
skillopt-eval --config <config.yaml> --skill <skill.md>
```
### Arguments
@@ -54,8 +42,7 @@ skillopt-eval --config <config.yaml> --skill <skill.md>
|---|---|
| `--config` | Path to YAML config file (required) |
| `--skill` | Path to skill document to evaluate (required) |
| `--split` | `train`, `valid_seen`, `valid_unseen`, or `all` (default) |
| `--cfg-options` | One or more `section.key=value` overrides |
| `--split` | Evaluation split: `test` (default), `valid`, `train` |
### Examples
@@ -63,64 +50,15 @@ skillopt-eval --config <config.yaml> --skill <skill.md>
# Evaluate best skill on test set
python scripts/eval_only.py \
--config configs/searchqa/default.yaml \
--skill outputs/searchqa_run/best_skill.md \
--split valid_unseen
--skill outputs/searchqa/run_001/skills/best_skill.md
# Evaluate on validation set
python scripts/eval_only.py \
--config configs/searchqa/default.yaml \
--skill outputs/searchqa_run/best_skill.md \
--split valid_seen
--skill outputs/searchqa/run_001/skills/best_skill.md \
--split valid
```
`--skill` consumes the artifact produced by training. Unless `--out_root` is
set for evaluation, `eval_only.py` creates a separate timestamped
`outputs/eval_<env>_<model>_<timestamp>/` directory and writes
`eval_summary.json` there; it does not modify the training run directory.
For the generic OpenAI-compatible research backend, select the role backends
explicitly:
```bash
python scripts/train.py \
--config configs/searchqa/default.yaml \
--cfg-options \
model.optimizer_backend=openai_compatible \
model.target_backend=openai_compatible \
model.optimizer=deepseek-chat \
model.target=deepseek-chat
```
## SkillOpt-Sleep
```bash
skillopt-sleep <action> [options]
# Equivalent from a source checkout:
python -m skillopt_sleep <action> [options]
```
Actions are `run`, `dry-run`, `status`, `adopt`, `harvest`, `schedule`, and
`unschedule`. Common options include:
| Argument | Description |
|---|---|
| `--project PATH` | Project to evolve (default: current directory) |
| `--scope invoked\|all` | Harvest this project or all projects |
| `--source claude\|codex\|auto` | Transcript source |
| `--backend mock\|claude\|codex\|copilot\|handoff\|azure_openai` | Replay/optimizer backend |
| `--model NAME` | Backend-specific model override |
| `--preferences TEXT` | House rules supplied to reflection |
| `--lookback-hours N` | Initial transcript lookback; `0` scans all history |
| `--max-sessions N` / `--max-tasks N` | Bound the harvested workload |
| `--target-skill-path PATH` | Explicit skill document to stage/adopt |
| `--tasks-file PATH` | Replay a reviewed task JSON file instead of harvesting |
| `--edit-budget N` | Maximum bounded edits for the night |
| `--progress` / `--json` | Progress or machine-readable output |
| `--auto-adopt` | Apply an accepted staged proposal automatically |
Backend-specific setup for compatible endpoints is documented in
[OpenAI-compatible endpoints for SkillOpt-Sleep](../sleep/openai-compatible-endpoints.md).
## WebUI
```bash
@@ -130,8 +68,4 @@ python -m skillopt_webui.app [--port PORT] [--share]
| Argument | Default | Description |
|---|---|---|
| `--port` | 7860 | Port number |
| `--host` | `0.0.0.0` | Server bind address |
| `--share` | false | Create public Gradio link |
The default host binds every network interface. Use `--host 127.0.0.1` when
the dashboard should be reachable only from the local machine.
docs/reference/config.md
+53 -142
View File
@@ -1,174 +1,85 @@
# Configuration Reference
SkillOpt loads structured YAML, resolves `_base_` inheritance, and flattens
the result for the trainer. Shipped defaults live in
`configs/_base_/default.yaml`; benchmark configs override them.
Complete reference for all SkillOpt configuration parameters.
## Model and Backend Selection
Use explicit optimizer and target backends when the two roles differ or when
selecting the generic OpenAI-compatible backend.
| Backend | Optimizer | Target |
|---|:---:|:---:|
| `openai_chat` | ✓ | ✓ |
| `openai_compatible` | ✓ | ✓ |
| `claude_chat` | ✓ | ✓ |
| `qwen_chat` | ✓ | ✓ |
| `minimax_chat` | ✓ | ✓ |
| `codex_exec` | ✓ | ✓ |
| `claude_code_exec` | — | ✓ |
MiniMax currently has one shared deployment. `model.minimax_model` is applied
when MiniMax is the target; mixed-backend runs cannot independently choose a
MiniMax optimizer model and a different target model.
## Model
| Parameter | Type | Default | Description |
|---|---|---|---|
| `model.backend` | str | `azure_openai` | Backward-compatible high-level run label |
| `model.optimizer` | str | `gpt-5.5` | Optimizer deployment/model |
| `model.target` | str | `gpt-5.5` | Target deployment/model |
| `model.optimizer_backend` | str | `openai_chat` | Optimizer client path; chat backends plus `codex_exec` |
| `model.target_backend` | str | `openai_chat` | Target client path; chat or exec backend |
| `model.reasoning_effort` | str | `medium` | Shared reasoning effort |
| `model.rewrite_reasoning_effort` | str | empty | Optional full-rewrite effort override |
| `model.rewrite_max_completion_tokens` | int | `64000` | Full-rewrite output cap |
### Azure/OpenAI `openai_chat`
| Parameter | Default | Description |
|---|---|---|
| `model.azure_openai_endpoint` | empty | Shared Azure resource URL or compatibility-mode base URL |
| `model.azure_openai_api_version` | `2024-12-01-preview` | Azure API version |
| `model.azure_openai_api_key` | empty | Key for `api_key` or compatibility auth |
| `model.azure_openai_auth_mode` | empty | Config value; empty falls back to env, whose default is `azure_cli` |
| `model.azure_openai_ad_scope` | Azure Cognitive Services scope | AAD token scope |
| `model.azure_openai_managed_identity_client_id` | empty | Optional user-assigned identity client ID |
Every shared key also has an `optimizer_azure_openai_*` and
`target_azure_openai_*` form.
### Claude `claude_chat`
`claude_chat` launches an installed, authenticated Claude Code CLI with
`claude -p`; it does not instantiate an Anthropic API client. The executable
defaults to `claude` and can be overridden with `CLAUDE_CLI_BIN`.
`ANTHROPIC_API_KEY` is one authentication option understood by the CLI.
### Qwen, MiniMax, and Exec Backends
| Parameter family | Description |
|---|---|
| `model.qwen_chat_*` | Shared `base_url`, `api_key`, `temperature`, `timeout_seconds`, `max_tokens`, and `enable_thinking` |
| `model.optimizer_qwen_chat_*` / `model.target_qwen_chat_*` | Per-role Qwen overrides |
| `model.minimax_*` | MiniMax `base_url`, `api_key`, shared `minimax_model`, `temperature`, `max_tokens`, and `enable_thinking`; `minimax_model` applies when MiniMax is the target |
| `model.codex_exec_*` | Codex path, sandbox, profile, SDK mode, reasoning, network/search, and approval policy |
| `model.claude_code_exec_*` | Claude path, profile, SDK mode, effort, and thinking-token cap |
| `model.backend` | str | `azure_openai` | Backend: `azure_openai` / `openai_chat` / `claude_code_exec` / `qwen` |
| `model.optimizer` | str | `gpt-5.5` | Optimizer model (for reflection & slow update) |
| `model.target` | str | `gpt-5.5` | Target model (for rollout execution) |
| `model.reasoning_effort` | str | `medium` | Reasoning effort level |
| `model.optimizer_backend` | str | `openai_chat` | Optimizer backend: `openai_chat` / `claude_chat` / `qwen_chat` / `minimax_chat` |
| `model.target_backend` | str | `openai_chat` | Target backend: chat backends plus execution harnesses |
| `model.qwen_chat_base_url` | str | `http://localhost:8000/v1` | Shared Qwen/vLLM OpenAI-compatible endpoint |
| `model.qwen_chat_enable_thinking` | bool | `false` | Shared Qwen thinking flag |
| `model.optimizer_qwen_chat_base_url` | str | — | Optimizer-specific Qwen/vLLM endpoint; overrides shared `qwen_chat_base_url` |
| `model.target_qwen_chat_base_url` | str | — | Target-specific Qwen/vLLM endpoint; overrides shared `qwen_chat_base_url` |
## Training (`train`)
| Parameter | Type | Default | Description |
|---|---|---|---|
| `train.num_epochs` | int | `4` | Training epochs |
| `train.train_size` | int | `0` | `0` derives the size from the dataset split |
| `train.steps_per_epoch` | int | derived | Runtime field recomputed from train size, batch size, and accumulation; configured values are overwritten |
| `train.batch_size` | int | `40` | Tasks sampled per step |
| `train.accumulation` | int | `1` | Accumulation rounds per step |
| `train.seed` | int | `42` | Random seed |
| Parameter | Type | Default | DL Analogy | Description |
|---|---|---|---|---|
| `train.num_epochs` | int | 4 | Epochs | Number of training epochs |
| `train.batch_size` | int | 40 | Batch size | Tasks sampled per step |
| `train.accumulation` | int | 1 | Gradient accumulation | Accumulation rounds per step |
| `train.seed` | int | 42 | Random seed | Reproducibility seed |
## Gradient / Reflection (`gradient`)
| Parameter | Type | Default | Description |
|---|---|---|---|
| `gradient.minibatch_size` | int | `8` | Reflect minibatch size |
| `gradient.merge_batch_size` | int | `8` | Patch merge batch size |
| `gradient.analyst_workers` | int | `16` | Parallel reflection workers |
| `gradient.max_analyst_rounds` | int | `3` | Maximum analyst rounds |
| `gradient.failure_only` | bool | `false` | Reflect only on failures |
| `gradient.minibatch_size` | int | 8 | Reflect minibatch size |
| `gradient.merge_batch_size` | int | 8 | Patch merge batch size |
| `gradient.analyst_workers` | int | 16 | Parallel reflection workers |
| `gradient.max_analyst_rounds` | int | 3 | Max rounds of analyst reflection |
| `gradient.failure_only` | bool | `false` | Only reflect on failures |
## Optimizer (`optimizer`)
| Parameter | Type | Default | Description |
|---|---|---|---|
| `optimizer.learning_rate` | int | `4` | Maximum edit patches per step |
| `optimizer.min_learning_rate` | int | `2` | Floor for decaying schedules |
| `optimizer.lr_scheduler` | str | `cosine` | `constant`, `linear`, `cosine`, or `autonomous` |
| `optimizer.lr_control_mode` | str | `fixed` | `fixed`, `autonomous`, or `none` |
| `optimizer.skill_update_mode` | str | `patch` | `patch`, `rewrite_from_suggestions`, or `full_rewrite_minibatch` |
| `optimizer.use_slow_update` | bool | `true` | Epoch-boundary longitudinal update |
| `optimizer.slow_update_samples` | int | `20` | Longitudinal evaluation samples |
| `optimizer.slow_update_gate_with_selection` | bool | `false` | Gate slow-update guidance on the selection split |
| `optimizer.longitudinal_pair_policy` | str | `mixed` | `mixed`, `changed`, or `unchanged` |
| `optimizer.use_meta_skill` | bool | `true` | Cross-epoch optimizer memory |
| `optimizer.use_skill_aware_reflection` | bool | `false` | Enable skill-defect vs execution-lapse routing |
| `optimizer.skill_aware_appendix_source` | str | `both` | `both` or `failure_only` |
| `optimizer.skill_aware_consolidate_threshold` | int | `0` | Appendix compaction threshold; `0` disables it |
| Parameter | Type | Default | DL Analogy | Description |
|---|---|---|---|---|
| `optimizer.learning_rate` | int | 4 | Learning rate | Max edit patches per step (edit budget) |
| `optimizer.min_learning_rate` | int | 2 | Min LR | Min edits for decay schedulers |
| `optimizer.lr_scheduler` | str | `cosine` | LR schedule | `constant` / `linear` / `cosine` / `autonomous` |
| `optimizer.skill_update_mode` | str | `patch` | — | `patch` / `rewrite_from_suggestions` / `full_rewrite_minibatch` |
| `optimizer.use_slow_update` | bool | `true` | Momentum | Epoch-boundary longitudinal comparison & guidance |
| `optimizer.slow_update_samples` | int | 20 | — | Samples for slow update evaluation |
| `optimizer.use_meta_skill` | bool | `true` | Meta-learning | Cross-epoch optimizer-side strategy memory |
| `optimizer.longitudinal_pair_policy` | str | `mixed` | — | `mixed` / `changed` / `unchanged` |
## Evaluation (`evaluation`)
| Parameter | Type | Default | Description |
|---|---|---|---|
| `evaluation.use_gate` | bool | `true` | Accept only improvements when enabled; `false` records validation but force-accepts each candidate |
| `evaluation.gate_metric` | str | `hard` | `hard`, `soft`, or `mixed` |
| `evaluation.gate_mixed_weight` | float | `0.5` | Soft-score weight for `mixed` |
| `evaluation.use_semantic_density` | bool | `false` | Add the optional instruction-density bonus |
| `evaluation.semantic_density_weight` | float | `0.05` | Density bonus weight |
| `evaluation.leading_words` | list/str | built in | Optional custom high-influence words |
| `evaluation.sel_env_num` | int | `0` | Selection size; `0` uses the full split |
| `evaluation.test_env_num` | int | `0` | Test size; `0` uses the full split |
| `evaluation.eval_test` | bool | `true` | Run final test evaluation |
| `evaluation.use_gate` | bool | `true` | Enable validation gating (accept/reject updates) |
| `evaluation.eval_test` | bool | `true` | Run test evaluation after training |
## Environment (`env`)
| Parameter | Type | Default | Description |
|---|---|---|---|
| `env.name` | str | empty | Benchmark name |
| `env.skill_init` | str | empty | Initial skill document |
| `env.name` | str | — | Benchmark name (e.g., `searchqa`, `docvqa`) |
| `env.data_path` | str | — | Path to dataset |
| `env.skill_init` | str | — | Path to initial seed skill (optional) |
| `env.split_mode` | str | `ratio` | `ratio` or `split_dir` |
| `env.split_ratio` | str | benchmark/default | Train:validation:test ratio |
| `env.split_seed` | int | `42` | Deterministic split seed |
| `env.split_dir` | str | empty | Materialized train/val/test directory |
| `env.data_path` | str | empty | Raw data path for ratio mode |
| `env.split_output_dir` | str | empty | Optional materialized split output |
| `env.exec_timeout` | int | `120` | Per-task timeout in seconds |
| `env.out_root` | str | generated by the train/eval CLIs | Output directory |
| `env.split_ratio` | str | `2:1:7` | Train:val:test ratio |
| `env.exec_timeout` | int | 120 | Per-task timeout in seconds |
| `env.out_root` | str | — | Output directory |
Benchmark-specific `env` keys are passed through to the adapter.
## Credential Environment Variables
### Azure-family backend
## Azure OpenAI Credentials
| Variable | Description |
|---|---|
| `AZURE_OPENAI_ENDPOINT` | Shared Azure endpoint or compatibility base URL |
| `AZURE_OPENAI_API_VERSION` | Azure API version |
| `AZURE_OPENAI_AUTH_MODE` | `api_key`, `azure_cli`, `managed_identity`, or `openai_compatible` |
| `AZURE_OPENAI_API_KEY` | Key for `api_key` or `openai_compatible` mode |
| `AZURE_OPENAI_AD_SCOPE` | Optional AAD scope |
| `AZURE_OPENAI_MANAGED_IDENTITY_CLIENT_ID` | Optional managed-identity client ID |
Use `OPTIMIZER_AZURE_OPENAI_*` and `TARGET_AZURE_OPENAI_*` for role-specific
overrides.
### Generic OpenAI-compatible backend
| Variable suffix | Shared / per-role forms |
|---|---|
| `BASE_URL` | `OPENAI_COMPATIBLE_BASE_URL`, `OPTIMIZER_OPENAI_COMPATIBLE_BASE_URL`, `TARGET_OPENAI_COMPATIBLE_BASE_URL` |
| `API_KEY` | Corresponding shared/optimizer/target `*_API_KEY` names |
| `MODEL` | Corresponding shared/optimizer/target `*_MODEL` names |
| `TEMPERATURE` | Corresponding shared/optimizer/target `*_TEMPERATURE` names |
| `MAX_TOKENS` | Corresponding shared/optimizer/target `*_MAX_TOKENS` names |
| `TIMEOUT_SECONDS` | Corresponding shared/optimizer/target `*_TIMEOUT_SECONDS` names |
The train/eval entry points set deployments from YAML `model.optimizer` and
`model.target` after backend initialization. For selected OpenAI-compatible or
Qwen roles, those values override the corresponding `*_MODEL` environment
variables; the environment model names mainly seed direct library use.
Other backend families use the authenticated Claude CLI (`CLAUDE_CLI_BIN`;
optionally `ANTHROPIC_API_KEY`), `QWEN_CHAT_*`, and `MINIMAX_*`.
SkillOpt-Sleep's compatible endpoint uses `AZURE_OPENAI_*`, not the research
backend's `OPENAI_COMPATIBLE_*`; see
[the Sleep endpoint guide](../sleep/openai-compatible-endpoints.md).
| `AZURE_OPENAI_ENDPOINT` / `model.azure_openai_endpoint` | Azure resource endpoint |
| `AZURE_OPENAI_API_KEY` / `model.azure_openai_api_key` | Azure API key |
| `OPENAI_API_KEY` | OpenAI API key (for `openai_chat` backend) |
| `ANTHROPIC_API_KEY` | Anthropic API key (for `claude_code_exec` backend) |
| `QWEN_CHAT_BASE_URL` | Shared local vLLM endpoint for `qwen_chat` |
| `QWEN_CHAT_MODEL` | Shared served model name for `qwen_chat` |
| `QWEN_CHAT_API_KEY` | Optional API key for the shared Qwen endpoint |
| `OPTIMIZER_QWEN_CHAT_BASE_URL` | Optimizer-specific local vLLM endpoint |
| `OPTIMIZER_QWEN_CHAT_MODEL` | Optimizer-specific served model name |
| `TARGET_QWEN_CHAT_BASE_URL` | Target-specific local vLLM endpoint |
| `TARGET_QWEN_CHAT_MODEL` | Target-specific served model name |
docs/sleep/README.md
+17 -39
View File
@@ -4,11 +4,11 @@
local coding agent a nightly **sleep cycle** that reviews your past sessions, replays
your recurring tasks on your own API budget, and consolidates what it learns into
**validated** long-term memory and skills — behind a held-out gate, staged for your
review. It requires **no weight training** and adds no separate optimization loop to
normal agent requests.
review. The agent gets better the more you use it, with **no weight training** and
**zero inference-time overhead**.
> **Preview.** This is an early preview we are actively iterating on; interfaces and
> defaults may change. The engine lives in the top-level [`skillopt_sleep/`](https://github.com/microsoft/SkillOpt/tree/main/skillopt_sleep)
> defaults may change. The engine lives in the top-level [`skillopt_sleep/`](../../skillopt_sleep)
> package with **zero dependency** on the paper's `skillopt/` code (the validation gate
> is vendored).
@@ -26,49 +26,29 @@ It synthesizes **SkillOpt** (validation-gated bounded text edits), **Claude Drea
(offline consolidation; review-then-adopt), and the **agent-sleep** idea (short-term
experience → long-term competence).
> **Data boundary.** Harvesting is local and read-only. The `mock` backend makes no
> provider calls. A real backend, however, sends truncated excerpts from harvested
> sessions and derived tasks to the provider you select for mining, replay, judging,
> and reflection. Outbound prompts are not currently guaranteed to be secret-free;
> review your transcript source and provider policy before running on sensitive
> projects. For a reviewable workflow, harvest to a task file, inspect/redact it, mark
> it `"reviewed": true`, and then replay that file with the real backend.
## How to use it
### Quickest path: the `skillopt-sleep` CLI (pip)
```bash
pip install skillopt # installs the engine + the `skillopt-sleep` command
skillopt-sleep dry-run # harvest + mine + replay, report only; stages nothing
skillopt-sleep dry-run # harvest + mine + replay, report only (changes nothing)
skillopt-sleep run # a full nightly cycle; the proposal is staged for review
skillopt-sleep status # show state + the latest staged proposal
skillopt-sleep adopt # apply the latest staged proposal
skillopt-sleep schedule # install a nightly cron entry for this project
```
> **Version note.** This page tracks `main`. PyPI 0.2.0 provides the base
> commands above. Sleep handoff, non-Azure OpenAI-compatible endpoints, and
> `--preferences` landed later and require a source install from `main` until
> the next release.
The per-agent plugin shells below (Claude Code / Codex / Copilot) still come from the
repo; the CLI above is the standalone, pip-only way to run a cycle.
The per-agent integrations below still come from the repo; the CLI above is the
standalone, pip-only way to run a cycle. Claude Code, Codex, Copilot, and Devin wrap
the shared engine. OpenClaw is a separate reference adaptation and has its own setup.
One engine, thin per-agent shells (see [`plugins/`](https://github.com/microsoft/SkillOpt/tree/main/plugins)):
One engine, thin per-agent shells (see [`plugins/`](../../plugins)):
| Platform | Folder | Install |
|---|---|---|
| **Claude Code** | [`plugins/claude-code`](https://github.com/microsoft/SkillOpt/tree/main/plugins/claude-code) | `/plugin marketplace add ./plugins/claude-code` → `/skillopt-sleep` |
| **Codex** | [`plugins/codex`](https://github.com/microsoft/SkillOpt/tree/main/plugins/codex) | `bash plugins/codex/install.sh` → `skillopt-sleep` skill |
| **Copilot** | [`plugins/copilot`](https://github.com/microsoft/SkillOpt/tree/main/plugins/copilot) | register `plugins/copilot/mcp_server.py` as an MCP server |
| **Devin** | [`plugins/devin`](https://github.com/microsoft/SkillOpt/tree/main/plugins/devin) | register `plugins/devin/mcp_server.py` as an MCP server |
| **OpenClaw** | [`plugins/openclaw`](https://github.com/microsoft/SkillOpt/tree/main/plugins/openclaw) | adapt the reference wrapper and paths for your installation |
To use DeepSeek, vLLM, Ollama, or another Chat Completions server, see
**[OpenAI-compatible endpoints](openai-compatible-endpoints.md)**. That guide also
documents the separate HTTPS-only boundary for Azure managed-identity credentials.
| **Claude Code** | [`plugins/claude-code`](../../plugins/claude-code) | `/plugin marketplace add ./plugins/claude-code` → `/skillopt-sleep` |
| **Codex** | [`plugins/codex`](../../plugins/codex) | `bash plugins/codex/install.sh` → `skillopt-sleep` skill |
| **Copilot** | [`plugins/copilot`](../../plugins/copilot) | register `plugins/copilot/mcp_server.py` as an MCP server |
Deterministic proof (no API key):
`python -m skillopt_sleep.experiments.run_experiment --persona researcher --assert-improves`.
@@ -91,12 +71,11 @@ correctness signal; the validation gate still governs what ships.
> scaling, and the dream-diversity ablation — are in
> [`docs/sleep/RESULTS.md`](RESULTS.md).** The highlights:
**Controlled experiment recipe (not the shipping CLI defaults).** 5 nights × 10 new
real "today" tasks per night; the full held-out **test** split is scored before night
1 (baseline) and after night 5 (after); optimizer = GPT-5.5; single seed (42). The
experiments use the shipped consolidation and gate components, while the nightly CLI
and benchmark harnesses remain separate entry points. Numbers are absolute held-out
accuracy; **Δ** = `after − baseline` in percentage points.
**Protocol (identical for every row below).** 5 nights × 10 new real "today" tasks
per night; the full held-out **test** split is scored before night 1 (baseline) and
after night 5 (after); optimizer = GPT-5.5; single seed (42); run through the exact
shipped engine (`skillopt_sleep.dream.dream_consolidate`). Numbers are absolute
held-out accuracy; **Δ** = `after − baseline` in percentage points.
**(a) End-to-end on real agents — [gbrain-evals](https://github.com/garrytan/gbrain-evals) `skillopt-v1`.**
Deficient seed skills go **0.00 → 1.00** on the held-out set with **both Claude Code
@@ -127,6 +106,5 @@ gate keeps the worst case bounded; keep it **on** by default.
## Learn more
See the [SkillOpt documentation index](../index.md), the
[CLI reference](../reference/cli.md), and the integration-specific READMEs under
[`plugins/`](https://github.com/microsoft/SkillOpt/tree/main/plugins).
Full reference (pipeline, the three plugins, the experience-replay knobs) is in the
**[Documentation & Reproduction Guide](https://microsoft.github.io/SkillOpt/docs/guideline.html#sleep)**.
docs/sleep/RESULTS.md
+17 -23
View File
@@ -2,9 +2,8 @@
This is the evidence behind SkillOpt-Sleep: does a nightly, offline sleep cycle
actually make a *deployed* agent better, and is it safe to run unattended? We
answer with a controlled deployment-scale study built from the same shipped
consolidation and gate components. Its multi-night benchmark recipe is an
experiment configuration, not the default configuration of the nightly CLI.
answer with a controlled deployment-scale study — the same protocol the plugin
runs in production, scored on full held-out test sets.
## Setup
@@ -12,10 +11,9 @@ experiment configuration, not the default configuration of the nightly CLI.
**10 new real "today" tasks**; the skill carries over and is refined night to
night. The full held-out **test** split is scored before night 1 (*baseline*) and
after night 5 (*after*); **Δ = after − baseline** in percentage points. Optimizer
model = **GPT-5.5**; single seed (42). The measurements use the shipped replay,
consolidation, and gate implementations. The nightly CLI and the checked-in
benchmark convenience harnesses are separate entry points and do not all call one
shared wrapper function.
model = **GPT-5.5**; single seed (42); every number is produced by the exact
shipped engine `skillopt_sleep.dream.dream_consolidate` (the experiment harness and
the plugin cycle call the same function).
**Benchmarks** (real evaluators, not format heuristics):
@@ -108,31 +106,27 @@ Replay-policy ablation (SearchQA, GPT-5.5):
| Replay policy | Gate-free Δ | Gated Δ |
|---|---|---|
| none (tonight's tasks only) | +3.9 | +2.0 |
| **recall k=10 (opt-in experiment)** | +5.1 | +4.4 |
| **recall k=10 (shipped default-able)** | +5.1 | +4.4 |
| cumulative (full history) | +4.8 | +6.0 |
Recall captures most of cumulative's benefit at a fraction of the per-night cost.
---
## 4. Sensitivity around the experiment recipe
## 4. Default hyperparameters are the sweet spot
We swept `dream_factor`, `rollouts`, `per_night`, and `nights` on the nano cell
(SearchQA, gated) around the study recipe: `dream_factor=2`, `rollouts=5`,
`per_night=10`, and `nights=5`. These are **experiment values**, not the shipping
defaults (`dream_factor=0`, `dream_rollouts=1`, and `recall_k=0`):
(SearchQA, gated) to verify the shipped defaults are well-tuned:
| Variant | Δ | vs experiment baseline (+11.9) |
| Variant | Δ | vs default (+11.9) |
|---|---|---|
| dream_factor=4 (baseline 2) | +8.8 | −3.1 |
| rollouts=10 (baseline 5) | +9.5 | −2.4 |
| per_night=15 (baseline 10) | +2.7 | −9.2 |
| nights=8 (baseline 5) | +9.5 | −2.4 |
| dream_factor=4 (default 2) | +8.8 | −3.1 |
| rollouts=10 (default 5) | +9.5 | −2.4 |
| per_night=15 (default 10) | +2.7 | −9.2 |
| nights=8 (default 5) | +9.5 | −2.4 |
Every tested direction away from that baseline reduced the measured gain in this
cell. The result supports that particular study recipe; it does not establish a
universal optimum. Shipping stays conservative, and users must opt in to additional
dream rollouts or recall after considering task quality and provider cost.
Every direction away from the default hurts. This means users get the best result
**out of the box** without tuning — the recipe is robust by design.
---
@@ -149,7 +143,7 @@ gains in Sections 1–2. Measured across an 18-cell deployment sweep (3 benchmar
|---|---|---|---|---|
| single-sample reflection (degraded) | −2.66 | **−52.8** | 7 / 18 | 5 / 18 |
| diverse rollouts (K=5), no recall | +0.24 | −4.0 | 6 / 18 | 7 / 18 |
| **diverse rollouts + recall (experiment recipe)** | **+0.53** | **−2.4** | 7 / 18 | 7 / 18 |
| **diverse rollouts + recall (shipped)** | **+0.53** | **−2.4** | 7 / 18 | 7 / 18 |
The catastrophic −52.8 is removed **at its source** by diverse rollouts: the same
gate-free nano-SearchQA cell goes 0.554 → **0.586 (+2.7)** with no gate at all once
@@ -188,4 +182,4 @@ cross-verify each other's consolidated skills.
---
Back to the module overview: [`docs/sleep/README.md`](README.md) ·
documentation index: [SkillOpt documentation](../index.md).
full reference: [Documentation & Reproduction Guide](https://microsoft.github.io/SkillOpt/docs/guideline.html#sleep).
docs/sleep/examples/runner.py
-105
View File
@@ -1,105 +0,0 @@
#!/usr/bin/env python3
"""Reference launcher for running SkillOpt-Sleep against an OpenAI-compatible
endpoint (DeepSeek shown here), plus an Antigravity `session-end` hook.
This is a *sanitized example*, not a supported entry point. Adapt the paths and
provider details to your environment. No API keys are hardcoded — the key is read
from an .env file or the process environment.
Usage:
python runner.py run # run a full sleep cycle against DeepSeek
python runner.py dry-run # harvest + replay, report only
python runner.py session-end # Antigravity Stop-hook: append rollout evidence
"""
import os
import re
import sys
import json
import subprocess
import datetime
from pathlib import Path
# --- Configure these for your environment -----------------------------------
# Path to a file containing your provider key as `sk-...` (kept out of source).
PROVIDER_ENV_FILE = Path(os.environ.get("SKILLOPT_PROVIDER_ENV_FILE", "provider.env"))
# Endpoint + model for the OpenAI-compatible provider.
PROVIDER_ENDPOINT = os.environ.get("SKILLOPT_PROVIDER_ENDPOINT", "https://api.deepseek.com")
PROVIDER_MODEL = os.environ.get("SKILLOPT_PROVIDER_MODEL", "deepseek-v4-pro")
# Project whose SKILL.md files the sleep cycle should evolve.
PROJECT_DIR = os.environ.get("SKILLOPT_PROJECT_DIR", os.getcwd())
# Where the session-end hook appends rollout evidence.
ROLLOUT_LOG = Path(os.environ.get("SKILLOPT_ROLLOUT_LOG", "brain/rollout-evidence.jsonl"))
# ----------------------------------------------------------------------------
def load_provider_key(env: dict) -> None:
"""Ensure DEEPSEEK_API_KEY is set, reading it from PROVIDER_ENV_FILE if needed."""
if env.get("DEEPSEEK_API_KEY"):
return
try:
text = PROVIDER_ENV_FILE.read_text(encoding="utf-8")
except OSError:
return
m = re.search(r"sk-[A-Za-z0-9]+", text)
if m:
env["DEEPSEEK_API_KEY"] = m.group(0)
def main() -> None:
if len(sys.argv) < 2:
print("Usage: runner.py [dry-run|run|status|adopt|session-end]")
sys.exit(1)
command = sys.argv[1]
# Antigravity Stop-hook: enrich future nights with task-outcome metadata.
if command == "session-end":
ROLLOUT_LOG.parent.mkdir(parents=True, exist_ok=True)
outcome = {
"timestamp": datetime.datetime.now().isoformat(),
"event": "SessionEnd",
"metadata": "Appended task outcome metadata",
}
with open(ROLLOUT_LOG, "a", encoding="utf-8") as f:
f.write(json.dumps(outcome) + "\n")
print("Rollout evidence metadata appended.")
return
env = os.environ.copy()
load_provider_key(env)
if env.get("DEEPSEEK_API_KEY"):
# OpenAI-compatible path — see docs/sleep/openai-compatible-endpoints.md
backend = "azure_openai"
env["PYTHONIOENCODING"] = "utf-8"
env["AZURE_OPENAI_AUTH_MODE"] = "openai_compatible"
env["AZURE_OPENAI_ENDPOINT"] = PROVIDER_ENDPOINT
env["AZURE_OPENAI_API_KEY"] = env["DEEPSEEK_API_KEY"]
# Provider-specific request fields are opt-in, never inferred from the
# model name. For DeepSeek reasoning models, enable the thinking channel:
env.setdefault("SKILLOPT_SLEEP_CHAT_EXTRA_BODY",
json.dumps({"thinking": {"type": "enabled"}}))
env.setdefault("SKILLOPT_SLEEP_COMPAT_MAX_TOKENS", "8192")
else:
# OPTIONAL, UNVERIFIED fallback: route the `claude` CLI backend through a
# local Anthropic-compatible proxy (e.g. LiteLLM) to reach Gemini. There
# is no native Gemini backend; this path was not validated. See the doc.
backend = "claude"
if "ANTHROPIC_API_KEY" not in env and "GEMINI_API_KEY" in env:
env["ANTHROPIC_API_KEY"] = env["GEMINI_API_KEY"]
env.setdefault("ANTHROPIC_BASE_URL", "http://127.0.0.1:4000")
args = ["skillopt-sleep", command]
if command in ("run", "dry-run"):
args = ["skillopt-sleep", command, "--backend", backend,
"--model", PROVIDER_MODEL, "--project", PROJECT_DIR]
print(f"Running: {' '.join(args)}")
# Propagate the child's exit code so supervisors (watchdog.py, systemd,
# Task Scheduler) see a failed sleep run as a failure, not a success.
proc = subprocess.run(args, env=env, check=False)
sys.exit(proc.returncode)
if __name__ == "__main__":
main()
docs/sleep/examples/watchdog.py
-56
View File
@@ -1,56 +0,0 @@
#!/usr/bin/env python3
"""Minimal supervisor that runs the SkillOpt-Sleep cycle on a fixed interval.
Sanitized example (see docs/sleep/openai-compatible-endpoints.md). On Windows,
register this under a Scheduled Task so it survives logout; on Linux/macOS a
systemd timer or cron entry serves the same purpose and is usually preferable to
a long-lived process.
"""
import os
import sys
import time
import subprocess
import datetime
import traceback
INTERVAL_SECONDS = int(os.environ.get("SKILLOPT_WATCHDOG_INTERVAL", str(4 * 3600)))
RUNNER = os.environ.get("SKILLOPT_RUNNER", os.path.join(os.path.dirname(__file__), "runner.py"))
LOG_FILE = os.environ.get("SKILLOPT_WATCHDOG_LOG", "brain/watchdog.log")
def log(msg: str) -> None:
os.makedirs(os.path.dirname(LOG_FILE) or ".", exist_ok=True)
line = f"[{datetime.datetime.now().isoformat()}] {msg}"
with open(LOG_FILE, "a", encoding="utf-8") as f:
f.write(line + "\n")
print(line)
def run_once() -> None:
log("Invoking skillopt-sleep run via runner.py...")
try:
result = subprocess.run([sys.executable, RUNNER, "run"],
capture_output=True, text=True)
if result.returncode == 0:
log("Successfully completed run.")
else:
log(f"Run failed (exit {result.returncode}).")
log(f"STDERR: {result.stderr}")
except Exception as e:
log(f"Exception while running skillopt: {e}")
log(traceback.format_exc())
def main() -> None:
log(f"Watchdog started. Interval: {INTERVAL_SECONDS}s.")
while True:
try:
run_once()
except Exception as e:
log(f"Unexpected error in watchdog loop: {e}")
log(f"Sleeping for {INTERVAL_SECONDS}s...")
time.sleep(INTERVAL_SECONDS)
if __name__ == "__main__":
main()
docs/sleep/openai-compatible-endpoints.md
-173
View File
@@ -1,173 +0,0 @@
# OpenAI-compatible endpoints for SkillOpt-Sleep (DeepSeek, local vLLM, …)
This document describes the `azure_openai` backend in
`skillopt_sleep/backend.py`, which can drive servers that implement the expected
OpenAI-compatible Chat Completions request shape — for example DeepSeek's hosted
API or a self-hosted vLLM/Ollama server — in addition to native Azure OpenAI
deployments. The included runner is a sanitized unattended-launch example that
was originally used alongside Antigravity; it is not an Antigravity transcript
integration.
> **Version requirement.** This capability landed after v0.2.0. Until the next
> release, install SkillOpt from the latest `main`; the current PyPI 0.2.0
> package does not provide this compatible-endpoint path.
## What changed
All changes are backward-compatible — the default managed-identity Azure path
is unchanged:
1. **CLI acceptance.** `skillopt-sleep run --backend azure_openai` is now an
accepted choice in `skillopt_sleep/__main__.py` (it was previously rejected
by argparse even though `get_backend()` understood the name).
2. **Endpoint resolution honors `AZURE_OPENAI_ENDPOINT`.**
`AzureOpenAIBackend.__init__` resolves the endpoint as `explicit arg` →
`AZURE_OPENAI_ENDPOINT` env → the built-in `_AZURE_ENDPOINTS` table.
Previously a non-Azure endpoint could not be supplied at all.
3. **`openai_compatible` auth mode.** When
`AZURE_OPENAI_AUTH_MODE=openai_compatible` (also accepts `compat`/`openai`),
`_get_client()` builds a plain `openai.OpenAI(base_url=…)` client with
`AZURE_OPENAI_API_KEY` instead of an `AzureOpenAI` client. This mirrors the
auth mode already supported by the sibling `skillopt/model/azure_openai.py`
module. (The `AzureOpenAI` client rewrites request URLs with Azure-only
`?api-version=…` query params and deployment path segments, which non-Azure
servers reject with `404 Resource not found` — the sleep cycle then scores
every rollout `0.0` with no diagnostic.)
4. **Managed-identity credential guard.** The managed-identity path attaches an
Azure AD bearer token to every request. It therefore accepts only an **HTTPS**
endpoint whose hostname ends in `*.openai.azure.com` or
`*.cognitiveservices.azure.com`. An HTTP endpoint — even one with an
Azure-looking hostname — and any host outside those suffixes are rejected
before a credential-bearing client is created.
5. **Provider-neutral request shape.** In compat mode the backend sends only the
standard OpenAI-compatible contract (`model`, `messages`, `max_tokens`).
Provider-specific request fields are **opt-in** via environment variables
(below) and are attached only in compat mode — nothing is inferred from
model-name substrings, and the native Azure request remains unchanged.
6. **Reliable error state.** `_call()` records the last exception in
`self.last_call_error` (surfaced in `diagnostics.json`), clears it when a
retry recovers, and sets an explicit `"empty response on all N attempts"`
diagnostic when every attempt returns empty text.
## Configuration reference
SkillOpt-Sleep's `azure_openai` backend reads these environment variables
(unprefixed only — the `OPTIMIZER_*`/`TARGET_*` dual-role variables belong to
the separate `skillopt.model.azure_openai` module and are **not** used by the
sleep cycle):
| Variable | Meaning |
|---|---|
| `AZURE_OPENAI_AUTH_MODE` | `openai_compatible` (or `compat`/`openai`) selects the plain OpenAI client. Unset/other = Azure managed identity (default). |
| `AZURE_OPENAI_ENDPOINT` | Base URL of the server, e.g. `https://api.deepseek.com`. Azure managed identity requires HTTPS plus an approved Azure hostname. |
| `AZURE_OPENAI_API_KEY` | API key sent by the compat client to the configured base URL. |
| `SKILLOPT_SLEEP_COMPAT_MAX_TOKENS` | Optional int (default `8192`): `max_tokens` sent in compat mode. |
| `SKILLOPT_SLEEP_CHAT_EXTRA_BODY` | Optional JSON object passed as `extra_body` for provider-specific fields in compat mode only. It is ignored in native Azure mode. |
## Data and transport boundaries
- Harvesting reads local transcripts without modifying them, and the `mock`
backend makes no provider calls. A real backend sends **truncated transcript
excerpts and derived task content** to the selected provider for mining,
replay, judging, and reflection.
- Outbound prompts are not currently guaranteed to be free of secrets. Review
the provider's data policy and avoid a third-party endpoint for sensitive
transcripts unless you have first inspected and redacted the task material.
One reviewable path is `skillopt-sleep harvest --output tasks.json`, followed
by a reviewed `--tasks-file` run.
- Use HTTPS for every remote compatible provider. Plain HTTP is appropriate only
for an explicitly trusted loopback development server such as
`http://127.0.0.1:8000/v1`; the compat client sends its API key to the configured
URL.
- Azure managed-identity credentials have the stricter invariant described
above: HTTPS **and** an approved Azure hostname are both mandatory.
## How to use it
```bash
export AZURE_OPENAI_AUTH_MODE=openai_compatible
export AZURE_OPENAI_ENDPOINT=https://api.deepseek.com # DeepSeek base URL
export AZURE_OPENAI_API_KEY=sk-... # your provider key
# DeepSeek reasoning models: enable the thinking channel (opt-in, not inferred)
export SKILLOPT_SLEEP_CHAT_EXTRA_BODY='{"thinking": {"type": "enabled"}}'
export SKILLOPT_SLEEP_COMPAT_MAX_TOKENS=8192
skillopt-sleep run \
--backend azure_openai \
--model deepseek-v4-pro \
--project /path/to/your/project
```
The same pattern works for a server that implements this Chat Completions
contract: point `AZURE_OPENAI_ENDPOINT` at the provider-specific base URL, set a
matching `--model`, and omit `SKILLOPT_SLEEP_CHAT_EXTRA_BODY` unless the provider
needs extra request fields. Self-hosted vLLM and Ollama commonly use a `/v1` base
path, for example `http://127.0.0.1:8000/v1` or
`http://127.0.0.1:11434/v1`.
`--project` selects the project/transcript scope and the project `CLAUDE.md`; it
does **not** by itself select an arbitrary project `SKILL.md`. Pass
`--target-skill-path path/to/SKILL.md` when a specific skill is the optimization
target. Without that flag, SkillOpt-Sleep uses its configured managed skill.
## Unattended runner example (originally used with Antigravity)
The [`examples/`](https://github.com/microsoft/SkillOpt/tree/main/docs/sleep/examples) directory contains a sanitized reference for running
the compatible backend unattended:
- **`examples/runner.py`** — a thin launcher that loads a provider key from an
`.env` file, exports the variables above, invokes `skillopt-sleep run` with
the DeepSeek backend, and **exits with the child's return code** so
supervisors see failures as failures. Its `session-end` action writes a small
local rollout-evidence event as an example hook target.
- **`examples/watchdog.py`** — a minimal supervisor loop that invokes the runner
on a fixed interval (e.g. every 4 hours) and logs non-zero exits as failures.
On Windows this is registered as a Scheduled Task so it survives logout; on
Linux/macOS a `systemd` timer or cron entry serves the same role.
The current engine does **not** read `brain/rollout-evidence.jsonl`, and it does
not harvest Antigravity transcripts. That hook output is illustrative metadata,
not additional training evidence. A real run must use a supported Claude
Code/Codex transcript source or a reviewed task file converted by the operator.
### Contributor-reported validation
The contributor reported the following results from a private Windows 11 setup
driving the cycle against `deepseek-v4-pro` in `openai_compatible` mode. They are
useful integration evidence, but the private session set is not a reproducible
benchmark bundled with this repository:
- A direct backend smoke test returns a live completion (no `404`,
`last_call_error` empty, client type `OpenAI`).
- A full nightly cycle using the configured session source moved the held-out
validation gate from `0.250 → 1.000`, **accepting** a DeepSeek-authored
skill edit (`accept_new_best`). `diagnostics.json` for that night reports
`"backend": "azure_openai"` with a non-empty token count and an empty
`call_error` — i.e. a genuine optimization night, versus the prior all-`0.0`
nights that the endpoint bug produced.
- A subsequent unattended night triggered by the watchdog completed the full
chain (watchdog → runner → `skillopt-sleep` → DeepSeek) and the gate correctly
**rejected** a non-improving proposal (`0.3 → 0.3`), confirming the validation
gate behaves normally on the new backend.
Deterministic no-network coverage for the new behavior lives in
`tests/test_azure_openai_compat.py` (CLI acceptance, client selection,
endpoint/auth guard, request kwargs, retry error-state, empty-response
diagnostics, and runner exit-code propagation).
## Unsupported Gemini proxy branch in the example
`examples/runner.py` still contains an illustrative branch that routes the
**`claude` CLI backend** through a loopback Anthropic-compatible proxy such as
[LiteLLM](https://github.com/BerriAI/litellm). It is not a native Gemini backend,
has no validated model mapping in this example, and is not part of the supported
path documented here. The sample currently enters that branch whenever no
DeepSeek key is found, so a production adaptation should remove it or replace it
with an explicit opt-in, a separately configured model, and a trusted isolated
loopback proxy. Do not treat this branch as tested Gemini support.
docs/superpowers/specs/2026-06-07-skillopt-sleep-claude-code-plugin-design.md
+1 -8
View File
@@ -1,13 +1,5 @@
# SkillOpt Sleep — Claude Code self-evolving plugin (design)
> **Historical design proposal.** This document records the June 2026 design
> target and includes planned controls that are not part of the current nightly
> CLI. It is not an installation or configuration reference. For implemented
> behavior, flags, defaults, and data boundaries, use
> [`docs/sleep/README.md`](../../sleep/README.md),
> [`docs/reference/cli.md`](../../reference/cli.md), and
> [`plugins/README.md`](https://github.com/microsoft/SkillOpt/blob/main/plugins/README.md).
**Status:** approved-for-build (autonomous offline session, 2026-06-07)
**Author:** generated for Yifan Yang, executed autonomously while user is asleep
**Branch:** `feat/claude-code-sleep-plugin` (worktree `my_repo/SkillOpt-sleep`)
@@ -242,3 +234,4 @@ This session targets **Phase 0 + Phase 1 fully**, **Phase 2 scaffolded**, and th
3. **Real-API demo:** want me to spend live `ANTHROPIC_API_KEY` budget on the persona demo, or keep everything mock until you say go?
4. **Skill target:** evolve a *new* dedicated `skillopt-sleep`-managed skill, or also edit your existing hand-written skills in `~/.claude/skills`?
5. **Paper:** should this become a section/figure in the SkillOpt arXiv (Dream+Sleep framing as "deployment-time continual skill optimization")?
```
index.html
+2 -4
View File
@@ -1778,8 +1778,6 @@
<a href="#evolution">Evolution</a>
<a href="#transfer">Transfer</a>
<a href="#citation">Citation</a>
<a href="https://microsoft.github.io/SkillOpt/blog/">Blog</a>
<a href="https://github.com/microsoft/SkillOpt/blob/main/docs/index.md" target="_blank" rel="noopener">Docs</a>
<a href="https://github.com/microsoft/SkillOpt" target="_blank" rel="noopener">Code</a>
</nav>
</header>
@@ -1917,7 +1915,7 @@
<h3>A skill is external state for an agent.</h3>
<p>
Instead of fine-tuning a model or hand-maintaining prompts, SkillOpt runs
the frozen agent on scored batches, asks an optimizer model to
the frozen agent on scored batches, asks a separate optimizer model to
propose structured edits, and accepts a candidate only when validation
performance improves.
</p>
@@ -2429,7 +2427,7 @@
<footer class="footer">
<span>SkillOpt: Executive Strategy for Self-Evolving Agent Skills</span>
<span><a href="https://microsoft.github.io/SkillOpt/blog/">Blog</a> / <a href="https://github.com/microsoft/SkillOpt/blob/main/docs/index.md" target="_blank" rel="noopener">Docs</a> / <a href="https://github.com/microsoft/SkillOpt" target="_blank" rel="noopener">Code</a> / <a href="#citation">Citation</a></span>
<span><a href="https://github.com/microsoft/SkillOpt" target="_blank" rel="noopener">Code</a> / <a href="#citation">Citation</a></span>
</footer>
</main>
<script>
mkdocs.yml
-5
View File
@@ -39,7 +39,6 @@ theme:
nav:
- Home: index.md
- Technical Blog: https://microsoft.github.io/SkillOpt/blog/
- Getting Started:
- Installation: guide/installation.md
- First Experiment: guide/first-experiment.md
@@ -48,10 +47,6 @@ nav:
- Training Loop: guide/training-loop.md
- Skill Document: guide/skill-document.md
- Deep Learning Analogy: guide/dl-analogy.md
- SkillOpt-Sleep:
- Overview: sleep/README.md
- OpenAI-compatible Endpoints: sleep/openai-compatible-endpoints.md
- Results: sleep/RESULTS.md
- Extension Guides:
- Add a New Benchmark: guide/new-benchmark.md
- Local Environment Smoke Tests: guide/local-env-smoke.md
plugins/README.md
+217 -133
View File
@@ -1,178 +1,262 @@
# SkillOpt-Sleep integrations
# SkillOpt-Sleep — plugins for Claude Code, Codex, Copilot, and Devin
**SkillOpt-Sleep** reviews recent agent sessions, mines recurring tasks, replays
them, and proposes bounded updates to memory and skills. A held-out validation
gate decides whether a proposal is worth staging, and nothing live changes until
the user explicitly adopts it.
**Your coding agent forgets everything between sessions. SkillOpt-Sleep fixes
that.** While you sleep, it reviews what you did today, notices the rules you
keep repeating ("always add a LIMIT", "answers in `\boxed{}`", "cite the
source"), and writes them into your agent's long-term memory and skills — but
only the rules that actually make it score better on *your own* past tasks. You
wake up to an agent that's better at *your* work, and you approve every change
before it sticks.
The shared engine lives in [`skillopt_sleep/`](../skillopt_sleep) and has no
runtime dependency on the paper's `skillopt/` experiment package.
One engine, four thin shells. It synthesizes **SkillOpt** (validation-gated
bounded text optimization — the research in this repo), **Claude Dreams**
(offline consolidation; input never mutated; review-then-adopt), and the **agent
sleep** idea (short-term experience → long-term competence).
## Available integrations
> **Open-source tool, decoupled from the research.** The engine lives in the
> top-level [`skillopt_sleep/`](../skillopt_sleep) package with **zero
> dependency** on the paper's `skillopt/` experiment code (the validation gate is
> vendored). Use it without the research stack.
Four integrations wrap the shared `skillopt_sleep` CLI. OpenClaw is a separate
reference adaptation with its own backend and setup assumptions.
---
| Platform | Folder | Mechanism | Status |
|---|---|---|---|
| **Claude Code** | [`claude-code/`](claude-code) | marketplace plugin, commands, skill, and hooks | installable shared-engine integration |
| **Codex** | [`codex/`](codex) | user-level skill and shared runner | installable shared-engine integration |
| **GitHub Copilot** | [`copilot/`](copilot) | MCP server exposing seven `sleep_*` tools | shared-engine MCP integration |
| **Devin** | [`devin/`](devin) | MCP server plus Devin transcript conversion | shared-engine MCP integration |
| **OpenClaw** | [`openclaw/`](openclaw) | custom DeepSeek/Ollama wrapper | independent reference adaptation; review and adapt before use |
| **Claude Code** | [`claude-code/`](claude-code) | `.claude-plugin` + `/skillopt-sleep` + `/skillopt-sleep-handoff` commands + skill + hooks | full, installable |
| **Codex** | [`codex/`](codex) | user-level `skillopt-sleep` skill + shared runner | full |
| **Copilot** | [`copilot/`](copilot) | MCP server (`sleep_*` tools) + `copilot-instructions` | full (MCP) |
| **Devin** | [`devin/`](devin) | MCP server (`sleep_*` tools) + Devin ATIF-v1.7 harvest + `.devin/rules` | full (MCP) |
## Install
Clone the repository first unless an installed `skillopt-sleep` CLI is sufficient
for your workflow.
## Install (pick your agent)
| Platform | Install | Then |
|---|---|---|
| **Claude Code** | from the repository root, `/plugin marketplace add ./plugins/claude-code`, then `/plugin install skillopt-sleep@skillopt-sleep` | `/skillopt-sleep status` |
| **Codex** | `bash plugins/codex/install.sh` | ask Codex to use the `skillopt-sleep` skill |
| **Copilot** | register `plugins/copilot/mcp_server.py` using its example MCP config | ask Copilot to run `sleep_status` |
| **Devin** | register `plugins/devin/mcp_server.py` using its example MCP config | ask Devin to run `sleep_status` |
| **OpenClaw** | follow and adapt [`openclaw/README.md`](openclaw/README.md) | validate paths, credentials, and tasks locally |
| **Claude Code** | `/plugin marketplace add microsoft/SkillOpt` → `/plugin install skillopt-sleep` | `/skillopt-sleep status` |
| **Codex** | `git clone` → `bash plugins/codex/install.sh` | `/skillopt-sleep status` |
| **Copilot** | `git clone` → register `plugins/copilot/mcp_server.py` as an MCP server | ask "run the sleep cycle" |
| **Devin** | `git clone` → `devin mcp add skillopt-sleep -- python3 plugins/devin/mcp_server.py` | ask "run the sleep cycle" |
Python 3.10 or newer is required. Real CLI backends also require the selected
agent CLI to be installed and authenticated.
Requirements: Python ≥ 3.10 and the agent's CLI on PATH. All three call the same
[`run-sleep.sh`](run-sleep.sh) → `python -m skillopt_sleep`, so behaviour is
identical everywhere. Default backend is `mock` (no API spend); `--backend
claude|codex|copilot` uses your own budget.
The shared [`run-sleep.sh`](run-sleep.sh) supports both source checkouts and
installed packages. If it cannot find the repository, it tries the
`skillopt-sleep` executable on `PATH` (including `uv tool`/`pipx` installs), then
an importable `skillopt_sleep` module. Install with `uv tool install skillopt` or
`pip install skillopt` when using that fallback.
---
> **Version note.** This integration reference tracks `main`. PyPI 0.2.0
> supports the base Sleep CLI, while handoff, Sleep support for non-Azure
> OpenAI-compatible endpoints, and `--preferences` require a source checkout
> from `main` until the next release.
## How it works: one "night", in plain terms
## One sleep cycle
```text
harvest supported local sessions → mine recurring tasks → replay tasks
→ reflect and propose bounded edits → validate on held-out real tasks
→ stage proposal → (you) review and adopt
```
harvest your past sessions → mine the tasks you keep doing → replay them offline
→ reflect on failures → propose a few rule edits → KEEP only edits that raise
your held-out score → stage a proposal → (you) review & adopt
```
The default backend is `mock`: it makes no provider calls and is useful for
checking plumbing. A real backend is required for model-driven mining and genuine
optimization.
Nothing live changes until you `adopt`; every adopt backs up the prior file.
## Data boundary
### The split that keeps it honest: dream-train / real-val / real-test
- Harvesting is local and read-only. The `mock` backend has no model-provider
data path and no API spend.
- A real backend sends truncated transcript excerpts and derived task content to
the provider selected for mining, replay, judging, and reflection.
- Outbound prompts are not currently guaranteed to be free of secrets. Do not
use a third-party provider on sensitive transcripts without reviewing the data
source and the provider's retention policy.
- For a reviewable workflow, export tasks first, inspect and redact the JSON, set
its top-level `"reviewed"` field to `true`, and then use the task file with a
real backend:
This is the heart of the design, borrowed from the SkillOpt paper's
train/selection/test protocol:
```bash
python -m skillopt_sleep harvest --project "$(pwd)" --output reviewed-tasks.json
python -m skillopt_sleep dry-run --project "$(pwd)" --backend codex \
--tasks-file reviewed-tasks.json --progress
```
Real backends reject task files that are still marked unreviewed.
For the separate API-key and Azure managed-identity transport boundaries, see
[OpenAI-compatible endpoints](../docs/sleep/openai-compatible-endpoints.md).
## Supported CLI surface
Actions:
| Action | Behavior |
|---|---|
| `status` | show state and the latest staged proposal |
| `dry-run` | harvest, mine, replay, and report; stage nothing |
| `run` | run the full cycle and stage a proposal |
| `adopt` | apply the latest staged proposal, with backups |
| `harvest` | inspect or export mined tasks |
| `schedule` / `unschedule` | install or remove the managed nightly cron entry |
Common implemented flags include:
| Flag | Default | Purpose |
| Split | Where it comes from | What it's for |
|---|---|---|
| `--backend mock\|claude\|codex\|copilot\|handoff\|azure_openai` | `mock` | select who performs model calls |
| `--model NAME` | backend default | select a backend-specific model |
| `--source claude\|codex\|auto` | `claude` | select the transcript source |
| `--project PATH` | current directory | select the project and invoked harvest scope |
| `--scope invoked\|all` | `invoked` | limit transcript harvesting |
| `--target-skill-path PATH` | managed skill | select a specific `SKILL.md` to stage/adopt |
| `--tasks-file PATH` | none | replay a reviewed task file instead of harvesting |
| `--max-sessions N` / `--max-tasks N` | unset → `3 × tasks` / `40` tasks | bound harvested work; these are not hard token or wall-clock budgets |
| `--edit-budget N` | `4` | cap bounded edits per cycle |
| `--preferences "..."` | empty | add house rules to the reflection prior |
| `--progress` | off | print phase progress to stderr |
| `--auto-adopt` | off | adopt an accepted proposal without a separate command |
| `--json` | off | emit machine-readable output where supported |
| **train** | your real tasks **+ optional "dreamed" variants** | what the optimizer *learns from*. Over-dreaming here is fine — it's imagination. |
| **val** (selection) | **your real tasks only**, held out | the **gate**: an edit is kept only if it raises this score. Stops overfitting. |
| **test** | **your real tasks only**, held out, never seen during optimization | the **final score** we report. Kept as close to your real usage as possible. |
The nightly CLI does **not** currently expose `--gate`, `--rollouts-k`,
`--optimizer-model`, `--target-model`, `--budget-tokens`, or `--budget-minutes`.
Do not pass experiment-harness flags to the main CLI.
So you can **dream up extra training examples** to learn a rule robustly, while
the rule is still **judged on real, unseen tasks**. A `dream` task can *never*
land in val or test — that invariant is unit-tested.
### Preferences
---
`--preferences` is the main user-facing steering knob:
## What each feature does **for you** (with examples)
Every control below works on all three platforms (pass it after the action,
e.g. `/skillopt-sleep run --rollouts-k 3`).
### `--preferences "..."` — tell it your house rules
The single most useful knob. Free text that steers what the optimizer writes,
as a prior. Use it to encode the conventions you're tired of repeating.
```bash
python -m skillopt_sleep run --backend codex --project "$(pwd)" \
--preferences "Prefer pytest. Keep commit subjects imperative and concise."
# A backend engineer:
/skillopt-sleep run --preferences "Always use async/await, never callbacks. \
Prefer pytest over unittest. Commit subjects in imperative mood under 50 chars."
# A data analyst:
/skillopt-sleep run --preferences "Every SQL query must end with LIMIT 1000 unless \
I say otherwise. Money in USD with 2 decimals. Prefer CTEs over nested subqueries."
# A researcher:
/skillopt-sleep run --preferences "Cite sources as [Author, Year]. Math answers in \
\\boxed{}. Keep explanations under 150 words unless I ask for depth."
```
*What it does for you:* the next morning your agent already follows these
without you re-typing them, and the rules are validated against your real tasks
(if a "preference" actually hurts your held-out score, the gate drops it).
Preferences guide reflection but remain subject to the validation gate.
### `--gate on|off` — strict vs. greedy
### Advanced config
- `on` (default): an edit is kept **only if it raises your held-out score**.
Safe — blocks plausible-but-wrong rules and reward-hacking.
- `off`: greedy — keep edits without the strict check (still reports whether
quality moved).
The JSON/YAML config under `~/.skillopt-sleep/` supports additional engine keys,
including `gate_mode`, `gate_metric`, `dream_rollouts`, `dream_factor`, `recall_k`,
`evolve_memory`, and `evolve_skill`. These are config keys, not aliases for the
unsupported CLI flags listed above. Shipping defaults are conservative:
`gate_mode="on"`, `dream_rollouts=1`, `dream_factor=0`, and `recall_k=0`.
*What it does for you:* leave it `on` for trust. Flip it `off` when you're
exploring and want to see everything the optimizer proposes.
### Handoff backend
### `--rollouts-k K` — learn from contrast, not just failure
`--backend handoff` keeps model subprocesses out of the engine. It writes pending
model calls to `.skillopt-sleep-handoff/PROMPTS.md` and `pending.json`, exits with
code 3, and resumes after answers are placed in `answers/<id>.md`:
Re-runs each task `K` times and learns from the difference between the **good**
and **bad** attempts, not just a single failure.
```bash
/skillopt-sleep run --rollouts-k 3
```
*What it does for you:* a much stronger signal. If your agent gets a task right 1
time in 3, the optimizer figures out *what the winning attempt did* and makes it
reliable.
### `--optimizer-model` / `--target-model` — optimize cheap, deploy anywhere
Use a strong model to *write* the rules and a cheap model to *run* your tasks.
The learned skill then helps the cheap model — or any model.
```bash
/skillopt-sleep run --optimizer-model sonnet --target-model haiku
```
*What it does for you:* spend a little on a smart optimizer overnight; your
everyday cheap/fast agent inherits the upgrade. (Verified: a skill optimized on
one model lifts a different one — cross-model and even cross-runtime
Codex↔Claude.)
### `--budget-tokens N` / `--budget-minutes M` — cap the spend
You decide how much the nightly "dreaming" costs; it auto-plans how many nights
× how many rollouts fit.
```bash
/skillopt-sleep run --backend claude --budget-tokens 60000
```
*What it does for you:* predictable cost. It stops cleanly when the budget is hit
and tells you what it skipped.
### multi-objective (accuracy ↑, tokens ↓, latency ↓)
The reward can weight not just correctness but **cost and speed**, so a skill can
learn to be cheaper and faster, not only more accurate. *What it does for you:*
"answer directly instead of opening five files" becomes a learned habit.
### `--backend handoff` — session-executed calls (no API subprocess)
For subscription seats and environments where the engine shouldn't spawn
`claude -p` / API calls itself. The engine still runs every deterministic
stage (harvest → mine → replay scoring → gate → stage), but each model call
(attempt / judge / reflect) is written to a prompt file that **your own agent
session answers between engine runs**:
```bash
python -m skillopt_sleep run --backend handoff --project "$(pwd)"
# answer each prompt in a fresh context, then run the same command again
# exit 3 => .skillopt-sleep-handoff/PROMPTS.md + pending.json were written
# answer each prompt (each in a FRESH context) into answers/<id>.md
# re-run the same command => it resumes from the answers and either
# finishes (exit 0) or stages the next prompt batch (exit 3)
```
Answering held-out prompts from a context that has already seen their references
contaminates the validation gate. Claude Code's `/skillopt-sleep-handoff` command
automates the loop with isolated fresh-context subagents.
A typical night converges in 3–6 rounds: baseline attempts → reflect →
candidate re-scoring per accepted edit. Resume is stateless — replay is
deterministic and answers are cached by prompt hash, so re-running skips
everything already answered. Mined tasks are pinned to
`.skillopt-sleep-handoff/tasks.json` on the first round, so the sessions that
answer the prompts can't shift the task set and invalidate earlier answers.
On a completed real run the handoff directory is archived to
`.skillopt-sleep-handoff.night<N>.done`.
## Validation
On Claude Code, `/skillopt-sleep-handoff run` drives the whole loop for you,
answering each prompt in an isolated fresh-context subagent.
The deterministic no-provider check exercises consolidation and the gate:
**Integrity rule:** answer every prompt in a fresh context (a subagent with no
conversation history). Answering from a session that has already seen the
mined tasks and their references contaminates the held-out gate and fakes the
improvement score.
*What it does for you:* the sleep cycle runs entirely on your interactive
session's subscription budget — no API key, no headless subprocess — while the
gate, splits, and staging discipline stay in the engine.
Limitations: `--rollouts-k > 1` gives no contrastive spread (identical prompt
→ identical answer file), and tool-loop tasks fall back to the single-shot
`TOOL_CALL:` marker convention.
### `schedule` / `unschedule` — set it and forget it
Built-in nightly scheduling (no manual cron):
```bash
python -m skillopt_sleep.experiments.run_experiment \
--persona researcher --assert-improves
/skillopt-sleep schedule --hour 3 --minute 17 # runs every night for this project
/skillopt-sleep unschedule # stop it
```
*What it does for you:* it just gets better while you sleep. The nightly run only
*stages* a proposal — adopting is still your call (or add `--auto-adopt` when you
schedule, if you trust it).
Real-model benchmark results and their limitations are documented in
[`docs/sleep/RESULTS.md`](../docs/sleep/RESULTS.md). The benchmark recipes are not
the shipping CLI defaults.
---
## Safety summary
## Full action / flag reference
- Session harvesting is read-only.
- `mock` replay makes no provider calls.
- `run` stages proposals; `adopt` is the normal live-change boundary.
- Adoption backs up existing target files.
- `--max-sessions` and `--max-tasks` bound work, but the main CLI does not yet
enforce a hard token or elapsed-time budget.
- Treat real-backend transcript excerpts as data shared with the selected
provider.
| Action | Does |
|---|---|
| `status` | nights so far + the latest staged proposal (read-only) |
| `dry-run` | harvest→mine→replay→report; **stages nothing** |
| `run` | full cycle; **stages** a proposal; nothing live changes |
| `adopt` | apply the staged proposal to `CLAUDE.md`/`SKILL.md` (backs up first) |
| `harvest` | debug: print the recurring tasks it mined |
| `schedule` / `unschedule` | install/remove the nightly cron entry |
| Flag | Default | Meaning |
|---|---|---|
| `--backend mock\|claude\|codex\|copilot\|handoff` | `mock` | who runs/optimizes (mock = free; handoff = your own session answers) |
| `--preferences "..."` | – | your house rules, as a prior |
| `--gate on\|off` | `on` | strict held-out gate vs. greedy |
| `--rollouts-k K` | `1` | multi-rollout contrastive reflection |
| `--optimizer-model` / `--target-model` | – | split the optimizer from the target |
| `--budget-tokens` / `--budget-minutes` | – | cap the nightly spend |
| `--scope invoked\|all` | `invoked` | this project only, or all projects |
| `--auto-adopt` | off | apply without manual review (power users) |
Deep dive: [the SkillOpt-Sleep guide section](https://microsoft.github.io/SkillOpt/docs/guideline.html#sleep).
---
## Does it actually work?
Yes — measured with **real models on both Claude and Codex**, scored on held-out
tasks the optimizer never trained on:
- **gbrain-evals `skillopt-v1`** (the public suite gbrain scores SkillOpt on):
deficient skills go **0.00 → 1.00** on all 4 seeds, including a real tool-use
loop; cross-model transfer is positive; the gate blocks regressions.
→ [the SkillOpt-Sleep guide section](https://microsoft.github.io/SkillOpt/docs/guideline.html#sleep)
- **Academic daily-cases** (math / spreadsheet / search-QA, the paper's 4:1:5
split with dream-augmented train): see
[the SkillOpt-Sleep guide section](https://microsoft.github.io/SkillOpt/docs/guideline.html#sleep).
- **Fresh load-test** (a "SQL must always include LIMIT" analyst, built from
scratch): held-out **0.00 → 1.00** on both backends.
→ [the SkillOpt-Sleep guide section](https://microsoft.github.io/SkillOpt/docs/guideline.html#sleep)
Try the deterministic proof yourself (no API key, no spend):
```bash
python -m skillopt_sleep.experiments.run_experiment --persona researcher --assert-improves
```
It prints the held-out score rising to 1.0 as the gate accepts the right rules,
and confirms the gate **rejects** an injected harmful edit.
---
## Safety
- **Read-only** harvest of your sessions. `mock` replay has no side effects.
- Proposals are **staged**, never auto-applied (unless you opt in with `--auto-adopt`).
- Every adopt writes a backup. Per-night token/time budget caps. Secrets redacted.
plugins/claude-code/README.md
+19 -45
View File
@@ -10,7 +10,7 @@ SkillOpt-Sleep is the **deployment-time** companion to
[SkillOpt](https://github.com/microsoft/SkillOpt). SkillOpt trains a skill
offline on a benchmark; SkillOpt-Sleep applies the same discipline to *your own
daily usage*: bounded text edits, accepted only through a held-out validation
gate, with rejected candidates recorded in the cycle report for review.
gate, with rejected edits kept as negative feedback.
It synthesizes three ideas:
@@ -32,8 +32,7 @@ then adopt or discard" contract). Every adopt backs up the prior file first.
## Install
**Requirements:** Python ≥ 3.10. A real CLI backend additionally requires its
corresponding `claude` or `codex` executable on `PATH` and authenticated.
**Requirements:** Python ≥ 3.10, and the `claude` CLI (and/or `codex` CLI) on PATH.
```bash
# 1) get the code (the plugin ships inside the SkillOpt repo)
@@ -41,7 +40,7 @@ git clone https://github.com/microsoft/SkillOpt.git
cd SkillOpt
# 2) add the plugin to Claude Code as a local marketplace
/plugin marketplace add ./plugins/claude-code
/plugin marketplace add ./skillopt-sleep-plugin
/plugin install skillopt-sleep@skillopt-sleep
# 3) verify
@@ -49,21 +48,15 @@ cd SkillOpt
```
The plugin's bundled runner (`scripts/sleep.sh`) auto-selects a Python ≥ 3.10
interpreter and calls the `skillopt_sleep` engine. A source checkout needs no
`pip install`. If the marketplace cache does not contain a usable source tree,
the shared runner falls back first to a `skillopt-sleep` executable on `PATH`
(including `uv tool`/`pipx` installs), then to an importable Python module. Use
`uv tool install skillopt` or `pip install skillopt` for that fallback.
> **Version note.** This page tracks `main`. PyPI 0.2.0 provides the base Sleep
> CLI, but handoff mode and `--preferences` require a source checkout from
> `main` until the next release.
interpreter and calls the `skillopt_sleep` engine in the repo. No `pip install`
is required for the default `mock` backend or for `claude`/`codex` backends —
they shell out to the CLIs you already have.
## Quick start
```bash
# from inside any project you use with Claude Code:
/skillopt-sleep dry-run # preview what it would learn; no changes staged
/skillopt-sleep dry-run # safe preview: what it would learn, no changes staged
/skillopt-sleep run # full cycle: stages a reviewed proposal (still no live edits)
/skillopt-sleep status # see history + the latest staged proposal
/skillopt-sleep adopt # apply the staged proposal to CLAUDE.md / SKILL.md (with backup)
@@ -81,19 +74,8 @@ python -m skillopt_sleep run --project "$(pwd)" --backend codex # real lift v
```
Default backend is **`mock`** — deterministic, no API spend — so you can try the
plumbing for free. Switch to `--backend claude` or `--backend codex` for
model-driven mining and optimization on your own budget; an accepted gain is
task- and model-dependent, not guaranteed.
### Data boundary for real backends
Harvesting `~/.claude` is local and read-only, and the `mock` backend makes no
provider calls. A real backend sends truncated transcript excerpts and derived
tasks to the selected provider for mining, replay, judging, and reflection.
Outbound prompts are not currently guaranteed to be secret-free. Review your
session data and provider policy before using a real backend on a sensitive
project; the [shared integration guide](../README.md#data-boundary) describes a
reviewed task-file workflow.
plumbing for free. Switch to `--backend claude` or `--backend codex` for genuine
improvement on your own budget.
### Handoff mode (session answers the model calls)
@@ -113,7 +95,7 @@ python -m skillopt_sleep run --backend handoff --project "$(pwd)" # resume
Answer every prompt in a **fresh context** — a session that has already seen
the mined tasks and their references would contaminate the held-out gate.
Details: [the plugins README](../README.md#handoff-backend).
Details: [the plugins README](../README.md#--backend-handoff--session-executed-calls-no-api-subprocess).
## Does it actually improve? (real models, public benchmark)
@@ -132,8 +114,8 @@ Both took a brief-writer with no risks section / no confidence level and, within
1–2 nights, proposed gated edits that lifted the held-out score to perfect —
into the protected `LEARNED` block, nothing else touched. The Codex 2-night
trace even shows the optimizer **diagnosing its own residual failure** and
adding a meta-rule to fix it. See the recorded results and limitations in
[`docs/sleep/RESULTS.md`](../../docs/sleep/RESULTS.md).
adding a meta-rule to fix it. Full writeup + reproduction:
[the SkillOpt-Sleep guide section](https://microsoft.github.io/SkillOpt/docs/guideline.html#sleep).
Reproduce:
@@ -156,32 +138,24 @@ python -m skillopt_sleep.experiments.run_experiment --persona programmer --asse
Each prints the held-out score rising from baseline toward 1.0 as the gate
accepts the general rules your tasks need, and confirms the gate **rejects** an
injected harmful edit. Context for the measured experiments is in
[`docs/sleep/RESULTS.md`](../../docs/sleep/RESULTS.md).
injected harmful edit. Recorded output: [the SkillOpt-Sleep guide section](https://microsoft.github.io/SkillOpt/docs/guideline.html#sleep).
## Schedule it nightly
```bash
/skillopt-sleep schedule --hour 3 --minute 17
/skillopt-sleep unschedule
"${CLAUDE_PLUGIN_ROOT}/scripts/install-cron.sh" "$(pwd)" # prints a crontab line; installs nothing
```
The built-in scheduler creates a managed cron entry and logs under the project.
The scheduled run stages proposals unless `--auto-adopt` is explicitly selected.
## Safety
- **Read-only** harvest of `~/.claude`. `mock` replay has no side effects.
- Proposals are **staged**, never auto-applied (unless you opt in with `--auto-adopt`).
- Every adopt writes a backup under the staging dir's `backup/`.
- `--max-sessions` and `--max-tasks` bound work, but the main CLI does not enforce
a hard token or wall-clock budget.
- Real backends share truncated session/task content with the selected provider;
do not assume outbound prompts have been fully redacted.
- Per-night **token/task budget caps**; secrets redacted from prompts.
- `fresh` replay (Phase 3) runs only in throwaway git worktrees.
## Status
The engine, deterministic experiment, Claude/Codex CLI backends, handoff mode,
and staged adoption flow are implemented. Advanced experiment-harness flags are
not automatically available on the nightly CLI; see the
[shared integration reference](../README.md#supported-cli-surface).
Phase 1 (engine + deterministic experiment + plugin surface) is complete.
Phase 3 adds the real-API miner/judge and `fresh` worktree replay. See
[`docs/superpowers/specs/2026-06-07-skillopt-sleep-claude-code-plugin-design.md`](../docs/superpowers/specs/2026-06-07-skillopt-sleep-claude-code-plugin-design.md).
plugins/claude-code/commands/skillopt-sleep-handoff.md
+6 -13
View File
@@ -21,11 +21,10 @@ isolated context so the validation gate stays honest.
Repeat until the engine exits 0 (done) — at most 8 rounds:
1. **Run the engine** via the bundled runner. Split `$ARGUMENTS` into the action
and remaining options, and preserve those options on every resumed round:
1. **Run the engine** via the bundled runner:
```bash
"${CLAUDE_PLUGIN_ROOT}/scripts/sleep.sh" <action> --backend handoff --project "$(pwd)" --scope invoked <remaining options>
"${CLAUDE_PLUGIN_ROOT}/scripts/sleep.sh" <action> --backend handoff --project "$(pwd)" --scope invoked
```
- exit 0 → the night is complete; go to "Finish" below.
@@ -50,13 +49,10 @@ Repeat until the engine exits 0 (done) — at most 8 rounds:
## Finish
- For `run`, if the engine prints a staging directory, `Read` its `report.md`
and show the user: held-out baseline → candidate score, the gate decision,
the proposed edits, and where the proposal is staged. If an accepted proposal
was staged, tell the user nothing live changed and offer
`/skillopt-sleep adopt`.
- For `dry-run`, no staging directory or `report.md` is created; summarize the
final stdout instead.
- `Read` the `report.md` in the staging dir the engine printed and show
the user: held-out baseline → candidate score, the gate decision, the
proposed edits, and where the proposal is staged.
- Tell the user nothing live changed; offer `/skillopt-sleep adopt`.
- The engine archives `.skillopt-sleep-handoff/` on a completed real run;
do not delete it yourself.
@@ -69,6 +65,3 @@ Repeat until the engine exits 0 (done) — at most 8 rounds:
set. Do not edit that file.
- If a batch looks like it contains secrets or content the user would not
want re-processed, stop and ask before answering.
- Handoff files apply pattern-based secret redaction, but that is not a
guarantee that prompts are free of sensitive data. Treat the pending batch as
private user data and do not copy it into chat, logs, or commits.
plugins/claude-code/commands/skillopt-sleep.md
+24 -40
View File
@@ -1,5 +1,5 @@
---
description: Run or manage the SkillOpt-Sleep self-evolution cycle (review past sessions, replay tasks through a selected backend, consolidate validated memory + skills, or schedule nightly runs)
description: Run or manage the SkillOpt-Sleep self-evolution cycle (review past sessions, replay tasks offline, consolidate validated memory + skills; can also schedule nightly runs)
argument-hint: "[run | dry-run | status | adopt | harvest | schedule | unschedule] (default: status)"
allowed-tools: Bash, Read
---
@@ -7,11 +7,10 @@ allowed-tools: Bash, Read
# /skillopt-sleep — SkillOpt-Sleep nightly self-evolution
You are driving **SkillOpt-Sleep**: a tool that lets this user's Claude agent
improve from past usage by reviewing sessions, replaying recurring tasks, and
improve offline by reviewing past sessions, replaying recurring tasks, and
consolidating what it learns into **validated** memory (`CLAUDE.md`) and skills
(`SKILL.md`). With the default gate enabled, a change is kept only if it improves
a held-out replay score. Nothing live is modified until adoption unless the
user explicitly requests `--auto-adopt`.
(`SKILL.md`). It is gated like SkillOpt: a change is kept only if it improves a
held-out replay score, and nothing live is modified until the user adopts it.
## Requested action: $ARGUMENTS
@@ -19,14 +18,11 @@ user explicitly requests `--auto-adopt`.
## How to run it
The engine is the `skillopt_sleep` Python package in this repo. Split
`$ARGUMENTS` into the first action token and its remaining options, then use the
**plugin's bundled runner** so the right interpreter and repo are on the path.
Preserve the user's remaining options (for example `--preferences`, `--backend`,
or `--target-skill-path`) instead of silently dropping them:
The engine is the `skillopt_sleep` Python package in this repo. Use the
**plugin's bundled runner** so the right interpreter and repo are on the path:
```bash
"${CLAUDE_PLUGIN_ROOT}/scripts/sleep.sh" <action> --project "$(pwd)" --scope invoked <remaining options>
"${CLAUDE_PLUGIN_ROOT}/scripts/sleep.sh" <action> --project "$(pwd)" --scope invoked
```
`<action>` is one of:
@@ -34,49 +30,37 @@ or `--target-skill-path`) instead of silently dropping them:
| action | what it does |
|--------------|--------------|
| `status` | show how many nights have run + the latest staged proposal (READ-ONLY) |
| `dry-run` | harvest → mine → replay → report, but **stage nothing** (no-staging preview) |
| `run` | full cycle: **stage** a validation report and any accepted proposal; only explicit `--auto-adopt` may also update live files |
| `dry-run` | harvest → mine → replay → report, but **stage nothing** (safe preview) |
| `run` | full cycle: also **stage** a reviewed proposal (still does NOT touch live files) |
| `adopt` | apply the latest staged proposal to live `CLAUDE.md` / `SKILL.md` (backs up first) |
| `harvest` | debug: print the recurring tasks mined from recent sessions |
| `schedule` | install a nightly cron entry for this project (`--hour --minute`, off-:00 by default) |
| `unschedule` | remove the nightly cron entry (`--all` to remove every managed entry) |
Default backend is `mock` (deterministic, no API spend). To use real budget for
model-driven optimization, add `--backend claude` or `--backend codex`. An
accepted gain is evidence on this run's held-out tasks, not a guarantee of
general improvement; results depend on the tasks, model, and checks. To steer
what the optimizer writes, add `--preferences "<your house rules>"`.
genuine improvement, add `--backend claude` or `--backend codex`. To steer what
the optimizer writes, add `--preferences "<your house rules>"`.
## Steps to follow
1. **Run the requested action** via the bundled runner above. Capture stdout and
stderr.
2. **For `run`:** if it prints a staging directory, `Read` its `report.md` and
show the user:
- held-out score: baseline → candidate (evidence on this run's held-out tasks)
1. **Run the requested action** via the bundled runner above. Capture stdout.
2. **For `run` / `dry-run`:** after it completes, `Read` the generated
`report.md` in the staging dir it prints, and show the user:
- held-out score: baseline → candidate (the proof it helped)
- the gate decision (accept/reject) and the exact edits it proposes
- where the proposal is staged
3. **For `dry-run`:** no staging directory or `report.md` is created. Summarize
the score, gate decision, and edits from stdout (or request `--json` when
machine-readable output is useful).
4. **For `run` that produced an accepted proposal:** inspect whether stdout says
it was auto-adopted. If not, tell the user nothing live changed and offer
`/skillopt-sleep adopt`; if it was, report the updated paths explicitly.
5. **For `adopt`:** confirm which live files were updated and that backups were
3. **For `run` that produced an accepted proposal:** tell the user the diff is
staged and that **nothing live changed yet**. Offer to run `/skillopt-sleep adopt`.
4. **For `adopt`:** confirm which live files were updated and that backups were
written under the staging dir's `backup/`.
6. **Never** edit `CLAUDE.md` or `SKILL.md` yourself — let the engine's explicit
`adopt` or user-requested `--auto-adopt` path apply its manifest and backup
behavior. Respect the review gate.
5. **Never** edit `CLAUDE.md` or `SKILL.md` yourself — only the `adopt` action
does that, with a backup. Respect the review gate.
## Safety reminders
- Harvest is **read-only** over `~/.claude`. Replay in `mock` mode runs no
shell side effects.
- The cycle stages proposals by default; auto-adoption requires explicit opt-in.
- A real backend sends truncated transcript excerpts and derived tasks to its
provider for mining, replay, judging, and reflection. Pattern-based redaction
is not a guarantee that outbound prompts are secret-free. For sensitive data,
use `mock` or first run `harvest --output <file>`, review/redact the file, set
`"reviewed": true`, and then pass it with `--tasks-file`.
- `schedule` manages a cron entry when `crontab` is available; otherwise it
prints a line for manual installation.
- The cycle stages proposals; the user is in control of adoption.
- If the user asks to schedule this nightly, point them at
`${CLAUDE_PLUGIN_ROOT}/scripts/install-cron.sh` (prints a crontab line; does
not install anything without confirmation).
plugins/claude-code/scripts/install-cron.sh
+1 -2
View File
@@ -24,7 +24,6 @@ cat <<EOF
${MIN} ${HOUR} * * * "${RUNNER}" run --project "${PROJECT}" --scope invoked --backend ${BACKEND} >> "${PROJECT}/.skillopt-sleep/cron.log" 2>&1
#
# For fully-autonomous adoption (power users), append: --auto-adopt
# To use the authenticated Claude CLI for model-driven optimization, set
# BACKEND=claude above.
# To spend real API budget for genuine lift, set BACKEND=anthropic above.
# ────────────────────────────────────────────────────────────────────────────
EOF
plugins/claude-code/scripts/run-sleep.sh
+6 -34
View File
@@ -24,10 +24,11 @@ else
d="$(dirname "$d")"
done
fi
if [ "$#" -eq 0 ]; then set -- status; fi
if [ -z "${REPO_ROOT:-}" ]; then
echo "[sleep] ERROR: could not locate the skillopt_sleep package. Set SKILLOPT_SLEEP_REPO to the repo root." >&2
exit 1
fi
if [ -n "${REPO_ROOT:-}" ]; then
# Source checkout: run from repo root so skillopt_sleep/ is importable.
PY=""
# Allow explicit Python override (useful on macOS with old system Python).
if [ -n "${SKILLOPT_SLEEP_PYTHON:-}" ]; then
@@ -44,36 +45,7 @@ if [ -n "${REPO_ROOT:-}" ]; then
echo "[sleep] ERROR: need Python >= 3.10 (found none)." >&2
exit 1
fi
if [ "$#" -eq 0 ]; then set -- status; fi
cd "$REPO_ROOT"
exec "$PY" -m skillopt_sleep "$@"
fi
# No source checkout found — fall back to an installed engine.
# Fallback 1: skillopt-sleep CLI on PATH (uv tool install / pipx / pip install).
# Checked before the import fallback because uv tool install / pipx isolate the
# package from the system Python's import path, so `python -c "import
# skillopt_sleep"` would fail even though the CLI is available.
if command -v skillopt-sleep >/dev/null 2>&1; then
exec skillopt-sleep "$@"
fi
# Fallback 2: importable as a module (pip install into the active Python).
# Pick a Python >= 3.10 and check importability.
PY=""
for cand in python3.12 python3.11 python3.10 python3; do
if command -v "$cand" >/dev/null 2>&1; then
ver="$("$cand" -c 'import sys; print("%d%d" % sys.version_info[:2])' 2>/dev/null || echo 0)"
if [ "${ver:-0}" -ge 310 ] && "$cand" -c "import skillopt_sleep" >/dev/null 2>&1; then
PY="$cand"; break
fi
fi
done
if [ -n "$PY" ]; then
exec "$PY" -m skillopt_sleep "$@"
fi
echo "[sleep] ERROR: could not locate the skillopt_sleep package." >&2
echo "[sleep] Install it with 'uv tool install skillopt' or 'pip install skillopt'," >&2
echo "[sleep] or set SKILLOPT_SLEEP_REPO to a clone of the SkillOpt repo." >&2
exit 1
plugins/claude-code/skills/skillopt-sleep/SKILL.md
+31 -60
View File
@@ -1,29 +1,25 @@
---
name: skillopt-sleep
description: "Use when the user wants their Claude agent to self-improve from past usage, asks about a nightly/offline 'sleep' or 'dream' cycle, memory/skill consolidation, or says things like 'make my agent better the more I use it', 'review my past sessions', 'learn my preferences', 'consolidate what you learned', 'run the sleep cycle', or wants to schedule background self-optimization. Drives the skillopt_sleep engine: harvest past sessions -> mine recurring tasks -> replay through a selected backend -> consolidate validated CLAUDE.md/SKILL.md behind a held-out gate."
description: "Use when the user wants their Claude agent to self-improve from past usage, asks about a nightly/offline 'sleep' or 'dream' cycle, memory/skill consolidation, or says things like 'make my agent better the more I use it', 'review my past sessions', 'learn my preferences', 'consolidate what you learned', 'run the sleep cycle', or wants to schedule offline self-optimization. Drives the skillopt_sleep engine: harvest past sessions -> mine recurring tasks -> replay offline -> consolidate validated CLAUDE.md/SKILL.md behind a held-out gate."
---
# SkillOpt-Sleep: usage-driven self-evolution for a local Claude agent
# SkillOpt-Sleep: offline self-evolution for a local Claude agent
SkillOpt-Sleep gives the user's agent a **sleep cycle**. On demand or on a
nightly schedule, it reviews real past Claude Code sessions, re-runs recurring
tasks through the selected backend, and consolidates what it
learns into **memory** (`CLAUDE.md`) and **skills** (`SKILL.md`). With the
default validation gate enabled, it keeps only changes that improve a held-out
score. Live files change only through explicit adoption or a user-requested
`--auto-adopt`. It aims to improve this user's recurring work, while making
each accepted proposal measurable on the run's held-out tasks,
SkillOpt-Sleep gives the user's agent a **sleep cycle**. While the user is
offline (e.g. nightly), it reviews their real past Claude Code sessions,
re-runs recurring tasks on their own API budget, and consolidates what it
learns into **memory** (`CLAUDE.md`) and **skills** (`SKILL.md`) — but only
keeps changes that pass a held-out validation gate, and only after the user
adopts them. The agent gets measurably better at *this* user's recurring work,
with no model-weight training. It is the deployment-time analogue of training:
short-term experience → long-term competence.
It synthesizes three ideas:
- **SkillOpt** — the skill/memory doc is trainable text; bounded add/delete/replace
edits; accepted only through a held-out gate; rejected edits are recorded in
the run report for review.
- **Claude Dreams** — consolidation that reads past sessions and proposes changes
inside protected learned blocks; the input is never mutated, and output is
reviewed before adoption.
- **Agent sleep** — periodic background replay turns episodes into durable skill.
edits; accepted only through a held-out gate; rejected edits become negative feedback.
- **Claude Dreams** — offline consolidation that reads past sessions and rebuilds
memory (dedup/merge/resolve); the input is never mutated; output is reviewed then adopted.
- **Agent sleep** — periodic offline replay turns episodes into durable skill.
## When to use this skill
@@ -38,14 +34,9 @@ Trigger when the user wants any of:
1. **Harvest** — read `~/.claude/projects/*/<session>.jsonl` + `~/.claude/history.jsonl` (READ-ONLY) → session digests.
2. **Mine** — digests → `TaskRecord`s (recurring intents + outcome labels + checkable refs where possible).
3. **Replay** — re-run tasks through the selected backend under the *current*
skill+memory → (hard, soft) scores.
4. **Consolidate** — reflect on failures → propose bounded edits → **gate** on a held-out slice; with the default gate enabled, accept only if it strictly improves.
5. **Stage** — write the accepted `proposed_CLAUDE.md` and/or
`proposed_SKILL.md`, plus `report.md`, `report.json`, `manifest.json`, and
`diagnostics.json` into `<project>/.skillopt-sleep/staging/<timestamp>/`.
**Nothing live changes.** A rejected run still has a report but no proposed
live-file replacement.
3. **Replay** — re-run tasks offline under the *current* skill+memory → (hard, soft) scores.
4. **Consolidate** — reflect on failures → propose bounded edits → **gate** on a held-out slice; accept only if it strictly improves.
5. **Stage** — write `proposed_CLAUDE.md`, `proposed_SKILL.md`, a diff, and `report.md` into `<project>/.skillopt-sleep/staging/<date>/`. **Nothing live changes.**
6. **Adopt** — explicit (or opt-in auto): copy staged files over live ones, backing up first.
## How to drive it
@@ -54,20 +45,14 @@ Prefer the `/skillopt-sleep` command. Under the hood it calls the bundled runner
```bash
"${CLAUDE_PLUGIN_ROOT}/scripts/sleep.sh" status # what's happened
"${CLAUDE_PLUGIN_ROOT}/scripts/sleep.sh" dry-run --project "$(pwd)" # no-staging preview
"${CLAUDE_PLUGIN_ROOT}/scripts/sleep.sh" dry-run --project "$(pwd)" # safe preview
"${CLAUDE_PLUGIN_ROOT}/scripts/sleep.sh" run --project "$(pwd)" # full cycle, stages a proposal
"${CLAUDE_PLUGIN_ROOT}/scripts/sleep.sh" adopt --project "$(pwd)" # apply staged proposal (with backup)
```
- Default backend is `mock` (deterministic, **no API spend**) — good for trying the plumbing.
- Add `--backend claude` or `--backend codex` to spend the user's real budget
for model-driven optimization. A held-out gain is run-specific evidence, not
a guarantee of broader improvement; results depend on the tasks, model, and
checks.
- Scope defaults to the invoked project; `--scope all` harvests every Claude
project into the current run's configured targets.
- A real backend sends truncated transcript/task content to its provider. See
the data-boundary rules below before using one with sensitive sessions.
- Add `--backend claude` or `--backend codex` to spend the user's real budget for genuine improvement.
- Scope defaults to the invoked project; `--scope all` harvests every project.
### Scheduling
@@ -78,30 +63,25 @@ Prefer the `/skillopt-sleep` command. Under the hood it calls the bundled runner
Installs a nightly cron entry. `unschedule --all` removes every managed entry.
## Common CLI flags
## All CLI flags
| Flag | Default | Description |
|------|---------|-------------|
| `--project PATH` | cwd | Project directory to evolve |
| `--scope all\|invoked` | invoked | Harvest scope |
| `--backend mock\|claude\|codex\|copilot\|handoff\|azure_openai` | mock | Backend (mock = no provider calls) |
| `--backend mock\|claude\|codex\|copilot` | mock | Replay backend (mock = no API spend) |
| `--model NAME` | backend default | Override the model used for replay |
| `--source claude\|codex\|auto` | claude | Transcript source |
| `--lookback-hours N` | 72 | Harvest window |
| `--max-sessions N` | derived | Cap harvested sessions; defaults to 3 × max tasks (120 with current defaults) |
| `--max-sessions N` | unlimited | Cap harvested sessions |
| `--max-tasks N` | 40 | Cap mined tasks |
| `--target-skill-path PATH` | `~/.claude/skills/skillopt-sleep-learned/SKILL.md` | Explicit SKILL.md to evolve |
| `--target-skill-path PATH` | auto | Explicit SKILL.md to evolve |
| `--tasks-file PATH` | — | Reviewed TaskRecord JSON (skip harvest) |
| `--progress` | off | Print phase progress to stderr |
| `--auto-adopt` | off | Auto-adopt if gate passes |
| `--edit-budget N` | 4 | Max bounded edits per night |
| `--preferences TEXT` | empty | Add house rules to the optimizer's reflection prior |
| `--json` | off | Machine-readable JSON output |
The CLI also has source/runtime path overrides (`--claude-home`, `--codex-home`,
and `--codex-path`) and action-specific flags. Use
`python -m skillopt_sleep <action> --help` as the authoritative surface.
## Config keys (`~/.skillopt-sleep/config.json`)
Beyond the CLI flags, advanced behavior is controlled via config:
@@ -119,37 +99,28 @@ The sleep cycle can consolidate both:
- **SKILL.md** — the managed skill file (bounded edits: add/delete/replace)
- **CLAUDE.md** — the project memory (same bounded edits)
With the default gate enabled, both are evaluated by the same held-out score.
Set `evolve_memory: false` to consolidate only skills, or `evolve_skill: false`
for only memory.
Both are gated by the same held-out validation score. Set `evolve_memory: false` to consolidate only skills, or `evolve_skill: false` for only memory.
## Hard rules
- **Never** hand-edit the user's `CLAUDE.md` / `SKILL.md` as part of this skill.
Let the engine's explicit `adopt` or user-requested `--auto-adopt` path apply
the staging manifest and back up existing live files first.
Only the `adopt` action changes live files, and it backs them up first.
- Harvest is read-only. `mock` replay has no side effects.
- Real backends send truncated transcript excerpts and derived tasks to the
selected provider for mining, replay, judging, and reflection. The Claude
transcript path is not guaranteed to remove every secret before those calls.
Review provider policy and session contents first. For sensitive data, use
`mock` or run `harvest --output <file>`, inspect/redact the JSON, set
`"reviewed": true`, and replay it with `--tasks-file`; real backends refuse an
unreviewed task file.
- Always show the user the **held-out baseline → candidate** score and the
exact proposed edits before suggesting adoption. Evidence before adoption.
- If asked to demonstrate the mechanism without provider calls, run
- If asked whether it really helps, run
`python -m skillopt_sleep.experiments.run_experiment --persona researcher --json`
— a deterministic synthetic demo of held-out lift and gate rejection. It
validates the mechanism, not effectiveness on the user's own tasks.
— a deterministic demo that proves held-out lift and that the gate blocks
harmful edits.
## Validate / demo
```bash
# deterministic synthetic demo (no API): score rises and the gate blocks a regression
# deterministic proof (no API): held-out score rises, gate blocks regressions
python -m skillopt_sleep.experiments.run_experiment --persona researcher --assert-improves
python -m skillopt_sleep.experiments.run_experiment --persona programmer --assert-improves
```
See the [SkillOpt-Sleep documentation](https://github.com/microsoft/SkillOpt/tree/main/docs/sleep)
for recorded results, limitations, and the supported integration surface.
See the SkillOpt-Sleep guide section for recorded output and
`docs/superpowers/specs/2026-06-07-skillopt-sleep-claude-code-plugin-design.md`
for the full design.
plugins/codex/README.md
+8 -27
View File
@@ -9,8 +9,7 @@ as the Claude Code plugin (`skillopt_sleep`), wrapped for Codex.
> [gbrain-evals](https://github.com/garrytan/gbrain-evals) `skillopt-v1`
> benchmark, a deliberately deficient skill goes **0.00 → 1.00** on a held-out
> set with the Codex backend (incl. the tool-use seed via a real tool loop).
> See the recorded results and limitations in
> [`docs/sleep/RESULTS.md`](../../docs/sleep/RESULTS.md).
> See [the SkillOpt-Sleep guide section](https://microsoft.github.io/SkillOpt/docs/guideline.html#sleep).
## What Codex supports (and what we use)
@@ -22,20 +21,11 @@ rules. The shared runner remains a plain shell entrypoint that the skill calls.
## Install
On Linux/macOS:
```bash
git clone https://github.com/microsoft/SkillOpt.git
cd SkillOpt
bash plugins/codex/install.sh # installs the skill
export SKILLOPT_SLEEP_REPO="$(pwd)" # so the runner is found from anywhere
```
On Windows (PowerShell):
```powershell
git clone <repo-url> SkillOpt-Sleep
cd SkillOpt-Sleep
powershell -File plugins/codex/install.ps1
[System.Environment]::SetEnvironmentVariable("SKILLOPT_SLEEP_REPO", "$(pwd)", "User")
bash plugins/codex/install.sh # installs the skill
export SKILLOPT_SLEEP_REPO="$(pwd)" # so the runner is found from anywhere
```
If a previous install created `~/.codex/prompts/sleep.md`, the installer moves
@@ -70,19 +60,15 @@ python -m skillopt_sleep run --project "$(pwd)" --source codex --backend codex \
`~/.codex/archived_sessions`. Use `--codex-home /path/to/.codex` to point at a
different Codex home, or `--source auto` to try Codex archives first and fall
back to Claude Code transcripts. Default backend is `mock` (no API spend).
`--backend codex` uses your Codex budget for model-driven optimization; an
accepted gain is task-dependent, not guaranteed. Bound live runs
`--backend codex` uses your Codex budget for real improvement. Bound live runs
with `--max-sessions` and `--max-tasks`; add `--progress` because Codex-backed
mining, replay, and reflection can be slow and otherwise quiet. Use
`--target-skill-path` to stage/adopt into a repo-scoped Codex skill such as
`.agents/skills/<name>/SKILL.md`; target runs over-sample mined tasks and
prefer tasks that match the target skill's path, headings, and content. The
implemented main-CLI flags work the same across the shared integrations, and
`--preferences "..."` is available for house rules. Advanced keys such as
`gate_mode`, `dream_rollouts`, and `recall_k` belong in the Sleep config; the
nightly CLI does not expose `--gate`, `--rollouts-k`, token/time-budget, or
optimizer/target-split flags. See the
[shared CLI reference](../README.md#supported-cli-surface).
prefer tasks that match the target skill's path, headings, and content. All the
controllable knobs (`--gate on|off`, `--rollouts-k`, `--budget-tokens`,
`--preferences`, optimizer/target split) work identically — see
[the SkillOpt-Sleep guide section](https://microsoft.github.io/SkillOpt/docs/guideline.html#sleep).
For privacy-sensitive projects, split the run into reviewable steps:
@@ -100,11 +86,6 @@ Inspect/redact the JSON and set `"reviewed": true` before using a real backend.
`--tasks-file` skips archive harvest/mining and replays only the reviewed JSON
tasks; real backends refuse task files still marked `"reviewed": false`.
This review step matters even though the Codex transcript converter removes
known secret-shaped strings: pattern-based redaction is not a guarantee. A real
backend sends truncated transcript/task content to the selected provider, while
`--backend mock` makes no provider calls.
## Notes / status
- Codex's `exec` runs shell, so the real-tool-loop replay (e.g. the
plugins/codex/install.ps1
-47
View File
@@ -1,47 +0,0 @@
# Install the SkillOpt-Sleep Codex integration as a user-level Codex skill on Windows.
# Idempotent; prints what it does.
$ErrorActionPreference = "Stop"
$RepoRoot = Resolve-Path (Join-Path $PSScriptRoot "..\..")
$CodexHome = if ($env:CODEX_HOME) { $env:CODEX_HOME } else { Join-Path $env:USERPROFILE ".codex" }
$AgentsSkills = Join-Path $env:USERPROFILE ".agents\skills"
$LegacyPrompt = Join-Path $CodexHome "prompts\sleep.md"
Write-Output "[install] repo: $RepoRoot"
# 1) user-level skill
$SkillDir = Join-Path $AgentsSkills "skillopt-sleep"
if (-not (Test-Path $SkillDir)) {
New-Item -ItemType Directory -Path $SkillDir -Force | Out-Null
}
Copy-Item (Join-Path $RepoRoot "plugins\codex\skills\skillopt-sleep\SKILL.md") (Join-Path $SkillDir "SKILL.md") -Force
Write-Output "[install] skill -> $(Join-Path $SkillDir 'SKILL.md')"
# 2) retire the old custom prompt entrypoint from previous installs
if (Test-Path $LegacyPrompt) {
$Backup = "${LegacyPrompt}.skillopt-legacy.bak"
if (Test-Path $Backup) {
$DateStr = Get-Date -Format "yyyyMMddHHmmss"
$Backup = "${LegacyPrompt}.skillopt-legacy.${DateStr}.bak"
}
Move-Item $LegacyPrompt $Backup -Force
Write-Output "[install] legacy prompt -> $Backup"
}
# 3) record the repo location so the runner is found from anywhere
Write-Output "[install] add to your environment variables:"
Write-Output " [System.Environment]::SetEnvironmentVariable('SKILLOPT_SLEEP_REPO', '$RepoRoot', 'User')"
Write-Output " Or set it via System Properties."
# 4) optional: append an AGENTS.md hint (only if the user opts in)
Write-Output ""
Write-Output "[install] Optional — add this to ~/.codex/AGENTS.md so Codex always knows the tool:"
Write-Output ""
Write-Output " ## SkillOpt-Sleep"
Write-Output " Use the skillopt-sleep skill when I ask to run a sleep/dream/offline"
Write-Output " self-improvement cycle. The runner is:"
Write-Output " \`powershell -File `"$RepoRoot\plugins\run-sleep.ps1`" status --project `"\$(pwd)\`"\`."
Write-Output ""
Write-Output "Done. Try asking Codex:"
Write-Output " Use the skillopt-sleep skill to run status for this project."
plugins/codex/skills/skillopt-sleep/SKILL.md
+30 -83
View File
@@ -1,23 +1,16 @@
---
name: skillopt-sleep
description: "Use when the user wants Codex to self-improve from past usage, asks about a nightly/offline 'sleep' or 'dream' cycle, wants Codex to review past sessions, learn preferences, consolidate memory/skills, run dry-run/run/adopt/status for SkillOpt-Sleep, or schedule background self-optimization. Drives the skillopt_sleep engine: harvest past sessions -> mine recurring tasks -> replay through a selected backend -> consolidate validated memory + skills behind a held-out gate."
description: "Use when the user wants Codex to self-improve from past usage, asks about a nightly/offline 'sleep' or 'dream' cycle, wants Codex to review past sessions, learn preferences, consolidate memory/skills, run dry-run/run/adopt/status for SkillOpt-Sleep, or schedule offline self-optimization. Drives the skillopt_sleep engine: harvest past sessions -> mine recurring tasks -> replay offline -> consolidate validated memory + skills behind a held-out gate."
---
# SkillOpt-Sleep: usage-driven self-evolution for a local Codex agent
# SkillOpt-Sleep: offline self-evolution for a local Codex agent
SkillOpt-Sleep gives the user's Codex agent a sleep cycle. On demand or on a
nightly schedule, it reviews past local sessions, re-runs recurring tasks
through the selected backend, and proposes changes to a configured skill and to
the project's `CLAUDE.md`. With the default validation gate enabled, it keeps
only changes that improve a held-out score. Live files change only through
explicit adoption or a user-requested `--auto-adopt`. There is no model-weight
training.
The current shared engine does **not** write `AGENTS.md`. For a Codex-visible
result, always select a Codex skill explicitly with `--target-skill-path` (for
example `.agents/skills/<name>/SKILL.md`). If project `CLAUDE.md` is not a
desired secondary target, set `"evolve_memory": false` in
`~/.skillopt-sleep/config.json` before running.
SkillOpt-Sleep gives the user's Codex agent a sleep cycle. While the user is
offline or on demand, it reviews past local sessions, re-runs recurring tasks
on the user's own budget, and consolidates what it learns into memory and
skills. It keeps only changes that pass a held-out validation gate, and live
files change only after the user explicitly adopts a staged proposal. There is
no model-weight training.
## When to use
@@ -35,15 +28,13 @@ Trigger when the user wants any of:
configuration and normalize them into session digests.
2. **Mine** - turn digests into recurring `TaskRecord`s with outcomes and
checkable references where possible.
3. **Replay** - re-run mined tasks through the selected backend under the
current skill and memory.
3. **Replay** - re-run mined tasks offline under the current skill and memory.
4. **Consolidate** - reflect on failures and propose bounded edits.
5. **Gate** - with the default gate enabled, accept edits only when the held-out
validation score improves.
5. **Gate** - accept edits only when the held-out validation score improves.
6. **Stage** - write the proposal under
`<project>/.skillopt-sleep/staging/<date>/`; nothing live changes.
7. **Adopt** - explicitly, or through user-requested auto-adopt, copy staged
files over live files with backups for existing targets.
7. **Adopt** - only after explicit user approval, copy staged files over live
files with backups.
## How to drive it
@@ -52,59 +43,32 @@ finds the engine and a Python >= 3.10 automatically.
```bash
# point at the repo if it isn't auto-detected from CWD:
export SKILLOPT_SLEEP_REPO=/path/to/SkillOpt
TARGET_SKILL=.agents/skills/example/SKILL.md
export SKILLOPT_SLEEP_REPO=/path/to/SkillOpt-Sleep
bash "$SKILLOPT_SLEEP_REPO/plugins/run-sleep.sh" status --project "$(pwd)"
bash "$SKILLOPT_SLEEP_REPO/plugins/run-sleep.sh" harvest --project "$(pwd)" \
--source codex --target-skill-path "$TARGET_SKILL"
bash "$SKILLOPT_SLEEP_REPO/plugins/run-sleep.sh" dry-run --project "$(pwd)" \
--source codex --target-skill-path "$TARGET_SKILL" --backend mock
bash "$SKILLOPT_SLEEP_REPO/plugins/run-sleep.sh" run --project "$(pwd)" \
--source codex --target-skill-path "$TARGET_SKILL" --backend codex \
--max-sessions 5 --max-tasks 3 --progress
bash "$SKILLOPT_SLEEP_REPO/plugins/run-sleep.sh" harvest --project "$(pwd)"
bash "$SKILLOPT_SLEEP_REPO/plugins/run-sleep.sh" dry-run --project "$(pwd)" --backend mock
bash "$SKILLOPT_SLEEP_REPO/plugins/run-sleep.sh" run --project "$(pwd)" --backend codex
bash "$SKILLOPT_SLEEP_REPO/plugins/run-sleep.sh" run --project "$(pwd)" --source codex # harvest from Codex Desktop
bash "$SKILLOPT_SLEEP_REPO/plugins/run-sleep.sh" adopt --project "$(pwd)"
```
On Windows (CMD / PowerShell):
```cmd
:: CMD
set SKILLOPT_SLEEP_REPO=C:\path\to\SkillOpt-Sleep
"%SKILLOPT_SLEEP_REPO%\plugins\run-sleep.cmd" status --project "%CD%"
```
```powershell
# PowerShell
$env:SKILLOPT_SLEEP_REPO = "C:\path\to\SkillOpt-Sleep"
powershell -File "$env:SKILLOPT_SLEEP_REPO\plugins\run-sleep.ps1" status --project "$(pwd)"
```
Actions are `status`, `harvest`, `dry-run`, `run`, `adopt`, `schedule`, and `unschedule`.
- Default backend is `mock`, which is deterministic and spends no API budget.
- `--backend codex` uses the user's Codex budget for model-driven optimization.
An accepted held-out gain is run-specific evidence, not a guarantee of
broader improvement; results depend on the tasks, model, and checks.
- `--backend codex` uses the user's Codex budget for real improvement.
- `--source codex` reads Codex Desktop archived sessions from `~/.codex/archived_sessions`;
use `--codex-home /path/to/.codex` if the archive lives elsewhere.
- `--target-skill-path` is required for a Codex skill target. Without it, the
shared default is a Claude-managed skill under `~/.claude/skills/`, not an
`.agents` skill.
- Keep `dry-run --backend mock` as the first smoke check unless the user
explicitly asked for a real optimization run.
### Scheduling
```bash
bash "$SKILLOPT_SLEEP_REPO/plugins/run-sleep.sh" schedule --project "$(pwd)" \
--backend codex --hour 3 --minute 17
bash "$SKILLOPT_SLEEP_REPO/plugins/run-sleep.sh" schedule --project "$(pwd)" --hour 3 --minute 17
bash "$SKILLOPT_SLEEP_REPO/plugins/run-sleep.sh" unschedule --project "$(pwd)"
```
The scheduler persists the project, backend, time, and optional auto-adopt flag;
it does not persist `--source` or `--target-skill-path` from this command. Before
scheduling a Codex-targeted run, set `"transcript_source": "codex"` and an
absolute `"target_skill_path"` in `~/.skillopt-sleep/config.json`. On systems
without `crontab`, `schedule` prints a line for manual installation.
`unschedule --all` removes every managed entry.
Installs a nightly cron entry. `unschedule --all` removes every managed entry.
### All backends
@@ -112,8 +76,6 @@ without `crontab`, `schedule` prints a line for manual installation.
- `--backend claude` — uses the Claude CLI
- `--backend codex` — uses the Codex CLI
- `--backend copilot` — uses the GitHub Copilot CLI
- `--backend handoff` — emits prompt/answer files for an interactive session
- `--backend azure_openai` — uses the configured Azure OpenAI endpoint
### Additional flags
@@ -134,10 +96,7 @@ without `crontab`, `schedule` prints a line for manual installation.
### Memory consolidation
The shared sleep cycle consolidates project **memory** (`CLAUDE.md`) and the
selected **skill** (`SKILL.md`) by default. It does not update `AGENTS.md`.
Each target is independently toggleable through `evolve_memory` /
`evolve_skill`, and both are gated by the same held-out validation score.
The sleep cycle consolidates both **memory** (AGENTS.md / CLAUDE.md) and **skills** (SKILL.md) by default. Each is independently toggleable via `evolve_memory` / `evolve_skill` config keys. Both are gated by the same held-out validation score.
## Steps
@@ -145,23 +104,16 @@ Each target is independently toggleable through `evolve_memory` /
2. For `dry-run` and `run`, report the held-out baseline -> candidate score,
gate action, task count, session count, and exact proposed edits.
3. If a staging directory is printed, read `report.md` before summarizing.
4. `run` stages by default; if `--auto-adopt` was explicitly supplied, report
the paths it updated instead of claiming nothing changed.
5. Offer adoption only after the user has reviewed a still-staged proposal.
6. Never hand-edit the configured `CLAUDE.md` or target skill as a substitute
for the engine's adopt path; adoption is the safety boundary and backs up
existing targets first.
4. `run` only stages a proposal; nothing live changes until `adopt`.
5. Offer adoption only after the user has reviewed the staged proposal.
6. Never hand-edit the user's `AGENTS.md`, memory, or skills as a substitute
for `adopt`; adoption is the safety boundary and writes backups first.
## Hard rules
- Harvest is read-only. Do not edit archived sessions or raw transcripts.
- Codex transcript harvesting removes known secret-shaped strings, developer
instructions, and raw tool payloads, but pattern-based redaction is not a
guarantee. A real backend still sends truncated transcript/task content to
its provider. Review sensitive sessions and provider policy first; prefer a
reviewed `--tasks-file` workflow when the data boundary matters.
- Keep raw secrets, credentials, private user data, and transcript contents out
of messages, logs, generated artifacts, and commits.
- Keep raw secrets, credentials, private user data, and unsanitized transcript
contents out of messages, logs, generated artifacts, and commits.
- Show validation evidence before recommending adoption.
- Treat generated edits as proposals, not as source of truth.
- Do not rely on deprecated custom prompts or `/sleep` slash commands for this
@@ -170,16 +122,11 @@ Each target is independently toggleable through `evolve_memory` /
## Validate
```bash
python -m skillopt_sleep dry-run --project "$(pwd)" --source codex \
--target-skill-path .agents/skills/example/SKILL.md --backend mock --json
python -m skillopt_sleep dry-run --project "$(pwd)" --backend mock --json
python -m skillopt_sleep.experiments.run_gbrain --backend codex \
--seeds brief-writer --data-root /path/to/gbrain-evals/eval/data/skillopt-v1 \
--nights 2 --limit-replay 3 --limit-holdout 3
```
In the recorded `brief-writer` gbrain run, the deliberately deficient fixture
went 0.00 -> 1.00 on that run's held-out set. Treat this as reproducible
benchmark evidence for that configuration, not a guarantee for other skills,
tasks, or models; see the
[recorded results](https://github.com/microsoft/SkillOpt/blob/main/docs/sleep/RESULTS.md)
for context and limitations.
A deficient skill goes 0.00 -> 1.00 on a held-out set; the optimizer's edits
are gated on real-task performance.
plugins/copilot/README.md
+9 -17
View File
@@ -27,8 +27,8 @@ Requires Python ≥ 3.10. No third-party packages — the server is pure stdlib.
"mcpServers": {
"skillopt-sleep": {
"command": "python3",
"args": ["/abs/path/SkillOpt/plugins/copilot/mcp_server.py"],
"env": { "SKILLOPT_SLEEP_REPO": "/abs/path/SkillOpt" }
"args": ["/abs/path/SkillOpt-Sleep/plugins/copilot/mcp_server.py"],
"env": { "SKILLOPT_SLEEP_REPO": "/abs/path/SkillOpt-Sleep" }
}
}
}
@@ -42,21 +42,14 @@ Requires Python ≥ 3.10. No third-party packages — the server is pure stdlib.
## Use
Ask Copilot things like *"run the sleep cycle"*, *"what did the last sleep
propose?"*, *"adopt the staged sleep proposal"*. The server exposes seven MCP
tools: `sleep_status`, `sleep_dry_run`, `sleep_run`, `sleep_adopt`,
`sleep_harvest`, `sleep_schedule`, and `sleep_unschedule`.
propose?"*, *"adopt the staged sleep proposal"*. Copilot calls the MCP tools:
`sleep_status`, `sleep_dry_run`, `sleep_run`, `sleep_adopt`, `sleep_harvest`.
Each tool takes optional `project`, `backend` (`mock`/`claude`/`codex`/`copilot`), and
`scope` arguments. Default backend is `mock` (no API spend). The `copilot`
backend drives the GitHub Copilot CLI (`copilot -p ... --output-format json`)
and requires the `copilot` CLI to be installed and authenticated.
Harvesting is local and read-only, and the default `mock` backend makes no
provider calls. A real backend sends truncated transcript excerpts and derived
tasks to the selected provider. Outbound prompts are not currently guaranteed
to be secret-free; review sensitive data and provider policy first. See the
[shared data-boundary guidance](../README.md#data-boundary).
For speed, the `copilot` backend runs each call against an isolated
`COPILOT_HOME` with built-in MCP servers and custom instructions disabled, so
your user MCP servers (including this project's own) are not spawned per call
@@ -72,13 +65,12 @@ printf '%s\n' \
'{"jsonrpc":"2.0","id":2,"method":"tools/list"}' \
| SKILLOPT_SLEEP_REPO="$(pwd)" python3 plugins/copilot/mcp_server.py
```
You should see the server info and all seven `sleep_*` tools.
You should see the server info and the five `sleep_*` tools.
## Notes / status
- MCP is the stable, official Copilot extension surface, so this is the most
portable shared-engine integration (one server → CLI + IDE).
- The MCP schema exposes the main CLI's implemented controls, including task and
session caps, target-skill selection, scheduling, and staged adoption. It does
not add experiment-only gate, rollout, token/time-budget, or optimizer/target
split flags. See the [shared CLI reference](../README.md#supported-cli-surface).
portable of the three integrations (one server → CLI + IDE).
- The engine and all its controls (gate on/off, multi-rollout, budget,
preferences, optimizer/target split) are identical across platforms — see
[the SkillOpt-Sleep guide section](https://microsoft.github.io/SkillOpt/docs/guideline.html#sleep).
plugins/copilot/copilot-instructions.snippet.md
+9 -22
View File
@@ -8,29 +8,26 @@ automatically as ambient guidance.)
This project has SkillOpt-Sleep available via an MCP server (`skillopt-sleep`).
It gives the agent a nightly "sleep cycle": it reviews past sessions, replays
recurring tasks through a selected backend, and stages validation-gated changes
to project `CLAUDE.md` and a configured `SKILL.md`.
recurring tasks offline, and consolidates validated memory + skills behind a
held-out gate.
When the user asks to "run the sleep cycle", "review my past sessions", "learn
my preferences", or "make the agent improve from past usage", use the MCP tools:
- `sleep_status` — what's happened + the latest staged proposal
- `sleep_dry_run` — no-staging preview; a real backend still makes provider calls
- `sleep_run` — full cycle, stages a validation-gated proposal by default;
explicit `auto_adopt` may update live files
- `sleep_adopt` — apply the staged proposal (backs up an existing live file first)
- `sleep_dry_run` — safe preview, stages nothing
- `sleep_run` — full cycle, stages a reviewed proposal (nothing live changes)
- `sleep_adopt` — apply the staged proposal (backs up first)
- `sleep_harvest` — list mined recurring tasks
- `sleep_schedule` — install a nightly cron entry (set `hour`/`minute`)
- `sleep_unschedule` — remove the nightly cron entry
### Key parameters (pass as MCP tool arguments)
- `backend` — `mock` (default, no provider calls), `claude`, `codex`, or `copilot`
- `backend` — `mock` (default, free), `claude`, `codex`, or `copilot`
- `source` — `claude`, `codex`, or `auto` (where to read transcripts)
- `target_skill_path` — explicit SKILL.md to evolve; use this for a skill that
the current agent actually loads
- `tasks_file` — reviewed TaskRecord JSON (skip harvest); real backends require
its metadata to contain `"reviewed": true`
- `target_skill_path` — explicit SKILL.md to evolve
- `tasks_file` — pre-built TaskRecord JSON (skip harvest)
- `max_tasks` / `max_sessions` — cap workload
- `auto_adopt` — auto-adopt if the gate passes
- `json` — machine-readable output for programmatic use
@@ -43,14 +40,4 @@ my preferences", or "make the agent improve from past usage", use the MCP tools:
Always show the user the held-out baseline → candidate score and the proposed
edits before suggesting `sleep_adopt`. Never hand-edit the user's memory/skill
files; use `sleep_adopt` (or an explicitly requested `auto_adopt`) so the engine
applies its staging manifest and backup behavior.
Harvesting is local and read-only, and `backend: "mock"` makes no provider
calls. A real backend sends truncated transcript excerpts and derived tasks to
the selected provider; outbound prompts are not guaranteed to be secret-free.
Review sensitive data and provider policy before selecting a real backend.
`sleep_schedule` persists only the project, backend, time, and optional
auto-adopt setting. Put a non-default transcript source or target skill in
`~/.skillopt-sleep/config.json` before scheduling it.
files; only `sleep_adopt` does that, with a backup.
plugins/copilot/skillopt/README.md
+3 -12
View File
@@ -68,18 +68,9 @@ skill"*, or *"evaluate this skill on the dataset"*. Copilot calls the MCP tools:
Common optional args (both train and eval): `env`, `backend`,
`optimizer_model`, `target_model`, `out_root`, `cfg_options` (space-separated
`section.key=value` YAML overrides), and `extra_args` (raw passthrough flags
for the underlying script). `skillopt_train` also accepts `num_epochs`,
`batch_size`, `seed`, and `use_gate`. `use_gate` defaults to `true`; setting it
to `false` still records validation scores but force-accepts every candidate,
which changes the optimization semantics.
The MCP schema's `backend` argument follows the underlying script's
`--backend` choices. Role-specific and generic OpenAI-compatible backends can
be selected through config or `extra_args` (for example,
`--optimizer_backend openai_compatible --target_backend openai_compatible`);
see the repository's [backend guide](../../../docs/guide/new-backend.md) for
the required environment variables.
`KEY=VALUE` YAML overrides), and `extra_args` (raw passthrough flags for the
underlying script). `skillopt_train` also accepts `num_epochs`, `batch_size`,
`seed`, and `use_gate`.
Runs can be very long. The server's subprocess timeout defaults to 6 hours;
override it with the `SKILLOPT_RUN_TIMEOUT` environment variable (seconds).
plugins/copilot/skillopt/copilot-instructions.snippet.md
+3 -4
View File
@@ -23,10 +23,9 @@ Guidance:
- Always run `skillopt_list_configs` first if you don't already know a valid `config` path.
- `skillopt_train` and `skillopt_eval` are long-running and consume the user's
model backend/budget — confirm the `config`, `backend`, and model choices
with the user before launching. Surface the held-out gate result for training,
or the evaluation score and output directory for eval-only runs.
- For one-off YAML overrides use dotted `cfg_options` for structured configs
(e.g. `train.seed=123 train.batch_size=40`);
with the user before launching, and surface the held-out gate result when the
run finishes.
- For one-off YAML overrides use `cfg_options` (e.g. `seed=123 batch_size=40`);
for any other underlying flag use `extra_args`.
This is distinct from the **SkillOpt-Sleep** MCP server (`skillopt-sleep`,
plugins/devin/README.md
+4 -19
View File
@@ -54,28 +54,13 @@ Requires Python ≥ 3.10. No third-party packages — the server is pure stdlib.
| Tool | What it does |
|---|---|
| `sleep_status` | nights run so far + latest staged proposal |
| `sleep_dry_run` | preview cycle — no staging; a real backend still makes provider calls |
| `sleep_dry_run` | preview cycle — no staging, no changes |
| `sleep_run` | full cycle; stages a proposal for review |
| `sleep_adopt` | apply the staged proposal; syncs skill to the workspace |
| `sleep_harvest` | debug: list the recurring tasks mined |
| `sleep_schedule` | install a nightly cron entry (`--hour` / `--minute`) |
| `sleep_unschedule` | remove the nightly cron entry |
Default backend is `mock` (no API spend); the `claude`, `codex`, and `copilot`
backends use the corresponding authenticated CLI and budget. The `handoff`
backend runs the cycle with no model subprocess or API key — the engine writes
pending model calls to `.skillopt-sleep-handoff/PROMPTS.md` + `pending.json`
(exit code 3) and resumes after answers are placed in `answers/<id>.md`; re-run
`sleep_run` with the same arguments to resume. The seven tools call the same
`python -m skillopt_sleep` actions as the other shared-engine integrations.
## Data boundary
The Devin harvester reads local ATIF transcripts, agentmemory, and skill files
and converts them into the engine's session format. The `mock` backend keeps
that workflow local. A real backend sends truncated excerpts and derived tasks
to the selected provider for mining, replay, judging, and reflection. The
conversion step is not a guarantee that outbound prompts contain no secrets;
review sensitive sources and provider policy before enabling a real backend.
See the [shared data-boundary guidance](../README.md#data-boundary) and
[implemented CLI reference](../README.md#supported-cli-surface).
Default backend is `mock` (no API spend); `--backend claude|codex` uses your own
budget. Same engine and `sleep_*` interface as the other plugins — all call
`python -m skillopt_sleep`.
plugins/devin/devin-rules.snippet.md
+9 -31
View File
@@ -3,39 +3,17 @@
You have access to a nightly self-evolution cycle via the `skillopt-sleep` MCP
server. Use these tools to improve your long-term skills over time:
- **`sleep_status`** — refresh the converted local cache, then show how many
nights have run and the latest staged proposal
- **`sleep_dry_run`** — refresh the converted local cache and preview a cycle
without engine staging/adoption; a real backend still makes provider calls
- **`sleep_run`** — run a full cycle; stages a proposal by default, while an
explicit `auto_adopt` may also update live files
- **`sleep_adopt`** — apply the staged proposal, then sync the managed skill to
`.devin/skills/skillopt-sleep-learned/SKILL.md` when `project` is the Devin
workspace and that workspace already contains a `.devin/` directory
- **`sleep_status`** — how many nights have run + the latest staged proposal
- **`sleep_dry_run`** — preview a cycle without changing anything
- **`sleep_run`** — run a full cycle; stages a proposal for review
- **`sleep_adopt`** — apply the staged proposal to `.devin/skills/skillopt-sleep-learned/SKILL.md`
- **`sleep_harvest`** — debug: list the recurring tasks mined from recent sessions
- **`sleep_schedule`** / **`sleep_unschedule`** — low-level shared-engine cron
controls; the current scheduled command does not run Devin's conversion step,
so do not use it as an unattended Devin-harvest workflow
- **`sleep_schedule`** / **`sleep_unschedule`** — install/remove a nightly cron run
When a user asks about the sleep cycle or skill evolution, prefer calling these
tools over explaining the concept.
When a user asks about the sleep cycle, skill evolution, or improving your
long-term memory, prefer calling these tools over explaining the concept.
Always pass the absolute Devin workspace as `project`, especially for
`sleep_adopt`. Default backend is `mock` (no provider calls). The `claude`,
`codex`, and `copilot` backend values use the corresponding installed and
authenticated CLI; they do not require this plugin to implement a separate
API-key flow. The `handoff` backend runs the cycle with no model subprocess
or API key — the engine writes pending model calls to
`.skillopt-sleep-handoff/` and exits; answer each prompt in a fresh context
and re-run `sleep_run` to resume (typically 3–6 rounds).
The Devin conversion and mock workflow stay local. A real backend sends
truncated transcript excerpts and derived tasks to the selected provider for
mining, replay, judging, and reflection; conversion is not a guarantee that
outbound prompts contain no secrets. Review local sources and provider policy
before selecting a real backend.
For a reviewed task file, pass `tasks_file`; before using it with a real backend,
inspect/redact it and ensure its metadata contains `"reviewed": true`.
Default backend is `mock` (no API spend). Pass `backend: "claude"` or
`backend: "codex"` with your own API key for real LLM-driven optimization.
Place this file at `.devin/rules/skillopt-sleep.md` in your workspace.
plugins/devin/mcp_server.py
+2 -2
View File
@@ -62,8 +62,8 @@ _TOOL_SCHEMA = {
"properties": {
"project": {"type": "string",
"description": "Project dir to evolve (default: cwd)."},
"backend": {"type": "string", "enum": ["mock", "claude", "codex", "copilot", "handoff"],
"description": "mock = no API spend (default); claude/codex/copilot = real; handoff = session answers prompts, no API subprocess."},
"backend": {"type": "string", "enum": ["mock", "claude", "codex", "copilot"],
"description": "mock = no API spend (default); claude/codex/copilot = real."},
"scope": {"type": "string", "enum": ["invoked", "all"],
"description": "Harvest scope (default: invoked project only)."},
"source": {"type": "string", "enum": ["claude", "codex", "auto"],
plugins/openclaw/README.md
+84 -87
View File
@@ -1,115 +1,112 @@
# OpenClaw reference adaptation for SkillOpt-Sleep
# OpenClaw Plugin for SkillOpt-Sleep
This directory is a contributed reference for connecting
[SkillOpt-Sleep](https://github.com/microsoft/SkillOpt) to
[OpenClaw](https://github.com/openclaw/openclaw) with a custom DeepSeek/Ollama
backend.
Thin shell for running [SkillOpt-Sleep](https://github.com/microsoft/SkillOpt) on [OpenClaw](https://github.com/openclaw/openclaw).
> **Reference status.** This is not one of the shared, plug-and-play
> `skillopt_sleep` wrappers. Several scripts and the sample config contain
> environment-specific absolute paths and assumptions from the original setup,
> and the contributed wrapper has unresolved Python 3.10 syntax and backend
> factory-signature gaps. The current checkout is not directly runnable; treat
> it as porting source material, not an installation.
## What it does
## Included components
Adds a nightly "sleep cycle" to any OpenClaw agent. The cycle:
| File | Purpose |
|---|---|
| `run_sleep.py` | custom cycle entry point |
| `skillopt_sleep_openclaw.py` | DeepSeek Chat Completions backend plus local Ollama embeddings |
| `run_sleep_cron.sh` | category-oriented cron wrapper |
| `slash_sleep.py` | experimental `/sleep` command helper |
| `config.json` | example engine configuration |
| `SKILL.md` | OpenClaw skill manifest |
| `tests/*.json` | example task sets for research, DevOps, and wiki workflows |
1. **Harvests** recent session transcripts from `~/.openclaw/agents/<name>/sessions/*.jsonl`
2. **Mines** recurring task patterns using the optimizer LLM
3. **Replays** each pattern with the current `SKILL.md` (baseline) and a candidate `SKILL.md` (with proposed edits)
4. **Gates** the candidate against the held-out score (rejects regressions)
5. **Stages** the accepted proposal in `~/.skillopt-sleep/staging/<night>/`
6. Leaves adoption to the operator (Ethan)
The adaptation imports the shared engine but registers its own backend and
maintains its own wrapper behavior. Changes to the shared CLI documentation do
not automatically make every option available through these custom scripts.
Nothing live changes until you adopt. Every adopt backs up first.
## Intended cycle
## Install
```text
harvest supported session data or load a task file
→ replay with the current skill
→ propose bounded edits
→ validate the candidate on held-out tasks
→ stage a proposal for operator review
```
The intended safety boundary is manual adoption: review the generated report and
staged files before changing a live skill.
## Adapt before use
1. Clone SkillOpt into a location you control:
The plugin is a thin wrapper around the engine at `~/.openclaw/workspace/SkillOpt/skillopt_sleep/`:
```bash
# 1. Clone the engine (one-time)
cd ~/.openclaw/workspace
git clone https://github.com/microsoft/SkillOpt.git
cd SkillOpt/plugins/openclaw
# 2. Install the OpenClaw skill (this folder)
ln -s /path/to/openclaw ~/.openclaw/workspace/skills/skillopt-sleep
# 3. Configure
cp ~/.openclaw/workspace/skills/skillopt-sleep/config.json ~/.skillopt-sleep/config.json
$EDITOR ~/.skillopt-sleep/config.json
# Set backend = "openclaw-deepseek"
# Set model = "deepseek-v4-pro" (or "deepseek-v4-flash" for budget)
# 4. Set API key
echo 'export DEEPSEEK_API_KEY="sk-..."' >> ~/.openclaw/.env
# 5. Add the nightly cron
(crontab -l 2>/dev/null; echo "0 3 * * * cd ~/.openclaw/workspace/skills/skillopt-sleep && bash run_sleep_cron.sh >> ~/.skillopt-sleep/nightly.log 2>&1") | crontab -
```
2. Inspect and replace the sample absolute paths in `run_sleep.py`,
`slash_sleep.py`, `run_sleep_cron.sh`, and `config.json`. Confirm the engine
checkout, OpenClaw workspace, state directory, skill directory, and task-file
paths all point to isolated test locations.
## Use
3. Review `config.json`. In particular, do not assume that values such as
`max_tokens_per_night` or `replay_mode` are enforced by this custom wrapper
merely because they appear in the example config.
4. Supply credentials through your normal secret-management mechanism. Do not
commit a DeepSeek key or place it in a world-readable file.
5. Resolve every known porting gap listed in [`SKILL.md`](SKILL.md), add
isolated tests for your adapted backend, and verify that `--help` imports
cleanly on Python 3.10+. Only then start with a dry run and one reviewed
task file. The target command should be shaped like:
### Manual trigger
```bash
cd /path/to/SkillOpt/plugins/openclaw
python3 run_sleep.py --config /path/to/reviewed-config.json \
--tasks tests/research-cron-tasks.json --dry-run
# Run one cycle now
python3 ~/.openclaw/workspace/skills/skillopt-sleep/run_sleep.py
# Dry run (report only)
python3 ~/.openclaw/workspace/skills/skillopt-sleep/run_sleep.py --dry-run
# One category only
python3 ~/.openclaw/workspace/skills/skillopt-sleep/run_sleep.py --tasks tests/research-cron-tasks.json
```
6. Inspect the report, paths, network destinations, and proposed edits before
considering a non-dry run or scheduling.
### Slash command
## Data boundary
```bash
# In any OpenClaw session
/sleep status
/sleep run
/sleep run research-cron
/sleep dry-run
/sleep adopt # adopt most recent accepted proposal
/sleep reject # discard most recent
/sleep cost
```
The custom `openclaw-deepseek` backend sends task, skill, response, rubric, and
reflection content to the configured DeepSeek endpoint. Its embedding helper can
send truncated text to the configured local Ollama service. Do not assume these
outbound prompts have been fully redacted; inspect transcript/task inputs and the
provider's retention policy before using real data.
## Architecture
Use HTTPS for a remote DeepSeek-compatible endpoint. Keep any plaintext Ollama
endpoint on a trusted loopback interface. For a network-free engine smoke test,
use the shared SkillOpt-Sleep CLI with `--backend mock` rather than assuming this
custom wrapper is isolated.
```
plugins/openclaw/
├── README.md # this file
├── run_sleep_cron.sh # wrapper for cron invocation
├── run_sleep.py # main entry point
├── slash_sleep.py # /sleep command implementation
├── skillopt_sleep_openclaw.py # DeepSeek + Ollama backend
├── config.json # engine config
├── SKILL.md # OpenClaw skill manifest
└── tests/ # held-out test sets
├── research-cron-tasks.json
├── devops-tasks.json
└── wiki-tasks.json
```
## Scheduling
The OpenClaw shell is one engine (skillopt_sleep/) + one backend (DeepSeek/Ollama) + four thin wrappers (cron, slash, skill, tests).
`run_sleep_cron.sh` and the scheduling helpers are examples, not portable
installers. Adapt their paths, create log directories, verify their environment,
and run the exact command manually before adding a cron entry. Scheduled runs
must preserve the same manual-adoption and credential boundaries as interactive
runs.
## Why this matters for OpenClaw
## Validation scope
OpenClaw currently has no built-in "self-evolving skills" mechanism. The community has:
The bundled JSON files are example held-out task sets, not a universal OpenClaw
benchmark. Provider cost and quality depend on the selected model, task content,
number of calls, and pricing at run time; this reference does not promise a fixed
nightly cost. Validate the adapted workflow in an isolated workspace before
using it on live skills.
- **Manual skills** — Ethan writes them
- **LLM-generated skills** — one-shot, no validation
- **Self-revision** — unbounded, no quality bar
For the supported shared-engine CLI and its current flags, see the
[integration reference](../README.md#supported-cli-surface). For measured
SkillOpt-Sleep results and limitations, see
[`docs/sleep/RESULTS.md`](../../docs/sleep/RESULTS.md).
SkillOpt-Sleep adds a 4th option: **validated self-evolution**. The skill is the training target, the engine is the optimizer, the gate is the quality bar, the operator is the human-in-the-loop.
## Validation
Validated on the public [gbrain-evals](https://github.com/garrytan/gbrain-evals) `skillopt-v1` benchmark with real Claude and Codex (deficient skills 0.00 → 1.00 on held-out, all 4 seeds).
End-to-end test on our own 14-task held-out set: pipeline runs, gate correctly rejects non-improvements, staging artifacts land in `~/.skillopt-sleep/staging/<night>/`.
## Cost
Measured: ~$0.02/night with `deepseek-v4-pro` at 12 tasks/night. ~$0.59/month, $7.18/year.
## License
MIT, consistent with SkillOpt core.
MIT (same as SkillOpt core).
plugins/openclaw/SKILL.md
+107 -81
View File
@@ -1,103 +1,129 @@
---
name: skillopt-sleep
description: Reference-only OpenClaw adaptation of SkillOpt-Sleep. Use it to study or port the contributed DeepSeek wrapper, not as a ready-to-run installation.
description: Validate and refine agent skills through nightly sleep cycles with held-out gates. Wraps Microsoft's SkillOpt-Sleep engine for the OpenClaw/DeepSeek stack.
---
# SkillOpt-Sleep OpenClaw reference adaptation
# skillopt-sleep — OpenClaw Adaptation of Microsoft SkillOpt-Sleep
This directory is a contributed **reference**, not a supported, plug-and-play
OpenClaw integration. It illustrates one way to connect the shared
`skillopt_sleep` cycle to a custom DeepSeek Chat Completions backend and a set of
environment-specific task fixtures.
A nightly self-improvement loop that reads our session transcripts, mines recurring workflow patterns, replays them with proposed skill edits, and gates the proposals against a held-out test set. Only improvements that beat baseline are staged for human adoption.
Do not run or schedule the files unchanged. Several scripts and the sample
configuration preserve assumptions from the contributor's original machine,
and parts of the wrapper have not yet been ported to the current shared-engine
interfaces. Start with the directory's [README.md](README.md), which is the
authoritative status and adaptation guide.
## When To Use
## What is included
- After Hermes's Weekly Skill Review (or as its replacement)
- When a skill is being used 10+ times/week and could be tighter
- Before promoting a new skill from `skill-proposals/` to `skills/`
- When a skill regresses in observed quality
- `skillopt_sleep_openclaw.py` — a contributed DeepSeek backend prototype. It
also contains an Ollama embedding helper, but that helper is not wired into
the current shared sleep cycle.
- `run_sleep.py` — a custom cycle wrapper with environment-specific paths and a
backend-registration shim.
- `slash_sleep.py` — an experimental command helper written for an older
staging-manifest shape.
- `run_sleep_cron.sh` — a machine-specific category runner, not a portable cron
installer.
- `config.json` — a sample configuration, not a set of guaranteed or enforced
runtime limits.
- `tests/*.json` — example task fixtures from one environment, not a universal
OpenClaw benchmark.
## What It Does (One Cycle)
## Known porting gaps
```
harvest session transcripts -> mine recurring task patterns
-> replay each pattern (current skill vs proposed)
-> GATE: must improve held-out score
-> stage proposal
-> Ethan adopts (manual)
```
Before treating this as an integration, a maintainer must at least:
Nothing live changes until Ethan adopts. Every adopt backs up first.
1. Replace every absolute workspace, repository, state, skill, log, and task
path with explicit user configuration.
2. Update the custom backend factory to the current `get_backend` call contract,
including the project directory, and update its backend methods and edit
records to the current protocol.
3. Replace the experimental adoption logic with the current staging manifest
and `skillopt_sleep.staging.adopt` behavior. Current staging artifacts use
`proposed_SKILL.md` / `proposed_CLAUDE.md`, `manifest.json`, and report files;
they do not expose the old `manifest.proposed_skill` field.
4. Decide how real OpenClaw transcripts are converted into a supported session
format. Pointing `claude_home` at an arbitrary agent directory does not by
itself make its files Claude Code-compatible JSONL.
5. Build scheduling around the adapted wrapper. The shared scheduler launches
the shared CLI; it does not automatically preserve this custom backend or
its category task-file flow.
6. Add isolated end-to-end tests for dry-run, accepted/rejected gates, staging,
adoption and backup, credential failure, and scheduled execution.
## Architecture
Until those gaps are resolved, use the supported shared
`python -m skillopt_sleep` CLI with `--backend mock` to test SkillOpt-Sleep itself,
and treat this directory only as source material for a future OpenClaw port.
```
skills/skillopt-sleep/
├── SKILL.md # this file
├── config.json # engine config (backend, budgets, etc.)
├── run_sleep.py # entry point
└── skillopt_sleep_openclaw.py # DeepSeek/Ollama backend
```
## Shared-engine features are not wrapper features
The engine itself is at `~/.openclaw/workspace/SkillOpt/skillopt_sleep/` (cloned from microsoft/SkillOpt).
At this revision the supported shared CLI backends are `mock`, `claude`,
`codex`, `copilot`, `handoff`, and `azure_openai`; the
[plugin integration reference](../README.md#supported-cli-surface) is the
authoritative list. The shared engine can consolidate a selected skill and
project `CLAUDE.md` memory (controlled by `evolve_skill` and `evolve_memory`),
and its `schedule` / `unschedule` actions manage shared-engine cron entries.
Those capabilities do **not** make the custom OpenClaw wrapper portable: the
shared scheduler will not invoke the prototype backend or its category
fixtures. Use the shared documentation for those features, not this reference
SKILL.
## Usage
## Data and credential boundary
```bash
# Run one cycle with current config
cd ~/.openclaw/workspace/skills/skillopt-sleep
python3 run_sleep.py
The prototype DeepSeek backend sends task, skill, memory, response, rubric, and
reflection content to its configured Chat Completions endpoint. Its source also
contains a helper that can send text to an Ollama service if a future port wires
that helper into the cycle. Neither path should be assumed to remove every
secret or private detail.
# Dry run (report only, no staging)
python3 run_sleep.py --dry-run
Before any port is tested with real data:
# Use a pre-built task set (recommended for testing)
python3 run_sleep.py --tasks tests/research-cron-tasks.json
```
- use isolated, synthetic or explicitly reviewed task files;
- replace sample business names, personal references, URLs, and machine paths;
- load credentials through the operator's secret-management mechanism;
- verify TLS and retention policy for every remote endpoint; and
- inspect all staged artifacts before adoption.
## Scheduling
The bundled fixtures are examples only. Their scores and any old cost estimates
do not establish effectiveness, safety, or a stable nightly price for another
OpenClaw deployment.
```bash
python3 slash_sleep.py schedule --hour 3 --minute 17
python3 slash_sleep.py unschedule
python3 slash_sleep.py unschedule --all
```
## Further information
Installs a nightly cron entry using the shared SkillOpt-Sleep scheduler. This is an alternative to the external `run_sleep_cron.sh` script.
- [OpenClaw README](README.md) — current reference status and adaptation checklist
- [plugin integration reference](../README.md) — supported shared-engine CLI
surface and data boundary
- [SkillOpt-Sleep documentation](../../docs/sleep/README.md) — concepts,
results, and limitations
## Alternative backends
Contributions that turn this reference into a portable integration should add
tests and update all three documents together.
While OpenClaw defaults to `openclaw-deepseek` (DeepSeek V4 Pro + Ollama), the shared engine also supports:
- `--backend mock` — deterministic, no API spend (for testing)
- `--backend claude` — uses the Claude CLI
- `--backend codex` — uses the Codex CLI
- `--backend copilot` — uses the GitHub Copilot CLI
These can be used via the engine directly (`python -m skillopt_sleep`).
## Shared-engine flags
When invoking the engine directly, all standard flags are available:
- `--source codex` / `--source auto` — harvest from Codex Desktop sessions
- `--tasks-file PATH` — use a pre-built task set
- `--target-skill-path PATH` — explicit SKILL.md target
- `--max-tasks N` / `--max-sessions N` — cap workload
- `--progress` — print phase progress
- `--json` — machine-readable output
- `--auto-adopt` — auto-adopt if gate passes
Config keys: `preferences`, `gate_mode`, `gate_metric`, `dream_rollouts`, `recall_k`, `evolve_memory`, `evolve_skill`.
## Config (config.json)
Key knobs:
- `backend: "openclaw-deepseek"` — our custom backend
- `model: "deepseek-v4-pro"` — optimizer model
- `edit_budget: 3` — max bounded edits per night
- `gate_mode: "on"` — validation-gated (rejects regressions)
- `auto_adopt: false` — require Ethan to adopt manually
- `max_tasks_per_night: 12` — cap to control cost
## Cost Estimate
Per night: 12 tasks × (1 attempt + 1 judge + 1 reflect) × ~$0.005/1K tokens × ~3K tokens/call ≈ **$0.50-2.00/night**.
## Outputs
- Report: `~/.skillopt-sleep/state.json` (running totals)
- Staging: `~/.skillopt-sleep/staging/<night>/`
- `report.md` — readable summary
- `best_skill.md` — proposed skill
- `edits.json` — bounded edit list
- `before.md` / `after.md` — diffs
## Held-Out Test Sets (Phase 2)
Located at `tests/<category>-tasks.json`. Each task has:
- `prompt` — the recurring task
- `reference` — exact-match gold answer
- `rubric` — soft score rubric (0-1)
- `domain` — research/devops/wiki/etc.
Currently building for 3 categories:
- research-cron-output
- devops-infrastructure-check
- wiki-canonical-guide
## When NOT To Use
- For a one-off workflow (not a recurring pattern)
- During a crisis/incident (humans must lead)
- When session transcripts are < 24h old (not enough signal)
- For skills < 300 tokens (over-optimization risk)
plugins/run-sleep.cmd
-122
View File
@@ -1,122 +0,0 @@
@echo off
setlocal enabledelayedexpansion
:: Resolve REPO_ROOT
set "SCRIPT_DIR=%~dp0"
:: Strip trailing backslash
set "SCRIPT_DIR=%SCRIPT_DIR:~0,-1%"
set "REPO_ROOT="
if exist "%SCRIPT_DIR%\..\skillopt_sleep" (
cd /d "%SCRIPT_DIR%\.."
set "REPO_ROOT=%CD%"
goto root_resolved
)
if not "%CLAUDE_PLUGIN_ROOT%"=="" if exist "%CLAUDE_PLUGIN_ROOT%\..\..\skillopt_sleep" (
cd /d "%CLAUDE_PLUGIN_ROOT%\..\.."
set "REPO_ROOT=%CD%"
goto root_resolved
)
if not "%SKILLOPT_SLEEP_REPO%"=="" if exist "%SKILLOPT_SLEEP_REPO%\skillopt_sleep" (
set "REPO_ROOT=%SKILLOPT_SLEEP_REPO%"
goto root_resolved
)
:: Search upward from current directory
set "d=%CD%"
:loop
if exist "!d!\skillopt_sleep" (
set "REPO_ROOT=!d!"
goto root_resolved
)
for %%I in ("!d!") do set "parent=%%~dpI"
:: Strip trailing backslash from parent if it's not root
if "!parent!"=="!d!" goto root_resolved
set "parent=!parent:~0,-1!"
if "!parent!"=="" goto root_resolved
set "d=!parent!"
goto loop
:root_resolved
if "%REPO_ROOT%"=="" goto fallback_mode
:: ── Source Checkout Mode ───────────────────────────────────────────
set "PY="
if not "%SKILLOPT_SLEEP_PYTHON%"=="" (
set "PY=%SKILLOPT_SLEEP_PYTHON%"
goto py_found
)
for %%p in (python3.exe python.exe py.exe) do (
where %%p >nul 2>nul
if !errorlevel! equ 0 (
%%p -c "import sys; sys.exit(0 if sys.version_info >= (3, 10) else 1)" >nul 2>nul
if !errorlevel! equ 0 (
set "PY=%%p"
goto py_found
)
)
)
:py_found
if "%PY%"=="" (
echo [sleep] ERROR: need Python >= 3.10 (found none). >&2
exit /b 1
)
cd /d "%REPO_ROOT%"
if "%~1" == "" (
"%PY%" -m skillopt_sleep status
) else (
"%PY%" -m skillopt_sleep %*
)
exit /b !errorlevel!
:: ── Fallback Mode (No Source Checkout) ─────────────────────────────
:fallback_mode
:: Fallback 1: skillopt-sleep CLI on PATH (uv tool install / pipx / pip install).
where skillopt-sleep >nul 2>nul
if !errorlevel! neq 0 goto try_fallback_2
if "%~1" == "" (
skillopt-sleep status
) else (
skillopt-sleep %*
)
exit /b !errorlevel!
:try_fallback_2
:: Fallback 2: importable as a module (pip install into the active Python).
set "PY="
for %%p in (python3.exe python.exe py.exe) do (
where %%p >nul 2>nul
if !errorlevel! equ 0 (
%%p -c "import sys; sys.exit(0 if sys.version_info >= (3, 10) else 1)" >nul 2>nul
if !errorlevel! equ 0 (
%%p -c "import skillopt_sleep" >nul 2>nul
if !errorlevel! equ 0 (
set "PY=%%p"
goto py_import_found
)
)
)
)
:py_import_found
if "%PY%" == "" goto not_found
if "%~1" == "" (
"%PY%" -m skillopt_sleep status
) else (
"%PY%" -m skillopt_sleep %*
)
exit /b !errorlevel!
:not_found
echo [sleep] ERROR: could not locate the skillopt_sleep package. >&2
echo [sleep] Install it with 'uv tool install skillopt' or 'pip install skillopt', >&2
echo [sleep] or set SKILLOPT_SLEEP_REPO to a clone of the SkillOpt repo. >&2
exit /b 1
plugins/run-sleep.ps1
-94
View File
@@ -1,94 +0,0 @@
# SkillOpt-Sleep shared runner for Windows PowerShell
# Resolves the repo root, picks Python >= 3.10, and runs the engine CLI.
#
# Usage: .\run-sleep.ps1 [status|run|dry-run|adopt|...] [args...]
$ErrorActionPreference = "Stop"
$ScriptDir = $PSScriptRoot
$RepoRoot = $null
if (Test-Path (Join-Path $ScriptDir "..\skillopt_sleep")) {
$RepoRoot = Resolve-Path (Join-Path $ScriptDir "..")
} elseif ($env:CLAUDE_PLUGIN_ROOT -and (Test-Path (Join-Path $env:CLAUDE_PLUGIN_ROOT "..\..\skillopt_sleep"))) {
$RepoRoot = Resolve-Path (Join-Path $env:CLAUDE_PLUGIN_ROOT "..\..")
} elseif ($env:SKILLOPT_SLEEP_REPO -and (Test-Path (Join-Path $env:SKILLOPT_SLEEP_REPO "skillopt_sleep"))) {
$RepoRoot = Resolve-Path $env:SKILLOPT_SLEEP_REPO
} else {
# search upward from current location
$d = Get-Item .
while ($d -and $d.FullName -ne $d.Root.FullName) {
if (Test-Path (Join-Path $d.FullName "skillopt_sleep")) {
$RepoRoot = $d.FullName
break
}
$d = Split-Path -Parent $d.FullName -ErrorAction SilentlyContinue | Get-Item -ErrorAction SilentlyContinue
}
}
$argsList = @($args)
if ($argsList.Count -eq 0) {
$argsList = @("status")
}
if ($RepoRoot) {
$Py = ""
if ($env:SKILLOPT_SLEEP_PYTHON) {
$Py = $env:SKILLOPT_SLEEP_PYTHON
} else {
foreach ($cand in @("python3", "python", "py")) {
$cmd = Get-Command $cand -ErrorAction SilentlyContinue
if ($cmd) {
$ver = & $cand -c "import sys; print('%d%d' % sys.version_info[:2])" 2>$null
if ($ver -and [int]$ver -ge 310) {
$Py = $cand
break
}
}
}
}
if (-not $Py) {
[Console]::Error.WriteLine("[sleep] ERROR: need Python >= 3.10 (found none).")
exit 1
}
Set-Location $RepoRoot
& $Py -m skillopt_sleep $argsList
exit $LASTEXITCODE
}
# No source checkout found — fall back to an installed engine.
# Fallback 1: skillopt-sleep CLI on PATH (uv tool install / pipx / pip install).
$cliCmd = Get-Command "skillopt-sleep" -ErrorAction SilentlyContinue
if ($cliCmd) {
& skillopt-sleep $argsList
exit $LASTEXITCODE
}
# Fallback 2: importable as a module (pip install into the active Python).
$Py = ""
foreach ($cand in @("python3", "python", "py")) {
$cmd = Get-Command $cand -ErrorAction SilentlyContinue
if ($cmd) {
$ver = & $cand -c "import sys; print('%d%d' % sys.version_info[:2])" 2>$null
if ($ver -and [int]$ver -ge 310) {
$hasModule = & $cand -c "import skillopt_sleep" 2>$null
if ($LASTEXITCODE -eq 0) {
$Py = $cand
break
}
}
}
}
if ($Py) {
& $Py -m skillopt_sleep $argsList
exit $LASTEXITCODE
}
[Console]::Error.WriteLine("[sleep] ERROR: could not locate the skillopt_sleep package.")
[Console]::Error.WriteLine("[sleep] Install it with 'uv tool install skillopt' or 'pip install skillopt',")
[Console]::Error.WriteLine("[sleep] or set SKILLOPT_SLEEP_REPO to a clone of the SkillOpt repo.")
exit 1
plugins/run-sleep.sh
+6 -34
View File
@@ -24,10 +24,11 @@ else
d="$(dirname "$d")"
done
fi
if [ "$#" -eq 0 ]; then set -- status; fi
if [ -z "${REPO_ROOT:-}" ]; then
echo "[sleep] ERROR: could not locate the skillopt_sleep package. Set SKILLOPT_SLEEP_REPO to the repo root." >&2
exit 1
fi
if [ -n "${REPO_ROOT:-}" ]; then
# Source checkout: run from repo root so skillopt_sleep/ is importable.
PY=""
# Allow explicit Python override (useful on macOS with old system Python).
if [ -n "${SKILLOPT_SLEEP_PYTHON:-}" ]; then
@@ -44,36 +45,7 @@ if [ -n "${REPO_ROOT:-}" ]; then
echo "[sleep] ERROR: need Python >= 3.10 (found none)." >&2
exit 1
fi
if [ "$#" -eq 0 ]; then set -- status; fi
cd "$REPO_ROOT"
exec "$PY" -m skillopt_sleep "$@"
fi
# No source checkout found — fall back to an installed engine.
# Fallback 1: skillopt-sleep CLI on PATH (uv tool install / pipx / pip install).
# Checked before the import fallback because uv tool install / pipx isolate the
# package from the system Python's import path, so `python -c "import
# skillopt_sleep"` would fail even though the CLI is available.
if command -v skillopt-sleep >/dev/null 2>&1; then
exec skillopt-sleep "$@"
fi
# Fallback 2: importable as a module (pip install into the active Python).
# Pick a Python >= 3.10 and check importability.
PY=""
for cand in python3.12 python3.11 python3.10 python3; do
if command -v "$cand" >/dev/null 2>&1; then
ver="$("$cand" -c 'import sys; print("%d%d" % sys.version_info[:2])' 2>/dev/null || echo 0)"
if [ "${ver:-0}" -ge 310 ] && "$cand" -c "import skillopt_sleep" >/dev/null 2>&1; then
PY="$cand"; break
fi
fi
done
if [ -n "$PY" ]; then
exec "$PY" -m skillopt_sleep "$@"
fi
echo "[sleep] ERROR: could not locate the skillopt_sleep package." >&2
echo "[sleep] Install it with 'uv tool install skillopt' or 'pip install skillopt'," >&2
echo "[sleep] or set SKILLOPT_SLEEP_REPO to a clone of the SkillOpt repo." >&2
exit 1
pyproject.toml
-3
View File
@@ -73,9 +73,6 @@ Issues = "https://github.com/microsoft/SkillOpt/issues"
# skillopt_webui = the Gradio dashboard (installed via the `webui` extra)
include = ["skillopt", "skillopt.*", "skillopt_sleep", "skillopt_sleep.*", "skillopt_webui", "skillopt_webui.*", "scripts*"]
[tool.setuptools.package-data]
"*" = ["*.md"]
[tool.ruff]
line-length = 120
target-version = "py310"
scripts/eval_only.py
+2 -4
View File
@@ -320,10 +320,8 @@ def main() -> None:
cfg.setdefault("optimizer_backend", "claude_chat")
cfg.setdefault("target_backend", "claude_chat")
elif backend in {"codex", "codex_exec"}:
if not _has_model_override("model.optimizer_backend", "optimizer_backend"):
cfg["optimizer_backend"] = "codex_exec"
if not _has_model_override("model.target_backend", "target_backend"):
cfg["target_backend"] = "codex_exec"
cfg.setdefault("optimizer_backend", "openai_chat")
cfg.setdefault("target_backend", "codex_exec")
elif backend == "claude_code_exec":
cfg.setdefault("optimizer_backend", "openai_chat")
cfg.setdefault("target_backend", "claude_code_exec")
scripts/materialize_searchqa.py
-22
View File
@@ -44,27 +44,6 @@ def load_manifest_ids(manifest_dir: Path) -> dict[str, list[str]]:
return split_ids
def _reject_duplicate_manifest_ids(split_ids: Mapping[str, Iterable[str]]) -> None:
seen: dict[str, str] = {}
duplicates: dict[str, list[str]] = {}
for split, ids in split_ids.items():
for item_id in ids:
if item_id in seen:
duplicates.setdefault(item_id, [seen[item_id]]).append(split)
else:
seen[item_id] = split
if duplicates:
preview = ", ".join(
f"{item_id} ({'/'.join(splits)})"
for item_id, splits in sorted(duplicates.items())[:5]
)
raise ValueError(
"SearchQA split manifest contains duplicate IDs across splits. "
f"First IDs: {preview}"
)
def _iter_dataset_rows(dataset: Mapping[str, Iterable[dict]]) -> Iterable[dict]:
for source_split in dataset.values():
yield from source_split
@@ -99,7 +78,6 @@ def materialize_searchqa_splits(
manifest_dir = manifest_dir.resolve()
output_dir = output_dir.resolve()
split_ids = load_manifest_ids(manifest_dir)
_reject_duplicate_manifest_ids(split_ids)
wanted_ids = {item_id for ids in split_ids.values() for item_id in ids}
selected: dict[str, dict] = {}
scripts/train.py
+2 -4
View File
@@ -438,10 +438,8 @@ def load_config(args: argparse.Namespace) -> dict:
flat.setdefault("optimizer_backend", "claude_chat")
flat.setdefault("target_backend", "claude_chat")
elif backend in {"codex", "codex_exec"}:
if not _has_model_override("model.optimizer_backend", "optimizer_backend"):
flat["optimizer_backend"] = "codex_exec"
if not _has_model_override("model.target_backend", "target_backend"):
flat["target_backend"] = "codex_exec"
flat.setdefault("optimizer_backend", "openai_chat")
flat.setdefault("target_backend", "codex_exec")
elif backend == "claude_code_exec":
flat.setdefault("optimizer_backend", "openai_chat")
flat.setdefault("target_backend", "claude_code_exec")
skillopt.html
+2 -4
View File
@@ -1778,8 +1778,6 @@
<a href="#evolution">Evolution</a>
<a href="#transfer">Transfer</a>
<a href="#citation">Citation</a>
<a href="https://microsoft.github.io/SkillOpt/blog/">Blog</a>
<a href="https://github.com/microsoft/SkillOpt/blob/main/docs/index.md" target="_blank" rel="noopener">Docs</a>
<a href="https://github.com/microsoft/SkillOpt" target="_blank" rel="noopener">Code</a>
</nav>
</header>
@@ -1917,7 +1915,7 @@
<h3>A skill is external state for an agent.</h3>
<p>
Instead of fine-tuning a model or hand-maintaining prompts, SkillOpt runs
the frozen agent on scored batches, asks an optimizer model to
the frozen agent on scored batches, asks a separate optimizer model to
propose structured edits, and accepts a candidate only when validation
performance improves.
</p>
@@ -2429,7 +2427,7 @@
<footer class="footer">
<span>SkillOpt: Executive Strategy for Self-Evolving Agent Skills</span>
<span><a href="https://microsoft.github.io/SkillOpt/blog/">Blog</a> / <a href="https://github.com/microsoft/SkillOpt/blob/main/docs/index.md" target="_blank" rel="noopener">Docs</a> / <a href="https://github.com/microsoft/SkillOpt" target="_blank" rel="noopener">Code</a> / <a href="#citation">Citation</a></span>
<span><a href="https://github.com/microsoft/SkillOpt" target="_blank" rel="noopener">Code</a> / <a href="#citation">Citation</a></span>
</footer>
</main>
<script>
skillopt/engine/trainer.py
+2 -4
View File
@@ -674,10 +674,8 @@ class ReflACTTrainer:
optimizer_backend = optimizer_backend or "claude_chat"
target_backend = target_backend or "claude_chat"
elif backend in {"codex", "codex_exec"}:
if optimizer_backend in (None, "", "openai_chat"):
optimizer_backend = "codex_exec"
if target_backend in (None, "", "openai_chat"):
target_backend = "codex_exec"
optimizer_backend = optimizer_backend or "openai_chat"
target_backend = target_backend or "codex_exec"
elif backend == "claude_code_exec":
optimizer_backend = optimizer_backend or "openai_chat"
target_backend = target_backend or "claude_code_exec"
skillopt/envs/_template/README.md
+7 -10
View File
@@ -28,16 +28,13 @@ This directory provides scaffold files for adding a new benchmark to SkillOpt.
`TemplateBenchmarkLoader → YourBenchmarkLoader`)
and fix the cross-import in `adapter.py`.
3. **Implement the TODO blocks** inside `adapter.py:rollout` and the
`_normalize_item` helper in `dataloader.py`. In addition to returning
`id`/`hard`/`soft`, persist each non-empty trajectory at
`<out_dir>/predictions/<id>/conversation.json`; the inherited `reflect`
method reads those files. Override `reflect` only for custom reflection
logic.
4. **Register** the adapter — add matching `try / except ImportError` blocks
to `_register_builtins()` in both `scripts/train.py` and
`scripts/eval_only.py`, mapping the registry key to your
`YourBenchmarkAdapter` class. There is no `BENCHMARK_REGISTRY` dict in
`skillopt/envs/__init__.py`; each CLI keeps its own lazy `_ENV_REGISTRY`.
`_normalize_item` helper in `dataloader.py`. (`reflect` is inherited from
`EnvAdapter`; override it only for custom reflection logic.)
4. **Register** the adapter — add a `try / except ImportError` block in
`scripts/train.py`'s `_register_builtins()` mapping the registry key
to your `YourBenchmarkAdapter` class. There is no
`BENCHMARK_REGISTRY` dict in `skillopt/envs/__init__.py`; the live
registry is `_ENV_REGISTRY` in `scripts/train.py`.
5. **Create the config** at `configs/your_benchmark/default.yaml`
(start from `config_template.yaml`). `_base_` is a **string path**,
not a list.
skillopt/envs/_template/config_template.yaml
+2 -2
View File
@@ -50,6 +50,6 @@ evaluation:
# ── Model ────────────────────────────────────────
# Override only what differs from the inherited defaults.
model:
optimizer_backend: openai_chat # openai_chat | claude_chat | qwen_chat | minimax_chat | codex_exec
target_backend: openai_chat # chat backends plus codex_exec / claude_code_exec
optimizer_backend: openai_chat # openai_chat | claude_chat | qwen_chat | minimax_chat
target_backend: openai_chat # … plus codex_exec / claude_code_exec for target only
reasoning_effort: medium
skillopt/model/__init__.py
+7 -144
View File
@@ -6,9 +6,7 @@ from typing import Any
from skillopt.model import azure_openai as _openai
from skillopt.model import claude_backend as _claude
from skillopt.model import codex_backend as _codex
from skillopt.model import minimax_backend as _minimax
from skillopt.model import openai_compatible_backend as _openai_compat
from skillopt.model import qwen_backend as _qwen
from skillopt.model.backend_config import ( # noqa: F401
configure_claude_code_exec,
@@ -42,14 +40,10 @@ def set_backend(name: str | None) -> str:
set_target_backend("claude_chat")
return "claude_chat"
if normalized == "codex":
set_optimizer_backend("codex_exec")
set_optimizer_backend("openai_chat")
set_target_backend("codex_exec")
return "codex"
if normalized == "codex_exec":
set_optimizer_backend("codex_exec")
set_target_backend("codex_exec")
return normalized
if normalized == "claude_code_exec":
if normalized in {"codex_exec", "claude_code_exec"}:
set_optimizer_backend("openai_chat")
set_target_backend(normalized)
return normalized
@@ -61,10 +55,6 @@ def set_backend(name: str | None) -> str:
set_optimizer_backend("openai_chat")
set_target_backend("minimax_chat")
return "minimax_chat"
if normalized in {"openai_compatible", "openai_compatible_chat", "openai-compatible", "compat"}:
set_optimizer_backend("openai_compatible")
set_target_backend("openai_compatible")
return "openai_compatible"
raise ValueError(f"Unsupported legacy backend: {name!r}")
@@ -78,14 +68,12 @@ def get_backend_name() -> str:
return "qwen_chat"
if optimizer == "openai_chat" and target == "openai_chat":
return "azure_openai"
if optimizer == "codex_exec" and target == "codex_exec":
if optimizer == "openai_chat" and target == "codex_exec":
return "codex"
if optimizer == "openai_chat" and target == "qwen_chat":
return "qwen_chat"
if optimizer == "openai_chat" and target == "minimax_chat":
return "minimax_chat"
if optimizer == "openai_compatible" and target == "openai_compatible":
return "openai_compatible"
return f"{optimizer}+{target}"
@@ -127,25 +115,6 @@ def chat_optimizer(
reasoning_effort=reasoning_effort,
timeout=timeout,
)
if get_optimizer_backend() == "openai_compatible":
return _openai_compat.chat_optimizer(
system=system,
user=user,
max_completion_tokens=max_completion_tokens,
retries=retries,
stage=stage,
reasoning_effort=reasoning_effort,
timeout=timeout,
)
if get_optimizer_backend() == "codex_exec":
return _codex.chat_optimizer(
system=system,
user=user,
max_completion_tokens=max_completion_tokens,
retries=retries,
stage=stage,
timeout=timeout,
)
return _openai.chat_optimizer(
system=system,
user=user,
@@ -194,20 +163,10 @@ def chat_target(
stage=stage,
reasoning_effort=reasoning_effort,
)
if get_target_backend() == "openai_compatible":
return _openai_compat.chat_target(
system=system,
user=user,
max_completion_tokens=max_completion_tokens,
retries=retries,
stage=stage,
reasoning_effort=reasoning_effort,
timeout=timeout,
)
if not is_target_chat_backend():
raise NotImplementedError(
"chat_target is only supported with target_backend=openai_chat, claude_chat, qwen_chat, minimax_chat, "
"or openai_compatible. Exec backends are handled in environment-specific rollout code."
"chat_target is only supported with target_backend=openai_chat, claude_chat, qwen_chat, or minimax_chat. "
"Exec backends are handled in environment-specific rollout code."
)
return _openai.chat_target(
system=system,
@@ -267,29 +226,6 @@ def chat_optimizer_messages(
return_message=return_message,
timeout=timeout,
)
if get_optimizer_backend() == "openai_compatible":
return _openai_compat.chat_optimizer_messages(
messages=messages,
max_completion_tokens=max_completion_tokens,
retries=retries,
stage=stage,
reasoning_effort=reasoning_effort,
tools=tools,
tool_choice=tool_choice,
return_message=return_message,
timeout=timeout,
)
if get_optimizer_backend() == "codex_exec":
return _codex.chat_optimizer_messages(
messages=messages,
max_completion_tokens=max_completion_tokens,
retries=retries,
stage=stage,
tools=tools,
tool_choice=tool_choice,
return_message=return_message,
timeout=timeout,
)
return _openai.chat_optimizer_messages(
messages=messages,
max_completion_tokens=max_completion_tokens,
@@ -349,22 +285,10 @@ def chat_target_messages(
tool_choice=tool_choice,
return_message=return_message,
)
if get_target_backend() == "openai_compatible":
return _openai_compat.chat_target_messages(
messages=messages,
max_completion_tokens=max_completion_tokens,
retries=retries,
stage=stage,
reasoning_effort=reasoning_effort,
tools=tools,
tool_choice=tool_choice,
return_message=return_message,
timeout=timeout,
)
if not is_target_chat_backend():
raise NotImplementedError(
"chat_target_messages is only supported with target_backend=openai_chat, claude_chat, qwen_chat, "
"minimax_chat, or openai_compatible. Exec backends are handled in environment-specific rollout code."
"chat_target_messages is only supported with target_backend=openai_chat, claude_chat, qwen_chat, or minimax_chat. "
"Exec backends are handled in environment-specific rollout code."
)
return _openai.chat_target_messages(
messages=messages,
@@ -463,28 +387,6 @@ def get_token_summary() -> dict:
summary[stage]["prompt_tokens"] += values["prompt_tokens"]
summary[stage]["completion_tokens"] += values["completion_tokens"]
summary[stage]["total_tokens"] += values["total_tokens"]
openai_compat_summary = _openai_compat.get_token_summary()
for stage, values in openai_compat_summary.items():
if stage == "_total":
continue
if stage not in summary:
summary[stage] = values
continue
summary[stage]["calls"] += values["calls"]
summary[stage]["prompt_tokens"] += values["prompt_tokens"]
summary[stage]["completion_tokens"] += values["completion_tokens"]
summary[stage]["total_tokens"] += values["total_tokens"]
codex_summary = _codex.get_token_summary()
for stage, values in codex_summary.items():
if stage == "_total":
continue
if stage not in summary:
summary[stage] = values
continue
summary[stage]["calls"] += values["calls"]
summary[stage]["prompt_tokens"] += values["prompt_tokens"]
summary[stage]["completion_tokens"] += values["completion_tokens"]
summary[stage]["total_tokens"] += values["total_tokens"]
total = {
"calls": 0,
"prompt_tokens": 0,
@@ -507,8 +409,6 @@ def reset_token_tracker() -> None:
_claude.reset_token_tracker()
_qwen.reset_token_tracker()
_minimax.reset_token_tracker()
_openai_compat.reset_token_tracker()
_codex.reset_token_tracker()
def configure_azure_openai(
@@ -622,44 +522,11 @@ def configure_minimax_chat(
)
def configure_openai_compatible(
*,
base_url: str | None = None,
api_key: str | None = None,
model: str | None = None,
temperature: float | str | None = None,
timeout_seconds: float | str | None = None,
max_tokens: int | str | None = None,
optimizer_base_url: str | None = None,
optimizer_api_key: str | None = None,
optimizer_model: str | None = None,
target_base_url: str | None = None,
target_api_key: str | None = None,
target_model: str | None = None,
) -> None:
_openai_compat.configure_openai_compatible(
base_url=base_url,
api_key=api_key,
model=model,
temperature=temperature,
timeout_seconds=timeout_seconds,
max_tokens=max_tokens,
optimizer_base_url=optimizer_base_url,
optimizer_api_key=optimizer_api_key,
optimizer_model=optimizer_model,
target_base_url=target_base_url,
target_api_key=target_api_key,
target_model=target_model,
)
def set_reasoning_effort(effort: str | None) -> None:
_openai.set_reasoning_effort(effort)
_claude.set_reasoning_effort(effort)
_qwen.set_reasoning_effort(effort)
_minimax.set_reasoning_effort(effort)
_openai_compat.set_reasoning_effort(effort)
_codex.set_reasoning_effort(effort)
def set_target_deployment(deployment: str) -> None:
@@ -667,13 +534,9 @@ def set_target_deployment(deployment: str) -> None:
_claude.set_target_deployment(deployment)
_qwen.set_target_deployment(deployment)
_minimax.set_target_deployment(deployment)
_openai_compat.set_target_deployment(deployment)
_codex.set_target_deployment(deployment)
def set_optimizer_deployment(deployment: str) -> None:
_openai.set_optimizer_deployment(deployment)
_claude.set_optimizer_deployment(deployment)
_qwen.set_optimizer_deployment(deployment)
_openai_compat.set_optimizer_deployment(deployment)
_codex.set_optimizer_deployment(deployment)
skillopt/model/backend_config.py
+6 -21
View File
@@ -49,18 +49,10 @@ CLAUDE_CODE_EXEC_MAX_THINKING_TOKENS = max(
def set_optimizer_backend(backend: str) -> None:
global OPTIMIZER_BACKEND
OPTIMIZER_BACKEND = normalize_backend_name(backend or "openai_chat")
if OPTIMIZER_BACKEND not in {
"openai_chat",
"claude_chat",
"qwen_chat",
"minimax_chat",
"openai_compatible",
"codex_exec",
}:
if OPTIMIZER_BACKEND not in {"openai_chat", "claude_chat", "qwen_chat", "minimax_chat"}:
raise ValueError(
f"Unsupported optimizer backend: {OPTIMIZER_BACKEND!r}. "
"Supported values are 'openai_chat', 'claude_chat', 'qwen_chat', 'minimax_chat', "
"'openai_compatible', and 'codex_exec'."
"Supported values are 'openai_chat', 'claude_chat', 'qwen_chat', and 'minimax_chat'."
)
os.environ["OPTIMIZER_BACKEND"] = OPTIMIZER_BACKEND
@@ -72,10 +64,10 @@ def get_optimizer_backend() -> str:
def set_target_backend(backend: str) -> None:
global TARGET_BACKEND
TARGET_BACKEND = normalize_backend_name(backend or "openai_chat")
if TARGET_BACKEND not in {"openai_chat", "claude_chat", "qwen_chat", "minimax_chat", "openai_compatible", "codex_exec", "claude_code_exec"}:
if TARGET_BACKEND not in {"openai_chat", "claude_chat", "qwen_chat", "minimax_chat", "codex_exec", "claude_code_exec"}:
raise ValueError(
f"Unsupported target backend: {TARGET_BACKEND!r}. "
"Supported values are 'openai_chat', 'claude_chat', 'qwen_chat', 'minimax_chat', 'openai_compatible', 'codex_exec', and 'claude_code_exec'."
"Supported values are 'openai_chat', 'claude_chat', 'qwen_chat', 'minimax_chat', 'codex_exec', and 'claude_code_exec'."
)
os.environ["TARGET_BACKEND"] = TARGET_BACKEND
@@ -89,18 +81,11 @@ def is_target_exec_backend() -> bool:
def is_optimizer_chat_backend() -> bool:
return OPTIMIZER_BACKEND in {
"openai_chat",
"claude_chat",
"qwen_chat",
"minimax_chat",
"openai_compatible",
"codex_exec",
}
return OPTIMIZER_BACKEND in {"openai_chat", "claude_chat", "qwen_chat", "minimax_chat"}
def is_target_chat_backend() -> bool:
return TARGET_BACKEND in {"openai_chat", "claude_chat", "qwen_chat", "minimax_chat", "openai_compatible"}
return TARGET_BACKEND in {"openai_chat", "claude_chat", "qwen_chat", "minimax_chat"}
def configure_codex_exec(
skillopt/model/claude_backend.py
+1 -6
View File
@@ -243,12 +243,7 @@ def _assistant_message_schema_wrapper() -> str:
def _run_claude_print(*, system: str, prompt: str, model: str, tools: list[dict[str, Any]] | None, tool_choice: str | dict[str, Any] | None, return_message: bool, timeout: int | None, attachments: list[dict[str, Any]] | None = None) -> tuple[str, dict[str, Any], dict[str, int]]:
effort = _normalize_reasoning_effort(REASONING_EFFORT)
# A lingering claude/node handle can make Windows cleanup raise WinError 32.
# Python 3.10+ supports suppressing cleanup failures while retaining
# TemporaryDirectory's normal lifecycle and best-effort removal behavior.
with tempfile.TemporaryDirectory(
prefix="skillopt_claude_", ignore_cleanup_errors=True,
) as temp_dir:
with tempfile.TemporaryDirectory(prefix="skillopt_claude_") as temp_dir:
copied_attachments = _copy_attachments_to_temp(attachments or [], temp_dir)
prompt_for_cli = _append_attachment_instructions(prompt, copied_attachments)
cmd = [CLAUDE_BIN, "-p", "--output-format", "json", "--permission-mode", CLAUDE_PERMISSION_MODE, "--add-dir", temp_dir]
skillopt/model/codex_backend.py
+8 -14
View File
@@ -12,12 +12,11 @@ import uuid
from typing import Any
from urllib.parse import unquote, urlparse
from skillopt.model.backend_config import get_codex_exec_config
from skillopt.model.common import (
CompatAssistantMessage,
CompatToolCall,
CompatToolFunction,
TokenTracker,
tracker,
)
@@ -29,7 +28,6 @@ OPTIMIZER_DEPLOYMENT = os.environ.get("OPTIMIZER_DEPLOYMENT", "gpt-4o")
TARGET_DEPLOYMENT = os.environ.get("TARGET_DEPLOYMENT", "gpt-4o")
REASONING_EFFORT: str | None = None
tracker = TokenTracker()
def _default_working_directory() -> str:
@@ -288,21 +286,20 @@ def _run_codex_exec(
timeout: int | None,
) -> tuple[str, dict[str, int]]:
with tempfile.TemporaryDirectory(prefix="skillopt_codex_") as temp_dir:
config = get_codex_exec_config()
output_path = os.path.join(temp_dir, "last_message.txt")
image_paths = _materialize_attachments(attachments, temp_dir)
profile = str(config.get("profile") or os.environ.get("CODEX_PROFILE", "")).strip()
reasoning_effort = str(REASONING_EFFORT or config.get("reasoning_effort") or "").strip()
command = [
str(config.get("path") or CODEX_BIN),
CODEX_BIN,
"exec",
"--json",
"--ephemeral",
"--profile",
CODEX_PROFILE,
"-c",
f"approval_policy={json.dumps(str(config.get('approval_policy') or 'never'))}",
"approval_policy=\"never\"",
"--sandbox",
str(config.get("sandbox") or CODEX_SANDBOX_MODE),
CODEX_SANDBOX_MODE,
"--skip-git-repo-check",
"--cd",
_default_working_directory(),
@@ -312,11 +309,8 @@ def _run_codex_exec(
output_path,
]
if profile:
command.extend(["--profile", profile])
if reasoning_effort and reasoning_effort != "none":
command.extend(["-c", f"model_reasoning_effort={json.dumps(reasoning_effort)}"])
if REASONING_EFFORT:
command.extend(["-c", f"model_reasoning_effort={json.dumps(REASONING_EFFORT)}"])
schema_path = None
if output_schema is not None:
skillopt/model/common.py
-5
View File
@@ -26,7 +26,6 @@ _BACKEND_DEFAULT_MODELS = {
"claude_code_exec": "claude-sonnet-4-6",
"qwen_chat": "Qwen/Qwen3.5-4B",
"minimax_chat": "MiniMax-M2.7",
"openai_compatible": "gpt-4o-mini",
}
_BACKEND_ALIASES = {
@@ -45,10 +44,6 @@ _BACKEND_ALIASES = {
"qwen_chat": "qwen_chat",
"minimax": "minimax_chat",
"minimax_chat": "minimax_chat",
"openai_compatible": "openai_compatible",
"openai_compatible_chat": "openai_compatible",
"openai-compatible": "openai_compatible",
"compat": "openai_compatible",
}
skillopt/model/openai_compatible_backend.py
-447
View File
@@ -1,447 +0,0 @@
"""Generic OpenAI-compatible chat backend for optimizer and target paths.
This backend talks to *any* service that exposes an OpenAI-compatible
``/chat/completions`` endpoint through the official ``openai`` SDK. A single
implementation therefore covers a large family of providers, for example:
* DeepSeek (``https://api.deepseek.com``)
* Groq (``https://api.groq.com/openai/v1``)
* Together AI (``https://api.together.xyz/v1``)
* Mistral / Fireworks / OpenRouter / Perplexity / xAI Grok
* Ollama (``http://localhost:11434/v1``)
* vLLM / SGLang / TGI self-hosted servers
* LiteLLM proxy (``http://localhost:4000``)
* Azure OpenAI and OpenAI itself
Unlike the Azure backend it never assumes Azure-specific auth or the Responses
API — it only needs a ``base_url`` and an ``api_key`` (some local servers accept
any key, so the key is optional and falls back to a harmless placeholder).
The module mirrors the callable surface of the other chat backends
(:mod:`skillopt.model.qwen_backend`, :mod:`skillopt.model.minimax_backend`) so
it can be selected as the optimizer and/or target backend and routed through
:mod:`skillopt.model`.
"""
from __future__ import annotations
import os
import threading
import time
from dataclasses import dataclass
from typing import Any
from openai import OpenAI
from skillopt.model.common import (
TokenTracker,
compat_message_from_chat_message,
default_model_for_backend,
usage_from_openai_usage,
)
BACKEND_NAME = "openai_compatible"
# A neutral, widely-available default. Real deployments should set the model
# explicitly (e.g. "deepseek-chat", "llama-3.3-70b-versatile", "qwen2.5:7b").
_DEFAULT_BASE_URL = "https://api.openai.com/v1"
@dataclass
class OpenAICompatibleConfig:
base_url: str
api_key: str
deployment: str
timeout_seconds: float
max_tokens: int
temperature: float | None
def _parse_optional_float(value: Any) -> float | None:
if value is None:
return None
raw = str(value).strip()
return float(raw) if raw else None
def _parse_int(value: Any, default: int) -> int:
if value is None:
return default
raw = str(value).strip()
return int(raw) if raw else default
def _role_env(role: str, key: str, default: str) -> str:
"""Resolve a config value, preferring role-specific over shared env vars."""
role_key = f"{role.upper()}_OPENAI_COMPATIBLE_{key}"
generic_key = f"OPENAI_COMPATIBLE_{key}"
return os.environ.get(role_key) or os.environ.get(generic_key) or default
def _initial_config(role: str) -> OpenAICompatibleConfig:
role_upper = role.upper()
deployment_env = "OPTIMIZER_DEPLOYMENT" if role == "optimizer" else "TARGET_DEPLOYMENT"
return OpenAICompatibleConfig(
base_url=_role_env(role, "BASE_URL", _DEFAULT_BASE_URL),
api_key=_role_env(role, "API_KEY", ""),
deployment=(
os.environ.get(f"{role_upper}_OPENAI_COMPATIBLE_MODEL")
or os.environ.get("OPENAI_COMPATIBLE_MODEL")
or os.environ.get(deployment_env)
or default_model_for_backend(BACKEND_NAME)
),
timeout_seconds=float(_role_env(role, "TIMEOUT_SECONDS", "300") or 300),
max_tokens=_parse_int(_role_env(role, "MAX_TOKENS", "8000"), 8000),
temperature=_parse_optional_float(_role_env(role, "TEMPERATURE", "")),
)
OPTIMIZER_CONFIG = _initial_config("optimizer")
TARGET_CONFIG = _initial_config("target")
_config_lock = threading.Lock()
_client_lock = threading.Lock()
tracker = TokenTracker()
_optimizer_client: OpenAI | None = None
_target_client: OpenAI | None = None
def _config_for(role: str) -> OpenAICompatibleConfig:
return OPTIMIZER_CONFIG if role == "optimizer" else TARGET_CONFIG
def _build_client(config: OpenAICompatibleConfig) -> OpenAI:
return OpenAI(
base_url=config.base_url.rstrip("/") or _DEFAULT_BASE_URL,
# Some OpenAI-compatible servers (Ollama, vLLM, local proxies) do not
# require an API key. The SDK still expects a non-empty string, so fall
# back to a harmless placeholder when none is configured.
api_key=config.api_key or "dummy",
timeout=config.timeout_seconds,
)
def _get_client(role: str) -> OpenAI:
global _optimizer_client, _target_client
with _client_lock:
if role == "optimizer":
if _optimizer_client is None:
_optimizer_client = _build_client(OPTIMIZER_CONFIG)
return _optimizer_client
if _target_client is None:
_target_client = _build_client(TARGET_CONFIG)
return _target_client
def _reset_clients() -> None:
global _optimizer_client, _target_client
with _client_lock:
_optimizer_client = None
_target_client = None
def count_tokens(text: str, model: str | None = None) -> int:
"""Best-effort token count for a string.
Uses ``tiktoken`` when available (per-model encoding, falling back to the
``cl100k_base`` encoding). If ``tiktoken`` is not installed or fails — which
is common for non-OpenAI models served through compatible APIs — it falls
back to a character-based estimate of roughly four characters per token.
"""
if not text:
return 0
try:
import tiktoken
try:
encoding = tiktoken.encoding_for_model(model or "gpt-4o")
except Exception: # noqa: BLE001 - unknown/non-OpenAI model name
encoding = tiktoken.get_encoding("cl100k_base")
return len(encoding.encode(text))
except Exception: # noqa: BLE001 - tiktoken missing or encoding failure
# Rough heuristic: ~4 characters per token for English-like text.
return max(1, (len(text) + 3) // 4)
def _chat_messages_impl(
messages: list[dict[str, Any]],
max_completion_tokens: int,
retries: int,
stage: str,
*,
role: str,
tools: list[dict[str, Any]] | None = None,
tool_choice: str | dict[str, Any] | None = None,
return_message: bool = False,
deployment: str | None = None,
timeout: float | None = None,
) -> tuple[Any, dict[str, int]]:
config = _config_for(role)
client = _get_client(role)
kwargs: dict[str, Any] = {
"model": deployment or config.deployment,
"messages": messages,
# ``max_tokens`` (rather than ``max_completion_tokens``) is the field
# understood by the broadest set of OpenAI-compatible providers.
"max_tokens": min(max_completion_tokens, config.max_tokens),
}
if config.temperature is not None:
kwargs["temperature"] = config.temperature
if tools:
kwargs["tools"] = tools
if tool_choice is not None:
kwargs["tool_choice"] = tool_choice
if timeout is not None:
kwargs["timeout"] = timeout
last_err: Exception | None = None
for attempt in range(retries):
try:
resp = client.chat.completions.create(**kwargs)
choices = getattr(resp, "choices", None) or []
if not choices:
raise RuntimeError(
f"OpenAI-compatible API returned no choices: {resp!r}"
)
message = choices[0].message
text = message.content or ""
usage_info = usage_from_openai_usage(getattr(resp, "usage", None))
tracker.record(
stage,
usage_info["prompt_tokens"],
usage_info["completion_tokens"],
)
if return_message:
return compat_message_from_chat_message(message), usage_info
return text, usage_info
except Exception as e: # noqa: BLE001
last_err = e
time.sleep(min(2 ** attempt, 30))
raise RuntimeError(
f"OpenAI-compatible chat call failed after {retries} retries: {last_err}"
)
# ── Public API (mirrors the other chat backends) ─────────────────────────────
def chat_optimizer(
system: str,
user: str,
max_completion_tokens: int = 16384,
retries: int = 5,
stage: str = "optimizer",
reasoning_effort: str | None = None,
timeout: float | None = None,
) -> tuple[str, dict[str, int]]:
del reasoning_effort # not forwarded — kept for a uniform signature
messages = [
{"role": "system", "content": system},
{"role": "user", "content": user},
]
return _chat_messages_impl(
messages,
max_completion_tokens,
retries,
stage,
role="optimizer",
timeout=timeout,
)
def chat_target(
system: str,
user: str,
max_completion_tokens: int = 16384,
retries: int = 5,
stage: str = "target",
reasoning_effort: str | None = None,
timeout: float | None = None,
) -> tuple[str, dict[str, int]]:
del reasoning_effort
messages = [
{"role": "system", "content": system},
{"role": "user", "content": user},
]
return _chat_messages_impl(
messages,
max_completion_tokens,
retries,
stage,
role="target",
timeout=timeout,
)
def chat_optimizer_messages(
messages: list[dict[str, Any]],
max_completion_tokens: int = 16384,
retries: int = 5,
stage: str = "optimizer",
reasoning_effort: str | None = None,
*,
tools: list[dict[str, Any]] | None = None,
tool_choice: str | dict[str, Any] | None = None,
return_message: bool = False,
timeout: float | None = None,
) -> tuple[Any, dict[str, int]]:
del reasoning_effort
return _chat_messages_impl(
messages,
max_completion_tokens,
retries,
stage,
role="optimizer",
tools=tools,
tool_choice=tool_choice,
return_message=return_message,
timeout=timeout,
)
def chat_target_messages(
messages: list[dict[str, Any]],
max_completion_tokens: int = 16384,
retries: int = 5,
stage: str = "target",
reasoning_effort: str | None = None,
*,
tools: list[dict[str, Any]] | None = None,
tool_choice: str | dict[str, Any] | None = None,
return_message: bool = False,
timeout: float | None = None,
) -> tuple[Any, dict[str, int]]:
del reasoning_effort
return _chat_messages_impl(
messages,
max_completion_tokens,
retries,
stage,
role="target",
tools=tools,
tool_choice=tool_choice,
return_message=return_message,
timeout=timeout,
)
# ── Configuration / lifecycle ────────────────────────────────────────────────
def _update_config(
config: OpenAICompatibleConfig,
role: str,
*,
base_url: str | None = None,
api_key: str | None = None,
deployment: str | None = None,
temperature: float | str | None = None,
timeout_seconds: float | str | None = None,
max_tokens: int | str | None = None,
) -> None:
env_prefix = role.upper()
if base_url is not None:
config.base_url = str(base_url).strip() or config.base_url
os.environ[f"{env_prefix}_OPENAI_COMPATIBLE_BASE_URL"] = config.base_url
if api_key is not None:
config.api_key = str(api_key).strip()
os.environ[f"{env_prefix}_OPENAI_COMPATIBLE_API_KEY"] = config.api_key
if deployment is not None:
config.deployment = str(deployment).strip() or config.deployment
os.environ[f"{env_prefix}_OPENAI_COMPATIBLE_MODEL"] = config.deployment
if temperature is not None:
raw = str(temperature).strip()
config.temperature = float(raw) if raw else None
os.environ[f"{env_prefix}_OPENAI_COMPATIBLE_TEMPERATURE"] = raw
if timeout_seconds is not None:
config.timeout_seconds = float(timeout_seconds)
os.environ[f"{env_prefix}_OPENAI_COMPATIBLE_TIMEOUT_SECONDS"] = str(timeout_seconds)
if max_tokens is not None:
config.max_tokens = int(max_tokens)
os.environ[f"{env_prefix}_OPENAI_COMPATIBLE_MAX_TOKENS"] = str(max_tokens)
def configure_openai_compatible(
*,
base_url: str | None = None,
api_key: str | None = None,
model: str | None = None,
temperature: float | str | None = None,
timeout_seconds: float | str | None = None,
max_tokens: int | str | None = None,
optimizer_base_url: str | None = None,
optimizer_api_key: str | None = None,
optimizer_model: str | None = None,
target_base_url: str | None = None,
target_api_key: str | None = None,
target_model: str | None = None,
) -> None:
"""Configure the generic OpenAI-compatible backend at runtime.
Shared values apply to both the optimizer and target roles; the
``optimizer_*`` / ``target_*`` variants override them per role.
"""
with _config_lock:
if base_url is not None:
os.environ["OPENAI_COMPATIBLE_BASE_URL"] = str(base_url).strip()
if api_key is not None:
os.environ["OPENAI_COMPATIBLE_API_KEY"] = str(api_key).strip()
if model is not None:
os.environ["OPENAI_COMPATIBLE_MODEL"] = str(model).strip()
if temperature is not None:
os.environ["OPENAI_COMPATIBLE_TEMPERATURE"] = str(temperature).strip()
if timeout_seconds is not None:
os.environ["OPENAI_COMPATIBLE_TIMEOUT_SECONDS"] = str(timeout_seconds)
if max_tokens is not None:
os.environ["OPENAI_COMPATIBLE_MAX_TOKENS"] = str(max_tokens)
_update_config(
OPTIMIZER_CONFIG,
"optimizer",
base_url=optimizer_base_url if optimizer_base_url is not None else base_url,
api_key=optimizer_api_key if optimizer_api_key is not None else api_key,
deployment=optimizer_model if optimizer_model is not None else model,
temperature=temperature,
timeout_seconds=timeout_seconds,
max_tokens=max_tokens,
)
_update_config(
TARGET_CONFIG,
"target",
base_url=target_base_url if target_base_url is not None else base_url,
api_key=target_api_key if target_api_key is not None else api_key,
deployment=target_model if target_model is not None else model,
temperature=temperature,
timeout_seconds=timeout_seconds,
max_tokens=max_tokens,
)
_reset_clients()
def get_max_tokens() -> int:
return TARGET_CONFIG.max_tokens
def get_token_summary() -> dict[str, dict[str, int]]:
return tracker.summary()
def reset_token_tracker() -> None:
tracker.reset()
def set_reasoning_effort(effort: str | None) -> None:
# Reasoning effort is provider-specific and not universally supported by
# OpenAI-compatible endpoints, so it is intentionally a no-op here.
del effort
def set_target_deployment(deployment: str) -> None:
TARGET_CONFIG.deployment = deployment or default_model_for_backend(BACKEND_NAME)
os.environ["TARGET_DEPLOYMENT"] = TARGET_CONFIG.deployment
_reset_clients()
def set_optimizer_deployment(deployment: str) -> None:
OPTIMIZER_CONFIG.deployment = deployment or default_model_for_backend(BACKEND_NAME)
os.environ["OPTIMIZER_DEPLOYMENT"] = OPTIMIZER_CONFIG.deployment
_reset_clients()
skillopt/optimizer/scheduler.py
+2 -12
View File
@@ -61,12 +61,7 @@ class ConstantScheduler(LRScheduler):
class LinearScheduler(LRScheduler):
"""Linear decay from ``max_lr`` to ``min_lr`` over ``total_steps``.
For multi-step runs, the first call evaluates the first decay interval
(``t = 1 / total_steps``) and the ``total_steps``-th call returns
``min_lr``. Values are rounded to the nearest integer.
"""
"""Linear decay from ``max_lr`` to ``min_lr`` over ``total_steps``."""
def _compute_lr(self, step: int) -> int:
if self.total_steps <= 1:
@@ -77,12 +72,7 @@ class LinearScheduler(LRScheduler):
class CosineScheduler(LRScheduler):
"""Cosine annealing from ``max_lr`` to ``min_lr`` over ``total_steps``.
For multi-step runs, the first call evaluates the first decay interval
(``t = 1 / total_steps``) and the ``total_steps``-th call returns
``min_lr``. Intermediate steps follow a half-cosine curve.
"""
"""Cosine annealing from ``max_lr`` to ``min_lr`` over ``total_steps``."""
def _compute_lr(self, step: int) -> int:
if self.total_steps <= 1:
skillopt/types.py
+5 -5
View File
@@ -11,12 +11,12 @@ BatchSpec — from skillopt.datasets.base
"""
from __future__ import annotations
from dataclasses import dataclass, field
from dataclasses import fields as dc_fields
from dataclasses import dataclass, field, fields as dc_fields
from typing import Any, Literal
from skillopt.datasets.base import BatchSpec # noqa: F401
from skillopt.evaluation.gate import GateAction, GateResult # noqa: F401
from skillopt.datasets.base import BatchSpec # noqa: F401
# ── Atomic types ─────────────────────────────────────────────────────────
@@ -109,7 +109,7 @@ class RolloutResult:
"""
id: str
hard: float
hard: int
soft: float
n_turns: int = 0
fail_reason: str = ""
@@ -142,7 +142,7 @@ class RolloutResult:
extras = {k: v for k, v in d.items() if k not in known}
return cls(
id=str(d.get("id", "")),
hard=float(d.get("hard", 0)),
hard=int(d.get("hard", 0)),
soft=float(d.get("soft", 0.0)),
n_turns=int(d.get("n_turns", 0)),
fail_reason=str(d.get("fail_reason", "")),
skillopt/utils/json_utils.py
+4 -99
View File
@@ -71,96 +71,6 @@ def _top_level_brace_objects(text: str) -> list[str]:
return spans
def _top_level_bracket_arrays(text: str) -> list[str]:
"""Return balanced top-level ``[...]`` spans in ``text``.
This mirrors :func:`_top_level_brace_objects` for array responses. Arrays
nested inside a JSON object are not top-level array answers, so they are
ignored while the outer scanner is inside a ``{...}`` span.
"""
spans: list[str] = []
i, n = 0, len(text)
outer_in_str = False
outer_esc = False
# Precompute balanced object spans once. Looking ahead to EOF for every
# unmatched ``{`` makes malformed model output quadratic, while a stack
# keeps the scan linear and still lets a later valid array remain visible.
brace_ends = [-1] * n
brace_stack: list[int] = []
brace_in_str = False
brace_esc = False
for pos, current in enumerate(text):
if brace_in_str:
if brace_esc:
brace_esc = False
elif current == "\\":
brace_esc = True
elif current == '"':
brace_in_str = False
continue
if current == '"':
brace_in_str = True
elif current == "{":
brace_stack.append(pos)
elif current == "}" and brace_stack:
brace_ends[brace_stack.pop()] = pos
while i < n:
ch = text[i]
if outer_in_str:
if outer_esc:
outer_esc = False
elif ch == "\\":
outer_esc = True
elif ch == '"':
outer_in_str = False
i += 1
continue
if ch == '"':
outer_in_str = True
i += 1
continue
if ch == "{":
# Skip balanced objects, including arrays nested inside them. An
# unmatched brace may be ordinary prose, though, so do not let it
# poison the rest of the scan and hide a later valid array.
object_end = brace_ends[i]
i = object_end + 1 if object_end >= 0 else i + 1
continue
if ch != "[":
i += 1
continue
depth = 0
in_str = False
esc = False
start = i
while i < n:
ch = text[i]
if in_str:
if esc:
esc = False
elif ch == "\\":
esc = True
elif ch == '"':
in_str = False
elif ch == '"':
in_str = True
elif ch == "[":
depth += 1
elif ch == "]":
depth -= 1
if depth == 0:
spans.append(text[start:i + 1])
i += 1
break
i += 1
else:
break # unterminated final array
return spans
def _looks_json_like(span: str) -> bool:
"""Heuristic: does ``span`` look like an intended JSON object (vs. prose)?
@@ -253,15 +163,10 @@ def extract_json_array(text: str) -> list | None:
return json.loads(m.group(1))
except json.JSONDecodeError:
pass
parsed_arrays = []
for candidate in _top_level_bracket_arrays(text):
m = re.search(r"\[.*\]", text, re.DOTALL)
if m:
try:
parsed = json.loads(candidate)
return json.loads(m.group(0))
except json.JSONDecodeError:
continue
if isinstance(parsed, list):
parsed_arrays.append(parsed)
if len(parsed_arrays) == 1:
return parsed_arrays[0]
pass
return None
skillopt_sleep/__main__.py
+1 -6
View File
@@ -70,8 +70,7 @@ def _add_common(p: argparse.ArgumentParser) -> None:
p.add_argument("--project", default="")
p.add_argument("--scope", default="", choices=["", "all", "invoked"])
p.add_argument("--backend", default="",
choices=["", "mock", "claude", "codex", "copilot", "handoff",
"azure_openai"])
choices=["", "mock", "claude", "codex", "copilot", "handoff"])
p.add_argument("--model", default="")
p.add_argument("--codex-path", default="", help="path to the real @openai/codex binary")
p.add_argument("--claude-home", default="", help="override ~/.claude (also isolates state)")
@@ -92,8 +91,6 @@ def _add_common(p: argparse.ArgumentParser) -> None:
p.add_argument("--progress", action="store_true",
help="print phase progress to stderr")
p.add_argument("--auto-adopt", action="store_true")
p.add_argument("--preferences", default="",
help="free-text house rules injected into the optimizer's reflect step")
p.add_argument("--json", action="store_true")
@@ -121,8 +118,6 @@ def _cfg_from_args(args, task_meta: Dict[str, Any] | None = None) -> Any:
overrides["lookback_hours"] = lh
if getattr(args, "edit_budget", 0):
overrides["edit_budget"] = args.edit_budget
if getattr(args, "preferences", ""):
overrides["preferences"] = args.preferences
if getattr(args, "max_sessions", 0):
overrides["max_sessions_per_night"] = args.max_sessions
if getattr(args, "max_tasks", 0):
skillopt_sleep/backend.py
+18 -178
View File
@@ -615,6 +615,9 @@ class ClaudeCliBackend(CliBackend):
# --exclude-dynamic-... drop per-machine cwd/env/memory/git sections
# cwd=<clean temp> no project CLAUDE.md
import tempfile
# Reset per call so a prior failure does not make a later successful
# call look failed in persisted diagnostics (matches CodexCliBackend).
self.last_call_error = ""
cmd = [self.claude_path, "-p", "--output-format", "text"]
if os.environ.get("ANTHROPIC_API_KEY"):
cmd.append("--bare")
@@ -657,6 +660,7 @@ class ClaudeCliBackend(CliBackend):
# validated honestly: we detect the call from the shim's log, not from
# a self-reported marker. Other tools are stubbed the same way.
import tempfile, shutil, stat
self.last_call_error = ""
work = tempfile.mkdtemp(prefix="skillopt_sleep_tools_")
calllog = os.path.join(work, "_tool_calls.log")
try:
@@ -735,30 +739,9 @@ def resolve_codex_path(explicit: str = "") -> str:
env = os.environ.get("SKILLOPT_SLEEP_CODEX_PATH")
if env:
return env
import sys
import shutil
candidates = []
# Try shutil.which("codex") first so PATH, pnpm, Volta, etc. work.
which_codex = shutil.which("codex")
if which_codex:
candidates.append(which_codex)
if sys.platform == "win32":
import ntpath
appdata = os.environ.get("APPDATA")
if appdata:
candidates.append(ntpath.join(appdata, "npm", "codex.cmd"))
userprofile = os.environ.get("USERPROFILE")
if userprofile:
candidates.append(ntpath.join(userprofile, "AppData", "Roaming", "npm", "codex.cmd"))
nvm_home = os.environ.get("NVM_HOME")
if nvm_home:
candidates.append(ntpath.join(nvm_home, "codex.cmd"))
else:
candidates.extend([
candidates = [
os.path.expanduser("~/.nvm/versions/node/v22.22.3/bin/codex"),
])
]
# any nvm node version
nvm = os.path.expanduser("~/.nvm/versions/node")
if os.path.isdir(nvm):
@@ -776,7 +759,7 @@ def resolve_codex_path(explicit: str = "") -> str:
except Exception:
pass
return c
return "codex"
return "codex" # last resort (may be the wrapper)
class CodexCliBackend(CliBackend):
@@ -907,19 +890,8 @@ class CodexCliBackend(CliBackend):
work = tempfile.mkdtemp(prefix="skillopt_sleep_codextools_")
calllog = os.path.join(work, "_tool_calls.log")
out_path = os.path.join(work, "_last.txt")
tool_names = tools or ["search"]
is_windows = os.name == "nt"
try:
for tname in tool_names:
if is_windows:
shim = os.path.join(work, f"{tname}.cmd")
with open(shim, "w") as f:
f.write(
"@echo off\n"
f'echo %~n0>>"{calllog}"\n'
"echo (search results: 3 relevant notes found; use them to answer)\n"
)
else:
for tname in (tools or ["search"]):
shim = os.path.join(work, tname)
with open(shim, "w") as f:
f.write(
@@ -928,20 +900,9 @@ class CodexCliBackend(CliBackend):
'echo "(search results: 3 relevant notes found; use them to answer)"\n'
)
os.chmod(shim, os.stat(shim).st_mode | stat.S_IEXEC | stat.S_IXGRP | stat.S_IXOTH)
if is_windows:
tool_hint = (
"Shell tools are available in the working directory: "
+ ", ".join(f"{t}.cmd" for t in tool_names)
+ " (each callable as `" + tool_names[0] + "` or `.\\"
+ tool_names[0] + "`). When the skill says to look something "
"up or search before answering, you MUST actually run the "
"tool (e.g. `" + tool_names[0] + " \"query\"`) before giving "
"your final answer."
)
else:
tool_hint = (
"Shell tools are available in the working directory: "
+ ", ".join(f"./{t}" for t in tool_names)
+ ", ".join(f"./{t}" for t in (tools or ["search"]))
+ ". When the skill says to look something up or search before "
"answering, you MUST actually run the tool (e.g. `./search \"query\"`) "
"before giving your final answer."
@@ -1265,52 +1226,24 @@ _AZURE_MI_CLIENT_ID = "8cafa2b1-a2a7-4ad9-814a-ffe4aed7e800"
class AzureOpenAIBackend(CliBackend):
"""Drives Azure OpenAI gpt-5.x deployments via managed identity, or any
OpenAI-compatible chat-completions server via explicit compat auth.
"""Drives Azure OpenAI gpt-5.x deployments via managed identity.
Mirrors the intern's blog_1 setup (avail_api.md): managed-identity auth, the
same endpoints/deployments. Reuses CliBackend's attempt/judge/reflect prompts
and JSON parsing; only _call() differs. openai + azure-identity are lazy
imported so the mock/CLI paths stay dependency-free.
OpenAI-compatible mode (opt-in, matching skillopt.model.azure_openai):
* AZURE_OPENAI_AUTH_MODE=openai_compatible selects a plain ``openai.OpenAI``
client with AZURE_OPENAI_API_KEY against AZURE_OPENAI_ENDPOINT.
* SKILLOPT_SLEEP_COMPAT_MAX_TOKENS (int, default 8192) caps completion
length via the standard ``max_tokens`` parameter in compat mode.
* SKILLOPT_SLEEP_CHAT_EXTRA_BODY (JSON object) is passed as ``extra_body``
for provider-specific request fields (e.g. DeepSeek's
``{"thinking": {"type": "enabled"}}``). Nothing provider-specific is
inferred from model names.
"""
name = "azure"
_COMPAT_MODES = {"openai_compatible", "compat", "openai"}
# Hosts the managed-identity (AAD bearer token) path may talk to. A custom
# endpoint outside these domains must use explicit compat auth — we never
# send Azure credentials to an arbitrary host.
_AZURE_HOST_SUFFIXES = (".openai.azure.com", ".cognitiveservices.azure.com")
def __init__(self, deployment: str = "", endpoint: str = "", timeout: int = 180,
api_version: str = "2024-12-01-preview") -> None:
super().__init__(model=deployment or "gpt-5.5", timeout=timeout)
self.deployment = deployment or "gpt-5.5"
# Endpoint resolution order: explicit arg > AZURE_OPENAI_ENDPOINT env >
# the built-in Azure endpoint table. Honoring the env var lets callers
# point this backend at any OpenAI-compatible server (DeepSeek, a local
# vLLM, etc.) without editing the hardcoded _AZURE_ENDPOINTS map.
self.endpoint = (
endpoint
or os.environ.get("AZURE_OPENAI_ENDPOINT", "").strip()
or self._endpoint_for(self.deployment)
)
self.endpoint = endpoint or self._endpoint_for(self.deployment)
self.api_version = api_version
self.name = f"azure:{self.deployment}"
self._client = None
# Opt-in request knobs (read once; see class docstring).
self.compat_max_tokens = self._read_compat_max_tokens()
self.chat_extra_body = self._read_chat_extra_body()
@staticmethod
def _endpoint_for(deployment: str) -> str:
@@ -1319,75 +1252,8 @@ class AzureOpenAIBackend(CliBackend):
return ep
return "https://oaidr9.openai.azure.com/"
@classmethod
def _compat_mode(cls) -> bool:
return os.environ.get("AZURE_OPENAI_AUTH_MODE", "").strip().lower() in cls._COMPAT_MODES
@staticmethod
def _read_compat_max_tokens() -> int:
raw = os.environ.get("SKILLOPT_SLEEP_COMPAT_MAX_TOKENS", "").strip()
try:
val = int(raw) if raw else 8192
return val if val > 0 else 8192
except ValueError:
import logging
logging.getLogger("skillopt_sleep").warning(
"ignoring non-integer SKILLOPT_SLEEP_COMPAT_MAX_TOKENS=%r", raw)
return 8192
@staticmethod
def _read_chat_extra_body() -> Optional[Dict[str, Any]]:
raw = os.environ.get("SKILLOPT_SLEEP_CHAT_EXTRA_BODY", "").strip()
if not raw:
return None
try:
parsed = json.loads(raw)
except ValueError:
parsed = None
if isinstance(parsed, dict) and parsed:
return parsed
import logging
logging.getLogger("skillopt_sleep").warning(
"ignoring SKILLOPT_SLEEP_CHAT_EXTRA_BODY: not a non-empty JSON object: %r",
raw[:100])
return None
def _is_azure_host(self) -> bool:
from urllib.parse import urlparse
parsed = urlparse(self.endpoint)
host = (parsed.hostname or "").lower()
return (
parsed.scheme.lower() == "https"
and host.endswith(self._AZURE_HOST_SUFFIXES)
)
def _get_client(self):
if self._client is None:
# AZURE_OPENAI_AUTH_MODE=openai_compatible (matching the sibling
# skillopt.model.azure_openai module) selects a plain OpenAI client
# against a raw base_url. This is what makes any OpenAI-compatible
# endpoint work: the AzureOpenAI client would otherwise rewrite the
# URL with Azure-only query params (?api-version=...) and deployment
# path segments, which non-Azure servers reject with a 404.
if self._compat_mode():
from openai import OpenAI
self._client = OpenAI(
base_url=self.endpoint.rstrip("/"),
api_key=os.environ.get("AZURE_OPENAI_API_KEY"),
max_retries=4,
)
else:
# Security guard: the managed-identity path attaches an Azure AD
# bearer token to every request. Refuse to do that for a custom
# non-Azure endpoint — leaking a live AAD token to an arbitrary
# host must be impossible by (mis)configuration.
if not self._is_azure_host():
raise ValueError(
"azure_openai backend: refusing to send Azure managed-identity "
f"credentials to non-Azure endpoint {self.endpoint!r}. For "
"OpenAI-compatible servers set AZURE_OPENAI_AUTH_MODE="
"openai_compatible and AZURE_OPENAI_API_KEY."
)
from azure.identity import ManagedIdentityCredential, get_bearer_token_provider
from openai import AzureOpenAI
cred = ManagedIdentityCredential(client_id=_AZURE_MI_CLIENT_ID)
@@ -1413,26 +1279,13 @@ class AzureOpenAIBackend(CliBackend):
client = self._get_client()
last_exc = None
n_attempts = max(1, retries)
for attempt in range(n_attempts):
for attempt in range(max(1, retries)):
try:
kwargs: Dict[str, Any] = {
"model": self.deployment,
"messages": [{"role": "user", "content": prompt}],
}
# Provider-neutral request shape: compat mode speaks the
# standard OpenAI-compatible contract (`max_tokens`); the Azure
# gpt-5.x deployments require `max_completion_tokens`. Any
# provider-specific body fields are opt-in via
# SKILLOPT_SLEEP_CHAT_EXTRA_BODY (see class docstring) — never
# inferred from the model name.
if self._compat_mode():
kwargs["max_tokens"] = self.compat_max_tokens
else:
kwargs["max_completion_tokens"] = 16384
if self._compat_mode() and self.chat_extra_body:
kwargs["extra_body"] = self.chat_extra_body
resp = client.chat.completions.create(**kwargs)
resp = client.chat.completions.create(
model=self.deployment,
messages=[{"role": "user", "content": prompt}],
max_completion_tokens=16384,
)
text = (resp.choices[0].message.content or "").strip()
try:
u = resp.usage
@@ -1440,28 +1293,15 @@ class AzureOpenAIBackend(CliBackend):
except Exception:
pass
if text:
# A recovered retry must not leave a stale error behind:
# last_call_error always reflects the LATEST outcome.
self.last_call_error = ""
return text
# empty but no exception: model genuinely returned nothing — one
# quick retry can help (reasoning models occasionally yield empty)
last_exc = "empty-response"
except Exception as e: # noqa: BLE001
last_exc = e
# Surface the error so a 0.0 night is diagnosable (e.g. a 404
# from a mis-pointed endpoint) instead of a silent empty->0.
self.last_call_error = str(e)[:500]
# backoff before next try (skip after the final attempt)
if attempt < n_attempts - 1:
if attempt < retries - 1:
_t.sleep(min(8.0, (2 ** attempt) * 0.5) + _r.random() * 0.4)
if last_exc == "empty-response":
# All attempts "succeeded" HTTP-wise but carried no text — say so,
# otherwise a run of empty completions is indistinguishable from a
# never-called backend in diagnostics.
self.last_call_error = (
f"{self.deployment}: empty response on all {n_attempts} attempts"
)
return ""
skillopt_sleep/config.py
-1
View File
@@ -41,7 +41,6 @@ DEFAULTS: Dict[str, Any] = {
"gate_mode": "on", # "on" (validation-gated) | "off" (greedy, no hard filter)
"codex_path": "", # "" => auto-detect the real @openai/codex binary
"edit_budget": 4, # textual learning rate (max edits/night)
"preferences": "", # free-text house rules injected into reflect as a prior
"gate_metric": "mixed", # hard | soft | mixed (mixed best for tiny holdouts)
"gate_mixed_weight": 0.5,
"replay_mode": "mock", # "mock" (sandboxed prompt) | "fresh" (worktree)
skillopt_sleep/cycle.py
-1
View File
@@ -120,7 +120,6 @@ def run_sleep_cycle(
codex_path=cfg.get("codex_path", ""),
project_dir=project,
)
backend.preferences = cfg.get("preferences", "")
_progress(cfg, f"night {night}: project={project} backend={backend.name}")
# ── live skill/memory docs ───────────────────────────────────────────
skillopt_sleep/scheduler.py
+19 -110
View File
@@ -1,7 +1,19 @@
"""SkillOpt-Sleep — built-in nightly scheduler.
Installs/removes a crontab entry (on Unix) or a Scheduled Task (on Windows) that
runs the sleep cycle automatically, so the user doesn't have to wire it manually.
Installs/removes a crontab entry that runs the sleep cycle automatically, so the
user doesn't have to wire cron themselves. Idempotent: a managed block delimited
by marker comments is added/replaced/removed in the user's crontab.
Design choices:
* Off-:00 minute (3:17 local by default) so many users don't all hit the API
at the same instant.
* The entry runs `python -m skillopt_sleep run` for a specific project and
appends to <project>/.skillopt-sleep/cron.log.
* `schedule` is additive per project (keyed by project path); `unschedule`
removes the project's line (or the whole managed block with --all).
cron is the portable mechanism on Linux/macOS. On systems without `crontab`,
`schedule` prints the line and instructions instead of failing.
"""
from __future__ import annotations
@@ -53,74 +65,12 @@ def _split_managed(crontab: str) -> Tuple[str, List[str]]:
return "\n".join(outside).rstrip(), managed
def _have_schtasks() -> bool:
return shutil.which("schtasks") is not None
def _win_task_name(project: str) -> str:
project = os.path.abspath(project)
safe = project.replace(":\\", "_").replace("\\", "_").replace("/", "_").replace(" ", "_")
return f"SkillOpt-Sleep-{safe}"
def _create_win_task(task_name: str, command: str, hour: int, minute: int) -> bool:
try:
st = f"{hour:02d}:{minute:02d}"
cmd = ["schtasks", "/create", "/tn", task_name, "/tr", command, "/sc", "daily", "/st", st, "/f"]
proc = subprocess.run(cmd, capture_output=True, text=True)
return proc.returncode == 0
except Exception:
return False
def _delete_win_task(task_name: str) -> bool:
try:
cmd = ["schtasks", "/delete", "/tn", task_name, "/f"]
proc = subprocess.run(cmd, capture_output=True, text=True)
return proc.returncode == 0
except Exception:
return False
def _list_win_tasks() -> List[str]:
try:
proc = subprocess.run(["schtasks", "/query", "/fo", "csv"], capture_output=True, text=True)
if proc.returncode != 0:
return []
tasks = []
for line in proc.stdout.splitlines():
if not line.startswith('"'):
continue
parts = line.split(",")
if len(parts) > 0:
name = parts[0].strip('"')
if name.startswith("SkillOpt-Sleep-"):
tasks.append(name)
return tasks
except Exception:
return []
def _runner_cmd(project: str, backend: str, extra: str, python: str) -> str:
logdir = os.path.join(project, ".skillopt-sleep")
log = os.path.join(logdir, "cron.log")
# use absolute python + -m so cron's/scheduler's minimal env still works
cmd = (f'"{python}" -m skillopt_sleep run --project "{project}" '
# use absolute python + -m so cron's minimal env still works
cmd = (f'{python} -m skillopt_sleep run --project "{project}" '
f'--scope invoked --backend {backend} {extra}'.rstrip())
if sys.platform == "win32":
helper_script = os.path.join(logdir, "run.cmd")
try:
os.makedirs(logdir, exist_ok=True)
content = (
"@echo off\n"
f'cd /d "{_repo_root()}"\n'
f'{cmd} >> "{log}" 2>&1\n'
)
with open(helper_script, "w", encoding="utf-8") as f:
f.write(content)
except Exception:
pass
return f'"{helper_script}"'
return f'mkdir -p "{logdir}"; cd "{_repo_root()}" && {cmd} >> "{log}" 2>&1'
@@ -137,26 +87,12 @@ def schedule(project: str, *, backend: str = "mock", hour: int = 3, minute: int
extra: str = "", python: Optional[str] = None) -> Tuple[bool, str]:
"""Install (or replace) the nightly entry for ``project``.
Returns (installed, message). If the scheduler backend is unavailable, installed=False and
the message contains instructions to add manually.
Returns (installed, message). If crontab is unavailable, installed=False and
the message contains the line to add manually.
"""
project = os.path.abspath(project)
python = python or sys.executable or "python3"
runner_cmd = _runner_cmd(project, backend, extra, python)
if sys.platform == "win32":
if not _have_schtasks():
return False, "schtasks.exe not found on this system. Add this command to your scheduler manually:\n" + runner_cmd
tn = _win_task_name(project)
ok = _create_win_task(tn, runner_cmd, hour, minute)
if ok:
return True, (f"Scheduled nightly at {hour:02d}:{minute:02d} for {project} "
f"(backend={backend}) via Windows Task Scheduler. Task Name: {tn}\n"
f"Logs -> {project}/.skillopt-sleep/cron.log\n"
f"Runs `skillopt_sleep run`; it only STAGES a proposal — adopt is still manual.")
return False, f"Failed to write scheduled task. Command to run manually:\nschtasks /create /tn \"{tn}\" /tr \"{runner_cmd}\" /sc daily /st {hour:02d}:{minute:02d} /f"
cron_line = f"{minute} {hour} * * * {runner_cmd} {_project_marker(project)}"
cron_line = f"{minute} {hour} * * * {_runner_cmd(project, backend, extra, python)} {_project_marker(project)}"
if not _have_crontab():
return False, ("crontab not found on this system. Add this line to your "
@@ -180,29 +116,6 @@ def schedule(project: str, *, backend: str = "mock", hour: int = 3, minute: int
def unschedule(project: Optional[str] = None, *, all_projects: bool = False) -> Tuple[bool, str]:
"""Remove the entry for ``project`` (or the whole managed block with all_projects)."""
if sys.platform == "win32":
if not _have_schtasks():
return False, "schtasks.exe not found on this system."
if all_projects:
tasks = _list_win_tasks()
ok = True
for t in tasks:
if not _delete_win_task(t):
ok = False
return ok, ("Removed all scheduled tasks." if ok else "Failed to remove some tasks.")
elif project:
tn = _win_task_name(project)
ok = _delete_win_task(tn)
try:
logdir = os.path.join(project, ".skillopt-sleep")
helper = os.path.join(logdir, "run.cmd")
if os.path.exists(helper):
os.remove(helper)
except Exception:
pass
return ok, ("Removed." if ok else "Failed to remove scheduled task (does it exist?).")
return False, "No project specified to unschedule."
if not _have_crontab():
return False, "crontab not found; nothing to remove."
outside, managed = _split_managed(_read_crontab())
@@ -221,9 +134,5 @@ def unschedule(project: Optional[str] = None, *, all_projects: bool = False) ->
def list_scheduled() -> List[str]:
if sys.platform == "win32":
if not _have_schtasks():
return []
return _list_win_tasks()
_outside, managed = _split_managed(_read_crontab())
return [ln for ln in managed if ln.strip()]
tests/test_alfworld_paths.py
+1 -1
View File
@@ -9,7 +9,7 @@ def test_resolve_alfworld_gamefile_uses_alfworld_data_for_relative_paths(monkeyp
resolved = _resolve_alfworld_gamefile("json_2.1.1/valid_seen/task/game.tw-pddl")
assert resolved == os.path.normpath(os.path.join(str(data_root), "json_2.1.1/valid_seen/task/game.tw-pddl"))
assert resolved == os.path.join(str(data_root), "json_2.1.1/valid_seen/task/game.tw-pddl")
def test_resolve_alfworld_gamefile_keeps_absolute_paths(monkeypatch, tmp_path):
tests/test_azure_openai_compat.py
-268
View File
@@ -1,268 +0,0 @@
"""Tests for the azure_openai backend's OpenAI-compatible mode.
Pure-stdlib (unittest), deterministic, NO network, no API key. Covers the
integration points requested in review of the openai-compatible feature:
* CLI acceptance of --backend azure_openai
* compat-vs-Azure client selection
* endpoint resolution and the managed-identity credential guard
* provider-neutral request kwargs (opt-in extra body / token cap)
* retry-success error clearing + empty-response diagnostics
* example runner exit-code propagation
Run: python -m unittest tests.test_azure_openai_compat
"""
from __future__ import annotations
import argparse
import contextlib
import importlib.util
import io
import json
import os
import sys
import unittest
from pathlib import Path
from types import SimpleNamespace
from unittest import mock
from skillopt_sleep.__main__ import _add_common
from skillopt_sleep.backend import AzureOpenAIBackend
try:
import openai # noqa: F401
HAVE_OPENAI = True
except ImportError:
HAVE_OPENAI = False
COMPAT_ENV = {
"AZURE_OPENAI_AUTH_MODE": "openai_compatible",
"AZURE_OPENAI_ENDPOINT": "https://compat.example.test",
"AZURE_OPENAI_API_KEY": "sk-test-not-a-real-key",
}
def _resp(text):
return SimpleNamespace(
choices=[SimpleNamespace(message=SimpleNamespace(content=text))],
usage=None,
)
class _FakeCompletions:
"""Scripted chat.completions.create: replies are strings or Exceptions."""
def __init__(self, replies):
self.replies = list(replies)
self.calls = []
def create(self, **kwargs):
self.calls.append(kwargs)
item = self.replies.pop(0)
if isinstance(item, Exception):
raise item
return _resp(item)
def _fake_client(replies):
return SimpleNamespace(chat=SimpleNamespace(completions=_FakeCompletions(replies)))
def _backend_with(replies, env):
"""Build a backend under `env` with a scripted fake client injected."""
with mock.patch.dict(os.environ, env, clear=True):
be = AzureOpenAIBackend(deployment="any-model")
be._client = _fake_client(replies)
return be
class TestCliAcceptance(unittest.TestCase):
def test_backend_azure_openai_is_accepted(self):
p = argparse.ArgumentParser()
_add_common(p)
ns = p.parse_args(["--backend", "azure_openai"])
self.assertEqual(ns.backend, "azure_openai")
class TestClientSelection(unittest.TestCase):
@unittest.skipUnless(HAVE_OPENAI, "openai package not installed")
def test_compat_mode_builds_plain_openai_client(self):
from openai import AzureOpenAI, OpenAI
# Overlay (clear=False): constructing a real OpenAI client builds an
# SSL context, which needs the ambient cert-related env vars intact.
with mock.patch.dict(os.environ, COMPAT_ENV, clear=False):
be = AzureOpenAIBackend(deployment="some-model", endpoint=COMPAT_ENV["AZURE_OPENAI_ENDPOINT"])
client = be._get_client()
self.assertIsInstance(client, OpenAI)
self.assertNotIsInstance(client, AzureOpenAI)
self.assertIn("compat.example.test", str(client.base_url))
def test_managed_identity_refuses_non_azure_endpoint(self):
# No compat auth mode + a custom non-Azure endpoint: the backend must
# raise instead of sending an Azure AD bearer token to that host. The
# guard fires before azure.identity is imported, so this test needs
# neither azure-identity nor any network.
env = {"AZURE_OPENAI_ENDPOINT": "https://api.deepseek.com"}
with mock.patch.dict(os.environ, env, clear=True):
be = AzureOpenAIBackend(deployment="some-model")
with self.assertRaises(ValueError) as ctx:
be._get_client()
self.assertIn("openai_compatible", str(ctx.exception))
def test_managed_identity_refuses_insecure_azure_endpoint(self):
# A matching Azure hostname is insufficient: AAD bearer credentials
# must never be sent over plaintext HTTP.
env = {"AZURE_OPENAI_ENDPOINT": "http://foo.openai.azure.com"}
with mock.patch.dict(os.environ, env, clear=True):
be = AzureOpenAIBackend(deployment="some-model")
self.assertFalse(be._is_azure_host())
with self.assertRaises(ValueError) as ctx:
be._get_client()
self.assertIn("openai_compatible", str(ctx.exception))
def test_azure_host_detection(self):
with mock.patch.dict(os.environ, {}, clear=True):
be = AzureOpenAIBackend(deployment="gpt-5.5") # table endpoint
self.assertTrue(be._is_azure_host())
be2 = AzureOpenAIBackend(
deployment="gpt-5.5", endpoint="https://api.deepseek.com")
self.assertFalse(be2._is_azure_host())
class TestEndpointResolution(unittest.TestCase):
def test_env_endpoint_is_honored(self):
env = {"AZURE_OPENAI_ENDPOINT": "https://compat.example.test"}
with mock.patch.dict(os.environ, env, clear=True):
be = AzureOpenAIBackend(deployment="some-model")
self.assertEqual(be.endpoint, "https://compat.example.test")
def test_explicit_arg_beats_env(self):
env = {"AZURE_OPENAI_ENDPOINT": "https://env.example.test"}
with mock.patch.dict(os.environ, env, clear=True):
be = AzureOpenAIBackend(deployment="m", endpoint="https://arg.example.test")
self.assertEqual(be.endpoint, "https://arg.example.test")
def test_table_fallback_without_env(self):
with mock.patch.dict(os.environ, {}, clear=True):
be = AzureOpenAIBackend(deployment="gpt-5.5")
self.assertTrue(be.endpoint.endswith(".openai.azure.com/"))
class TestRequestKwargs(unittest.TestCase):
def test_compat_mode_sends_standard_max_tokens(self):
be = _backend_with(["hi"], COMPAT_ENV)
with mock.patch.dict(os.environ, COMPAT_ENV, clear=True):
out = be._call("p", retries=1)
self.assertEqual(out, "hi")
(call,) = be._client.chat.completions.calls
self.assertEqual(call["max_tokens"], 8192)
self.assertNotIn("max_completion_tokens", call)
self.assertNotIn("extra_body", call)
def test_compat_max_tokens_env_override(self):
env = dict(COMPAT_ENV, SKILLOPT_SLEEP_COMPAT_MAX_TOKENS="4096")
be = _backend_with(["hi"], env)
with mock.patch.dict(os.environ, env, clear=True):
be._call("p", retries=1)
(call,) = be._client.chat.completions.calls
self.assertEqual(call["max_tokens"], 4096)
def test_extra_body_is_opt_in_via_env_not_model_name(self):
# A deepseek-named model WITHOUT the env knob gets a pure standard
# request (nothing inferred from the name)...
with mock.patch.dict(os.environ, COMPAT_ENV, clear=True):
be = AzureOpenAIBackend(deployment="deepseek-v4-pro")
be._client = _fake_client(["hi"])
be._call("p", retries=1)
(call,) = be._client.chat.completions.calls
self.assertNotIn("extra_body", call)
# ...and any model WITH the env knob gets exactly the configured body.
body = {"thinking": {"type": "enabled"}}
env = dict(COMPAT_ENV, SKILLOPT_SLEEP_CHAT_EXTRA_BODY=json.dumps(body))
be2 = _backend_with(["hi"], env)
with mock.patch.dict(os.environ, env, clear=True):
be2._call("p", retries=1)
(call2,) = be2._client.chat.completions.calls
self.assertEqual(call2["extra_body"], body)
def test_malformed_extra_body_is_ignored(self):
env = dict(COMPAT_ENV, SKILLOPT_SLEEP_CHAT_EXTRA_BODY="{not json")
be = _backend_with(["hi"], env)
with mock.patch.dict(os.environ, env, clear=True):
be._call("p", retries=1)
(call,) = be._client.chat.completions.calls
self.assertNotIn("extra_body", call)
def test_azure_mode_sends_max_completion_tokens(self):
be = _backend_with(["hi"], {}) # no compat auth mode
with mock.patch.dict(os.environ, {}, clear=True):
be._call("p", retries=1)
(call,) = be._client.chat.completions.calls
self.assertEqual(call["max_completion_tokens"], 16384)
self.assertNotIn("max_tokens", call)
def test_azure_mode_ignores_compat_extra_body(self):
body = {"thinking": {"type": "enabled"}}
env = {"SKILLOPT_SLEEP_CHAT_EXTRA_BODY": json.dumps(body)}
be = _backend_with(["hi"], env)
with mock.patch.dict(os.environ, env, clear=True):
be._call("p", retries=1)
(call,) = be._client.chat.completions.calls
self.assertNotIn("extra_body", call)
class TestErrorState(unittest.TestCase):
def test_recovered_retry_clears_last_call_error(self):
be = _backend_with([RuntimeError("transient boom"), "recovered"], COMPAT_ENV)
with mock.patch.dict(os.environ, COMPAT_ENV, clear=True), \
mock.patch("time.sleep"):
out = be._call("p", retries=2)
self.assertEqual(out, "recovered")
self.assertEqual(be.last_call_error, "")
def test_all_empty_responses_set_diagnostic(self):
be = _backend_with(["", ""], COMPAT_ENV)
with mock.patch.dict(os.environ, COMPAT_ENV, clear=True), \
mock.patch("time.sleep"):
out = be._call("p", retries=2)
self.assertEqual(out, "")
self.assertIn("empty response on all 2 attempts", be.last_call_error)
def test_persistent_exception_is_surfaced(self):
be = _backend_with([RuntimeError("boom-1"), RuntimeError("boom-2")], COMPAT_ENV)
with mock.patch.dict(os.environ, COMPAT_ENV, clear=True), \
mock.patch("time.sleep"):
out = be._call("p", retries=2)
self.assertEqual(out, "")
self.assertIn("boom-2", be.last_call_error)
class TestRunnerExitCode(unittest.TestCase):
RUNNER = Path(__file__).resolve().parent.parent / "docs" / "sleep" / "examples" / "runner.py"
def _load_runner(self):
spec = importlib.util.spec_from_file_location("example_runner", self.RUNNER)
mod = importlib.util.module_from_spec(spec)
spec.loader.exec_module(mod)
return mod
def _run_with_child_rc(self, rc):
env = {"DEEPSEEK_API_KEY": "sk-test-not-a-real-key"}
with mock.patch.dict(os.environ, env, clear=True):
mod = self._load_runner()
fake = SimpleNamespace(run=lambda *a, **k: SimpleNamespace(returncode=rc))
with mock.patch.object(mod, "subprocess", fake), \
mock.patch.object(sys, "argv", ["runner.py", "run"]), \
contextlib.redirect_stdout(io.StringIO()):
with self.assertRaises(SystemExit) as ctx:
mod.main()
return ctx.exception.code
def test_child_failure_propagates(self):
self.assertEqual(self._run_with_child_rc(7), 7)
def test_child_success_exits_zero(self):
self.assertEqual(self._run_with_child_rc(0), 0)
if __name__ == "__main__":
unittest.main()
tests/test_claude_backend_tempdir.py
-59
View File
@@ -1,59 +0,0 @@
"""Regression coverage for Claude CLI temporary-directory cleanup."""
from __future__ import annotations
import json
import os
import shutil
import tempfile
from types import SimpleNamespace
from skillopt.model import claude_backend
def test_claude_print_uses_cleanup_tolerant_temporary_directory(monkeypatch) -> None:
cleanup_modes = []
subprocess_cwd = ""
def simulated_windows_rmtree(cls, name, ignore_errors=False, repeated=False):
del cls, repeated
cleanup_modes.append(ignore_errors)
if not ignore_errors:
raise PermissionError(32, "directory is still in use", name)
shutil.rmtree(name, ignore_errors=True)
def fake_run(*args, **kwargs):
nonlocal subprocess_cwd
subprocess_cwd = kwargs["cwd"]
payload = {
"type": "result",
"result": "ok",
"usage": {"input_tokens": 2, "output_tokens": 3},
}
return SimpleNamespace(
returncode=0,
stdout=json.dumps(payload) + "\n",
stderr="",
)
monkeypatch.setattr(
tempfile.TemporaryDirectory,
"_rmtree",
classmethod(simulated_windows_rmtree),
)
monkeypatch.setattr(claude_backend.subprocess, "run", fake_run)
text, _, usage = claude_backend._run_claude_print(
system="system",
prompt="prompt",
model="",
tools=None,
tool_choice=None,
return_message=False,
timeout=10,
)
assert text == "ok"
assert usage == {"prompt_tokens": 2, "completion_tokens": 3, "total_tokens": 5}
assert cleanup_modes == [True]
assert subprocess_cwd
assert not os.path.exists(subprocess_cwd)
tests/test_codex_optimizer_backend.py
-371
View File
@@ -1,371 +0,0 @@
from __future__ import annotations
import importlib.util
import os
import sys
import types
from collections.abc import Iterator
from typing import Any
import pytest
class _OpenAIClientStub:
def __init__(self, *args: Any, **kwargs: Any) -> None:
self.args = args
self.kwargs = kwargs
def _install_openai_stub() -> None:
if "openai" in sys.modules or importlib.util.find_spec("openai") is not None:
return
openai_stub = types.ModuleType("openai")
openai_stub.AzureOpenAI = _OpenAIClientStub
openai_stub.OpenAI = _OpenAIClientStub
sys.modules["openai"] = openai_stub
def _import_model_modules() -> tuple[Any, Any, Any, Any]:
_install_openai_stub()
import skillopt.model as model_module
from skillopt.model import azure_openai, backend_config, codex_backend
return model_module, backend_config, codex_backend, azure_openai
@pytest.fixture(autouse=True)
def isolate_backend_state() -> Iterator[tuple[Any, Any, Any, Any]]:
model_module, backend_config, codex_backend, azure_openai = _import_model_modules()
optimizer_backend = backend_config.get_optimizer_backend()
target_backend = backend_config.get_target_backend()
env = {
key: os.environ.get(key)
for key in (
"OPTIMIZER_BACKEND",
"TARGET_BACKEND",
"OPTIMIZER_DEPLOYMENT",
"TARGET_DEPLOYMENT",
)
}
yield model_module, backend_config, codex_backend, azure_openai
backend_config.set_optimizer_backend(optimizer_backend)
backend_config.set_target_backend(target_backend)
for key, value in env.items():
if value is None:
os.environ.pop(key, None)
else:
os.environ[key] = value
def test_codex_exec_can_be_optimizer_backend(
isolate_backend_state: tuple[Any, Any, Any, Any],
) -> None:
_model_module, backend_config, _codex_backend, _azure_openai = isolate_backend_state
backend_config.set_optimizer_backend("codex_exec")
assert backend_config.get_optimizer_backend() == "codex_exec"
def test_set_backend_codex_uses_codex_for_optimizer_and_target(
isolate_backend_state: tuple[Any, Any, Any, Any],
) -> None:
model_module, backend_config, _codex_backend, _azure_openai = isolate_backend_state
assert model_module.set_backend("codex") == "codex"
assert backend_config.get_optimizer_backend() == "codex_exec"
assert backend_config.get_target_backend() == "codex_exec"
assert model_module.get_backend_name() == "codex"
def test_chat_optimizer_routes_to_codex_backend(
monkeypatch: pytest.MonkeyPatch,
isolate_backend_state: tuple[Any, Any, Any, Any],
) -> None:
model_module, backend_config, codex_backend, azure_openai = isolate_backend_state
codex_calls: list[dict[str, Any]] = []
def fake_codex_optimizer(**kwargs: Any) -> tuple[str, dict[str, int]]:
codex_calls.append(kwargs)
return "codex result", {
"prompt_tokens": 1,
"completion_tokens": 2,
"total_tokens": 3,
}
def fail_openai_optimizer(**_kwargs: Any) -> tuple[str, dict[str, int]]:
raise AssertionError("openai optimizer should not be called for codex_exec")
monkeypatch.setattr(codex_backend, "chat_optimizer", fake_codex_optimizer)
monkeypatch.setattr(azure_openai, "chat_optimizer", fail_openai_optimizer)
backend_config.set_optimizer_backend("codex_exec")
text, usage = model_module.chat_optimizer("system", "user", retries=1, timeout=5)
assert text == "codex result"
assert usage["total_tokens"] == 3
assert codex_calls[0]["system"] == "system"
assert codex_calls[0]["user"] == "user"
assert codex_calls[0]["timeout"] == 5
def test_openai_compatible_still_allowed_as_optimizer_backend(
isolate_backend_state: tuple[Any, Any, Any, Any],
) -> None:
_model_module, backend_config, _codex_backend, _azure_openai = isolate_backend_state
backend_config.set_optimizer_backend("openai_compatible")
assert backend_config.get_optimizer_backend() == "openai_compatible"
def test_chat_optimizer_routes_to_openai_compatible_when_selected(
monkeypatch: pytest.MonkeyPatch,
isolate_backend_state: tuple[Any, Any, Any, Any],
) -> None:
model_module, backend_config, codex_backend, azure_openai = isolate_backend_state
from skillopt.model import openai_compatible_backend
compat_calls: list[dict[str, Any]] = []
def fake_compat_optimizer(**kwargs: Any) -> tuple[str, dict[str, int]]:
compat_calls.append(kwargs)
return "compat result", {"prompt_tokens": 1, "completion_tokens": 1, "total_tokens": 2}
def fail_codex_optimizer(**_kwargs: Any) -> tuple[str, dict[str, int]]:
raise AssertionError("codex optimizer should not be called for openai_compatible")
def fail_openai_optimizer(**_kwargs: Any) -> tuple[str, dict[str, int]]:
raise AssertionError("openai optimizer should not be called for openai_compatible")
monkeypatch.setattr(openai_compatible_backend, "chat_optimizer", fake_compat_optimizer)
monkeypatch.setattr(codex_backend, "chat_optimizer", fail_codex_optimizer)
monkeypatch.setattr(azure_openai, "chat_optimizer", fail_openai_optimizer)
backend_config.set_optimizer_backend("openai_compatible")
text, usage = model_module.chat_optimizer("system", "user", retries=1, timeout=7)
assert text == "compat result"
assert usage["total_tokens"] == 2
assert compat_calls[0]["timeout"] == 7
def test_chat_optimizer_messages_routes_to_codex_backend(
monkeypatch: pytest.MonkeyPatch,
isolate_backend_state: tuple[Any, Any, Any, Any],
) -> None:
model_module, backend_config, codex_backend, azure_openai = isolate_backend_state
codex_calls: list[dict[str, Any]] = []
def fake_codex_messages(**kwargs: Any) -> tuple[str, dict[str, int]]:
codex_calls.append(kwargs)
return "codex messages", {"prompt_tokens": 2, "completion_tokens": 3, "total_tokens": 5}
def fail_openai_messages(**_kwargs: Any) -> tuple[str, dict[str, int]]:
raise AssertionError("openai messages should not be called for codex_exec")
monkeypatch.setattr(codex_backend, "chat_optimizer_messages", fake_codex_messages)
monkeypatch.setattr(azure_openai, "chat_optimizer_messages", fail_openai_messages)
backend_config.set_optimizer_backend("codex_exec")
text, usage = model_module.chat_optimizer_messages(
[{"role": "user", "content": "hi"}],
retries=1,
tools=[{"name": "lookup"}],
tool_choice="required",
return_message=True,
timeout=9,
)
assert text == "codex messages"
assert usage["total_tokens"] == 5
assert codex_calls[0]["tools"] == [{"name": "lookup"}]
assert codex_calls[0]["tool_choice"] == "required"
assert codex_calls[0]["return_message"] is True
assert codex_calls[0]["timeout"] == 9
def test_codex_optimizer_does_not_change_openai_target_routing(
monkeypatch: pytest.MonkeyPatch,
isolate_backend_state: tuple[Any, Any, Any, Any],
) -> None:
model_module, backend_config, codex_backend, azure_openai = isolate_backend_state
def fake_openai_target(**_kwargs: Any) -> tuple[str, dict[str, int]]:
return "openai target", {"prompt_tokens": 1, "completion_tokens": 0, "total_tokens": 1}
def fail_codex_target(**_kwargs: Any) -> tuple[str, dict[str, int]]:
raise AssertionError("codex target should not be called when target_backend=openai_chat")
monkeypatch.setattr(azure_openai, "chat_target", fake_openai_target)
monkeypatch.setattr(codex_backend, "chat_target", fail_codex_target)
backend_config.set_optimizer_backend("codex_exec")
backend_config.set_target_backend("openai_chat")
text, usage = model_module.chat_target("system", "user", retries=1)
assert text == "openai target"
assert usage["total_tokens"] == 1
def test_get_backend_name_keeps_openai_compatible(
isolate_backend_state: tuple[Any, Any, Any, Any],
) -> None:
model_module, backend_config, _codex_backend, _azure_openai = isolate_backend_state
backend_config.set_optimizer_backend("openai_compatible")
backend_config.set_target_backend("openai_compatible")
assert model_module.get_backend_name() == "openai_compatible"
def test_token_summary_merges_codex_once_with_existing_backends(
monkeypatch: pytest.MonkeyPatch,
isolate_backend_state: tuple[Any, Any, Any, Any],
) -> None:
model_module, _backend_config, codex_backend, azure_openai = isolate_backend_state
from skillopt.model import claude_backend, minimax_backend, openai_compatible_backend, qwen_backend
empty = {"_total": {"calls": 0, "prompt_tokens": 0, "completion_tokens": 0, "total_tokens": 0}}
monkeypatch.setattr(azure_openai, "get_token_summary", lambda: empty.copy())
monkeypatch.setattr(claude_backend, "get_token_summary", lambda: empty.copy())
monkeypatch.setattr(qwen_backend, "get_token_summary", lambda: empty.copy())
monkeypatch.setattr(minimax_backend, "get_token_summary", lambda: empty.copy())
monkeypatch.setattr(
openai_compatible_backend,
"get_token_summary",
lambda: {"optimizer": {"calls": 1, "prompt_tokens": 5, "completion_tokens": 7, "total_tokens": 12}},
)
monkeypatch.setattr(
codex_backend,
"get_token_summary",
lambda: {"optimizer": {"calls": 1, "prompt_tokens": 11, "completion_tokens": 13, "total_tokens": 24}},
)
summary = model_module.get_token_summary()
assert summary["optimizer"]["calls"] == 2
assert summary["optimizer"]["prompt_tokens"] == 16
assert summary["optimizer"]["completion_tokens"] == 20
assert summary["_total"]["total_tokens"] == 36
def test_codex_usage_is_not_double_counted_by_shared_tracker(
isolate_backend_state: tuple[Any, Any, Any, Any],
) -> None:
model_module, _backend_config, codex_backend, _azure_openai = isolate_backend_state
from skillopt.model import claude_backend
model_module.reset_token_tracker()
try:
assert codex_backend.tracker is not claude_backend.tracker
codex_backend.tracker.record("optimizer", 11, 13)
summary = model_module.get_token_summary()
assert summary["optimizer"] == {
"calls": 1,
"prompt_tokens": 11,
"completion_tokens": 13,
"total_tokens": 24,
}
assert summary["_total"] == {
"calls": 1,
"prompt_tokens": 11,
"completion_tokens": 13,
"total_tokens": 24,
}
finally:
model_module.reset_token_tracker()
def test_reset_token_tracker_resets_codex_and_existing_backends(
monkeypatch: pytest.MonkeyPatch,
isolate_backend_state: tuple[Any, Any, Any, Any],
) -> None:
model_module, _backend_config, codex_backend, azure_openai = isolate_backend_state
from skillopt.model import claude_backend, minimax_backend, openai_compatible_backend, qwen_backend
called: list[str] = []
for name, module in [
("openai", azure_openai),
("claude", claude_backend),
("qwen", qwen_backend),
("minimax", minimax_backend),
("compat", openai_compatible_backend),
("codex", codex_backend),
]:
monkeypatch.setattr(module, "reset_token_tracker", lambda name=name: called.append(name))
model_module.reset_token_tracker()
assert called == ["openai", "claude", "qwen", "minimax", "compat", "codex"]
def test_set_reasoning_effort_updates_codex_and_existing_backends(
monkeypatch: pytest.MonkeyPatch,
isolate_backend_state: tuple[Any, Any, Any, Any],
) -> None:
model_module, _backend_config, codex_backend, azure_openai = isolate_backend_state
from skillopt.model import claude_backend, minimax_backend, openai_compatible_backend, qwen_backend
called: list[tuple[str, str]] = []
for name, module in [
("openai", azure_openai),
("claude", claude_backend),
("qwen", qwen_backend),
("minimax", minimax_backend),
("compat", openai_compatible_backend),
("codex", codex_backend),
]:
monkeypatch.setattr(module, "set_reasoning_effort", lambda effort, name=name: called.append((name, effort)))
model_module.set_reasoning_effort("high")
assert called == [
("openai", "high"),
("claude", "high"),
("qwen", "high"),
("minimax", "high"),
("compat", "high"),
("codex", "high"),
]
def test_deployment_setters_update_codex_without_dropping_openai_compatible(
monkeypatch: pytest.MonkeyPatch,
isolate_backend_state: tuple[Any, Any, Any, Any],
) -> None:
model_module, _backend_config, codex_backend, _azure_openai = isolate_backend_state
from skillopt.model import openai_compatible_backend
called: list[tuple[str, str, str]] = []
monkeypatch.setattr(
openai_compatible_backend,
"set_target_deployment",
lambda deployment: called.append(("compat", "target", deployment)),
)
monkeypatch.setattr(
openai_compatible_backend,
"set_optimizer_deployment",
lambda deployment: called.append(("compat", "optimizer", deployment)),
)
monkeypatch.setattr(
codex_backend,
"set_target_deployment",
lambda deployment: called.append(("codex", "target", deployment)),
)
monkeypatch.setattr(
codex_backend,
"set_optimizer_deployment",
lambda deployment: called.append(("codex", "optimizer", deployment)),
)
model_module.set_target_deployment("target-model")
model_module.set_optimizer_deployment("optimizer-model")
assert ("compat", "target", "target-model") in called
assert ("codex", "target", "target-model") in called
assert ("compat", "optimizer", "optimizer-model") in called
assert ("codex", "optimizer", "optimizer-model") in called
tests/test_devin_plugin.py
+1 -5
View File
@@ -46,7 +46,7 @@ class TestDevinMcpSchema(unittest.TestCase):
def test_backends_in_enum(self):
backends = mcp_server._TOOL_SCHEMA["properties"]["backend"]["enum"]
for b in ["mock", "claude", "codex", "copilot", "handoff"]:
for b in ["mock", "claude", "codex", "copilot"]:
self.assertIn(b, backends)
def test_schema_has_key_engine_params(self):
@@ -64,10 +64,6 @@ class TestClaudeHomeExpansion(unittest.TestCase):
(the documented mcp-config sets SKILLOPT_DEVIN_CLAUDE_HOME="~/...")."""
def test_env_tilde_is_expanded(self):
# Re-insert the devin plugin path at position 0 so importlib.reload
# picks up this module, not plugins/copilot/mcp_server.py when both
# test modules are loaded in the same process.
sys.path.insert(0, PLUGIN)
os.environ["SKILLOPT_DEVIN_CLAUDE_HOME"] = "~/.skillopt-sleep-devin"
try:
importlib.reload(mcp_server)
tests/test_json_utils.py
-52
View File
@@ -5,7 +5,6 @@ import pytest
from skillopt.utils.json_utils import (
_top_level_brace_objects,
_top_level_bracket_arrays,
extract_json,
extract_json_array,
)
@@ -86,41 +85,6 @@ class TestTopLevelBraceObjects:
assert _top_level_brace_objects(text) == ['{"edit": "right"}']
class TestTopLevelBracketArrays:
"""_top_level_bracket_arrays — string/object-aware top-level array scan."""
def test_single_clean_array(self) -> None:
assert _top_level_bracket_arrays("[1, 2]") == ["[1, 2]"]
def test_two_top_level_arrays(self) -> None:
assert _top_level_bracket_arrays("[1]\n[2]") == ["[1]", "[2]"]
def test_array_inside_object_is_ignored(self) -> None:
assert _top_level_bracket_arrays('{"items": [1, 2]}') == []
def test_bracket_inside_quoted_prose_is_ignored(self) -> None:
assert _top_level_bracket_arrays('label is "set it to [x]" done') == []
def test_unmatched_prose_brace_does_not_hide_later_array(self) -> None:
text = "Explanation uses {placeholder, final answer [1, 2]"
assert _top_level_bracket_arrays(text) == ["[1, 2]"]
def test_many_unmatched_braces_scan_linearly(self) -> None:
class CountingText(str):
def __new__(cls, value: str):
instance = super().__new__(cls, value)
instance.reads = 0
return instance
def __getitem__(self, key):
self.reads += 1
return super().__getitem__(key)
text = CountingText("{" * 1_000 + " final answer [1, 2]")
assert _top_level_bracket_arrays(text) == ["[1, 2]"]
assert text.reads < len(text) * 4
class TestExtractJsonTolerantFallback:
"""extract_json — json_repair fallback for malformed non-OpenAI output."""
@@ -222,19 +186,3 @@ class TestExtractJsonArray:
"""extract_json_array should not match a bare JSON object."""
text = '{"this is an object": true}'
assert extract_json_array(text) is None
def test_ignores_prose_brackets_before_valid_array(self) -> None:
text = "Use [not JSON] as prose. Final answer: [1, 2, 3]"
assert extract_json_array(text) == [1, 2, 3]
def test_multiple_valid_arrays_are_ambiguous(self) -> None:
text = "First candidate [1], second candidate [2]"
assert extract_json_array(text) is None
def test_nested_object_array_not_confused_with_array_answer(self) -> None:
text = '{"items": [1, 2, 3]}'
assert extract_json_array(text) is None
def test_unmatched_prose_brace_before_valid_array(self) -> None:
text = "Explanation uses {placeholder, final answer [1, 2]"
assert extract_json_array(text) == [1, 2]
tests/test_materialize_searchqa.py
-13
View File
@@ -64,16 +64,3 @@ def test_materialize_searchqa_splits_fails_on_missing_manifest_id(tmp_path):
{"train": [_row("a")]},
dataset_name="example/searchqa",
)
def test_materialize_searchqa_splits_rejects_duplicate_manifest_ids(tmp_path):
manifest_dir = tmp_path / "manifest"
_write_manifest(manifest_dir, {"train": ["a"], "val": ["a"], "test": []})
with pytest.raises(ValueError, match="duplicate IDs"):
materialize_searchqa_splits(
manifest_dir,
tmp_path / "out",
{"train": [_row("a")]},
dataset_name="example/searchqa",
)
tests/test_openai_compatible_backend.py
-146
View File
@@ -1,146 +0,0 @@
"""Tests for the generic OpenAI-compatible model backend."""
from __future__ import annotations
from types import SimpleNamespace
from typing import Any
import pytest
import skillopt.model as model
from skillopt.model import backend_config
from skillopt.model import openai_compatible_backend as backend
class _CompletionRecorder:
def __init__(self) -> None:
self.calls: list[dict[str, Any]] = []
def create(self, **kwargs: Any) -> Any:
self.calls.append(kwargs)
message = SimpleNamespace(content="ok", tool_calls=[])
usage = SimpleNamespace(prompt_tokens=2, completion_tokens=3, total_tokens=5)
return SimpleNamespace(choices=[SimpleNamespace(message=message)], usage=usage)
class _Client:
def __init__(self, recorder: _CompletionRecorder) -> None:
self.chat = SimpleNamespace(completions=recorder)
@pytest.fixture(autouse=True)
def isolate_backend_state(monkeypatch: pytest.MonkeyPatch):
optimizer_backend = backend_config.get_optimizer_backend()
target_backend = backend_config.get_target_backend()
optimizer_config = vars(backend.OPTIMIZER_CONFIG).copy()
target_config = vars(backend.TARGET_CONFIG).copy()
backend.reset_token_tracker()
yield
backend.reset_token_tracker()
vars(backend.OPTIMIZER_CONFIG).update(optimizer_config)
vars(backend.TARGET_CONFIG).update(target_config)
backend_config.set_optimizer_backend(optimizer_backend)
backend_config.set_target_backend(target_backend)
backend._reset_clients()
def test_configure_preserves_role_specific_values() -> None:
model.configure_openai_compatible(
base_url="https://shared.example/v1",
api_key="shared-key",
model="shared-model",
optimizer_base_url="https://optimizer.example/v1",
optimizer_api_key="optimizer-key",
optimizer_model="optimizer-model",
target_base_url="https://target.example/v1",
target_api_key="target-key",
target_model="target-model",
)
assert backend.OPTIMIZER_CONFIG.base_url == "https://optimizer.example/v1"
assert backend.OPTIMIZER_CONFIG.api_key == "optimizer-key"
assert backend.OPTIMIZER_CONFIG.deployment == "optimizer-model"
assert backend.TARGET_CONFIG.base_url == "https://target.example/v1"
assert backend.TARGET_CONFIG.api_key == "target-key"
assert backend.TARGET_CONFIG.deployment == "target-model"
def test_optimizer_and_target_route_to_their_own_clients(monkeypatch: pytest.MonkeyPatch) -> None:
optimizer_calls = _CompletionRecorder()
target_calls = _CompletionRecorder()
monkeypatch.setattr(
backend,
"_get_client",
lambda role: _Client(optimizer_calls if role == "optimizer" else target_calls),
)
model.set_optimizer_backend("openai_compatible")
model.set_target_backend("openai_compatible")
backend.OPTIMIZER_CONFIG.deployment = "optimizer-model"
backend.TARGET_CONFIG.deployment = "target-model"
model.chat_optimizer("system", "user", retries=1)
model.chat_target_messages([{"role": "user", "content": "question"}], retries=1)
assert optimizer_calls.calls[0]["model"] == "optimizer-model"
assert target_calls.calls[0]["model"] == "target-model"
def test_client_creation_is_lazy(monkeypatch: pytest.MonkeyPatch) -> None:
builds: list[str] = []
def build(config: backend.OpenAICompatibleConfig) -> _Client:
builds.append(config.deployment)
return _Client(_CompletionRecorder())
backend._reset_clients()
monkeypatch.setattr(backend, "_build_client", build)
assert builds == []
backend._get_client("optimizer")
backend._get_client("optimizer")
assert builds == [backend.OPTIMIZER_CONFIG.deployment]
def test_combined_token_summary_counts_each_backend_once(
monkeypatch: pytest.MonkeyPatch,
) -> None:
def make(stage: str) -> dict[str, dict[str, int]]:
return {
stage: {
"calls": 1,
"prompt_tokens": 2,
"completion_tokens": 3,
"total_tokens": 5,
},
"_total": {
"calls": 1,
"prompt_tokens": 2,
"completion_tokens": 3,
"total_tokens": 5,
},
}
monkeypatch.setattr(model._openai, "get_token_summary", lambda: make("azure"))
monkeypatch.setattr(model._claude, "get_token_summary", lambda: make("claude"))
monkeypatch.setattr(model._qwen, "get_token_summary", lambda: make("qwen"))
monkeypatch.setattr(model._minimax, "get_token_summary", lambda: make("minimax"))
monkeypatch.setattr(model._openai_compat, "get_token_summary", lambda: make("openai_compatible"))
combined = model.get_token_summary()
assert set(combined) - {"_total"} == {"azure", "claude", "qwen", "minimax", "openai_compatible"}
expected_stage_total = {
"calls": 1,
"prompt_tokens": 2,
"completion_tokens": 3,
"total_tokens": 5,
}
for stage in {"azure", "claude", "qwen", "minimax", "openai_compatible"}:
assert combined[stage] == expected_stage_total
assert combined["_total"] == {
"calls": 5,
"prompt_tokens": 10,
"completion_tokens": 15,
"total_tokens": 25,
}
tests/test_run_sleep_fallback.py
-160
View File
@@ -1,160 +0,0 @@
"""Tests for plugins/run-sleep.sh engine resolution and fallback logic.
Verifies that the runner:
1. Uses the source checkout when skillopt_sleep/ is present (existing path).
2. Falls back to the skillopt-sleep CLI on PATH when no source dir is found.
3. Falls back to `python -m skillopt_sleep` when the package is importable
but no CLI is on PATH.
4. Errors with a helpful message when no engine can be found.
Pure-stdlib (unittest). No API key, no third-party deps.
Run: python3 -m pytest tests/test_run_sleep_fallback.py -v
"""
import os
import shutil
import subprocess
import sys
import tempfile
import unittest
REPO = os.path.abspath(os.path.join(os.path.dirname(__file__), ".."))
RUNNER = os.path.join(REPO, "plugins", "run-sleep.sh")
BUNDLED = os.path.join(REPO, "plugins", "claude-code", "scripts", "run-sleep.sh")
def _run(script, args, env, cwd):
"""Run a sleep runner script and return (returncode, stdout, stderr)."""
cmd = ["bash", script] + args
proc = subprocess.run(
cmd, capture_output=True, text=True, env=env, cwd=cwd, timeout=30,
)
return proc.returncode, proc.stdout, proc.stderr
@unittest.skipIf(sys.platform == "win32", "Unix bash runner tests require POSIX/bash environment")
class TestRunnerSourceCheckout(unittest.TestCase):
"""When skillopt_sleep/ is found in the repo, the source-checkout path is used."""
def test_source_checkout_runs_engine(self):
# The repo root contains skillopt_sleep/, so running from the repo
# should find it via the SCRIPT_DIR/../skillopt_sleep check.
env = {k: v for k, v in os.environ.items()
if k not in ("SKILLOPT_SLEEP_REPO", "CLAUDE_PLUGIN_ROOT")}
rc, out, err = _run(RUNNER, ["--help"], env, REPO)
self.assertEqual(rc, 0, f"stderr:\n{err}")
self.assertIn("skillopt_sleep", out)
self.assertIn("run", out)
self.assertIn("status", out)
@unittest.skipIf(sys.platform == "win32", "Unix bash runner tests require POSIX/bash environment")
class TestRunnerCliFallback(unittest.TestCase):
"""When no source dir is found, fall back to the skillopt-sleep CLI on PATH."""
def setUp(self):
# Copy the runner to a temp dir so SCRIPT_DIR/../skillopt_sleep doesn't
# resolve to the repo's source checkout (SCRIPT_DIR is the script's
# location, not CWD, so running the repo copy always finds the source).
self._tmp = tempfile.TemporaryDirectory()
self._script = os.path.join(self._tmp.name, "run-sleep.sh")
shutil.copy2(RUNNER, self._script)
# Build a fake skillopt-sleep CLI on PATH that records its args.
self._bindir = os.path.join(self._tmp.name, "bin")
os.makedirs(self._bindir)
self._fake_cli = os.path.join(self._bindir, "skillopt-sleep")
with open(self._fake_cli, "w") as f:
f.write("#!/usr/bin/env bash\n")
f.write('echo "fake-cli invoked: $@"\n')
os.chmod(self._fake_cli, 0o755)
def tearDown(self):
self._tmp.cleanup()
def _env_without_source(self):
"""Env with no source dir resolvable and fake CLI on PATH."""
env = {k: v for k, v in os.environ.items()
if k not in ("SKILLOPT_SLEEP_REPO", "CLAUDE_PLUGIN_ROOT")}
# Prepend fake bin dir so our skillopt-sleep is found first.
env["PATH"] = self._bindir + os.pathsep + env.get("PATH", "")
return env
def test_cli_fallback_used_when_no_source_dir(self):
env = self._env_without_source()
rc, out, err = _run(self._script, ["status", "--project", "/tmp"], env, "/tmp")
self.assertEqual(rc, 0, f"stderr:\n{err}")
self.assertIn("fake-cli invoked: status --project /tmp", out)
def test_cli_fallback_passes_through_all_args(self):
env = self._env_without_source()
rc, out, err = _run(
self._script, ["run", "--project", "/tmp", "--backend", "mock"], env, "/tmp",
)
self.assertEqual(rc, 0, f"stderr:\n{err}")
self.assertIn("run --project /tmp --backend mock", out)
def test_bundled_copy_also_uses_cli_fallback(self):
"""The byte-identical bundled copy in plugins/claude-code/scripts/ must
also fall back to the CLI — it's what the marketplace install ships."""
bundled_copy = os.path.join(self._tmp.name, "bundled-run-sleep.sh")
shutil.copy2(BUNDLED, bundled_copy)
env = self._env_without_source()
rc, out, err = _run(bundled_copy, ["status"], env, "/tmp")
self.assertEqual(rc, 0, f"stderr:\n{err}")
self.assertIn("fake-cli invoked: status", out)
@unittest.skipIf(sys.platform == "win32", "Unix bash runner tests require POSIX/bash environment")
class TestRunnerNoEngine(unittest.TestCase):
"""When no source dir, no CLI on PATH, and no importable package, error cleanly."""
def test_error_message_mentions_install_options(self):
# Copy the runner to a temp dir so SCRIPT_DIR/../skillopt_sleep doesn't
# resolve to the repo's source checkout.
with tempfile.TemporaryDirectory() as tmp:
script = os.path.join(tmp, "run-sleep.sh")
shutil.copy2(RUNNER, script)
# Build an env with a minimal PATH so skillopt-sleep isn't found,
# and no source dir resolvable.
env = {
"PATH": "/usr/bin:/bin",
"HOME": os.environ.get("HOME", "/tmp"),
}
# Use a Python that almost certainly doesn't have skillopt_sleep
# importable (system python3). If it does, skip — we can't easily
# simulate "not installed" in that case.
try:
subprocess.run(
[sys.executable, "-c", "import skillopt_sleep"],
capture_output=True, check=True, timeout=10,
)
has_package = True
except (subprocess.CalledProcessError, subprocess.TimeoutExpired):
has_package = False
if has_package:
self.skipTest("skillopt_sleep is importable in this env; "
"cannot test the no-engine error path")
rc, out, err = _run(script, ["status"], env, "/tmp")
self.assertNotEqual(rc, 0)
self.assertIn("could not locate the skillopt_sleep package", err)
self.assertIn("uv tool install skillopt", err)
self.assertIn("pip install skillopt", err)
self.assertIn("SKILLOPT_SLEEP_REPO", err)
class TestRunnerBundledMatchesShared(unittest.TestCase):
"""The bundled copy must stay in sync with the shared runner."""
def test_bundled_equals_shared(self):
with open(RUNNER) as f:
shared = f.read()
with open(BUNDLED) as f:
bundled = f.read()
self.assertEqual(shared, bundled,
"plugins/claude-code/scripts/run-sleep.sh has drifted "
"from plugins/run-sleep.sh — they must stay in sync")
if __name__ == "__main__":
unittest.main()
tests/test_runners.Tests.ps1
-295
View File
@@ -1,295 +0,0 @@
Describe "run-sleep.ps1 runner" {
BeforeAll {
$script = Join-Path $PSScriptRoot "..\plugins\run-sleep.ps1"
$tempDir = [System.IO.Path]::GetTempFileName()
Remove-Item $tempDir
New-Item -ItemType Directory -Path $tempDir | Out-Null
# Detect a working python to bypass Windows Store alias issues
if (-not $env:SKILLOPT_SLEEP_PYTHON) {
if (Test-Path "C:\Python314\python.exe") {
$env:SKILLOPT_SLEEP_PYTHON = "C:\Python314\python.exe"
} elseif (Test-Path "C:\Python313\python.exe") {
$env:SKILLOPT_SLEEP_PYTHON = "C:\Python313\python.exe"
} elseif (Test-Path "C:\Python312\python.exe") {
$env:SKILLOPT_SLEEP_PYTHON = "C:\Python312\python.exe"
} elseif (Test-Path "C:\Python311\python.exe") {
$env:SKILLOPT_SLEEP_PYTHON = "C:\Python311\python.exe"
} elseif (Test-Path "C:\Python310\python.exe") {
$env:SKILLOPT_SLEEP_PYTHON = "C:\Python310\python.exe"
}
}
}
AfterAll {
if (Test-Path $tempDir) {
Remove-Item -Recurse -Force $tempDir
}
}
It "runs successfully in source checkout mode" {
$env:SKILLOPT_SLEEP_REPO = Resolve-Path (Join-Path $PSScriptRoot "..")
# Run help
$result = powershell -File $script "--help" 2>&1
$LASTEXITCODE | Should Be 0
$result | Out-String | Should Match "skillopt_sleep"
}
It "falls back to CLI on PATH" {
# Create a temp dir for isolated testing
$sandbox = Join-Path $tempDir "cli_fallback"
New-Item -ItemType Directory -Path $sandbox | Out-Null
$scriptCopy = Join-Path $sandbox "run-sleep.ps1"
Copy-Item $script $scriptCopy
# Create fake CLI
$binDir = Join-Path $sandbox "bin"
New-Item -ItemType Directory -Path $binDir | Out-Null
$fakeCli = Join-Path $binDir "skillopt-sleep.cmd"
"@echo off`r`necho fake-cli invoked %*`r`nexit /b 0" | Out-File -FilePath $fakeCli -Encoding ascii
# Save existing env vars
$oldPath = $env:PATH
$oldRepo = $env:SKILLOPT_SLEEP_REPO
$oldPlugin = $env:CLAUDE_PLUGIN_ROOT
$env:PATH = "$binDir;$oldPath"
$env:SKILLOPT_SLEEP_REPO = $null
$env:CLAUDE_PLUGIN_ROOT = $null
$oldLocation = Get-Location
Set-Location $sandbox
try {
$result = powershell -File $scriptCopy "status" 2>&1
$LASTEXITCODE | Should Be 0
$result | Out-String | Should Match "fake-cli invoked status"
}
finally {
Set-Location $oldLocation
$env:PATH = $oldPath
$env:SKILLOPT_SLEEP_REPO = $oldRepo
$env:CLAUDE_PLUGIN_ROOT = $oldPlugin
}
}
It "propagates exit code on failure" {
$env:SKILLOPT_SLEEP_REPO = Resolve-Path (Join-Path $PSScriptRoot "..")
# Run a non-existent command to trigger python module failure
$result = powershell -File $script "non-existent-subcommand" 2>&1
$LASTEXITCODE | Should Not Be 0
}
It "handles paths containing spaces correctly" {
# Create path containing spaces
$spaceDir = Join-Path $tempDir "path with spaces"
New-Item -ItemType Directory -Path $spaceDir | Out-Null
$scriptCopy = Join-Path $spaceDir "run-sleep.ps1"
Copy-Item $script $scriptCopy
# Create fake CLI
$binDir = Join-Path $spaceDir "bin"
New-Item -ItemType Directory -Path $binDir | Out-Null
$fakeCli = Join-Path $binDir "skillopt-sleep.cmd"
"@echo off`r`necho fake-cli invoked %*`r`nexit /b 0" | Out-File -FilePath $fakeCli -Encoding ascii
# Save env
$oldPath = $env:PATH
$oldRepo = $env:SKILLOPT_SLEEP_REPO
$oldPlugin = $env:CLAUDE_PLUGIN_ROOT
$env:PATH = "$binDir;$oldPath"
$env:SKILLOPT_SLEEP_REPO = $null
$env:CLAUDE_PLUGIN_ROOT = $null
$oldLocation = Get-Location
Set-Location $spaceDir
try {
$result = powershell -File $scriptCopy "status" 2>&1
$LASTEXITCODE | Should Be 0
$result | Out-String | Should Match "fake-cli invoked status"
}
finally {
Set-Location $oldLocation
$env:PATH = $oldPath
$env:SKILLOPT_SLEEP_REPO = $oldRepo
$env:CLAUDE_PLUGIN_ROOT = $oldPlugin
}
}
}
Describe "run-sleep.cmd runner" {
BeforeAll {
$script = Join-Path $PSScriptRoot "..\plugins\run-sleep.cmd"
$tempDir = [System.IO.Path]::GetTempFileName()
Remove-Item $tempDir
New-Item -ItemType Directory -Path $tempDir | Out-Null
# Detect a working python to bypass Windows Store alias issues
if (-not $env:SKILLOPT_SLEEP_PYTHON) {
if (Test-Path "C:\Python314\python.exe") {
$env:SKILLOPT_SLEEP_PYTHON = "C:\Python314\python.exe"
} elseif (Test-Path "C:\Python313\python.exe") {
$env:SKILLOPT_SLEEP_PYTHON = "C:\Python313\python.exe"
} elseif (Test-Path "C:\Python312\python.exe") {
$env:SKILLOPT_SLEEP_PYTHON = "C:\Python312\python.exe"
} elseif (Test-Path "C:\Python311\python.exe") {
$env:SKILLOPT_SLEEP_PYTHON = "C:\Python311\python.exe"
} elseif (Test-Path "C:\Python310\python.exe") {
$env:SKILLOPT_SLEEP_PYTHON = "C:\Python310\python.exe"
}
}
}
AfterAll {
if (Test-Path $tempDir) {
Remove-Item -Recurse -Force $tempDir
}
}
It "runs successfully in source checkout mode" {
$env:SKILLOPT_SLEEP_REPO = Resolve-Path (Join-Path $PSScriptRoot "..")
# Run help
$result = cmd.exe /c $script "--help" 2>&1
$LASTEXITCODE | Should Be 0
$result | Out-String | Should Match "skillopt_sleep"
}
It "falls back to CLI on PATH" {
# Create a temp dir for isolated testing
$sandbox = Join-Path $tempDir "cmd_cli_fallback"
New-Item -ItemType Directory -Path $sandbox | Out-Null
$scriptCopy = Join-Path $sandbox "run-sleep.cmd"
Copy-Item $script $scriptCopy
# Create fake CLI
$binDir = Join-Path $sandbox "bin"
New-Item -ItemType Directory -Path $binDir | Out-Null
$fakeCli = Join-Path $binDir "skillopt-sleep.cmd"
"@echo off`r`necho fake-cli invoked %*`r`nexit /b 0" | Out-File -FilePath $fakeCli -Encoding ascii
# Save existing env vars
$oldPath = $env:PATH
$oldRepo = $env:SKILLOPT_SLEEP_REPO
$oldPlugin = $env:CLAUDE_PLUGIN_ROOT
$env:PATH = "$binDir;$oldPath"
$env:SKILLOPT_SLEEP_REPO = $null
$env:CLAUDE_PLUGIN_ROOT = $null
$oldLocation = Get-Location
Set-Location $sandbox
try {
$result = cmd.exe /c $scriptCopy "status" 2>&1
$LASTEXITCODE | Should Be 0
$result | Out-String | Should Match "fake-cli invoked status"
}
finally {
Set-Location $oldLocation
$env:PATH = $oldPath
$env:SKILLOPT_SLEEP_REPO = $oldRepo
$env:CLAUDE_PLUGIN_ROOT = $oldPlugin
}
}
It "falls back to importable module" {
$sandbox = Join-Path $tempDir "cmd_module_fallback"
New-Item -ItemType Directory -Path $sandbox | Out-Null
$scriptCopy = Join-Path $sandbox "run-sleep.cmd"
Copy-Item $script $scriptCopy
$oldRepo = $env:SKILLOPT_SLEEP_REPO
$oldPlugin = $env:CLAUDE_PLUGIN_ROOT
$env:SKILLOPT_SLEEP_REPO = $null
$env:CLAUDE_PLUGIN_ROOT = $null
$oldPythonPath = $env:PYTHONPATH
$env:PYTHONPATH = Resolve-Path (Join-Path $PSScriptRoot "..")
$oldLocation = Get-Location
Set-Location $sandbox
try {
$result = cmd.exe /c $scriptCopy "--help" 2>&1
$LASTEXITCODE | Should Be 0
$result | Out-String | Should Match "skillopt_sleep"
}
finally {
Set-Location $oldLocation
$env:SKILLOPT_SLEEP_REPO = $oldRepo
$env:CLAUDE_PLUGIN_ROOT = $oldPlugin
$env:PYTHONPATH = $oldPythonPath
}
}
It "fails when nothing is found" {
$sandbox = Join-Path $tempDir "cmd_failure"
New-Item -ItemType Directory -Path $sandbox | Out-Null
$scriptCopy = Join-Path $sandbox "run-sleep.cmd"
Copy-Item $script $scriptCopy
$oldRepo = $env:SKILLOPT_SLEEP_REPO
$oldPlugin = $env:CLAUDE_PLUGIN_ROOT
$oldPath = $env:PATH
$env:SKILLOPT_SLEEP_REPO = $null
$env:CLAUDE_PLUGIN_ROOT = $null
$emptyBin = Join-Path $sandbox "empty_bin"
New-Item -ItemType Directory -Path $emptyBin | Out-Null
$env:PATH = $emptyBin
$oldLocation = Get-Location
Set-Location $sandbox
try {
$result = cmd.exe /c $scriptCopy "status" 2>&1
$LASTEXITCODE | Should Not Be 0
$result | Out-String | Should Match "could not locate the skillopt_sleep package"
}
finally {
Set-Location $oldLocation
$env:SKILLOPT_SLEEP_REPO = $oldRepo
$env:CLAUDE_PLUGIN_ROOT = $oldPlugin
$env:PATH = $oldPath
}
}
It "propagates exit code on failure" {
$env:SKILLOPT_SLEEP_REPO = Resolve-Path (Join-Path $PSScriptRoot "..")
$result = cmd.exe /c $script "non-existent-subcommand" 2>&1
$LASTEXITCODE | Should Not Be 0
}
It "handles paths containing spaces correctly" {
$spaceDir = Join-Path $tempDir "cmd path with spaces"
New-Item -ItemType Directory -Path $spaceDir | Out-Null
$scriptCopy = Join-Path $spaceDir "run-sleep.cmd"
Copy-Item $script $scriptCopy
$binDir = Join-Path $spaceDir "bin"
New-Item -ItemType Directory -Path $binDir | Out-Null
$fakeCli = Join-Path $binDir "skillopt-sleep.cmd"
"@echo off`r`necho fake-cli invoked %*`r`nexit /b 0" | Out-File -FilePath $fakeCli -Encoding ascii
$oldPath = $env:PATH
$oldRepo = $env:SKILLOPT_SLEEP_REPO
$oldPlugin = $env:CLAUDE_PLUGIN_ROOT
$env:PATH = "$binDir;$oldPath"
$env:SKILLOPT_SLEEP_REPO = $null
$env:CLAUDE_PLUGIN_ROOT = $null
$oldLocation = Get-Location
Set-Location $spaceDir
try {
$result = cmd.exe /c $scriptCopy "status" 2>&1
$LASTEXITCODE | Should Be 0
$result | Out-String | Should Match "fake-cli invoked status"
}
finally {
Set-Location $oldLocation
$env:PATH = $oldPath
$env:SKILLOPT_SLEEP_REPO = $oldRepo
$env:CLAUDE_PLUGIN_ROOT = $oldPlugin
}
}
}
tests/test_scheduler.py
-346
View File
@@ -1,346 +0,0 @@
"""Tests for skillopt.optimizer.scheduler — edit-budget schedulers.
ReflACT trainers use an edit-budget scheduler at each optimisation step to
control how many skill edits are allowed (analogous to gradient clipping /
learning-rate annealing in neural-network training).
This module has zero LLM dependencies — all behaviour is deterministic pure
math — making it an ideal target for precise unit tests.
Scheduler contract
------------------
For multi-step runs, decay schedulers (Linear, Cosine) guarantee:
- First ``step()`` evaluates the first decay interval at
``t = 1 / total_steps`` (rounding can still produce ``max_lr``).
- ``total_steps``-th ``step()`` returns ``min_lr``.
- Intermediate values are monotonically non-increasing and clamped to
``[min_lr, max_lr]``.
- Beyond ``total_steps`` the value plateaus at ``min_lr``.
"""
from __future__ import annotations
import pytest
from skillopt.optimizer.scheduler import (
LRScheduler,
ConstantScheduler,
LinearScheduler,
CosineScheduler,
AutonomousScheduler,
build_scheduler,
)
# ── ConstantScheduler ────────────────────────────────────────────────────────
class TestConstantScheduler:
"""ConstantScheduler — fixed edit budget regardless of step."""
def test_always_returns_max_lr(self) -> None:
s = ConstantScheduler(max_lr=8, min_lr=2, total_steps=10)
for _ in range(10):
assert s.step() == 8
def test_step_advances_internal_counter(self) -> None:
s = ConstantScheduler(max_lr=5, min_lr=1, total_steps=5)
assert s._current_step == 0
s.step()
assert s._current_step == 1
s.step()
assert s._current_step == 2
def test_get_lr_returns_max_for_arbitrary_step(self) -> None:
s = ConstantScheduler(max_lr=12, min_lr=1, total_steps=100)
assert s.get_lr(1) == 12
assert s.get_lr(50) == 12
assert s.get_lr(999) == 12
def test_state_dict_and_load_state_dict_round_trip(self) -> None:
s = ConstantScheduler(max_lr=8, min_lr=2, total_steps=10)
for _ in range(3):
s.step()
assert s._current_step == 3
state = s.state_dict()
s2 = ConstantScheduler(max_lr=8, min_lr=2, total_steps=10)
s2.load_state_dict(state)
assert s2._current_step == 3
# Step after resume lands on the correct step
assert s2.step() == 8
assert s2._current_step == 4
def test_load_state_dict_with_missing_key_defaults_to_zero(self) -> None:
s = ConstantScheduler(max_lr=8, min_lr=2, total_steps=10)
s.load_state_dict({})
assert s._current_step == 0
# ── LinearScheduler ──────────────────────────────────────────────────────────
class TestLinearScheduler:
"""LinearScheduler — linear decay from max_lr to min_lr."""
def test_first_step_uses_first_decay_interval(self) -> None:
s = LinearScheduler(max_lr=10, min_lr=2, total_steps=10)
assert s.step() == 9
def test_last_step_returns_min_lr(self) -> None:
s = LinearScheduler(max_lr=10, min_lr=2, total_steps=10)
for _ in range(9):
s.step()
assert s.step() == 2
def test_all_steps_return_integers(self) -> None:
s = LinearScheduler(max_lr=8, min_lr=2, total_steps=6)
for _ in range(6):
lr = s.step()
assert isinstance(lr, int)
def test_total_steps_one_returns_max_lr(self) -> None:
s = LinearScheduler(max_lr=10, min_lr=2, total_steps=1)
assert s.step() == 10
def test_total_steps_zero_returns_max_lr(self) -> None:
"""Degenerate case: 0-step training still gets max_lr on the one call."""
s = LinearScheduler(max_lr=10, min_lr=2, total_steps=0)
assert s.step() == 10
def test_monotonically_non_increasing(self) -> None:
s = LinearScheduler(max_lr=20, min_lr=2, total_steps=100)
prev: int = 999
for _ in range(100):
lr = s.step()
assert lr <= prev
prev = lr
def test_never_below_min_lr(self) -> None:
s = LinearScheduler(max_lr=10, min_lr=2, total_steps=10)
for _ in range(20): # overshoot
assert s.step() >= 2
def test_never_above_max_lr(self) -> None:
s = LinearScheduler(max_lr=10, min_lr=2, total_steps=10)
for _ in range(20):
assert s.step() <= 10
def test_after_total_steps_stays_at_min_lr(self) -> None:
s = LinearScheduler(max_lr=8, min_lr=2, total_steps=5)
for _ in range(5):
s.step()
# Steps beyond total_steps should plateau at min_lr
for _ in range(5):
assert s.step() == 2
def test_known_decay_sequence(self) -> None:
"""Linear decay max_lr=10, min_lr=2, total_steps=4.
t = step/total_steps = 1/4, 1/2, 3/4, 1
lr = 10 + (2-10)*t = 10 - 8t
t=1/4: lr=8, t=1/2: lr=6, t=3/4: lr=4, t=1: lr=2
"""
s = LinearScheduler(max_lr=10, min_lr=2, total_steps=4)
assert s.step() == 8
assert s.step() == 6
assert s.step() == 4
assert s.step() == 2
def test_step_state_dict_resume_consistent(self) -> None:
"""After resume, the next step value is the same as without resume."""
s1 = LinearScheduler(max_lr=10, min_lr=2, total_steps=5)
for _ in range(3):
s1.step()
resumed_lr = s1.step() # step 4
s2 = LinearScheduler(max_lr=10, min_lr=2, total_steps=5)
s2.load_state_dict({"current_step": 3})
assert s2.step() == resumed_lr
def test_max_lr_equals_min_lr_yields_constant(self) -> None:
s = LinearScheduler(max_lr=5, min_lr=5, total_steps=10)
for _ in range(10):
assert s.step() == 5
# ── CosineScheduler ──────────────────────────────────────────────────────────
class TestCosineScheduler:
"""CosineScheduler — cosine annealing from max_lr to min_lr."""
def test_first_step_uses_first_decay_interval(self) -> None:
s = CosineScheduler(max_lr=100, min_lr=0, total_steps=4)
assert s.step() == 85
def test_last_step_returns_min_lr(self) -> None:
s = CosineScheduler(max_lr=10, min_lr=2, total_steps=10)
for _ in range(9):
s.step()
assert s.step() == 2
def test_all_steps_return_integers(self) -> None:
s = CosineScheduler(max_lr=8, min_lr=2, total_steps=6)
for _ in range(6):
lr = s.step()
assert isinstance(lr, int)
def test_total_steps_one_returns_max_lr(self) -> None:
s = CosineScheduler(max_lr=10, min_lr=2, total_steps=1)
assert s.step() == 10
def test_total_steps_zero_returns_max_lr(self) -> None:
s = CosineScheduler(max_lr=10, min_lr=2, total_steps=0)
assert s.step() == 10
def test_monotonically_non_increasing(self) -> None:
s = CosineScheduler(max_lr=20, min_lr=2, total_steps=100)
prev: int = 999
for _ in range(100):
lr = s.step()
assert lr <= prev
prev = lr
def test_never_below_min_lr(self) -> None:
s = CosineScheduler(max_lr=10, min_lr=2, total_steps=10)
for _ in range(20):
assert s.step() >= 2
def test_never_above_max_lr(self) -> None:
s = CosineScheduler(max_lr=10, min_lr=2, total_steps=10)
for _ in range(20):
assert s.step() <= 10
def test_after_total_steps_stays_at_min_lr(self) -> None:
s = CosineScheduler(max_lr=8, min_lr=2, total_steps=5)
for _ in range(5):
s.step()
for _ in range(5):
assert s.step() == 2
def test_midpoint_close_to_mean(self) -> None:
"""At the half-way neighbourhood, cosine is close to (max+min)/2.
total_steps=100, step=50 → t=50/100=0.5.
cos(0.5π)=0, lr = (20+2)/2 = 11.
"""
s = CosineScheduler(max_lr=20, min_lr=2, total_steps=100)
for _ in range(49):
s.step()
mid = s.step() # step 50
assert mid == 11
def test_step_state_dict_resume_consistent(self) -> None:
s1 = CosineScheduler(max_lr=10, min_lr=2, total_steps=5)
for _ in range(3):
s1.step()
resumed_lr = s1.step()
s2 = CosineScheduler(max_lr=10, min_lr=2, total_steps=5)
s2.load_state_dict({"current_step": 3})
assert s2.step() == resumed_lr
def test_max_lr_equals_min_lr_yields_constant(self) -> None:
s = CosineScheduler(max_lr=5, min_lr=5, total_steps=10)
for _ in range(10):
assert s.step() == 5
def test_early_steps_near_max(self) -> None:
"""Cosine annealing stays near max_lr early on."""
s = CosineScheduler(max_lr=100, min_lr=0, total_steps=100)
# step 1: t=0.01, cos(0.01π)≈0.9995 → lr≈99.98 → 100
assert s.step() == 100
# step 2: t=0.02, cos(0.02π)≈0.9980 → lr≈99.90 → 100
assert s.step() == 100
def test_known_decay_sequence(self) -> None:
s = CosineScheduler(max_lr=10, min_lr=2, total_steps=4)
assert [s.step() for _ in range(4)] == [9, 6, 3, 2]
def test_late_steps_near_min(self) -> None:
"""Cosine annealing flattens near min_lr at the end (cos(π)=-1)."""
s = CosineScheduler(max_lr=100, min_lr=0, total_steps=100)
for _ in range(99):
s.step()
# step 100: t=1, cos(π)=-1 → lr=0
assert s.step() == 0
# ── AutonomousScheduler ──────────────────────────────────────────────────────
class TestAutonomousScheduler:
"""AutonomousScheduler — no edit limit (model decides freely)."""
def test_always_returns_no_limit(self) -> None:
s = AutonomousScheduler(max_lr=8, min_lr=2, total_steps=10)
for _ in range(20):
assert s.step() == AutonomousScheduler.NO_LIMIT
def test_step_advances_counter(self) -> None:
s = AutonomousScheduler(max_lr=5, min_lr=1, total_steps=5)
assert s._current_step == 0
s.step()
assert s._current_step == 1
def test_get_lr_returns_no_limit(self) -> None:
s = AutonomousScheduler(max_lr=5, min_lr=1, total_steps=10)
assert s.get_lr(1) == AutonomousScheduler.NO_LIMIT
assert s.get_lr(50) == AutonomousScheduler.NO_LIMIT
def test_state_dict_round_trip(self) -> None:
s = AutonomousScheduler(max_lr=5, min_lr=1, total_steps=10)
for _ in range(4):
s.step()
s2 = AutonomousScheduler(max_lr=5, min_lr=1, total_steps=10)
s2.load_state_dict(s.state_dict())
assert s2._current_step == 4
# ── build_scheduler factory ──────────────────────────────────────────────────
class TestBuildScheduler:
"""build_scheduler factory — creates the right scheduler from a mode name."""
def test_constant(self) -> None:
s = build_scheduler("constant", max_lr=8, min_lr=2, total_steps=10)
assert isinstance(s, ConstantScheduler)
assert s.max_lr == 8
assert s.min_lr == 2
assert s.total_steps == 10
def test_linear(self) -> None:
s = build_scheduler("linear", max_lr=12, min_lr=3, total_steps=20)
assert isinstance(s, LinearScheduler)
def test_cosine(self) -> None:
s = build_scheduler("cosine", max_lr=16, min_lr=4, total_steps=30)
assert isinstance(s, CosineScheduler)
def test_autonomous(self) -> None:
s = build_scheduler("autonomous", max_lr=8, min_lr=2, total_steps=10)
assert isinstance(s, AutonomousScheduler)
def test_unknown_mode_raises_value_error(self) -> None:
with pytest.raises(ValueError, match="Unknown scheduler mode"):
build_scheduler("exponential", max_lr=8, min_lr=2, total_steps=10)
def test_default_mode_is_constant(self) -> None:
s = build_scheduler(max_lr=8, min_lr=2, total_steps=10)
assert isinstance(s, ConstantScheduler)
# ── Abstract base class ──────────────────────────────────────────────────────
class TestLRSchedulerBase:
"""LRScheduler — abstract base: cannot be instantiated directly."""
def test_cannot_instantiate_abstract(self) -> None:
with pytest.raises(TypeError):
LRScheduler(max_lr=8, min_lr=2, total_steps=10) # type: ignore[abstract]
def test_concrete_subclass_instantiates_fine(self) -> None:
"""ConstantScheduler (and others) work normally."""
s = ConstantScheduler(max_lr=8, min_lr=2, total_steps=10)
assert isinstance(s, LRScheduler)
assert s.step() == 8
tests/test_scheduler_windows.py
-77
View File
@@ -1,77 +0,0 @@
import os
import sys
import unittest
from unittest import mock
class TestSchedulerWindows(unittest.TestCase):
@mock.patch("sys.platform", "win32")
@mock.patch("shutil.which", return_value="C:\\Windows\\System32\\schtasks.exe")
def test_schedule_windows(self, mock_which):
from skillopt_sleep.scheduler import schedule
calls = []
def fake_run(cmd, **kwargs):
calls.append(cmd)
class Proc:
returncode = 0
stdout = "SUCCESS: The scheduled task ... has successfully been created."
stderr = ""
return Proc()
mock_open = mock.mock_open()
with mock.patch("subprocess.run", side_effect=fake_run), \
mock.patch("os.makedirs") as mock_makedirs, \
mock.patch("builtins.open", mock_open):
ok, msg = schedule("/p/my project", backend="mock", hour=3, minute=17)
self.assertTrue(ok)
self.assertIn("via Windows Task Scheduler", msg)
self.assertEqual(len(calls), 1)
cmd = calls[0]
self.assertEqual(cmd[0], "schtasks")
self.assertEqual(cmd[1], "/create")
self.assertEqual(cmd[2], "/tn")
self.assertTrue(cmd[3].startswith("SkillOpt-Sleep-"))
self.assertIn("my_project", cmd[3])
self.assertEqual(cmd[4], "/tr")
self.assertIn("run.cmd", cmd[5])
self.assertEqual(cmd[6], "/sc")
self.assertEqual(cmd[7], "daily")
self.assertEqual(cmd[8], "/st")
self.assertEqual(cmd[9], "03:17")
self.assertEqual(cmd[10], "/f")
mock_makedirs.assert_called_once()
mock_open.assert_called_once()
# Verify the content written to the helper script
handle = mock_open()
written = "".join(call[0][0] for call in handle.write.call_args_list)
self.assertIn("@echo off", written)
self.assertIn("run --project", written)
@mock.patch("sys.platform", "win32")
@mock.patch("shutil.which", return_value="C:\\Windows\\System32\\schtasks.exe")
def test_unschedule_windows(self, mock_which):
from skillopt_sleep.scheduler import unschedule
calls = []
def fake_run(cmd, **kwargs):
calls.append(cmd)
class Proc:
returncode = 0
stdout = "SUCCESS: The scheduled task ... was successfully deleted."
stderr = ""
return Proc()
with mock.patch("subprocess.run", side_effect=fake_run), \
mock.patch("os.path.exists", return_value=True), \
mock.patch("os.remove") as mock_remove:
ok, msg = unschedule("/p/my project")
self.assertTrue(ok)
self.assertEqual(len(calls), 1)
cmd = calls[0]
self.assertEqual(cmd[0], "schtasks")
self.assertEqual(cmd[1], "/delete")
self.assertEqual(cmd[2], "/tn")
self.assertTrue(cmd[3].startswith("SkillOpt-Sleep-"))
self.assertEqual(cmd[4], "/f")
mock_remove.assert_called_once()
tests/test_sleep_engine.py
+20 -129
View File
@@ -199,7 +199,6 @@ class TestHarvest(unittest.TestCase):
"max_sessions": 5,
"max_tasks": 3,
"target_skill_path": ".agents/skills/taste-skill/SKILL.md",
"preferences": "Always use async/await",
"progress": True,
"auto_adopt": False,
})
@@ -207,7 +206,6 @@ class TestHarvest(unittest.TestCase):
cfg = _cfg_from_args(Args())
self.assertEqual(cfg.get("backend"), "codex")
self.assertEqual(cfg.get("preferences"), "Always use async/await")
self.assertEqual(cfg.get("max_sessions_per_night"), 5)
self.assertEqual(cfg.get("max_tasks_per_night"), 3)
self.assertTrue(cfg.get("progress"))
@@ -769,60 +767,6 @@ class TestCodexBackend(unittest.TestCase):
self.assertIn("exited 1", be.last_call_error) # failure surfaced for diagnostics
self.assertEqual(called, []) # no tool actually ran
def test_codex_resolve_path_windows(self):
from skillopt_sleep.backend import resolve_codex_path
with mock.patch("sys.platform", "win32"), \
mock.patch("shutil.which", return_value=None), \
mock.patch.dict("os.environ", {
"APPDATA": r"C:\Users\Sparsh\AppData\Roaming",
"USERPROFILE": r"C:\Users\Sparsh",
"NVM_HOME": r"C:\Users\Sparsh\nvm"
}), \
mock.patch("os.path.exists", return_value=True):
path = resolve_codex_path("")
self.assertEqual(path, r"C:\Users\Sparsh\AppData\Roaming\npm\codex.cmd")
def test_codex_attempt_with_tools_windows(self):
from skillopt_sleep.backend import CodexCliBackend
be = CodexCliBackend(codex_path="codex")
task = TaskRecord(id="t", project="/p", intent="answer the question",
reference_kind="rule",
judge={"checks": [{"op": "tool_called", "arg": "search"}]})
calls = []
def fake_run(cmd, **kwargs):
calls.append((cmd, kwargs))
class Proc:
returncode = 0
stdout = ""
stderr = ""
return Proc()
with mock.patch("os.name", "nt"), \
mock.patch("shutil.rmtree"), \
mock.patch("skillopt_sleep.backend.subprocess.run", side_effect=fake_run):
orig_mkdtemp = tempfile.mkdtemp
temp_dirs = []
def fake_mkdtemp(*args, **kwargs):
d = orig_mkdtemp(*args, **kwargs)
temp_dirs.append(d)
return d
with mock.patch("tempfile.mkdtemp", side_effect=fake_mkdtemp):
be.attempt_with_tools(task, "", "", ["search"])
self.assertEqual(len(temp_dirs), 1)
work_dir = temp_dirs[0]
shim_path = os.path.join(work_dir, "search.cmd")
try:
self.assertTrue(os.path.exists(shim_path))
with open(shim_path, "r") as f:
content = f.read()
self.assertIn("@echo off", content)
self.assertIn("%~n0", content)
finally:
import shutil
shutil.rmtree(work_dir, ignore_errors=True)
class TestMultiRolloutAndBudget(unittest.TestCase):
def test_rolloutset_stats(self):
@@ -1204,6 +1148,26 @@ class TestClaudeCliBackendBare(unittest.TestCase):
self.assertEqual(resp, "")
self.assertIn("Claude CLI spawn failed", be.last_call_error)
def test_last_call_error_cleared_on_next_success(self):
"""A prior failure must not make a later successful call look failed:
_call resets last_call_error at the start of each call."""
from unittest import mock
from skillopt_sleep.backend import ClaudeCliBackend
be = ClaudeCliBackend(claude_path="claude", timeout=3)
# First call fails to spawn -> error recorded.
with mock.patch("skillopt_sleep.backend.subprocess.run",
side_effect=FileNotFoundError("no claude")):
self.assertEqual(be._call("p1"), "")
self.assertIn("Claude CLI spawn failed", be.last_call_error)
# Next call succeeds -> stale error must be cleared.
class _OK:
stdout = "answer"
stderr = ""
with mock.patch("skillopt_sleep.backend.subprocess.run", return_value=_OK()):
self.assertEqual(be._call("p2"), "answer")
self.assertEqual(be.last_call_error, "")
@@ -1259,45 +1223,6 @@ class MockBeneficialBackend(MockBackend):
break
return edits
class MockFractionalDegradationBackend(MockBackend):
"""Backend where an edit improves train but partially degrades held-out.
Train improves 0.0 → 1.0 (looks good!) but held-out degrades from
1.0 → 0.5 across two val tasks (one still passes, one breaks).
A naive optimizer might see the train improvement and miss the
partial regression — the gate must catch this subtle case.
"""
HACK_TEXT = "Always respond with a definitive answer"
def attempt(self, task: TaskRecord, skill: str, memory: str, sample_id: int = 0) -> str:
ctx = (skill or "") + "\n" + (memory or "")
if self.HACK_TEXT in ctx:
if "rule:__reward_hacking__" in task.tags:
return str(task.reference) # train improves
if "rule:real" in task.tags:
if task.id == "v1":
return str(task.reference) # val task 1 still passes
if task.id == "v2":
return "misleading answer" # val task 2 breaks
# Baseline (without edit)
if "rule:__reward_hacking__" in task.tags:
return "I am missing input" # train fails
if "rule:real" in task.tags:
return str(task.reference) # held-out passes
return super().attempt(task, skill, memory, sample_id)
def reflect(self, failures, successes, skill, memory, **kwargs):
edits = []
for task, _ in failures:
if "rule:__reward_hacking__" in task.tags:
edits.append(EditRecord("skill", "add", self.HACK_TEXT,
rationale="improve response confidence"))
break
return edits
class TestVerifierDiscipline(unittest.TestCase):
def test_gate_rejects_reward_hacking_edit(self):
be = MockRewardHackingBackend()
@@ -1329,40 +1254,6 @@ class TestVerifierDiscipline(unittest.TestCase):
self.assertGreater(len(res.applied_edits), 0)
self.assertIn("step-by-step", res.applied_edits[0].content)
def test_gate_rejects_fractional_degradation(self):
"""Gate must reject an edit that partially degrades held-out (1.0→0.5),
not just all-or-nothing collapses. Train improves (0.0→1.0) which makes
the regression easy to miss — the gate catches it anyway."""
from skillopt_sleep.replay import aggregate_scores, replay_batch
be = MockFractionalDegradationBackend()
train = TaskRecord(id="t3", project="/p", intent="train", reference="ABC",
reference_kind="exact", tags=["rule:__reward_hacking__"], split="train")
val1 = TaskRecord(id="v1", project="/p", intent="val", reference="DEF",
reference_kind="exact", tags=["rule:real"], split="val")
val2 = TaskRecord(id="v2", project="/p", intent="val", reference="GHI",
reference_kind="exact", tags=["rule:real"], split="val")
tasks = [train, val1, val2]
candidate_pairs = replay_batch(be, [val1, val2], be.HACK_TEXT, "")
candidate_hard, _candidate_soft = aggregate_scores(candidate_pairs)
self.assertEqual([result.hard for _task, result in candidate_pairs], [1.0, 0.0])
self.assertEqual(candidate_hard, 0.5)
res = consolidate(be, tasks, "", "", edit_budget=4, gate_metric="hard", night=1)
self.assertFalse(res.accepted)
self.assertEqual(res.gate_action, "reject")
# Baseline: both val tasks pass → 1.0
self.assertEqual(res.holdout_baseline, 1.0)
# After rejection the skill reverts; final replay also passes both
self.assertEqual(res.holdout_candidate, 1.0)
# Confirm we had two val tasks in the baseline
self.assertEqual(len(res.holdout_detail), 2)
self.assertGreater(len(res.rejected_edits), 0)
self.assertIn("definitive answer", res.rejected_edits[0].content)
class TestDiagnosticsRedaction(unittest.TestCase):
"""diagnostics.json surfaces backend stderr / optimizer replies / task
responses for debugging — but those can carry credentials (e.g. a codex 401
tests/test_types.py
+2 -15
View File
@@ -3,7 +3,8 @@ from __future__ import annotations
import pytest
from skillopt.types import Edit, Patch, RolloutResult
from skillopt.types import Edit, Patch
# ── Edit ────────────────────────────────────────────────────────────────────
@@ -246,17 +247,3 @@ class TestPatchEdgeCases:
assert isinstance(p.edits[0], Edit)
assert p.edits[0].op == "append"
assert p.edits[0].content == "hello"
# ── RolloutResult ────────────────────────────────────────────────────────────
class TestRolloutResultRoundTrip:
"""RolloutResult.to_dict() / RolloutResult.from_dict() round-trip."""
def test_preserves_fractional_hard_score(self) -> None:
"""Hard can be a continuous reward and must not be truncated."""
result = RolloutResult.from_dict({"id": "episode-1", "hard": 0.75, "soft": 0.5})
assert result.hard == pytest.approx(0.75)
assert result.to_dict()["hard"] == pytest.approx(0.75)