From f31bf8c06bb50df4d899efba2eb5c1c672761343 Mon Sep 17 00:00:00 2001 From: Yif-Yang Date: Tue, 14 Jul 2026 17:11:40 +0000 Subject: [PATCH] docs: sync documentation with post-v0.2 changes --- .env.example | 34 +- CHANGELOG.md | 60 +- CONTRIBUTING.md | 19 +- README.md | 22 +- ckpt/README.md | 36 +- docs/contributing.md | 29 +- docs/guide/configuration.md | 97 +- docs/guide/dl-analogy.md | 8 +- docs/guide/first-experiment.md | 109 +- docs/guide/installation.md | 121 +- docs/guide/local-env-smoke.md | 24 +- docs/guide/new-backend.md | 275 ++-- docs/guide/new-benchmark.md | 83 +- docs/guide/skill-document.md | 37 +- docs/guide/training-loop.md | 31 +- docs/guideline.html | 1422 ++++++----------- docs/index.md | 67 +- docs/reference/api.md | 37 +- docs/reference/cli.md | 78 +- docs/reference/config.md | 195 ++- docs/sleep/README.md | 56 +- docs/sleep/RESULTS.md | 40 +- docs/sleep/openai-compatible-endpoints.md | 115 +- ...killopt-sleep-claude-code-plugin-design.md | 9 +- index.html | 5 +- mkdocs.yml | 4 + plugins/README.md | 350 ++-- plugins/claude-code/README.md | 64 +- .../commands/skillopt-sleep-handoff.md | 19 +- .../claude-code/commands/skillopt-sleep.md | 64 +- plugins/claude-code/scripts/install-cron.sh | 3 +- .../skills/skillopt-sleep/SKILL.md | 91 +- plugins/codex/README.md | 26 +- plugins/codex/skills/skillopt-sleep/SKILL.md | 101 +- plugins/copilot/README.md | 26 +- .../copilot/copilot-instructions.snippet.md | 31 +- plugins/copilot/skillopt/README.md | 19 +- .../skillopt/copilot-instructions.snippet.md | 67 +- plugins/devin/README.md | 20 +- plugins/devin/devin-rules.snippet.md | 37 +- plugins/openclaw/README.md | 169 +- plugins/openclaw/SKILL.md | 188 +-- skillopt.html | 5 +- skillopt/envs/_template/README.md | 17 +- 44 files changed, 2285 insertions(+), 2025 deletions(-) diff --git a/.env.example b/.env.example index 7060b86..e56d40f 100644 --- a/.env.example +++ b/.env.example @@ -8,6 +8,7 @@ export AZURE_OPENAI_API_VERSION=2024-12-01-preview # Authentication: choose one method # Option 1: API Key export AZURE_OPENAI_API_KEY= +export AZURE_OPENAI_AUTH_MODE=api_key # Option 2: Azure CLI (no API key needed, recommended on Azure VMs) # export AZURE_OPENAI_AUTH_MODE=azure_cli # Option 3: Managed Identity @@ -15,20 +16,45 @@ export AZURE_OPENAI_API_KEY= # export AZURE_OPENAI_MANAGED_IDENTITY_CLIENT_ID=your-client-id # ── OpenAI-compatible endpoints ────────────────────────────────────── -# Set AUTH_MODE to openai_compatible and reuse AZURE_OPENAI_ENDPOINT / _API_KEY. -# The plain OpenAI client is used; no Azure auth, no api-version header. +# Path 1: generic research backend. Select openai_compatible explicitly as +# model.optimizer_backend and/or model.target_backend. +# export OPENAI_COMPATIBLE_BASE_URL=https://api.deepseek.com/v1 +# export OPENAI_COMPATIBLE_API_KEY=sk-... +# export OPENAI_COMPATIBLE_MODEL=deepseek-chat +# Per-role overrides use OPTIMIZER_OPENAI_COMPATIBLE_* and +# TARGET_OPENAI_COMPATIBLE_* (BASE_URL, API_KEY, MODEL, TEMPERATURE, +# MAX_TOKENS, TIMEOUT_SECONDS). +# For scripts/train.py and scripts/eval_only.py, also set model.optimizer and +# model.target in YAML (or via --cfg-options). Those role model values are +# applied after backend initialization and override *_MODEL environment values. + +# Path 2: research openai_chat compatibility mode. This reuses the Azure-family +# variables but creates a plain OpenAI client (no Azure auth or api-version). # export AZURE_OPENAI_ENDPOINT=https://api.openai.com/v1 # export AZURE_OPENAI_API_KEY=sk-... # export AZURE_OPENAI_AUTH_MODE=openai_compatible -# ── Anthropic / Claude (for claude_chat backend) ───────────────────── +# Path 3: SkillOpt-Sleep. `skillopt-sleep run --backend azure_openai` uses the +# same three AZURE_* variables from path 2. Optional Sleep-only controls: +# export SKILLOPT_SLEEP_COMPAT_MAX_TOKENS=8192 +# export SKILLOPT_SLEEP_CHAT_EXTRA_BODY='{"provider_option": true}' + +# ── Claude Code CLI (for claude_chat backend) ───────────────────────── +# Install and authenticate the `claude` CLI before use. For a non-default path: +# export CLAUDE_CLI_BIN=/path/to/claude +# ANTHROPIC_API_KEY is one authentication option understood by the CLI; SkillOpt +# does not create a direct Anthropic API client for this backend. # export ANTHROPIC_API_KEY=sk-ant-... # ── Qwen Local Model (for qwen_chat backend) ──────────────────────── # export QWEN_CHAT_BASE_URL=http://localhost:8000/v1 # export QWEN_CHAT_MODEL=Qwen/Qwen3.5-4B +# The train/eval entry points likewise override this model with +# model.optimizer/model.target for the selected Qwen roles. # ── MiniMax (for minimax_chat backend) ────────────────────────────── # export MINIMAX_BASE_URL=https://api.minimax.io/v1 # export MINIMAX_API_KEY=... -# export MINIMAX_MODEL=MiniMax-M2.7 +# When MiniMax is the target, set model.minimax_model in YAML. The current +# adapter shares one deployment across MiniMax roles; mixed-backend runs cannot +# independently select a MiniMax optimizer model and a different target model. diff --git a/CHANGELOG.md b/CHANGELOG.md index 1e42464..6a9dadc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,7 +15,65 @@ All notable changes to SkillOpt are documented here. This project adheres to per night). Mined tasks are pinned per night so answering sessions cannot shift the task set. Ships a `/skillopt-sleep-handoff` Claude Code command that automates the loop with fresh-context subagents to protect the - held-out gate. + held-out gate (thanks @dimitarvdenev, #125). +- **Generic OpenAI-compatible research backend** for optimizer and target + calls, with configurable base URL, API key, model, and timeout (thanks + @nankingjing, #115). +- **OpenAI-compatible SkillOpt-Sleep endpoint support** for providers such as + DeepSeek and self-hosted vLLM servers (thanks @Alphaxalchemy, #129; hardened + in #138). +- End-to-end wiring for the documented reflection `--preferences` option + (thanks @AKhozya, #131). + +### Changed +- Claude Code's Sleep plugin can now use a `pip`/`uv`-installed + `skillopt-sleep` when no repository checkout is present (thanks + @ichoosetoaccept, #107). +- Qwen reasoning-model requests now use `max_completion_tokens` and omit + unsupported temperature parameters (thanks @chirag127, #128). +- Configuration files are read explicitly as UTF-8 (thanks @nankingjing, + #124). + +### Fixed +- Preserve fractional rollout hard scores instead of coercing them to binary + values (thanks @zixuanguo786-ctrl, #104). +- Reject duplicate and overlapping IDs while materializing SearchQA manifests + (thanks @zixuanguo786-ctrl, #105). +- Make JSON-array extraction robust to unmatched braces and keep malformed + scans linear-time (thanks @zixuanguo786-ctrl, #103; follow-up #136). +- Package Markdown prompt assets in wheels and tolerate Windows temporary-file + cleanup failures (thanks @nankingjing, #135; follow-up #137). +- Exclude sub-agent transcripts and plugin-generated sessions from Sleep task + mining (thanks @codeL1985, #99). +- Normalize validation-gate density against the proposed edits and handle + zero-edit candidates safely (thanks @SparshGarg999, #102). +- Route optimizer-role MiniMax calls through the MiniMax backend (thanks + @jcforever1, #116). +- Surface Claude CLI spawn failures instead of silently turning them into zero + scores (thanks @Phoenix0531-sudo, #126). +- Improve Claude CLI behavior on Windows, including `.cmd` resolution and + long-prompt handling (thanks @codeL1985, #98). +- Preserve the scheduler's established annealing contract while expanding its + endpoint and sequence coverage (thanks @nankingjing, #123; follow-up #133). + +### Security +- Prevent managed-identity credentials from being sent to non-Azure or + non-HTTPS endpoints, and isolate compatible-provider request extensions + from native Azure mode in SkillOpt-Sleep (#138, following + @Alphaxalchemy's #129). + +### Tests +- Strengthen SkillOpt-Sleep verifier-discipline assertions, including recorded + scores and gate actions (thanks @Tanmay9223, #96). +- Add focused coverage for the validation-gate decision core and edit-budget + schedulers (thanks @nankingjing, #122, #123). + +### Acknowledgements 🙏 +Thank you to the contributors behind this unreleased work: +@AKhozya, @Alphaxalchemy, @Phoenix0531-sudo, @SparshGarg999, +@Tanmay9223, @chirag127, @codeL1985, @dimitarvdenev, +@ichoosetoaccept, @jcforever1, @nankingjing, and +@zixuanguo786-ctrl. ## [0.2.0] — 2026-07-02 diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 5a3f3c9..664cf8b 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -7,7 +7,7 @@ Thank you for your interest in contributing! SkillOpt welcomes contributions of ```bash git clone https://github.com/microsoft/SkillOpt.git cd SkillOpt -pip install -e ".[dev]" +python -m pip install -e ".[dev,docs]" ``` ## How to Contribute @@ -16,23 +16,28 @@ pip install -e ".[dev]" Open a GitHub issue with reproduction steps, expected/actual behavior, and your config file (remove API keys). ### 🔧 Add a Benchmark -See the [guide](docs/guide/new-benchmark.md) and use the scaffold at `skillopt/envs/_template/`. +See the [guide](docs/guide/new-benchmark.md) and use the scaffold at +`skillopt/envs/_template/`. Register the adapter lazily in both +`scripts/train.py` and `scripts/eval_only.py`, and add focused tests. ### 🤖 Add a Model Backend -See the [guide](docs/guide/new-backend.md). +First check whether the built-in `openai_compatible` backend covers the +provider. Otherwise follow the function-based backend contract in the +[backend guide](docs/guide/new-backend.md), including routing, configuration, +token accounting, and no-network tests. ### 📝 Improve Documentation ```bash -pip install -e ".[docs]" -mkdocs serve # Preview at http://localhost:8000 +python -m mkdocs serve # Preview at http://localhost:8000 ``` ## Pull Request Process 1. Fork the repo and create a feature branch -2. Make changes and test with an existing benchmark +2. Make changes and run focused tests plus `python -m pytest -q` 3. Submit a PR with a clear description -4. Ensure CI passes +4. For documentation changes, run `python -m mkdocs build --strict` +5. Ensure CI passes ## Code Style - Follow existing patterns in the codebase diff --git a/README.md b/README.md index 6f2c6ff..82d4781 100644 --- a/README.md +++ b/README.md @@ -9,12 +9,12 @@ microsoft%2FSkillOpt | Trendshift

-> 📖 **For installation, data preparation, training/eval commands, the full configuration reference, and framework internals, see the [Documentation & Reproduction Guide](https://microsoft.github.io/SkillOpt/docs/guideline.html)** (rendered on GitHub Pages). +> 📖 **For installation, data preparation, training/eval commands, configuration, and framework internals, start with the versioned [SkillOpt documentation](https://github.com/microsoft/SkillOpt/blob/main/docs/index.md). A concise rendered overview is available in the [Documentation & Reproduction Guide](https://microsoft.github.io/SkillOpt/docs/guideline.html). We also maintain a [Changelog](CHANGELOG.md) for released and unreleased changes.** --- ## News 🔥🔥🔥 -- **[2026-07-02]** 🚀 **SkillOpt [v0.2.0](https://github.com/microsoft/SkillOpt/releases/tag/v0.2.0) is out on [PyPI](https://pypi.org/project/skillopt/)!** Headline feature: **SkillOpt-Sleep**, a nightly offline self-evolution engine (harvest → mine → replay → consolidate, all behind a held-out validation gate) with multi-objective reward, experience replay + dream rollouts, and long-term memory — now shipped as the `skillopt-sleep` CLI. This release also adds cross-tool backends and plugin shells for **Claude, Codex, Copilot, Devin, and OpenClaw**, SearchQA split materialization, Windows robustness, and hardened JSON parsing. See the [release notes](https://github.com/microsoft/SkillOpt/releases/tag/v0.2.0) for the full changelog and contributor acknowledgements. +- **[2026-07-02]** 🚀 **SkillOpt [v0.2.0](https://github.com/microsoft/SkillOpt/releases/tag/v0.2.0) is out on [PyPI](https://pypi.org/project/skillopt/)!** Headline feature: **SkillOpt-Sleep**, a nightly offline self-evolution engine (harvest → mine → replay → consolidate behind a held-out validation gate), now shipped as the `skillopt-sleep` CLI. It also includes experimental multi-objective, replay, and dream-rollout controls; the main CLI keeps conservative defaults and does not expose every experiment-harness control as a flag. The release source adds integration shells for **Claude Code, Codex, Copilot, and Devin**, plus an **OpenClaw reference adaptation**; these plugin/MCP files live in the repository rather than the PyPI wheel. It also adds SearchQA split materialization, Windows robustness, and hardened JSON parsing. See the [release notes](https://github.com/microsoft/SkillOpt/releases/tag/v0.2.0) for full release details and contributor acknowledgements. - **[2026-06-15]** 😴 **SkillOpt-Sleep (preview)** — a nightly offline self-evolution companion for local coding agents (Claude Code / Codex / Copilot): review past sessions, replay recurring tasks, and consolidate validated skills behind a held-out gate. See **[`docs/sleep/README.md`](docs/sleep/README.md)** for what it is, how to use it, and results. - **[2026-06-03]** 🎉 **[gbrain](https://github.com/garrytan/gbrain), [gbrain-evals](https://github.com/garrytan/gbrain-evals/blob/main/docs/benchmarks/2026-06-03-skillopt.md), and [darwin-skill](https://github.com/alchaincyf/darwin-skill) have all integrated SkillOpt.** - **[2026-06-02]** 🎉 **SkillOpt [v0.1.0](https://github.com/microsoft/SkillOpt/releases/tag/v0.1.0) is now available on [PyPI](https://pypi.org/project/skillopt/)!** Install with `pip install skillopt`. This initial release includes the full training loop (rollout → reflect → aggregate → select → update → evaluate), multi-backend support (OpenAI / Azure / Claude / Qwen / MiniMax), six built-in benchmarks, and WebUI dashboard. @@ -31,9 +31,9 @@ which reliably improves over its starting point under feedback. **SkillOpt treats the skill document as the trainable state of a frozen agent**, and trains it with the discipline that makes weight-space optimization reproducible. A separate optimizer model turns scored rollouts -into bounded add / delete / replace edits on a single skill document; a -candidate edit is accepted only when it strictly improves a held-out -validation score. A textual learning-rate budget, a rejected-edit buffer, +into bounded add / delete / replace edits on a single skill document; in the +default paper-style path, a candidate edit is accepted only when it strictly +improves a held-out validation score. A textual learning-rate budget, a rejected-edit buffer, and an epoch-wise slow / meta update make skill training stable while adding **zero inference-time model calls** at deployment. @@ -64,7 +64,9 @@ https://github.com/user-attachments/assets/eb12d3bc-371c-467f-904d-91b61f339ed7 ### Adding a new backend A backend = a chat / exec target (e.g. `openai_chat`, `claude_chat`, -`qwen_chat`, `minimax_chat`, `codex_exec`, `claude_code_exec`). See +`qwen_chat`, `minimax_chat`, `openai_compatible`, `codex_exec`, +`claude_code_exec`). If a provider implements the OpenAI Chat Completions +protocol, try the built-in `openai_compatible` backend before adding code. See [`docs/guide/new-backend.md`](docs/guide/new-backend.md) for the full contract; in short you add a `skillopt/model/_backend.py` module, register it in `skillopt/model/common.py` + `backend_config.py`, and wire @@ -73,8 +75,9 @@ and `minimax_backend.py` are good templates. ### Adding a new benchmark -A benchmark = a `skillopt/envs//` package with a `dataloader.py`, a -`rollout.py`, and an `initial.md` seed skill. See +A benchmark = a `skillopt/envs//` package with an adapter, a data loader, +a scored rollout helper, a YAML config, and optionally an initial seed skill. +See [`docs/guide/new-benchmark.md`](docs/guide/new-benchmark.md) for the full contract; the simplest reference is `skillopt/envs/searchqa/`. @@ -93,6 +96,9 @@ python -m skillopt_webui.app | `--host` | `0.0.0.0` | Bind address | | `--share` | off | Create a public Gradio share link | +The default host listens on every network interface. Use +`--host 127.0.0.1` for local-only access. + --- ## Citation diff --git a/ckpt/README.md b/ckpt/README.md index b79f766..ecbc209 100644 --- a/ckpt/README.md +++ b/ckpt/README.md @@ -10,9 +10,10 @@ provided skills on a given split without re-running the training loop. > skills as portable artifacts. If you want to *train* your own skill, > use `scripts/train.py` per the top-level README. > -> This is the first artifact batch. We plan to continue uploading the -> remaining optimized skills and benchmark split manifests as they are -> cleaned and verified. +> This is the first optimized-skill artifact batch. We plan to continue +> uploading remaining paper artifacts as they are cleaned and verified. All +> six lightweight ID/path split manifests are already checked in under +> `data/`; most still require materializing their upstream benchmark payload. ## What's here @@ -35,12 +36,17 @@ longitudinal guidance — that's expected, not a formatting issue. invoking the optimizer. Example for SearchQA against the test split: ```bash +# The checked-in SearchQA split is ID-only; materialize full examples first. +python -m pip install -e ".[searchqa]" +python scripts/materialize_searchqa.py + python scripts/eval_only.py \ --config configs/searchqa/default.yaml \ --skill ckpt/searchqa/gpt5.5_skill.md \ --split valid_unseen \ - --split_dir data/searchqa_id_split \ + --split_dir data/searchqa_split \ --azure_openai_endpoint https://your-resource.openai.azure.com/ \ + --azure_openai_auth_mode api_key \ --target_model gpt-5.5 ``` @@ -52,13 +58,16 @@ is the selection / validation split, `train` is the training split, and ## On comparing to the paper numbers To compare against the paper-reported cells, use the same dataset split and -scorer. SearchQA's split is checked in at `data/searchqa_id_split/` (400 -train / 200 selection / 1400 test). For the other benchmarks, point -`--split_dir` at your own materialized split; the loader is deterministic -from `split_seed` (default `42`) + `split_ratio` (default `2:1:7`) when -`split_mode: ratio` is used, so a given `data_path` + seed reproduces -across machines. Explicit per-benchmark split manifests are being prepared -for upload — see issues #14 and #21. +scorer. SearchQA's ID manifest is checked in at `data/searchqa_id_split/` (400 +train / 200 selection / 1400 test); the materializer writes the runnable +payload to `data/searchqa_split/`. All six lightweight split manifests are +checked in under `data/`. ALFWorld's manifest records game-file paths; the +other ID manifests still require you to materialize the corresponding +upstream benchmark payload into the documented `split_dir`. See +[`data/README.md`](../data/README.md) for the exact status of each benchmark. +When using `split_mode: ratio` instead, the loader is deterministic from +`split_seed` (default `42`) + `split_ratio` (default `2:1:7`), so a given +`data_path` + seed reproduces across machines. ## Why force-accept vs. gated slow-update matters @@ -74,6 +83,5 @@ Current `main` defaults to `false` (force-accept mode), a newer post-submission behavior where the slow-update guidance is written into `current_skill` and `best_skill` unconditionally at the epoch boundary. If you re-train with the current default, you may produce a *different* -`best_skill.md` than the one checked in here. Both modes are supported; -see the top-level README's "Configuration -> Slow-update acceptance mode" -section. +`best_skill.md` than the one checked in here. Both modes are supported; see +the [configuration reference](../docs/reference/config.md). diff --git a/docs/contributing.md b/docs/contributing.md index 818a67e..8971693 100644 --- a/docs/contributing.md +++ b/docs/contributing.md @@ -15,6 +15,7 @@ pip install -e ".[dev]" ### 🐛 Bug Reports Open an issue with: + - Steps to reproduce - Expected vs actual behavior - Config file used (sanitize API keys) @@ -25,21 +26,29 @@ Open an issue with: See [Add a New Benchmark](guide/new-benchmark.md) for the implementation guide. **Checklist:** -- [ ] Data loader in `skillopt/envs//loader.py` -- [ ] Environment adapter in `skillopt/envs//env.py` + +- [ ] Data loader in `skillopt/envs//dataloader.py` +- [ ] Scored rollout implementation in `skillopt/envs//rollout.py` +- [ ] Per-item `predictions//conversation.json` artifacts for shared reflection +- [ ] Environment adapter in `skillopt/envs//adapter.py` - [ ] Config file in `configs//default.yaml` -- [ ] Registration in `skillopt/envs/__init__.py` -- [ ] Documentation page in `docs/` +- [ ] Lazy registration in `scripts/train.py` and `scripts/eval_only.py` +- [ ] Focused tests and an optional seed skill referenced by `env.skill_init` +- [ ] Documentation update ### 🤖 New Model Backend See [Add a New Model Backend](guide/new-backend.md) for the implementation guide. **Checklist:** -- [ ] Backend in `skillopt/model/.py` -- [ ] Registration in `skillopt/model/__init__.py` -- [ ] API key entry in `.env.example` -- [ ] Documentation update + +- [ ] Function-based backend module in `skillopt/model/_backend.py` +- [ ] Alias and default model in `skillopt/model/common.py` +- [ ] Optimizer/target whitelist entries in `skillopt/model/backend_config.py` +- [ ] Dispatch, token tracking, and setter forwarding in `skillopt/model/__init__.py` +- [ ] YAML/CLI wiring when the backend exposes structured config fields +- [ ] Focused routing, configuration, tool-call, and token-accounting tests +- [ ] `.env.example` and backend/configuration reference updates ### 📝 Documentation @@ -59,9 +68,9 @@ mkdocs serve # Preview at http://localhost:8000 ## Pull Request Process 1. Fork the repository -2. Create a feature branch: `git checkout -b feature/my-benchmark` +2. Create a feature branch: `git switch -c feature/my-benchmark` 3. Make your changes -4. Test with an existing benchmark config +4. Run focused tests, the full test suite, and `mkdocs build --strict` when docs change 5. Submit a PR with a clear description ## License diff --git a/docs/guide/configuration.md b/docs/guide/configuration.md index 57fffae..8600c44 100644 --- a/docs/guide/configuration.md +++ b/docs/guide/configuration.md @@ -20,15 +20,53 @@ Benchmark configs inherit from `_base_/default.yaml` and override specific value ## Key Parameters -### Model +### Model Backends + +`optimizer_backend` controls reflection and skill editing; +`target_backend` controls task rollout. The legacy `backend` field remains for +backward compatibility, but explicit role fields are the clearest configuration. ```yaml model: - backend: azure_openai # azure_openai | openai_chat | claude_code_exec | qwen - optimizer: gpt-5.5 # Optimizer model (for reflection) - target: gpt-5.5 # Target model (for rollout) + backend: azure_openai # High-level compatibility label + optimizer_backend: openai_chat + target_backend: openai_chat + optimizer: gpt-5.5 # Optimizer deployment/model + target: gpt-5.5 # Target deployment/model + azure_openai_auth_mode: api_key ``` +| Backend | Optimizer | Target | Configuration | +|---|:---:|:---:|---| +| `openai_chat` | ✓ | ✓ | Azure OpenAI, or its explicit compatibility auth mode | +| `openai_compatible` | ✓ | ✓ | Generic OpenAI Chat Completions endpoint | +| `claude_chat` | ✓ | ✓ | Claude Code CLI (`claude -p`) | +| `qwen_chat` | ✓ | ✓ | Qwen served through an OpenAI-compatible local endpoint | +| `minimax_chat` | ✓ | ✓ | MiniMax API | +| `codex_exec` | — | ✓ | Codex CLI execution harness | +| `claude_code_exec` | — | ✓ | Claude Code CLI execution harness | + +The current MiniMax adapter has one shared deployment. Set +`model.minimax_model` when MiniMax is the target; a mixed-backend run cannot +independently select a MiniMax optimizer model and a different target model. + +For a generic compatible provider, select the role backends explicitly rather +than relying on a high-level shorthand: + +```yaml +model: + optimizer_backend: openai_compatible + target_backend: openai_compatible + optimizer: deepseek-chat + target: deepseek-chat +``` + +The train/eval entry points apply `model.optimizer` and `model.target` after +backend initialization. For the selected roles, these YAML values override +`OPENAI_COMPATIBLE_MODEL`, `QWEN_CHAT_MODEL`, and their per-role environment +forms. The environment model variables mainly seed direct library use; always +set the role models in a training or evaluation config. + ### Training ```yaml @@ -96,9 +134,16 @@ Notes: ```yaml evaluation: use_gate: true # Validation gating (accept/reject updates) + gate_metric: hard # hard | soft | mixed + gate_mixed_weight: 0.5 # Soft-score weight when metric=mixed + use_semantic_density: false # Optional instruction-density bonus eval_test: true # Run test evaluation after training ``` +The default and paper-style setting is `use_gate: true`. Setting it to `false` +still records selection scores but force-accepts every candidate, so it changes +the optimization semantics and should be reported explicitly. + ### Environment (Data) ```yaml @@ -117,9 +162,10 @@ Override any config value from the command line: ```bash python scripts/train.py \ --config configs/searchqa/default.yaml \ - optimizer.learning_rate=16 \ - optimizer.lr_scheduler=linear \ - gradient.analyst_workers=8 + --cfg-options \ + optimizer.learning_rate=16 \ + optimizer.lr_scheduler=linear \ + gradient.analyst_workers=8 ``` ## Environment Variables @@ -128,11 +174,38 @@ Model credentials are loaded from environment variables: | Variable | Backend | Description | |---|---|---| -| `AZURE_OPENAI_ENDPOINT` | azure_openai | Azure resource endpoint | -| `AZURE_OPENAI_API_KEY` | azure_openai | Azure API key | -| `OPENAI_API_KEY` | openai | OpenAI API key | -| `ANTHROPIC_API_KEY` | claude | Anthropic API key | -| `QWEN_API_BASE` | qwen | Local Qwen vLLM endpoint | +| `AZURE_OPENAI_ENDPOINT` | `openai_chat` | Azure resource URL, or compatibility-mode base URL | +| `AZURE_OPENAI_API_VERSION` | `openai_chat` | Azure API version | +| `AZURE_OPENAI_AUTH_MODE` | `openai_chat` | `api_key`, `azure_cli`, `managed_identity`, or `openai_compatible` | +| `AZURE_OPENAI_API_KEY` | `openai_chat` | Required when auth mode is `api_key` or `openai_compatible` | +| `OPENAI_COMPATIBLE_BASE_URL` | `openai_compatible` | Generic Chat Completions base URL | +| `OPENAI_COMPATIBLE_API_KEY` | `openai_compatible` | Provider API key; optional for local servers | +| `OPENAI_COMPATIBLE_MODEL` | `openai_compatible` | Shared provider model ID for direct library use; train/eval YAML role models take precedence | +| `CLAUDE_CLI_BIN` | `claude_chat` | Optional path to the `claude` executable; defaults to `claude` | +| `ANTHROPIC_API_KEY` | `claude_chat` | Optional authentication method understood by the Claude CLI, not a direct SkillOpt API client | +| `QWEN_CHAT_BASE_URL` | `qwen_chat` | Local Qwen/vLLM endpoint | +| `QWEN_CHAT_MODEL` | `qwen_chat` | Served model name for direct library use; train/eval YAML role models take precedence | +| `MINIMAX_BASE_URL` | `minimax_chat` | MiniMax-compatible base URL | +| `MINIMAX_API_KEY` | `minimax_chat` | MiniMax API key | + +`OPTIMIZER_` and `TARGET_` prefixes provide per-role overrides for the +Azure, OpenAI-compatible, and Qwen variable families. See the +[Configuration Reference](../reference/config.md) for exact names. + +`claude_chat` launches the installed Claude Code CLI with `claude -p`; install +and authenticate that CLI before use. Setting `ANTHROPIC_API_KEY` is one way +the CLI may authenticate, but SkillOpt does not call the Anthropic API +directly through this backend. + +### Three OpenAI-compatible paths + +- Research, generic provider: select `openai_compatible` and use + `OPENAI_COMPATIBLE_*`. +- Research, Azure-family compatibility mode: keep `openai_chat`, set + `AZURE_OPENAI_AUTH_MODE=openai_compatible`, and use `AZURE_OPENAI_*`. +- SkillOpt-Sleep: run with `--backend azure_openai` and use the same + compatibility-mode `AZURE_OPENAI_*` variables. Sleep does not read the + research backend's role-specific variables. ## Full Reference diff --git a/docs/guide/dl-analogy.md b/docs/guide/dl-analogy.md index 758566f..d77919a 100644 --- a/docs/guide/dl-analogy.md +++ b/docs/guide/dl-analogy.md @@ -14,10 +14,9 @@ SkillOpt is designed around a core insight: **optimizing natural-language prompt | **Gradient aggregation** | Patch aggregation | Merge similar edits | | **Gradient clipping** | Edit selection | Cap max edits per step | | **Learning rate** | `learning_rate` | Max number of edits applied per step | -| **LR scheduler** | `lr_scheduler` | Decay schedule: cosine, linear, constant | +| **LR scheduler** | `lr_scheduler` | Edit-budget schedule: cosine, linear, constant, or autonomous | | **SGD step** | Skill update | Apply selected patches to document | | **Validation set** | Selection split | Gate checks improvement before accepting | -| **Early stopping** | Gate patience | Reject updates that don't improve | | **Training step** | Step | One rollout → reflect → update cycle | | **Epoch** | Epoch | Full pass with slow update + meta memory | | **Momentum** | Slow update | Longitudinal comparison at epoch boundary | @@ -34,7 +33,10 @@ SkillOpt is designed around a core insight: **optimizing natural-language prompt 1. **Familiar mental model**: ML practitioners immediately understand how to tune SkillOpt 2. **Principled hyperparameter search**: Grid search over `learning_rate` × `lr_scheduler` works just like in DL -3. **Proven mechanisms**: Gating ≈ validation-based selection, patience ≈ early stopping, slow update ≈ momentum — all with strong theoretical motivation +3. **Reusable mechanisms**: Gating provides validation-based model selection, while slow update plays a momentum-like role across epochs + +The gate is a per-candidate accept/reject decision. SkillOpt does not implement +a gate-patience counter or stop training after a run of rejected candidates. ## Hyperparameter Transfer Rules diff --git a/docs/guide/first-experiment.md b/docs/guide/first-experiment.md index 2a65589..1cda96c 100644 --- a/docs/guide/first-experiment.md +++ b/docs/guide/first-experiment.md @@ -4,17 +4,43 @@ This guide walks through running a complete SkillOpt training on SearchQA. ## 1. Choose a Benchmark -SkillOpt includes ready-to-use configs for several benchmarks: +SkillOpt includes ready-to-use configs for several benchmarks. End-to-end +runtime depends on the chosen models, provider latency, worker limits, and +dataset size, so the project does not promise fixed wall-clock estimates. -| Benchmark | Difficulty | Typical Runtime | +| Benchmark | Modality | Additional setup | |---|---|---| -| SearchQA | ⭐ Easy | ~30 min | -| DocVQA | ⭐⭐ Medium | ~2 hours | -| ALFWorld | ⭐⭐⭐ Hard | ~3 hours | +| SearchQA | Text QA | Materialize the released ID manifest | +| DocVQA | Document/image QA | Obtain and materialize images and examples | +| ALFWorld | Embodied agent | Install ALFWorld and download its assets | -We'll use **SearchQA** as it's the fastest to complete. +We'll use **SearchQA** because it is the simplest text-only walkthrough. -## 2. Configure +## 2. Install and Materialize SearchQA + +The repository contains a stable SearchQA ID manifest, not the full runnable +examples. From a source checkout, install the data extra and materialize the +split once: + +```bash +python -m pip install -e ".[searchqa]" +python scripts/materialize_searchqa.py +``` + +By default, the materializer reads `data/searchqa_id_split/` and writes the +train/validation/test payloads expected by the config to +`data/searchqa_split/`; both paths have command-line overrides. + +## 3. Configure + +Configure and export one model backend as described in +[Installation](installation.md#environment-variables). For example: + +```bash +cp .env.example .env +# Edit .env, choose one authentication mode, then export it: +set -a; source .env; set +a +``` Review the config file: @@ -42,56 +68,55 @@ evaluation: use_gate: true # (validation gating) ``` -## 3. Train +## 4. Train ```bash -python scripts/train.py --config configs/searchqa/default.yaml +python scripts/train.py \ + --config configs/searchqa/default.yaml \ + --out_root outputs/searchqa_first_run ``` -You'll see output like: +The command prints the resolved backend/data configuration, per-step rollout +and gate progress, and the generated output directory. + +## 5. Monitor + +The explicit `--out_root` above creates this run directory: ``` -[Step 1/8] Rollout: 20 items, 4 workers... -[Step 1/8] Score: 0.65 → Reflect... -[Step 1/8] 6 edit patches generated -[Step 1/8] Selected 4 edits (lr=8, cosine → 7.7) -[Step 1/8] Gate: val score 0.68 > 0.65 ✓ ACCEPT -[Step 2/8] ... -``` - -## 4. Monitor - -Training outputs are saved to `outputs///`: - -``` -outputs/searchqa/2024-01-15_10-30-00/ -├── steps/ -│ ├── step_0001/ -│ │ ├── candidate_skill.md -│ │ ├── step_record.json -│ │ └── trajectory_digest.json -│ └── step_0002/ -├── slow_update/ -│ └── epoch_02/ -├── meta_skill/ -│ └── epoch_02/ -├── skills/ -│ └── step_0001.md -├── best_skill.md +outputs/searchqa_first_run/ +├── config.json +├── runtime_state.json ├── history.json -└── config.yaml +├── best_skill.md +├── skills/ +│ └── skill_vXXXX.md +├── steps/ +│ └── step_XXXX/ +│ ├── candidate_skill.md +│ ├── step_record.json +│ └── trajectory_digest.json +├── slow_update/ +│ └── epoch_XX/ +└── meta_skill/ + └── epoch_XX/ ``` -## 5. Evaluate +## 6. Evaluate Evaluate the best skill on the test split: ```bash python scripts/eval_only.py \ --config configs/searchqa/default.yaml \ - --skill outputs/searchqa//skills/best_skill.md + --skill outputs/searchqa_first_run/best_skill.md \ + --split valid_unseen ``` +The `--skill` path above is the training artifact. Evaluation writes +`eval_summary.json` to its own timestamped `outputs/eval_.../` directory unless +you pass an explicit `--out_root`; it does not overwrite the training run. + ## WebUI Prefer a graphical interface? Launch the WebUI: @@ -101,7 +126,9 @@ pip install -e ".[webui]" python -m skillopt_webui.app ``` -Then open `http://localhost:7860` in your browser to configure parameters and launch training. +Then open `http://localhost:7860` in your browser to configure parameters and +launch training. The default host is `0.0.0.0`; pass `--host 127.0.0.1` for a +local-only dashboard. ## Next Steps diff --git a/docs/guide/installation.md b/docs/guide/installation.md index 0fd390e..1b24a2d 100644 --- a/docs/guide/installation.md +++ b/docs/guide/installation.md @@ -3,16 +3,44 @@ ## Requirements - Python ≥ 3.10 -- At least one model API key (Azure OpenAI, OpenAI, Anthropic, or local Qwen) +- For research training/evaluation, access to at least one configured model + backend (hosted API, local server, or an installed execution CLI) +- The SkillOpt-Sleep `mock` backend needs no credentials -## Quick Install +## Choose an Install + +### PyPI + +Use PyPI for the Python packages and installed commands: + +```bash +python -m pip install skillopt +skillopt-sleep --help +``` + +This installs `skillopt-train`, `skillopt-eval`, and `skillopt-sleep`. The wheel +does not include the repository's benchmark configs, data materializers, +agent-integration shells/MCP servers, or development tests; use a source +checkout for those files. + +!!! important "PyPI versus `main`" + These docs track the latest `main`. The current PyPI release is `0.2.0`. + The generic research `openai_compatible` backend, SkillOpt-Sleep handoff, + Sleep support for non-Azure OpenAI-compatible endpoints, and the Sleep + `--preferences` flag landed after that release and require a source install + from `main` until the next release. + +### Source checkout ```bash git clone https://github.com/microsoft/SkillOpt.git cd SkillOpt -pip install -e . +python -m pip install -e . ``` +Use the source checkout for paper reproduction, built-in benchmark configs, +and contributions. + ## Optional Dependencies Install extras for specific benchmarks or backends: @@ -20,68 +48,115 @@ Install extras for specific benchmarks or backends: === "ALFWorld" ```bash - pip install -e ".[alfworld]" + python -m pip install -e ".[alfworld]" ``` -=== "Claude Backend" +=== "Claude agent SDK (optional)" ```bash - pip install -e ".[claude]" + python -m pip install -e ".[claude]" ``` + This extra does not install the `claude` executable. The research + `claude_chat` backend launches `claude -p`, so install and authenticate the + Claude Code CLI separately. The SDK extra is only needed when selecting an + SDK-backed Claude Code exec path. + === "Qwen (Local)" ```bash - pip install -e ".[qwen]" + python -m pip install -e ".[qwen]" + ``` + +=== "SearchQA data" + + ```bash + python -m pip install -e ".[searchqa]" ``` === "WebUI" ```bash - pip install -e ".[webui]" + python -m pip install -e ".[webui]" ``` === "Development" ```bash - pip install -e ".[dev]" + python -m pip install -e ".[dev]" ``` === "All" ```bash - pip install -e ".[alfworld,claude,qwen,webui,dev]" + python -m pip install -e ".[alfworld,claude,qwen,searchqa,webui,docs,dev]" ``` ## Environment Variables -Copy the example `.env` file and fill in your credentials: +From a source checkout, copy the template and fill in only the backend you +will use: ```bash cp .env.example .env ``` -Edit `.env` with your API keys: +SkillOpt does not automatically load `.env`; export it into the current shell +before running commands: -```ini -# Azure OpenAI (default backend) -AZURE_OPENAI_ENDPOINT=https://your-resource.openai.azure.com/ -AZURE_OPENAI_API_KEY=your-key - -# Or use OpenAI directly -OPENAI_API_KEY=sk-... - -# Or Anthropic Claude -ANTHROPIC_API_KEY=sk-ant-... +```bash +set -a +source .env +set +a ``` +For Azure OpenAI with API-key authentication, the minimum settings are: + +```ini +AZURE_OPENAI_ENDPOINT=https://your-resource.openai.azure.com/ +AZURE_OPENAI_API_VERSION=2024-12-01-preview +AZURE_OPENAI_API_KEY=your-key +AZURE_OPENAI_AUTH_MODE=api_key +``` + +Use `AZURE_OPENAI_AUTH_MODE=azure_cli` for Azure CLI credentials, or +`managed_identity` with an optional +`AZURE_OPENAI_MANAGED_IDENTITY_CLIENT_ID`. + +The research `claude_chat` backend is a Claude Code CLI adapter, not a direct +Anthropic API client. Install and authenticate `claude`, and set +`CLAUDE_CLI_BIN` only if the executable is not available as `claude` on +`PATH`. `ANTHROPIC_API_KEY` is one authentication option the CLI may consume. + +OpenAI-compatible servers have three distinct entry points: + +1. The research engine's generic `openai_compatible` backend uses + `OPENAI_COMPATIBLE_BASE_URL`, `OPENAI_COMPATIBLE_API_KEY`, and + `OPENAI_COMPATIBLE_MODEL`. +2. The research `openai_chat` backend can use + `AZURE_OPENAI_AUTH_MODE=openai_compatible` with + `AZURE_OPENAI_ENDPOINT` and `AZURE_OPENAI_API_KEY`. +3. SkillOpt-Sleep uses the same Azure-family variables as item 2 with + `skillopt-sleep run --backend azure_openai`. + +For research train/eval commands, `model.optimizer` and `model.target` in the +YAML config are applied after backend initialization. They override model-name +environment variables such as `OPENAI_COMPATIBLE_MODEL` and +`QWEN_CHAT_MODEL`; set both role models explicitly when selecting those +backends. + !!! tip - You only need credentials for the backend you plan to use. Azure OpenAI is the default. + You only need to configure the backend you plan to use. See + [Configuration](configuration.md#model-backends) for exact backend names + and role-specific overrides. ## Verify Installation ```bash python -c "import skillopt; print('SkillOpt ready!')" +skillopt-train --help +skillopt-eval --help +skillopt-sleep --help ``` ## Next Steps diff --git a/docs/guide/local-env-smoke.md b/docs/guide/local-env-smoke.md index f1af13e..59ecd7a 100644 --- a/docs/guide/local-env-smoke.md +++ b/docs/guide/local-env-smoke.md @@ -35,9 +35,14 @@ Use the split names your adapter maps to SkillOpt phases: - `val` or `valid_seen` for selection/gating - `test` or `valid_unseen` for final evaluation -## 2. Support an offline mock mode +## 2. Support a genuinely offline mock mode -Add a configuration flag such as `mock: true` to your adapter. In mock mode, `rollout()` should return deterministic responses without calling external model APIs. +Add a configuration flag such as `mock: true` to your adapter. In mock mode, +`rollout()` should return deterministic responses without calling external +model APIs. The inherited `EnvAdapter.reflect()` does call the configured +optimizer backend, so a no-credential smoke test must also override +`reflect()` in mock mode to return a small, schema-valid deterministic patch +(and delegate to `super().reflect(...)` otherwise). This lets you verify the SkillOpt loop with a fast command such as: @@ -46,13 +51,14 @@ python scripts/train.py \ --config configs/myenv/tiny_mock.yaml ``` -Mock mode should still write the same artifacts as a real run, for example: +Mock mode should still exercise the trainer's normal artifact paths, including: -- `responses.json` -- `rollout_results.json` -- `ranked_edits.json` -- `candidate_skill.md` -- `summary.json` +- `config.json`, `runtime_state.json`, and `history.json` +- `skills/skill_vXXXX.md` +- `steps/step_XXXX/ranked_edits.json` +- `steps/step_XXXX/candidate_skill.md` +- `steps/step_XXXX/step_record.json` +- the final `summary.json` ## 3. Keep the smoke config tiny @@ -127,7 +133,7 @@ For the real tiny run, verify that: - the run completes - `summary.json` is written -- `ranked_edits.json` contains the expected ranking metadata +- the step directory's `ranked_edits.json` contains the expected ranking metadata - any optimizer bridge log marks the response schema as valid - no generated files are written outside `out_root` diff --git a/docs/guide/new-backend.md b/docs/guide/new-backend.md index c58ae24..6785527 100644 --- a/docs/guide/new-backend.md +++ b/docs/guide/new-backend.md @@ -1,11 +1,17 @@ # Add a New Model Backend -SkillOpt supports multiple LLM backends. This guide shows how to add your own. +SkillOpt's model layer is function-based: each chat backend is a Python module +that exposes the call, token-tracking, and deployment-setting functions used by +`skillopt.model`. There is no backend base class or registry object to subclass. ## Built-in: the generic OpenAI-compatible backend +!!! note "Version requirement" + This backend landed after v0.2.0. Install from the latest `main` until it is + included in the next release. + Before writing a new backend, check whether your provider already speaks the -OpenAI Chat Completions protocol. Most do — in which case you can use the +OpenAI Chat Completions protocol. Most do, in which case you can use the built-in **`openai_compatible`** backend (`skillopt/model/openai_compatible_backend.py`) with no code changes. @@ -21,15 +27,16 @@ A single `base_url` + `api_key` pair lets you point SkillOpt at, for example: | LiteLLM proxy | `http://localhost:4000` | any proxied model | | OpenRouter / Fireworks / xAI / … | provider base URL | provider model id | -Select it as the optimizer and/or target backend: +### Python API + +Select and configure the backend directly when embedding SkillOpt as a Python +library: ```python import skillopt.model as model -# Shorthand: use it for both optimizer and target. +# Use the generic backend for both optimizer and target calls. model.set_backend("openai_compatible") - -# Point it at a provider (shared, or per-role with optimizer_*/target_*). model.configure_openai_compatible( base_url="https://api.deepseek.com/v1", api_key="sk-...", @@ -37,147 +44,157 @@ model.configure_openai_compatible( ) ``` -Or configure it entirely through environment variables (role-specific -`OPTIMIZER_*` / `TARGET_*` variants override the shared ones): +`configure_openai_compatible()` also accepts `optimizer_*` and `target_*` +arguments when the two roles use different endpoints or models. + +### Environment variables + +The shared variables below configure both roles. Role-specific +`OPTIMIZER_OPENAI_COMPATIBLE_*` and `TARGET_OPENAI_COMPATIBLE_*` variables take +precedence: ```bash -export TARGET_BACKEND=openai_compatible export OPENAI_COMPATIBLE_BASE_URL="https://api.groq.com/openai/v1" export OPENAI_COMPATIBLE_API_KEY="gsk_..." export OPENAI_COMPATIBLE_MODEL="llama-3.3-70b-versatile" # Optional: OPENAI_COMPATIBLE_TEMPERATURE, _MAX_TOKENS, _TIMEOUT_SECONDS ``` -The backend uses the official `openai` SDK, records token usage through the -shared tracker, supports tool/function calling via -`chat_target_messages(..., tools=...)`, and exposes -`count_tokens()` (tiktoken with a character-based fallback for non-OpenAI -models). Only write a brand-new backend if your provider is *not* -OpenAI-compatible. - -## Backend Architecture - -``` -skillopt/model/ -├── base.py # Abstract base class -├── azure_openai.py # Azure OpenAI backend -├── openai_model.py # Direct OpenAI backend -├── claude.py # Anthropic Claude backend -├── qwen.py # Local Qwen (vLLM) backend -└── your_backend.py # Your new backend -``` - -## Step 1: Create the Backend - -Create `skillopt/model/your_backend.py`: - -```python -from skillopt.model.base import ModelBackend, ModelResponse - -class YourBackend(ModelBackend): - """Your custom model backend.""" - - def __init__(self, cfg: dict): - super().__init__(cfg) - self.model_name = cfg.get('model_name', 'your-default-model') - self.api_key = os.environ.get('YOUR_API_KEY', '') - self.client = self._init_client() - - def _init_client(self): - """Initialize API client.""" - # TODO: Set up your API client - pass - - async def generate( - self, - messages: list[dict], - temperature: float = 0.7, - max_tokens: int = 4096, - **kwargs - ) -> ModelResponse: - """ - Generate a completion. - - Args: - messages: Chat messages [{"role": "...", "content": "..."}] - temperature: Sampling temperature - max_tokens: Maximum tokens in response - - Returns: - ModelResponse with content, usage, and metadata - """ - response = await self.client.chat( - model=self.model_name, - messages=messages, - temperature=temperature, - max_tokens=max_tokens, - ) - - return ModelResponse( - content=response.text, - usage={ - 'prompt_tokens': response.usage.input, - 'completion_tokens': response.usage.output, - }, - model=self.model_name, - ) - - async def generate_with_tools( - self, - messages: list[dict], - tools: list[dict], - **kwargs - ) -> ModelResponse: - """Generate with tool/function calling support.""" - # Optional: implement if your model supports tool use - raise NotImplementedError("Tool use not supported") -``` - -## Step 2: Register the Backend - -Add to `skillopt/model/__init__.py`: - -```python -from .your_backend import YourBackend - -BACKEND_REGISTRY = { - # ... existing backends ... - 'your_backend': YourBackend, -} -``` - -## Step 3: Configure - -Use your backend in any config: +For direct library use, `OPTIMIZER_BACKEND=openai_compatible` and/or +`TARGET_BACKEND=openai_compatible` select the role. The training and evaluation +scripts resolve backend selection from their config, so set the split fields +explicitly there: ```yaml model: - backend: your_backend - model_name: your-model-id - temperature: 0.7 - max_tokens: 4096 + optimizer_backend: openai_compatible + target_backend: openai_compatible + optimizer: llama-3.3-70b-versatile + target: llama-3.3-70b-versatile ``` -Set credentials via environment variable: +Equivalently, override those fields on the command line: ```bash -export YOUR_API_KEY="your-key" +python scripts/train.py --config configs/searchqa/default.yaml \ + --cfg-options \ + model.optimizer_backend=openai_compatible \ + model.target_backend=openai_compatible \ + model.optimizer=llama-3.3-70b-versatile \ + model.target=llama-3.3-70b-versatile ``` -## Required Interface +Do not rely on the legacy high-level `model.backend` label to replace the two +role-specific fields in a structured config. -Your backend must implement these methods: +The generic backend uses the official `openai` SDK and the Chat Completions +API. It records token usage through the shared tracker, supports provider tool +calling through `chat_*_messages(..., tools=...)`, and exposes `count_tokens()` +(tiktoken when available, with a character-based fallback). Provider-specific +Responses API features are outside this backend's contract. -| Method | Required | Description | -|---|---|---| -| `generate()` | ✅ | Basic text generation | -| `generate_with_tools()` | Optional | Tool/function calling | -| `count_tokens()` | Optional | Token counting for context management | +Only write a new backend when the provider is not compatible with this surface +or requires behavior that cannot be expressed by its configuration. -## Tips +## Backend architecture -!!! tip - - Test your backend with `python -c "from skillopt.model.your_backend import YourBackend"` first - - Use `async` methods for all API calls — SkillOpt uses asyncio throughout - - Implement retry logic with exponential backoff for production use - - Add your API key to `.env.example` when submitting a PR +The active split optimizer/target dispatcher is the public +`skillopt/model/__init__.py` module: + +```text +skillopt/model/ +├── common.py # aliases, default models, token/response helpers +├── backend_config.py # optimizer/target whitelists and runtime selection +├── __init__.py # public API and split-role dispatch +├── openai_compatible_backend.py # generic Chat Completions example +├── qwen_backend.py # raw-HTTP chat example with per-role config +├── minimax_backend.py # compact raw-HTTP chat example +├── codex_harness.py # target-only exec harnesses +└── router.py # legacy single-backend compatibility surface +``` + +`router.py` is not the dispatcher used by the current training loop. Update it +only if the new backend must also be exposed through that legacy single-backend +API. + +## Step 1: implement the module contract + +Create a module such as `skillopt/model/your_backend.py`. Copy the signatures +from `openai_compatible_backend.py` or `qwen_backend.py`; model calls in the +current framework are synchronous. + +For a chat backend that supports both roles, the public module surface is: + +| Function | Purpose | +|---|---| +| `chat_optimizer(...)` | Optimizer system/user call; returns `(text, usage)` | +| `chat_target(...)` | Target system/user call; returns `(text, usage)` | +| `chat_optimizer_messages(...)` | Optimizer message-list call, including optional tools | +| `chat_target_messages(...)` | Target message-list call, including optional tools | +| `get_token_summary()` | Return per-stage counters plus `_total` | +| `reset_token_tracker()` | Clear this backend's counters | +| `set_optimizer_deployment(name)` | Change the optimizer model at runtime | +| `set_target_deployment(name)` | Change the target model at runtime | +| `set_reasoning_effort(effort)` | Apply or safely ignore the shared reasoning setting | + +Every call returns a usage dict with `prompt_tokens`, `completion_tokens`, and +`total_tokens`. Use `TokenTracker` from `skillopt.model.common` and record each +call exactly once. Message-list calls that accept tools should return the +compatibility message objects from `common.py` when `return_message=True`. + +Provider-specific configuration helpers and `count_tokens()` are optional, but +their state must be safe to update while calls may run concurrently. Keep +credentials out of logs and persisted artifacts. + +Exec-style targets do not implement this chat contract. They are target-only +and are integrated through `codex_harness.py` plus environment-specific rollout +code. + +## Step 2: register and route the backend + +A new backend normally requires all of the following: + +1. Add its canonical name, aliases, and default model to + `skillopt/model/common.py`. +2. Add the canonical name to the appropriate optimizer and/or target whitelist + in `skillopt/model/backend_config.py`. Do not advertise a role the module + cannot execute. +3. Import the module in `skillopt/model/__init__.py` and add dispatch branches + for every supported call surface. +4. Include its counters in `get_token_summary()` / `reset_token_tracker()` and + forward the shared deployment/reasoning setters where applicable. +5. If it has YAML settings, add structured-to-flat mappings in + `skillopt/config.py`, wire them through `scripts/train.py` and + `scripts/eval_only.py`, and document their precedence over environment + variables. +6. Update `router.py` only when legacy single-backend compatibility is part of + the intended feature. + +Backend selection in `scripts/train.py` must use +`model.optimizer_backend` and `model.target_backend`. A high-level +`model.backend` alias alone is not a substitute for this explicit split. + +## Step 3: test the integration + +Add focused tests under `tests/` that do not call a live provider. At minimum, +cover: + +- optimizer and target whitelist validation; +- routing for text and message-list calls; +- role-specific configuration precedence; +- tool-call compatibility, if supported; +- deployment/reasoning setters; +- token accounting, including a single correct `_total`; +- actionable errors for missing credentials or invalid responses. + +Then run the focused test, the full suite, and the documentation build: + +```bash +python -m pytest tests/test_your_backend.py -q +python -m pytest tests/ -q +mkdocs build --strict +``` + +Also update `.env.example`, the configuration reference, and the backend table +in the API reference. Add an optional dependency extra only when the backend +requires a package that is not already a core dependency. diff --git a/docs/guide/new-benchmark.md b/docs/guide/new-benchmark.md index 6dae9a1..183aed0 100644 --- a/docs/guide/new-benchmark.md +++ b/docs/guide/new-benchmark.md @@ -14,16 +14,18 @@ To add a benchmark you implement four things: 1. **A `SplitDataLoader` subclass** — knows how to load train / val / test item dicts from disk. -2. **A rollout helper** — runs the target model on a batch of items - under the current skill and scores each prediction. +2. **A rollout helper** — runs the target model on a batch of items, scores + each prediction, and persists the per-item conversation consumed by the + shared reflection stage. 3. **An `EnvAdapter` subclass** — wires the loader + rollout helper into - SkillOpt's lifecycle (`build_*_env`, `rollout`, `reflect`, - `get_task_types`). + SkillOpt's lifecycle (`build_*_env`, `rollout`, and `get_task_types`). + The shared `reflect()` implementation is inherited unless the benchmark + needs custom reflection logic. 4. **A YAML config** — references your env name plus the standard train / optimizer / gradient knobs. -Then one line in `scripts/train.py`'s `_register_builtins()` makes it -discoverable. +Then lazy registration in the training and evaluation scripts makes it +discoverable without importing optional dependencies at startup. --- @@ -99,8 +101,8 @@ def _score(prediction: str, ground_truth: str) -> tuple[int, float]: return hard, soft -def _rollout_one(item: dict, skill_content: str, - *, max_completion_tokens: int) -> dict: +def _rollout_one(item: dict, skill_content: str, *, prediction_dir: Path, + max_completion_tokens: int) -> dict: system = skill_content user = ( f"Question: {item['question']}\n\n" @@ -113,14 +115,33 @@ def _rollout_one(item: dict, skill_content: str, max_completion_tokens=max_completion_tokens, ) hard, soft = _score(prediction, item.get("ground_truth", "")) + + # EnvAdapter.reflect() reads this exact trajectory path. Keep item IDs + # unique and filesystem-safe. + task_dir = prediction_dir / str(item["id"]) + task_dir.mkdir(parents=True, exist_ok=True) + conversation = [ + {"role": "system", "content": system}, + {"role": "user", "content": user}, + {"role": "assistant", "content": prediction}, + ] + (task_dir / "conversation.json").write_text( + json.dumps(conversation, ensure_ascii=False, indent=2), + encoding="utf-8", + ) + return { "id": str(item["id"]), "hard": hard, "soft": soft, "predicted_answer": prediction, + "task_description": item.get("question", ""), "question": item.get("question", ""), "reference_text": item.get("reference_text", ""), "task_type": item.get("task_type", "docfaithful"), + "target_system_prompt": system, + "target_user_prompt": user, + "n_turns": 1, } @@ -128,15 +149,18 @@ def run_batch(*, items: list[dict], skill_content: str, out_root: str, workers: int = 4, max_completion_tokens: int = 4096) -> list[dict]: """Run a batch of episodes sequentially or with a thread pool.""" os.makedirs(out_root, exist_ok=True) + prediction_dir = Path(out_root, "predictions") # For brevity we go sequentially — swap in concurrent.futures.ThreadPoolExecutor # when network / model latency dominates. results = [ _rollout_one(item, skill_content, + prediction_dir=prediction_dir, max_completion_tokens=max_completion_tokens) for item in items ] Path(out_root, "rollouts.json").write_text( - json.dumps(results, ensure_ascii=False, indent=2) + json.dumps(results, ensure_ascii=False, indent=2), + encoding="utf-8", ) return results ``` @@ -150,9 +174,17 @@ Two design points worth flagging: - **Use `skillopt.model.chat_target`**, not raw OpenAI/Claude calls. That routes through whichever **chat** target backend the user configured (`openai_chat` / `claude_chat` / `qwen_chat` / - `minimax_chat`) without your adapter caring. Exec-style backends - (`codex_exec`, `claude_code_exec`) need env-specific rollout code — - see `skillopt/envs/swebench/` for an example. + `minimax_chat` / `openai_compatible`) without your adapter caring. + Exec-style backends (`codex_exec`, `claude_code_exec`) need + environment-specific rollout code — + see `skillopt/model/codex_harness.py` together with the rollout modules in + `skillopt/envs/searchqa/`, `skillopt/envs/docvqa/`, or + `skillopt/envs/officeqa/` for working examples. +- **Persist a conversation for reflection.** The shared `EnvAdapter.reflect()` + looks under `/predictions//conversation.json` and + skips results whose trajectory is absent or empty. Returning `hard`/`soft` + scores alone is sufficient for evaluation, but it cannot produce learning + patches. ## Step 4 — Implement the environment adapter @@ -277,9 +309,11 @@ the answer against `item["ground_truth"]`, and returns a list of dicts: ] ``` -The trainer only requires `id`, `hard`, `soft`. The rest is preserved on -`RolloutResult.extras` (see `skillopt/types.py`) and is what your -`reflect()` consumes via `run_minibatch_reflect`. +The trainer requires `id`, `hard`, and `soft` for scoring. The remaining fields +are preserved on `RolloutResult.extras` (see `skillopt/types.py`). The shared +reflection implementation combines those fields with each persisted +`predictions//conversation.json`; without that file the result is omitted +from reflection. ## Step 5 — Register the adapter @@ -294,9 +328,11 @@ and add to `_register_builtins()`: pass # docfaithful deps not installed — skip ``` -There is **no `BENCHMARK_REGISTRY` dict in `skillopt/envs/__init__.py`** — -the registry lives in `scripts/train.py` and is populated lazily so that -optional deps don't break `--help`. +Mirror the same lazy registration in +[`scripts/eval_only.py`](https://github.com/microsoft/SkillOpt/blob/main/scripts/eval_only.py) +so standalone evaluation can resolve the environment too. There is **no +`BENCHMARK_REGISTRY` dict in `skillopt/envs/__init__.py`**; both entry points +keep a small lazy registry so optional dependencies do not break `--help`. ## Step 6 — Create the YAML config @@ -322,9 +358,7 @@ optimizer: env: name: docfaithful - # Optional: a seed skill document. Create this file (or any markdown - # file) yourself before the first run, or omit the key to let SkillOpt - # start from an empty skill. + # Point to an existing Markdown file. Use an empty file to start blank. skill_init: skillopt/envs/docfaithful/skills/initial.md split_mode: split_dir split_dir: data/docfaithful_split @@ -341,7 +375,7 @@ env: ## Step 7 — Run ```bash -# If you set skill_init above, create the seed skill first: +# Create the file referenced by env.skill_init before the first run: # mkdir -p skillopt/envs/docfaithful/skills # echo "# DocFaithful initial skill" > skillopt/envs/docfaithful/skills/initial.md @@ -364,8 +398,11 @@ you forgot to implement one of the four abstract methods on `EnvAdapter`: `hard` / `soft`. - Noisy scoring kills the optimizer. Spend time on `run_batch`'s scoring before you spend time on prompts. +- If training repeatedly reports `skip_no_patches`, first verify that every + rollout result has a non-empty + `rollout/predictions//conversation.json` using the same `id` string. - If your benchmark needs heavy optional deps (selenium, vllm, ...), - wrap the registration block with `try / except ImportError` (Step 5) + wrap both registration blocks with `try / except ImportError` (Step 5) so people without those deps can still `--help`. - Copy `skillopt/envs/_template/` as a starting skeleton — it now implements the real abstract methods. diff --git a/docs/guide/skill-document.md b/docs/guide/skill-document.md index 62d1a34..1df89d3 100644 --- a/docs/guide/skill-document.md +++ b/docs/guide/skill-document.md @@ -38,23 +38,45 @@ During training, the skill document is modified by **edit patches**: 2. **Modifications**: Refining existing rules that are partially correct 3. **Deletions**: Removing rules that consistently lead to errors -Each edit is validated through the **gate** mechanism before being permanently accepted. +Selected edits are applied together to produce a candidate skill. With the +validation gate enabled, that candidate replaces the current skill only when +its score on the selection split strictly improves. + +SkillOpt may maintain two protected, machine-managed regions: + +```markdown + +... epoch-level longitudinal guidance ... + + + +... skill-aware execution reminders ... + +``` + +Normal edit patches cannot modify either region. Slow update owns the first; +optional skill-aware reflection owns the second. Preserve these markers when +copying or manually inspecting a trained skill. ## Initial Skill You can start training with: -- **Empty skill**: The system learns everything from scratch +- **Empty skill**: Point `env.skill_init` to an empty Markdown file - **Seed skill**: Provide initial instructions to bootstrap training - **Pre-trained skill**: Transfer a skill from a related benchmark Configure the initial skill in your YAML: ```yaml -train: - init_skill: "path/to/initial_skill.md" # or omit for empty +env: + skill_init: path/to/initial_skill.md ``` +To start from scratch, create an empty Markdown file and use its path. A missing +path currently also starts blank, so using an explicit file avoids silently +treating a typo as an empty skill. + ## Skill Quality Metrics Track your skill's evolution through: @@ -62,15 +84,16 @@ Track your skill's evolution through: - **Validation score**: Primary metric on the selection split - **Test score**: Final metric on held-out test data - **Skill length**: Total tokens in the document -- **Edit acceptance rate**: Fraction of proposed edits that pass gating +- **Candidate acceptance rate**: Fraction of candidate skill updates that pass + gating; multiple proposed edits can be combined into one candidate ## Best Practices !!! tip "Tips for better skills" 1. **Start with a seed skill** (`env.skill_init`) if you have domain knowledge — it converges faster 2. **Use cosine LR schedule** — aggressive early exploration + careful late refinement - 3. **Enable slow update** (`use_slow_update: true`) to prevent forgetting across epochs - 4. **Enable meta skill** (`use_meta_skill: true`) so the optimizer accumulates strategy memory + 3. **Enable slow update** (`optimizer.use_slow_update: true`) to counter forgetting across epochs + 4. **Enable meta skill** (`optimizer.use_meta_skill: true`) so the optimizer accumulates strategy memory ## Next Steps diff --git a/docs/guide/training-loop.md b/docs/guide/training-loop.md index 7922305..30393dc 100644 --- a/docs/guide/training-loop.md +++ b/docs/guide/training-loop.md @@ -37,12 +37,11 @@ scores = evaluate(predictions, ground_truth) ### 2. Reflect (Backward Pass) -The **optimizer** model analyzes failed trajectories and produces **edit patches** — structured suggestions for improving the skill document. - -Two modes: - -- **Shallow**: Analyze each trajectory independently -- **Deep**: Cross-reference multiple failures to find systemic issues +The **optimizer** model analyzes trajectory minibatches and produces **edit +patches** — structured suggestions for improving the skill document. Failure +minibatches are always eligible for analysis; successful trajectories are also +analyzed unless `gradient.failure_only` is enabled. Independent minibatches can +run concurrently according to `gradient.analyst_workers`. ```python # Analogy: computing gradients @@ -74,17 +73,31 @@ Selected edits are applied to the skill document, producing a new version. ### 6. Gate (Validation) -The updated skill is evaluated on a **selection split** (analogous to a validation set). The update is only accepted if performance improves. +The updated skill is evaluated on a **selection split** (analogous to a +validation set). With the gate enabled, the candidate is accepted only when its +configured gate score (`hard`, `soft`, or `mixed`) is strictly higher than the +current skill's score. With `evaluation.use_gate: false`, validation is still +recorded but candidates are force-accepted. ## Epoch Boundary Mechanisms ### Slow Update -At the end of each epoch (starting from epoch 2), the system performs a **longitudinal comparison**: it rolls out both the previous epoch's skill and the current skill on the same samples, categorizes items as improved/regressed/persistent_fail/stable_success, then generates high-level **guidance** that is injected into the skill document. This prevents catastrophic forgetting of earlier improvements. +At the end of each epoch (starting from epoch 2), the system performs a +**longitudinal comparison**: it rolls out both the previous epoch's skill and +the current skill on the same samples, categorizes items as +improved/regressed/persistent-fail/stable-success, then generates high-level +**guidance** for the skill document. Depending on +`optimizer.slow_update_gate_with_selection`, that guidance is either checked on +the selection split or applied unconditionally. Its purpose is to counter +cross-epoch forgetting. ### Meta Skill -A **meta-skill memory** accumulates high-level strategy notes across the entire training run. At the end of each epoch, the optimizer reflects on what changed between epochs and produces a compact memory that is provided as additional context during future reflection steps. +A **meta-skill memory** accumulates high-level strategy notes across the training +run. Starting at the end of epoch 2, the optimizer compares the previous and +current epoch, writes a compact memory, and provides the prior epoch's memory as +additional context during later reflection and update stages. ## Next Steps diff --git a/docs/guideline.html b/docs/guideline.html index 8712012..127162c 100644 --- a/docs/guideline.html +++ b/docs/guideline.html @@ -4,1039 +4,545 @@ SkillOpt — Documentation & Reproduction Guide - + - -
- - - SkillOpt - Documentation & Reproduction Guide +
+ + SkillOpt - GitHub ↗ - Paper ↗ + Paper + GitHub
+ - -
-

1.1 What is SkillOpt #

-

SkillOpt is a text-space optimizer that improves a frozen language agent by iteratively editing a natural-language skill document — never the model weights. The skill document is a Markdown file that conditions a target model as it executes tasks. SkillOpt treats this document as the "weights" and runs a training loop that mirrors deep-learning training: rollout (forward pass), reflect (backward pass / gradients), select & apply edits (optimizer step), and a validation gate (accept/reject).

-

Two roles split every model call:

-
    -
  • Target — executes tasks using the current skill document (the agent being improved).
    • -
  • Optimizer — analyzes the target's trajectories and proposes edits to the skill document.
    • -
-

The same loop drives six benchmarks out of the box (QA, document QA, embodied agents, math, spreadsheet code generation, and tool-augmented QA).

-
+
+ Microsoft Research · documentation hub +

SkillOpt Documentation & Reproduction Guide

+

Improve frozen agents by optimizing the Markdown skills that guide them—using reflective updates and held-out validation instead of weight training.

-
-

1.2 Deep-Learning ↔ SkillOpt Analogy #

-

Every concept below maps to a concrete code construct, so deep-learning intuitions transfer directly to hyperparameter tuning.

-
+
+ How this guide stays accurate + This page is a stable, concise entry point. Detailed commands, defaults, + and APIs live in the versioned + Markdown documentation + beside the code. For exact behavior in a checkout, the command's + --help, the selected YAML config, and the code are authoritative. +
+ +
+

Overview

+

SkillOpt treats a natural-language skill document as the + trainable state of an agent. A target model executes tasks, an optimizer + reflects on the resulting trajectories, bounded edits form a candidate + skill, and a validation gate decides whether to keep it.

+
+
+

Research engine

+

Run reproducible training and evaluation over benchmark splits. + Six released benchmark configurations cover QA, document QA, embodied + agents, math, spreadsheets, and tool-augmented QA.

+
+
+

SkillOpt-Sleep preview

+

Harvest supported coding-agent sessions, mine replayable tasks, and + stage proposed memory or skill updates for review. It is a separate, + evolving deployment companion—not the paper's benchmark runner.

+
+
+

The optimizer and target are separate roles and may use different + backends. Validation gating is the research default and the paper-style + setting; deliberately disabling it force-accepts candidates and changes the + experiment semantics. SkillOpt-Sleep stages updates by default; automatic + adoption is opt-in.

+
+ +
+

Choose the right workflow

+
- + - - - - - - - - - - - - - - - + + + + +
Deep learningSkillOptWhere it lives
GoalStart with
Model weightsSkill document (Markdown)skillopt/optimizer/skill.py
Forward passRollout — target runs tasksenvs/<bench>/rollout.py
Loss / scoreTask evaluatorenvs/<bench>/evaluator.py
Backprop / gradientsReflect → edit patchesgradient/reflect.py
Gradient aggregationHierarchical patch mergegradient/aggregate.py
Gradient clippingRank & select top-k editsoptimizer/clip.py
Learning rateoptimizer.learning_rate (edits/step)optimizer/scheduler.py
LR schedulerlr_scheduler (cosine/linear/…)optimizer/scheduler.py
Optimizer stepApply patches to the documentoptimizer/skill.py
Validation setSelection split (valid_seen)evaluation/gate.py
Early stopping / acceptValidation gateevaluation/gate.py
MomentumSlow update (epoch boundary)optimizer/slow_update.py
Meta-learningMeta skill (cross-epoch memory)optimizer/meta_skill.py
Batch / minibatchbatch_size / minibatch_sizeengine/trainer.py
EpochEpoch (+ slow update & meta skill)engine/trainer.py
Reproduce paper-style benchmark trainingResearch first experiment
Evaluate an existing skill without trainingEvaluation CLI reference
Add a benchmark adapterNew benchmark guide
Connect another model providerBackend guide
Improve a coding-agent skill from local sessionsSkillOpt-Sleep
-
-
What transfers from DL -

Cosine schedule tends to beat constant; moderate learning rates (≈4–16 edits/step) beat very high/low; slow update curbs cross-epoch forgetting; meta-skill memory improves reflection quality. Conversely, bigger rollout batches and many epochs show diminishing returns — skills converge in ~2–4 epochs.

-
-
+
+
-
-

1.3 Key Features #

-
-

Validation gating

Every candidate skill is scored on a held-out selection split and only accepted if it beats the current/best skill.

-

Slow update

Epoch-boundary longitudinal comparison writes guidance into a protected region — momentum against forgetting. Force-injected or selection-gated.

-

Meta skill

Optimizer-side memory that reflects on what worked across epochs and feeds back into reflection.

-

Pluggable backends

OpenAI / Azure OpenAI, Anthropic Claude, local Qwen (vLLM), plus Codex/Claude-Code exec backends for the target.

-

Six benchmarks

SearchQA, DocVQA, ALFWorld, LiveMathematicianBench, SpreadsheetBench, OfficeQA — each a self-contained env module.

-

Auto-resume

Every run is checkpointed step-by-step; re-running the same command continues from the last completed step.

-
-
+
+

Install

+

SkillOpt requires Python 3.10 or newer.

+
# Published package
+python -m pip install skillopt
 
-    

-      
1.4 Repository Layout #

-
# top level
-configs/            # YAML configs (_base_ + per-benchmark)
-scripts/            # train.py, eval_only.py CLIs
-ckpt/               # packaged reference skills (e.g. gpt5.5_skill.md)
-docs/               # this guide + mkdocs sources
-skillopt/           # the package
- ├─ config.py        # YAML loading, _base_ inheritance, flatten
- ├─ engine/trainer.py# the training loop (ReflACTTrainer)
- ├─ gradient/        # reflect.py (analyst), aggregate.py (merge)
- ├─ optimizer/       # skill edits, scheduler, clip, slow_update, meta_skill
- ├─ evaluation/      # gate.py (accept/reject logic)
- ├─ model/           # backend clients + routing
- └─ envs/<benchmark>/ # adapter, dataloader, rollout, evaluator, reflect

-    

+# Latest source and development workflow
+git clone https://github.com/microsoft/SkillOpt.git
+cd SkillOpt
+python -m pip install -e .
 
-    
-    

-      
2.1 Requirements #

-      
    
-        
  • Python ≥ 3.10
    • 
-        
  • Credentials for at least one model backend (Azure OpenAI, OpenAI-compatible, Anthropic, or a local Qwen server)
    • 
-        
  • Benchmark datasets are not bundled — prepare your own splits (see §4)
    • 
-      

-    

+# Install only the extras you need
+python -m pip install -e ".[searchqa]"  # SearchQA materialization
+python -m pip install -e ".[alfworld]"  # ALFWorld
+python -m pip install -e ".[claude]"    # optional Claude agent SDK support
+python -m pip install -e ".[webui]"     # Gradio dashboard
+python -m pip install -e ".[dev]"       # tests and linting
+
+ Release boundary + This guide tracks main. PyPI currently serves 0.2.0; the + generic research openai_compatible backend, Sleep handoff, + SkillOpt-Sleep support for non-Azure OpenAI-compatible endpoints, and the + Sleep --preferences flag require a source install from + main until the next release. +
+

See the installation guide + for platform notes and dependency boundaries.

+
-
-

2.2 Install the Package #

-

Option A — from PyPI:

-
pip install skillopt
+  

+    
Credentials and endpoint families

+    
Copy .env.example, fill only the backend you use, and load
+    it into your shell. Do not commit the resulting .env.

+
cp .env.example .env
+set -a
+source .env
+set +a

 
-# Optional extras:
-pip install skillopt[alfworld]   # ALFWorld benchmark
-pip install skillopt[webui]      # Gradio monitoring dashboard
-pip install skillopt[claude]     # Claude model backend
-
-

Option B — from source (for development):

-
git clone https://github.com/microsoft/SkillOpt.git
-cd SkillOpt
-pip install -e .
+    
Azure OpenAI

+
export AZURE_OPENAI_ENDPOINT="https://your-resource.openai.azure.com/"
+export AZURE_OPENAI_API_VERSION="2024-12-01-preview"
+export AZURE_OPENAI_AUTH_MODE="api_key"
+export AZURE_OPENAI_API_KEY="your-key"

+    
For keyless Azure authentication, use azure_cli or
+    managed_identity and follow the
+    configuration guide.
+    Setting an API key without AZURE_OPENAI_AUTH_MODE=api_key does
+    not change the default authentication mode.

 
-# Optional extras (install only what you need):
-pip install -e ".[alfworld]"   # ALFWorld benchmark
-pip install -e ".[claude]"     # Anthropic Claude backend
-pip install -e ".[qwen]"       # local Qwen backend
-pip install -e ".[webui]"      # monitoring dashboard
+    
Generic OpenAI-compatible research backend

+
export OPENAI_COMPATIBLE_BASE_URL="https://api.example.com/v1"
+export OPENAI_COMPATIBLE_API_KEY="your-key"
+export OPENAI_COMPATIBLE_MODEL="provider-model"
 
-# ALFWorld also needs its data assets:
-alfworld-download

-
+python scripts/train.py --config configs/searchqa/default.yaml \ + --cfg-options \ + model.optimizer_backend=openai_compatible \ + model.target_backend=openai_compatible \ + model.optimizer=provider-model \ + model.target=provider-model +

This provider-neutral backend is distinct from Azure OpenAI. Per-role + overrides use OPTIMIZER_OPENAI_COMPATIBLE_* and + TARGET_OPENAI_COMPATIBLE_*. Train/eval applies the YAML role + models after backend initialization, so they override model-name + environment variables.

-
-

2.3 Configure Credentials #

-

Copy the template and fill in whichever backend you will use:

-
cp .env.example .env
-# edit .env, then:
-set -a; source .env; set +a
-
One env-var family for all OpenAI modes -

SkillOpt reuses the AZURE_OPENAI_* variable names even for plain OpenAI — there is no separate OPENAI_API_KEY knob. AZURE_OPENAI_ENDPOINT is required for every OpenAI auth mode.

-
-

Azure OpenAI (default)

-
export AZURE_OPENAI_ENDPOINT="https://your-resource.openai.azure.com/"
-export AZURE_OPENAI_API_VERSION="2024-12-01-preview"
-# Auth option 1 — API key:
-export AZURE_OPENAI_API_KEY="your-key"
-# Auth option 2 — Azure CLI (no key; recommended on Azure VMs):
-export AZURE_OPENAI_AUTH_MODE=azure_cli
-# Auth option 3 — Managed Identity:
-export AZURE_OPENAI_AUTH_MODE=managed_identity
-export AZURE_OPENAI_MANAGED_IDENTITY_CLIENT_ID="your-client-id"
-

OpenAI-compatible endpoint

-
export AZURE_OPENAI_ENDPOINT="https://api.openai.com/v1"
-export AZURE_OPENAI_API_KEY="sk-..."
-export AZURE_OPENAI_AUTH_MODE=openai_compatible
-

Anthropic Claude / local Qwen

-
export ANTHROPIC_API_KEY="sk-ant-..."          # claude_chat backend
+    
OpenAI-compatible endpoints in SkillOpt-Sleep

+    
The Sleep CLI exposes this compatibility path through its
+    azure_openai backend for backward compatibility, so it uses a
+    different environment-variable family:

+
export AZURE_OPENAI_ENDPOINT="https://api.example.com/v1"
+export AZURE_OPENAI_API_KEY="your-key"
+export AZURE_OPENAI_AUTH_MODE="openai_compatible"
 
-export QWEN_CHAT_BASE_URL="http://localhost:8000/v1" # local vLLM
-export QWEN_CHAT_MODEL="Qwen/Qwen3.5-4B"

-
+skillopt-sleep run --backend azure_openai --model provider-model +

Do not mix this mode with Azure CLI or managed-identity settings. See + the dedicated + Sleep endpoint guide.

+ -
-

2.4 Verify Installation #

-
python -c "import skillopt; print('SkillOpt ready!')"
-
+
+

Research engine: first experiment

+

The repository ships deterministic ID manifests, not the benchmark + examples themselves. Materialize the SearchQA examples once, then run its + checked-in config:

+
python -m pip install -e ".[searchqa]"
+python scripts/materialize_searchqa.py
 
-    
-    

-      
3.1 Your First Demo #

-      
What ships in this repo: ready-to-use configs and
-      pretrained skills (ckpt/) for six benchmarks, plus
-      lightweight ID manifests under data/. The manifests
-      pin exactly which examples each split uses but do not
-      contain the example contents — so you materialize the data once before
-      the first run.

-      
Step 1 — materialize the SearchQA splits (one-time; downloads the ~6.5 GB source dataset). The manifest IDs match the key field of the
-      lucadiliello/searchqa
-      dataset:

-
pip install datasets
-python - <<'PY'
-import json, os
-from datasets import load_dataset
-
-ds = load_dataset("lucadiliello/searchqa")
-by_key = {r["key"]: r for split in ds.values() for r in split}
-
-for split in ["train", "val", "test"]:
-    ids = json.load(open(f"data/searchqa_id_split/{split}/items.json"))
-    items = []
-    for x in ids:
-        r = by_key[x["id"]]
-        items.append({"id": r["key"], "question": r["question"],
-                      "context": r["context"], "answers": r["answers"]})
-    os.makedirs(f"data/searchqa_split/{split}", exist_ok=True)
-    json.dump(items, open(f"data/searchqa_split/{split}/items.json", "w"))
-    print(split, len(items))
-PY

-      
Step 2 — train (4 epochs × batch 40; see §3.2
-      for the CLI reference):

-
python scripts/train.py \
-    --config configs/searchqa/default.yaml \
-    --split_dir data/searchqa_split \
-    --azure_openai_endpoint https://your-resource.openai.azure.com/ \
-    --optimizer_model gpt-5.5 \
-    --target_model gpt-5.5

-      
Other benchmarks follow the same pattern — materialize from the raw
-      source listed in
-      data/README.md
-      (it documents the lookup key per benchmark), then point
-      --split_dir at the result. The one exception is
-      ALFWorld, whose bundled
-      data/alfworld_path_split works directly: just
-      pip install -e ".[alfworld]" && alfworld-download and
-      set $ALFWORLD_DATA.

-      
To sanity-check your setup without training, evaluate a
-      packaged pretrained skill instead (§3.3 uses
-      ckpt/searchqa/gpt5.5_skill.md), or launch the monitoring
-      WebUI (§8.4).

-    

-
-    

-      
3.2 Train a Skill #

-
# Minimal SearchQA run
-python scripts/train.py \
-    --config configs/searchqa/default.yaml \
-    --split_dir /path/to/your/searchqa_split \
-    --azure_openai_endpoint https://your-resource.openai.azure.com/ \
-    --optimizer_model gpt-5.5 \
-    --target_model gpt-5.5

-      
Swap the config for another benchmark (e.g. configs/livemathematicianbench/default.yaml, configs/alfworld/default.yaml). Common CLI arguments:

-      

-        
-        
-          
-          
-          
-          
-          
-          
-          
-        
-      
ArgumentDescription
--configBenchmark config YAML (required)
--split_dirPath to the data split directory
--azure_openai_endpointAzure OpenAI endpoint URL
--optimizer_model / --target_modelDeployment names for optimizer / target
--num_epochs / --batch_sizeEpochs and rollout batch size
--out_rootOutput directory
--cfg-options k=v ...Override any config key (see §6.1)

-    

-
-    

-      
3.3 Evaluate a Skill #

-      
Evaluate any skill document (a packaged reference skill, or a trained run's best_skill.md) without training:

-
# Evaluate the packaged GPT-5.5 SearchQA skill on the test split
-python scripts/eval_only.py \
-  --config configs/searchqa/default.yaml \
-  --skill ckpt/searchqa/gpt5.5_skill.md \
-  --split valid_unseen \
-  --split_dir /path/to/searchqa_split \
-  --azure_openai_endpoint https://your-resource.openai.azure.com/

-      

-        
-        
-          
-          
-          
-          
-        
-      
--splitMeaning
valid_unseenTest set (held-out)
valid_seenValidation / selection set
trainTraining set
allAll splits combined (default)

-    

-
-    

-      
3.4 Output Structure #

-
outputs/<run_name>/
- ├─ config.json          # flattened runtime config
- ├─ history.json         # per-step training history
- ├─ runtime_state.json   # resume checkpoint
- ├─ best_skill.md        # best validated skill document
- ├─ skills/skill_vXXXX.md# skill snapshot per step
- ├─ steps/step_XXXX/     # per-step artifacts (patches, evals)
- ├─ slow_update/epoch_XX/# slow-update logs & rollouts
- └─ meta_skill/epoch_XX/ # meta-skill logs

-    

-
-    

-      
3.5 Auto-Resume #

-      
Each completed step persists its state to runtime_state.json and a steps/step_XXXX/ directory. Re-running the same command against the same out_root detects finished work and continues from the last completed step — including epoch-boundary slow-update and meta-skill stages.

-    

-
-    
-    

-      
4.1 Split Directory Format #

-      
Bringing your own dataset takes three steps:
-      (1) create a split directory with train/ val/ test/ item
-      files in the format below; (2) make sure each item carries the fields
-      the closest existing benchmark adapter expects (§4.2); (3) point
-      --split_dir at it and train with that benchmark's config.
-      If no existing adapter matches your task shape (different rollout or
-      scoring logic), write a new benchmark adapter instead — see §7.2.

-
-      
With env.split_mode: split_dir (the recommended, deterministic mode), SkillOpt reads a directory containing train/, val/, and test/ subfolders, each holding a JSON array of task items:

-
data/my_split/
- ├─ train/items.json   # used for rollout (the "train split")
- ├─ val/items.json     # selection split → validation gate (valid_seen)
- └─ test/items.json    # held-out final eval (valid_unseen)

-      
Split naming
-        
Internally the splits are referred to as train, valid_seen (validation/selection), and valid_unseen (test). The --split flag of eval_only.py uses these names.

-      

-    

-
-    

-      
4.2 Item JSON Schema #

-      
Required fields depend on the benchmark; consult skillopt/envs/<benchmark>/dataloader.py for the exact contract. A SearchQA item, for example:

-
[
-  {
-    "id":       "unique_item_id",
-    "question": "Who wrote the novel ...",
-    "context":  "[DOC] relevant passage text ...",
-    "answers":  ["expected answer"]
-  }
-]

-      
Datasets not included
-        
This repository ships no benchmark data. Prepare your own splits in the format above before training.

-      

-    

-
-    

-      
4.3 Split Modes #

-      

-        
-        
-          
-          
-        
-      
env.split_modeBehavior
split_dirUse a pre-built directory with explicit train/val/test folders (set env.split_dir). Deterministic and reproducible.
ratioBuild a deterministic split on the fly from a single env.data_path, using split_seed (and a train:val:test ratio). Convenient for quick experiments.

-    

-
-    
-    

-      
5.1 The Training Loop #

-      
The loop lives in ReflACTTrainer (skillopt/engine/trainer.py). Each epoch runs a series of optimization steps over rollout batches, then performs two epoch-boundary stages.

-
for epoch in epochs:
-    for step in steps:
-        1. Rollout    # target executes a batch of tasks
-        2. Reflect    # optimizer analyzes trajectories → edit patches
-        3. Aggregate  # hierarchically merge similar patches
-        4. Select     # rank & clip edits to the learning rate
-        5. Update     # apply patches → candidate skill
-        6. Gate       # score on selection split → accept / reject
-
-    # epoch boundary (from epoch 2 onward)
-    Slow update   # longitudinal comparison → protected guidance
-    Meta skill    # cross-epoch optimizer memory

-    

-
-    

-      
5.2 The Six Per-Step Stages #

-      

-        
-        
-          
-          
-          
-          
-          
-          
-        
-      
StageWhat happensSource
1. RolloutThe target model runs each task in the batch with the current skill as context, producing trajectories and scores.envs/<b>/rollout.py
2. ReflectThe optimizer runs an error analyst (and optional success analyst) over minibatches of trajectories, emitting structured edit patches. Runs in parallel across analyst_workers.gradient/reflect.py
3. AggregateSemantically similar patches are merged hierarchically to remove redundancy.gradient/aggregate.py → merge_patches
4. SelectPatches are ranked and clipped to the current learning rate (max edits this step), set by the scheduler.optimizer/clip.py → rank_and_select
5. UpdateSelected edits are applied to the skill document, producing a candidate skill (patch / rewrite modes).optimizer/skill.py, update_modes.py
6. GateThe candidate is scored on the selection split and accepted only if it improves (see §5.3).evaluation/gate.py → evaluate_gate

-    

-
-    

-      
5.3 Validation Gate #

-      
evaluate_gate is a pure decision function. It compares the candidate's selection-set score against the current and best skills:

-      
    
-        
  • accept_new_best — candidate > current and candidate > best → becomes both current and best.
    • 
-        
  • accept — candidate > current but ≤ best → becomes current only.
    • 
-        
  • reject — candidate ≤ current → discarded; current/best unchanged.
    • 
-      

-      
The comparison metric is configurable via evaluation.gate_metric:

-      

-        
-        
-          
-          
-          
-        
-      
MetricScore used
hard defaultExact-match / discrete score
softPartial-credit / continuous score
mixedWeighted blend, controlled by gate_mixed_weight

-      
When to use soft/mixed
-        
The soft/mixed metrics (contributed config configs/examples/soft_gate.yaml) help when the selection split is small and rewards are continuous, where a discrete hard gate may reject every candidate and stall training. Paper numbers use the default hard gate.

-      

-    

-
-    

-      
5.4 Slow Update (Momentum) #

-      
At each epoch boundary (from epoch 2), the slow update rolls out both the previous epoch's skill and the current skill on the same sampled tasks, categorizes items (improved / regressed / persistent-fail / stable-success), and asks the optimizer to write a free-form guidance block. This guidance lands in a protected region of the skill that step-level edits cannot touch — only the slow update overwrites it. It is SkillOpt's analogue of momentum, countering cross-epoch forgetting.

-      
Acceptance has two modes, selected by optimizer.slow_update_gate_with_selection:

-      

-        
-        
-          
-          
-        
-      
ModeBehavior
false default — force-injectedGuidance is injected into both current and best skills unconditionally. The longitudinal guidance always persists; it is not gated by step-level selection scores.
true — gatedThe slow-update candidate is scored on the selection split and accepted/rejected through the same validation gate as step-level updates.

-    

-
-    

-      
5.5 Meta Skill (Optimizer Memory) #

-      
The meta skill is optimizer-side memory — it never modifies the target skill document. At the end of each epoch (skipped for epoch 1), the optimizer compares the previous and current epoch's last-step skills on the same sampled tasks and writes a compact, evidence-based reflection on what kind of edits helped or hurt. That memory is then injected as extra context into the next epoch's reflect / merge / learning-rate / ranking stages, so the optimizer accumulates strategy across the run.

-    

-
-    

-      
5.6 Skill Document Anatomy #

-      
A skill document is plain Markdown. Initial skills can be empty (learn from scratch) or seeded with domain knowledge via env.skill_init. During training the document accrues rules, patterns, and edge-case handling through accepted edit patches. A dedicated protected region holds the slow-update guidance, delimited by HTML-comment markers:

-
# Question Answering Skill
-
-## Learned rules ...
-- When the context contains multiple candidates, prefer ...
-
-<!-- SLOW_UPDATE_START -->
-# (epoch-level longitudinal guidance — only the slow update writes here)
-<!-- SLOW_UPDATE_END -->

-      
Helpers in optimizer/slow_update.py manage this region: inject_empty_slow_update_field (placeholder at epoch 1), extract_slow_update_field (read), and replace_slow_update_field (overwrite). Step-level edits are blocked from modifying anything inside the markers.

-    

-
-    
-    

-      
6.1 Configuration System #

-      
Configs are structured YAML with section blocks (model, train, gradient, optimizer, evaluation, env) and _base_ inheritance. A benchmark config inherits the shared defaults and overrides only what differs:

-
# configs/searchqa/default.yaml
-_base_: ../_base_/default.yaml
-train:
-  train_size: 400
-  batch_size: 40
-optimizer:
-  learning_rate: 4
-env:
-  name: searchqa
-  split_dir: data/searchqa_split

-      
Override any key at the command line without editing files:

-
python scripts/train.py --config configs/searchqa/default.yaml \
-  --cfg-options optimizer.learning_rate=16 optimizer.lr_scheduler=linear

-      
Reading the tables below
-        
Each section lists the key (relative to its YAML block), type, default (from configs/_base_/default.yaml), allowed values, and meaning. Defaults shown are the shipped base defaults.

-      

-    

-
-    

-      
6.2 model.* #

-      

-        
-        
-          
-          
-          
-          
-          
-          
-          
-          
-          
-          
-          
-        
-      
KeyTypeDefaultDescription / options
backendstrazure_openaiHigh-level backend label for the run.
optimizerstrgpt-5.5Optimizer model deployment (writes skill edits).
targetstrgpt-5.5Target model deployment (executes tasks).
optimizer_backendstropenai_chatClient path for the optimizer: openai_chat or claude_chat.
target_backendstropenai_chatClient path for the target: openai_chat / claude_chat / qwen_chat / codex_exec / claude_code_exec.
reasoning_effortstrmediumlow / medium / high / xhigh / max (or empty).
rewrite_reasoning_effortstr""Override effort for full-rewrite calls (empty = inherit).
rewrite_max_completion_tokensint64000Token cap for full-rewrite optimizer calls.
azure_openai_endpointstr""Azure resource URL (or via AZURE_OPENAI_ENDPOINT).
azure_openai_api_versionstr2024-12-01-previewAzure API version header.
azure_openai_auth_modestr""api_key / azure_cli / managed_identity / openai_compatible (empty → env default).

-      
Separate optimizer / target endpoints
-        
Every azure_openai_* key also has optimizer_azure_openai_* and target_azure_openai_* variants, letting you point the optimizer and target at different Azure resources. Exec backends (codex_exec, claude_code_exec) add their own codex_exec_* / claude_code_exec_* knobs (sandbox, reasoning effort, SDK mode, etc.).

-      

-    

-
-    

-      
6.3 train.* #

-      

-        
-        
-          
-          
-          
-          
-          
-        
-      
KeyTypeDefaultDL analogyDescription
num_epochsint4EpochsNumber of training epochs.
train_sizeint0Train-set size0 = derive from the dataset split. (Fixed by split size when using split_dir.)
batch_sizeint40Batch sizeTasks rolled out per optimization step.
accumulationint1Grad accumulationAccumulation rounds per step.
seedint42Random seedReproducibility seed.

-    

-
-    

-      
6.4 gradient.* #

-      

-        
-        
-          
-          
-          
-          
-          
-        
-      
KeyTypeDefaultDescription
minibatch_sizeint8Trajectories per reflect minibatch.
merge_batch_sizeint8Patches per merge batch during aggregation.
analyst_workersint16Parallel reflection workers (data parallelism).
max_analyst_roundsint3Max rounds of analyst reflection per step.
failure_onlyboolfalseReflect only on failed trajectories when true.

-    

-
-    

-      
6.5 optimizer.* #

-      

-        
-        
-          
-          
-          
-          
-          
-          
-          
-          
-          
-          
-          
-          
-          
-        
-      
KeyTypeDefaultDL analogyDescription / options
learning_rateint4Learning rateMax edit patches applied per step (the "edit budget").
min_learning_rateint2Min LRFloor edit budget for decaying schedulers.
lr_schedulerstrcosineLR scheduleconstant / linear / cosine / autonomous.
lr_control_modestrfixed—fixed / autonomous / none.
skill_update_modestrpatch—patch / rewrite_from_suggestions / full_rewrite_minibatch.
use_slow_updatebooltrueMomentumEnable epoch-boundary slow update.
slow_update_samplesint20—Tasks sampled for the longitudinal comparison.
slow_update_gate_with_selectionboolfalse—false = force-inject guidance; true = gate it on the selection split (see §5.4).
longitudinal_pair_policystrmixed—mixed / changed / unchanged — which comparison pairs to keep.
use_meta_skillbooltrueMeta-learningEnable cross-epoch optimizer memory.
use_skill_aware_reflectionboolfalse—EmbodiSkill-style failure routing: SKILL_DEFECT (rule wrong/missing → gated body edit) vs EXECUTION_LAPSE (valid rule not followed → reminder appended to a protected appendix region that step-level edits never modify). Off = baseline-identical; resolved process-wide, works on every benchmark. Not supported with rewrite_from_suggestions / full-rewrite modes.
skill_aware_appendix_sourcestrboth—both (success analyst may also re-emphasize rules) / failure_only (paper-faithful S_app: failure side only).
skill_aware_consolidate_thresholdint0—>0: LLM-compact the appendix once it exceeds N notes (experimental); 0 = off.

-    

-
-    

-      
6.6 evaluation.* #

-      

-        
-        
-          
-          
-          
-          
-          
-          
-        
-      
KeyTypeDefaultDescription / options
use_gatebooltrueValidation gating is mandatory in this branch (must remain true).
gate_metricstrhardhard / soft / mixed — score used by the gate (see §5.3).
gate_mixed_weightfloat0.5Weight on the soft score when gate_metric = mixed.
sel_env_numint0Selection-split eval size (0 = use full split).
test_env_numint0Test-split eval size (0 = use full split).
eval_testbooltrueRun a final test evaluation after training.

-      
Gate is required
-        
Setting evaluation.use_gate: false raises an error — validation gating cannot be disabled in this branch.

-      

-    

-
-    

-      
6.7 env.* #

-      

-        
-        
-          
-          
-          
-          
-          
-          
-          
-          
-        
-      
KeyTypeDefaultDescription
namestr""Benchmark name (searchqa, docvqa, alfworld, …). Selects the env module.
skill_initstr""Path to a seed skill (empty = start from scratch).
split_modestrratioratio or split_dir (see §4.3).
split_dirstr""Pre-split directory (when split_mode = split_dir).
data_pathstr""Single dataset path (when split_mode = ratio).
split_seedint42Seed for deterministic ratio splitting.
exec_timeoutint120Per-task target/code-agent timeout (seconds).
out_rootstr""Output directory for the run.

-      
Benchmark-specific env keys
-        
Env blocks may carry extra benchmark-specific keys (e.g. max_turns, workers, max_completion_tokens, limit). Unmapped env keys are passed straight through to the benchmark adapter — check the relevant configs/<benchmark>/default.yaml.

-      

-    

-
-    
-    

-      
7.1 Supported Benchmarks #

-      

-        
-        
-          
-          
-          
-          
-          
-          
-        
-      
BenchmarkTypeConfig
SearchQAQuestion answeringconfigs/searchqa/default.yaml
DocVQADocument QAconfigs/docvqa/default.yaml
ALFWorldEmbodied agentconfigs/alfworld/default.yaml
LiveMathematicianBenchMath reasoningconfigs/livemathematicianbench/default.yaml
SpreadsheetBenchSpreadsheet code generationconfigs/spreadsheetbench/default.yaml
OfficeQATool-augmented QAconfigs/officeqa/default.yaml

-      
Each benchmark is a self-contained module under skillopt/envs/<benchmark>/ with an adapter.py, dataloader.py, rollout.py, and evaluator.py (some add a custom reflect.py). Packaged reference skills live in ckpt/<benchmark>/.

-    

-
-    

-      
7.2 Add a New Benchmark #

-      
Use skillopt/envs/_template/ as a starting point. At minimum, implement:

-      
    
-        
  1. Dataloader — read your item JSON into the framework's item dicts (dataloader.py).
    2. 
-        
  2. Rollout — run the target on one item with the current skill and return a trajectory + score (rollout.py).
    3. 
-        
  3. Evaluator — score predictions against ground truth (evaluator.py).
    4. 
-        
  4. Adapter — wire the above into the trainer's expected interface and register the env name (adapter.py).
    5. 
-      

-      
Then add a configs/<name>/default.yaml inheriting _base_/default.yaml and set env.name to your new benchmark.

-    

-
-    
-    

-      
8.1 Module Map #

-      

-        
-        
-          
-          
-          
-          
-          
-          
-          
-        
-      
ModuleResponsibility
skillopt/config.pyLoad structured YAML, resolve _base_ inheritance, flatten to the trainer's flat dict, apply CLI overrides.
skillopt/engine/trainer.pyReflACTTrainer — orchestrates the whole loop, gating, slow update, meta skill, resume, and artifact writing.
skillopt/gradient/Reflection ("backward pass"): reflect.py analysts, aggregate.py patch merging.
skillopt/optimizer/The "optimizer": edit application, learning-rate scheduling, edit selection, slow update, meta skill, rewrite modes.
skillopt/evaluation/gate.pyPure accept/reject decision and metric selection.
skillopt/model/Backend clients (OpenAI/Azure, Claude, Qwen, Codex/Claude-Code exec) and routing.
skillopt/envs/<b>/Per-benchmark dataloader, rollout, evaluator, adapter.

-    

-
-    

-      
8.2 Core Functions #

-      

-        
-        
-          
-          
-          
-          
-          
-          
-          
-          
-          
-          
-          
-          
-        
-      
FunctionFilePurpose
load_config / flatten_config / apply_overridesconfig.pyLoad YAML with inheritance; flatten sections; apply key=value overrides.
run_minibatch_reflectgradient/reflect.pyRun error/success analysts over trajectory minibatches → edit patches.
merge_patchesgradient/aggregate.pyHierarchically merge semantically similar patches.
rank_and_selectoptimizer/clip.pyRank edits and clip to the learning-rate budget.
build_scheduleroptimizer/scheduler.pyConstruct the LR (edit-budget) scheduler: constant/linear/cosine/autonomous.
decide_autonomous_learning_rateoptimizer/lr_autonomous.pyLet the optimizer pick the next learning rate (autonomous mode).
apply_patch / apply_editoptimizer/skill.pyApply edits to the skill document (respecting the protected region).
rewrite_skill_from_suggestionsoptimizer/rewrite.pyFull-rewrite update mode from accumulated suggestions.
evaluate_gate / select_gate_scoreevaluation/gate.pyAccept/reject decision; compute hard/soft/mixed score.
run_slow_updateoptimizer/slow_update.pyProduce epoch-boundary longitudinal guidance.
replace_slow_update_field / extract_slow_update_fieldoptimizer/slow_update.pyRead/overwrite the protected guidance region.
run_meta_skill / format_meta_skill_contextoptimizer/meta_skill.pyGenerate cross-epoch optimizer memory and render it into reflection context.

-    

-
-    

-      
8.3 CLI Scripts #

-      
scripts/train.py

-      
Runs a full training loop. Required: --config. Override config via --cfg-options section.key=value … or legacy flat flags (--num_epochs, --batch_size, --optimizer_model, --target_model, --lr_scheduler, --edit_budget, --split_dir, …).

-      
scripts/eval_only.py

-      
Evaluates a skill document without training. Required: --config and --skill. Use --split to choose train / valid_seen / valid_unseen / all.

-
python scripts/eval_only.py \
-  --config configs/searchqa/default.yaml \
-  --skill outputs/my_run/best_skill.md \
+# Load model credentials first, then:
+python scripts/train.py --config configs/searchqa/default.yaml

+    
The run directory contains best_skill.md,
+    runtime_state.json, history.json, versioned files
+    under skills/, and step-level artifacts. Re-running with the
+    same output root resumes from persisted state.

+
python scripts/eval_only.py \
+  --config configs/searchqa/default.yaml \
+  --skill outputs/<run>/best_skill.md \
   --split valid_unseen

-    

+    

+      Reproduction boundary
+      Use the released train/validation/test manifests and record the exact
+      model deployment, config, seed, and source revision. Provider behavior
+      can change independently of this repository.
+    

+    
Continue with the
+    first-experiment guide
+    and dataset manifest documentation.

+
-
-

8.4 WebUI #

-

An optional Gradio dashboard to configure parameters and monitor runs:

-
pip install -e ".[webui]"
-python -m skillopt_webui.app          # http://localhost:7860
-python -m skillopt_webui.app --share  # public share link
-
- +
+

Research model backends

+
+
FlagDefaultDescription
+ - - - + + + + + + + -
BackendOptimizerTargetNotes
--port7860Server port.
--host0.0.0.0Bind address.
--shareoffCreate a public Gradio share link.
openai_chatYesYesAzure OpenAI plus its explicit authentication modes.
openai_compatibleYesYesProvider-neutral chat-completions endpoint.
claude_chatYesYesRuns an installed, authenticated Claude Code CLI via claude -p; not a direct Anthropic API client.
qwen_chatYesYesLocal or hosted Qwen-compatible server.
minimax_chatYesYesMiniMax chat endpoint.
codex_execNoSupported adapters onlyExecutes Codex as a target agent.
claude_code_execNoSupported adapters onlyExecutes Claude Code as a target agent.
-
+ +
+

Prefer the structured model.optimizer_backend and + model.target_backend settings. Legacy --backend + aliases do not expose every role-specific combination. Exec backends are + not generic chat replacements and require adapter support.

+ -
-

9.1 SkillOpt-Sleep — the deployment-time companion (preview) #

-

SkillOpt-Sleep applies SkillOpt's discipline to your own daily usage. It gives a - local coding agent a nightly sleep cycle that reviews your past sessions, replays your - recurring tasks on your own API budget, and consolidates what it learns into validated - long-term memory and skills — behind a held-out gate, staged for your review. The agent gets better - the more you use it, with no weight training and zero inference-time overhead. It is an early - preview we are actively iterating on; interfaces and defaults may change.

-

One "night":

-
harvest Claude Code / Codex transcripts → mine recurring tasks → replay offline
-   → consolidate (reflect → bounded edit → GATE on real held-out tasks)
-   → stage proposal → (you) adopt
-

The engine lives in the top-level skillopt_sleep/ package with zero dependency - on the paper's skillopt/ experiment code (the validation gate is vendored). Deterministic - proof, no API key required: - python -m skillopt_sleep.experiments.run_experiment --persona researcher --assert-improves.

- -

9.2 Plugins (three agents) #

-

One engine, thin per-agent shells (see plugins/):

-
- +
+

Research documentation map

+
+
PlatformFolderInstall
+ - - - + + + + + + + -
ReferenceUse it for
Claude Codeplugins/claude-code/plugin marketplace add ./plugins/claude-code → /skillopt-sleep
Codexplugins/codexbash plugins/codex/install.sh → skillopt-sleep skill
Copilotplugins/copilotregister plugins/copilot/mcp_server.py as an MCP server
ConfigurationAuthentication, structured YAML, role-specific backends, and overrides.
CLICurrent train/eval entry points and exact output paths.
Config referenceSupported sections, defaults, and validation constraints.
Training loopRollout, reflection, edit selection, gating, slow update, and meta skill.
Skill documentSkill structure and protected regions.
Python APIStable public imports and low-level/internal boundaries.
ChangelogRecently merged capabilities, fixes, and contributor credits.
-

Transcript source and replay backend are separate knobs: --source claude for Claude Code - transcripts, --source codex for Codex Desktop archived sessions under - ~/.codex/archived_sessions, and --backend codex only when you want the - replay/optimizer to spend Codex budget.

+ + +
-

9.3 Experience replay & dream rollouts (opt-in) #

-

Two consolidation mechanisms, both default off (so behavior is unchanged unless - enabled). They strengthen the nightly update when your tasks have a clean correctness signal; the - validation gate still governs what ships.

-
- +
+

SkillOpt-Sleep: safe first run

+

SkillOpt-Sleep is a preview deployment companion. Its default + mock backend is useful for testing control flow without API + spend; it is not evidence that a real model's quality improved.

+
# Deterministic engine proof; no model credentials required
+python -m skillopt_sleep.experiments.run_experiment \
+  --persona researcher --assert-improves
+
+# Inspect local session handling without adopting any update
+skillopt-sleep dry-run \
+  --project "$PWD" \
+  --source auto \
+  --backend mock
+
+# A real run: explicitly identify the skill to evolve
+skillopt-sleep run \
+  --project "$PWD" \
+  --target-skill-path path/to/SKILL.md \
+  --source auto \
+  --backend claude
+
+skillopt-sleep status --project "$PWD"
+skillopt-sleep adopt --project "$PWD"
+

--project scopes collection but does not automatically + choose a project's skill file. Use --target-skill-path when + you intend to evolve a particular SKILL.md. Transcript source + (claude, codex, or auto) and replay + backend are independent settings.

+

For subscription-based workflows that should not launch an API or model + subprocess, use --backend handoff and follow the generated + prompt/answer loop. Read the + complete Sleep guide + before a real run.

+
+ +
+

Agent integrations

+
+
Config knobDefaultEffect
+ - - - + + + + + -
AgentIntegration statusGuide
dream_rollouts1Run each task K times and learn from the good-vs-bad contrast (contrastive reflection).
recall_k0Associative recall — pull the K most-similar past tasks (from a persisted archive) into tonight's dream.
dream_factor0Add N lightweight synthetic variants of each task.
Claude CodeShared-engine plugin and handoff commandREADME
CodexShared-engine skill shellREADME
GitHub CopilotShared-engine Sleep MCP plus a separate research MCPREADME
DevinShared-engine MCP with Devin transcript conversionREADME
OpenClawIndependent community/reference adaptation; review locally before useREADME
-

On a clean-signal benchmark the gain scales with recall depth (deployment protocol: 5 nights × - 10 new real tasks/night, full held-out test, GPT-5.5, gated): recall_k=10 → +3.1 pts, - recall_k=20 → +4.5 pts, full-history replay reference → +5.6 pts; a second benchmark - (SpreadsheetBench, GPT-5.4-nano, gate-free) gives +3.6 pts. On saturated or noisy tasks the effect is - flat within run-to-run noise (±1–2 pts). Keep the gate on; it bounds the downside.

+ + +

The plugin overview + records which integrations use the shared engine and which require local + adaptation.

+ -
- SkillOpt — Executive Strategy for Self-Evolving Agent Skills · - github.com/microsoft/SkillOpt · - arXiv:2605.23904
- This guide reflects the current configuration defaults in configs/_base_/default.yaml. When in doubt, the code is the source of truth. -
- +
+

Advanced Sleep controls

+

The main CLI exposes project/source selection, backend/model selection, + bounded task and edit counts, preferences, reviewed task files, and + staged adoption. Additional JSON configuration fields include:

+
+ + + + + + + +
FieldDefaultStatus
dream_rollouts1Single rollout by default; values above 1 enable experimental contrastive replay.
dream_factor0Synthetic task variants are off by default.
recall_k0Historical associative recall is off by default.
+
+

These are configuration fields, not current skillopt-sleep run + flags. Treat multi-rollout, recall, synthetic dreaming, and experimental + reward/budget controls as advanced features that require task-specific + validation. The reported experiments and their exact settings are in + RESULTS.md.

+
- +
+

Data, privacy, and adoption safety

+
    +
  • Real Sleep backends may send session-derived prompts, mined tasks, + trajectories, and candidate edits to the selected provider. Review the + source data and provider policy before use.
    • +
  • Secret redaction for persisted diagnostics is defense in depth; it + is not a guarantee that every outbound model prompt is free of sensitive + content. In particular, do not treat raw coding-agent transcripts as + pre-sanitized.
    • +
  • Updates are staged for review by default. Use + --auto-adopt only when you have an independent rollback and + validation process.
    • +
  • A held-out gate reduces regressions on its measured tasks; it is not + a security boundary or a proof of general improvement.
    • +
  • Use a temporary clone and synthetic transcripts when validating a + new backend or plugin integration.
    • +
+
- - +
+

Contributing and extending

+

Before proposing a change, run the focused tests for the affected area, + then the full suite where practical. Documentation changes should pass a + strict MkDocs build and should be checked against actual CLI + --help output.

+
python -m pip install -e ".[dev,docs]"
+python -m pytest -q
+python -m mkdocs build --strict
+

See CONTRIBUTING.md, + the documentation workflow, + and the focused guides for + benchmarks + and model backends.

+
+
+ SkillOpt · github.com/microsoft/SkillOpt + · arXiv:2605.23904
+ This public overview intentionally avoids duplicating the complete, + fast-changing configuration surface. Follow the linked versioned + references for details. +
+ - - diff --git a/docs/index.md b/docs/index.md index 2dfe6da..9bc1968 100644 --- a/docs/index.md +++ b/docs/index.md @@ -18,6 +18,19 @@ hide: --- +## Two Complementary Workflows + +| Workflow | Package / command | Use it for | +|---|---|---| +| **Research engine** | `skillopt`, `skillopt-train`, `skillopt-eval` | Train and evaluate skill documents on explicit benchmark splits. | +| **SkillOpt-Sleep (preview)** | `skillopt_sleep`, `skillopt-sleep` | Review supported coding-agent sessions and stage proposed memory/skill updates for human adoption. | + +They share the idea of bounded text updates and validation, but they are +separate entry points with different configs and safety boundaries. Start with +the [SkillOpt-Sleep overview](sleep/README.md) before using real session data. + +--- + ## How It Works
@@ -106,29 +119,52 @@ SkillOpt brings the familiar deep-learning training paradigm to agentic prompt o | **ALFWorld** | Embodied AI | `configs/alfworld/` | | **OfficeQA** | Enterprise QA | `configs/officeqa/` | | **SearchQA** | Open-domain QA | `configs/searchqa/` | -| **LiveMathBench** | Math reasoning | `configs/livemathematicianbench/` | -| **SWEBench** | Software Engineering | `configs/swebench/` | -| + 5 more | Various | See [docs](guide/first-experiment.md) | +| **LiveMathematicianBench** | Math reasoning | `configs/livemathematicianbench/` | +| **SpreadsheetBench** | Spreadsheet editing | `configs/spreadsheetbench/` | + +--- + +## Model Backends + +Optimizer and target roles are configured separately. Chat backends include +Azure OpenAI (`openai_chat`), the provider-neutral +`openai_compatible` backend, the Claude Code CLI (`claude_chat`), Qwen, and +MiniMax. Codex and Claude Code exec harnesses are target-only and require +adapter support. Despite its name, `claude_chat` launches `claude -p`; it is +not a direct Anthropic API client. + +If a provider implements OpenAI Chat Completions, begin with the +[built-in compatible backend](guide/new-backend.md#built-in-the-generic-openai-compatible-backend) +instead of adding a new integration. See [Configuration](guide/configuration.md) +for authentication and per-role overrides. --- ## Quick Example ```bash -# Install -pip install -e . +# Clone and install the research checkout plus the SearchQA data extra +git clone https://github.com/microsoft/SkillOpt.git +cd SkillOpt +python -m pip install -e ".[searchqa]" -# Configure credentials -export AZURE_OPENAI_ENDPOINT="https://your-resource.openai.azure.com/" -export AZURE_OPENAI_API_KEY="your-key" +# Configure credentials (choose one auth mode in .env) +cp .env.example .env +set -a; source .env; set +a -# Train on SearchQA -python scripts/train.py --config configs/searchqa/default.yaml +# Materialize the runnable split from the checked-in ID manifest +python scripts/materialize_searchqa.py + +# Train on SearchQA into a predictable output directory +python scripts/train.py \ + --config configs/searchqa/default.yaml \ + --out_root outputs/searchqa_quickstart # Evaluate best skill python scripts/eval_only.py \ --config configs/searchqa/default.yaml \ - --skill outputs/best_skill.md + --skill outputs/searchqa_quickstart/best_skill.md \ + --split valid_unseen ``` --- @@ -167,4 +203,13 @@ python scripts/eval_only.py \ [:octicons-arrow-right-24: WebUI Guide](guide/first-experiment.md#webui) +- :material-weather-night:{ .lg .middle } **SkillOpt-Sleep** + + --- + + Test the deployment companion with the no-provider mock path, then review + its data boundary before selecting a real backend. + + [:octicons-arrow-right-24: Sleep Overview](sleep/README.md) +
diff --git a/docs/reference/api.md b/docs/reference/api.md index 8e364c7..b9fd6e3 100644 --- a/docs/reference/api.md +++ b/docs/reference/api.md @@ -17,7 +17,9 @@ browse [`skillopt/envs/`](https://github.com/microsoft/SkillOpt/tree/main/skillo `skillopt/envs/base.py` — abstract adapter that connects the SkillOpt trainer to an environment (benchmark, simulator, REST API, ...). -Subclasses **must** implement the five abstract methods below. +Subclasses **must** implement the four abstract methods below. Reflection has a +shared default implementation and only needs to be overridden for +environment-specific behavior. ```python from abc import ABC, abstractmethod @@ -30,6 +32,10 @@ class EnvAdapter(ABC): def setup(self, cfg: dict) -> None: ... def get_dataloader(self) -> BaseDataLoader | None: ... def requires_ray(self) -> bool: ... # default False + def reflect(self, results: list[dict], skill_content: str, + out_dir: str, **kwargs) -> list[dict | None]: + """Delegate to the shared minibatch reflection pipeline.""" + ... # ── Abstract methods (subclasses MUST implement) ──────────────────── @@ -53,26 +59,21 @@ class EnvAdapter(ABC): It MAY contain env-specific extra keys (parsed into RolloutResult.extras). """ - @abstractmethod - def reflect(self, results: list[dict], skill_content: str, - out_dir: str, **kwargs) -> list[dict | None]: - """Turn rollout results into a list of raw patch dicts. - - Each dict (or None to drop the slot) MUST contain: - - "patch": {"edits": [...]} a Patch.to_dict() payload - - "source_type": "failure" | "success" - """ - @abstractmethod def get_task_types(self) -> list[str]: """Distinct task-type strings used for stratified sampling.""" ``` -The trainer also calls a few default-implemented helpers on every adapter: +The default `reflect()` delegates to `run_minibatch_reflect` and returns raw +patch dicts with a `patch` payload plus a `failure` or `success` source type. +It expects each rollout to persist a non-empty trajectory at +`/predictions//conversation.json`; results without that +file can be scored but are skipped during reflection. +The trainer also calls several default-implemented helpers on every adapter: `build_reference_text`, `get_reference_metadata`, `attach_reference_context`, `select_representative_items`, and `build_env_from_batch`. Read the docstrings in `skillopt/envs/base.py` if you need to override any of these — most -benchmarks don't. +benchmarks do not. ### `BaseDataLoader` / `SplitDataLoader` @@ -161,16 +162,17 @@ into `RolloutResult.extras`. ### `GateResult` / `GateAction` `skillopt/evaluation/gate.py` — the validation-gate decision types -returned each epoch. +returned for each candidate optimization step, and optionally for a separate +epoch-end slow-update candidate. --- ## Registering an environment Environments are not registered via decorators or a `BENCHMARK_REGISTRY` -dict. The trainer keeps a lazy registry inside `scripts/train.py` — -`_ENV_REGISTRY` — populated by `_register_builtins()`. To add a new env -you append a `try / except ImportError` block there. See +dict. The training and standalone-evaluation entry points each keep a lazy +`_ENV_REGISTRY`, populated by `_register_builtins()` in `scripts/train.py` and +`scripts/eval_only.py`. Add the environment to both entry points. See [Add a New Benchmark](../guide/new-benchmark.md) for the full step-by-step. --- @@ -187,6 +189,7 @@ not via a base class subclass. Supported values (as of this writing): | `claude_chat` | ✓ | ✓ | | `qwen_chat` | ✓ | ✓ | | `minimax_chat` | ✓ | ✓ | +| `openai_compatible` | ✓ | ✓ | | `codex_exec` | — | ✓ | | `claude_code_exec` | — | ✓ | diff --git a/docs/reference/cli.md b/docs/reference/cli.md index 24b5325..e4d277a 100644 --- a/docs/reference/cli.md +++ b/docs/reference/cli.md @@ -1,9 +1,17 @@ # CLI Reference +> **Version note.** This reference tracks `main`. PyPI 0.2.0 does not yet +> include the generic research `openai_compatible` backend, Sleep handoff, +> Sleep support for non-Azure OpenAI-compatible endpoints, or the Sleep +> `--preferences` flag; use a source install from `main` for those features +> until the next release. + ## Training ```bash python scripts/train.py --config [overrides...] +# Installed equivalent: +skillopt-train --config [overrides...] ``` ### Arguments @@ -11,13 +19,15 @@ python scripts/train.py --config [overrides...] | Argument | Description | |---|---| | `--config` | Path to YAML config file (required) | -| `key=value` | Override any config parameter | +| `--cfg-options key=value [...]` | Override structured config parameters | ### Examples ```bash # Basic training -python scripts/train.py --config configs/searchqa/default.yaml +python scripts/train.py \ + --config configs/searchqa/default.yaml \ + --out_root outputs/searchqa_run # With overrides python scripts/train.py \ @@ -34,6 +44,8 @@ python scripts/train.py \ ```bash python scripts/eval_only.py --config --skill +# Installed equivalent: +skillopt-eval --config --skill ``` ### Arguments @@ -42,7 +54,8 @@ python scripts/eval_only.py --config --skill |---|---| | `--config` | Path to YAML config file (required) | | `--skill` | Path to skill document to evaluate (required) | -| `--split` | Evaluation split: `test` (default), `valid`, `train` | +| `--split` | `train`, `valid_seen`, `valid_unseen`, or `all` (default) | +| `--cfg-options` | One or more `section.key=value` overrides | ### Examples @@ -50,15 +63,64 @@ python scripts/eval_only.py --config --skill # Evaluate best skill on test set python scripts/eval_only.py \ --config configs/searchqa/default.yaml \ - --skill outputs/searchqa/run_001/skills/best_skill.md + --skill outputs/searchqa_run/best_skill.md \ + --split valid_unseen # Evaluate on validation set python scripts/eval_only.py \ --config configs/searchqa/default.yaml \ - --skill outputs/searchqa/run_001/skills/best_skill.md \ - --split valid + --skill outputs/searchqa_run/best_skill.md \ + --split valid_seen ``` +`--skill` consumes the artifact produced by training. Unless `--out_root` is +set for evaluation, `eval_only.py` creates a separate timestamped +`outputs/eval___/` directory and writes +`eval_summary.json` there; it does not modify the training run directory. + +For the generic OpenAI-compatible research backend, select the role backends +explicitly: + +```bash +python scripts/train.py \ + --config configs/searchqa/default.yaml \ + --cfg-options \ + model.optimizer_backend=openai_compatible \ + model.target_backend=openai_compatible \ + model.optimizer=deepseek-chat \ + model.target=deepseek-chat +``` + +## SkillOpt-Sleep + +```bash +skillopt-sleep [options] +# Equivalent from a source checkout: +python -m skillopt_sleep [options] +``` + +Actions are `run`, `dry-run`, `status`, `adopt`, `harvest`, `schedule`, and +`unschedule`. Common options include: + +| Argument | Description | +|---|---| +| `--project PATH` | Project to evolve (default: current directory) | +| `--scope invoked\|all` | Harvest this project or all projects | +| `--source claude\|codex\|auto` | Transcript source | +| `--backend mock\|claude\|codex\|copilot\|handoff\|azure_openai` | Replay/optimizer backend | +| `--model NAME` | Backend-specific model override | +| `--preferences TEXT` | House rules supplied to reflection | +| `--lookback-hours N` | Initial transcript lookback; `0` scans all history | +| `--max-sessions N` / `--max-tasks N` | Bound the harvested workload | +| `--target-skill-path PATH` | Explicit skill document to stage/adopt | +| `--tasks-file PATH` | Replay a reviewed task JSON file instead of harvesting | +| `--edit-budget N` | Maximum bounded edits for the night | +| `--progress` / `--json` | Progress or machine-readable output | +| `--auto-adopt` | Apply an accepted staged proposal automatically | + +Backend-specific setup for compatible endpoints is documented in +[OpenAI-compatible endpoints for SkillOpt-Sleep](../sleep/openai-compatible-endpoints.md). + ## WebUI ```bash @@ -68,4 +130,8 @@ python -m skillopt_webui.app [--port PORT] [--share] | Argument | Default | Description | |---|---|---| | `--port` | 7860 | Port number | +| `--host` | `0.0.0.0` | Server bind address | | `--share` | false | Create public Gradio link | + +The default host binds every network interface. Use `--host 127.0.0.1` when +the dashboard should be reachable only from the local machine. diff --git a/docs/reference/config.md b/docs/reference/config.md index 0b39bd0..d057cf1 100644 --- a/docs/reference/config.md +++ b/docs/reference/config.md @@ -1,85 +1,174 @@ # Configuration Reference -Complete reference for all SkillOpt configuration parameters. +SkillOpt loads structured YAML, resolves `_base_` inheritance, and flattens +the result for the trainer. Shipped defaults live in +`configs/_base_/default.yaml`; benchmark configs override them. -## Model +## Model and Backend Selection + +Use explicit optimizer and target backends when the two roles differ or when +selecting the generic OpenAI-compatible backend. + +| Backend | Optimizer | Target | +|---|:---:|:---:| +| `openai_chat` | ✓ | ✓ | +| `openai_compatible` | ✓ | ✓ | +| `claude_chat` | ✓ | ✓ | +| `qwen_chat` | ✓ | ✓ | +| `minimax_chat` | ✓ | ✓ | +| `codex_exec` | — | ✓ | +| `claude_code_exec` | — | ✓ | + +MiniMax currently has one shared deployment. `model.minimax_model` is applied +when MiniMax is the target; mixed-backend runs cannot independently choose a +MiniMax optimizer model and a different target model. | Parameter | Type | Default | Description | |---|---|---|---| -| `model.backend` | str | `azure_openai` | Backend: `azure_openai` / `openai_chat` / `claude_code_exec` / `qwen` | -| `model.optimizer` | str | `gpt-5.5` | Optimizer model (for reflection & slow update) | -| `model.target` | str | `gpt-5.5` | Target model (for rollout execution) | -| `model.reasoning_effort` | str | `medium` | Reasoning effort level | -| `model.optimizer_backend` | str | `openai_chat` | Optimizer backend: `openai_chat` / `claude_chat` / `qwen_chat` / `minimax_chat` | -| `model.target_backend` | str | `openai_chat` | Target backend: chat backends plus execution harnesses | -| `model.qwen_chat_base_url` | str | `http://localhost:8000/v1` | Shared Qwen/vLLM OpenAI-compatible endpoint | -| `model.qwen_chat_enable_thinking` | bool | `false` | Shared Qwen thinking flag | -| `model.optimizer_qwen_chat_base_url` | str | — | Optimizer-specific Qwen/vLLM endpoint; overrides shared `qwen_chat_base_url` | -| `model.target_qwen_chat_base_url` | str | — | Target-specific Qwen/vLLM endpoint; overrides shared `qwen_chat_base_url` | +| `model.backend` | str | `azure_openai` | Backward-compatible high-level run label | +| `model.optimizer` | str | `gpt-5.5` | Optimizer deployment/model | +| `model.target` | str | `gpt-5.5` | Target deployment/model | +| `model.optimizer_backend` | str | `openai_chat` | Optimizer client path; chat backends only | +| `model.target_backend` | str | `openai_chat` | Target client path; chat or exec backend | +| `model.reasoning_effort` | str | `medium` | Shared reasoning effort | +| `model.rewrite_reasoning_effort` | str | empty | Optional full-rewrite effort override | +| `model.rewrite_max_completion_tokens` | int | `64000` | Full-rewrite output cap | + +### Azure/OpenAI `openai_chat` + +| Parameter | Default | Description | +|---|---|---| +| `model.azure_openai_endpoint` | empty | Shared Azure resource URL or compatibility-mode base URL | +| `model.azure_openai_api_version` | `2024-12-01-preview` | Azure API version | +| `model.azure_openai_api_key` | empty | Key for `api_key` or compatibility auth | +| `model.azure_openai_auth_mode` | empty | Config value; empty falls back to env, whose default is `azure_cli` | +| `model.azure_openai_ad_scope` | Azure Cognitive Services scope | AAD token scope | +| `model.azure_openai_managed_identity_client_id` | empty | Optional user-assigned identity client ID | + +Every shared key also has an `optimizer_azure_openai_*` and +`target_azure_openai_*` form. + +### Claude `claude_chat` + +`claude_chat` launches an installed, authenticated Claude Code CLI with +`claude -p`; it does not instantiate an Anthropic API client. The executable +defaults to `claude` and can be overridden with `CLAUDE_CLI_BIN`. +`ANTHROPIC_API_KEY` is one authentication option understood by the CLI. + +### Qwen, MiniMax, and Exec Backends + +| Parameter family | Description | +|---|---| +| `model.qwen_chat_*` | Shared `base_url`, `api_key`, `temperature`, `timeout_seconds`, `max_tokens`, and `enable_thinking` | +| `model.optimizer_qwen_chat_*` / `model.target_qwen_chat_*` | Per-role Qwen overrides | +| `model.minimax_*` | MiniMax `base_url`, `api_key`, shared `minimax_model`, `temperature`, `max_tokens`, and `enable_thinking`; `minimax_model` applies when MiniMax is the target | +| `model.codex_exec_*` | Codex path, sandbox, profile, SDK mode, reasoning, network/search, and approval policy | +| `model.claude_code_exec_*` | Claude path, profile, SDK mode, effort, and thinking-token cap | ## Training (`train`) -| Parameter | Type | Default | DL Analogy | Description | -|---|---|---|---|---| -| `train.num_epochs` | int | 4 | Epochs | Number of training epochs | -| `train.batch_size` | int | 40 | Batch size | Tasks sampled per step | -| `train.accumulation` | int | 1 | Gradient accumulation | Accumulation rounds per step | -| `train.seed` | int | 42 | Random seed | Reproducibility seed | +| Parameter | Type | Default | Description | +|---|---|---|---| +| `train.num_epochs` | int | `4` | Training epochs | +| `train.train_size` | int | `0` | `0` derives the size from the dataset split | +| `train.steps_per_epoch` | int | derived | Runtime field recomputed from train size, batch size, and accumulation; configured values are overwritten | +| `train.batch_size` | int | `40` | Tasks sampled per step | +| `train.accumulation` | int | `1` | Accumulation rounds per step | +| `train.seed` | int | `42` | Random seed | ## Gradient / Reflection (`gradient`) | Parameter | Type | Default | Description | |---|---|---|---| -| `gradient.minibatch_size` | int | 8 | Reflect minibatch size | -| `gradient.merge_batch_size` | int | 8 | Patch merge batch size | -| `gradient.analyst_workers` | int | 16 | Parallel reflection workers | -| `gradient.max_analyst_rounds` | int | 3 | Max rounds of analyst reflection | -| `gradient.failure_only` | bool | `false` | Only reflect on failures | +| `gradient.minibatch_size` | int | `8` | Reflect minibatch size | +| `gradient.merge_batch_size` | int | `8` | Patch merge batch size | +| `gradient.analyst_workers` | int | `16` | Parallel reflection workers | +| `gradient.max_analyst_rounds` | int | `3` | Maximum analyst rounds | +| `gradient.failure_only` | bool | `false` | Reflect only on failures | ## Optimizer (`optimizer`) -| Parameter | Type | Default | DL Analogy | Description | -|---|---|---|---|---| -| `optimizer.learning_rate` | int | 4 | Learning rate | Max edit patches per step (edit budget) | -| `optimizer.min_learning_rate` | int | 2 | Min LR | Min edits for decay schedulers | -| `optimizer.lr_scheduler` | str | `cosine` | LR schedule | `constant` / `linear` / `cosine` / `autonomous` | -| `optimizer.skill_update_mode` | str | `patch` | — | `patch` / `rewrite_from_suggestions` / `full_rewrite_minibatch` | -| `optimizer.use_slow_update` | bool | `true` | Momentum | Epoch-boundary longitudinal comparison & guidance | -| `optimizer.slow_update_samples` | int | 20 | — | Samples for slow update evaluation | -| `optimizer.use_meta_skill` | bool | `true` | Meta-learning | Cross-epoch optimizer-side strategy memory | -| `optimizer.longitudinal_pair_policy` | str | `mixed` | — | `mixed` / `changed` / `unchanged` | +| Parameter | Type | Default | Description | +|---|---|---|---| +| `optimizer.learning_rate` | int | `4` | Maximum edit patches per step | +| `optimizer.min_learning_rate` | int | `2` | Floor for decaying schedules | +| `optimizer.lr_scheduler` | str | `cosine` | `constant`, `linear`, `cosine`, or `autonomous` | +| `optimizer.lr_control_mode` | str | `fixed` | `fixed`, `autonomous`, or `none` | +| `optimizer.skill_update_mode` | str | `patch` | `patch`, `rewrite_from_suggestions`, or `full_rewrite_minibatch` | +| `optimizer.use_slow_update` | bool | `true` | Epoch-boundary longitudinal update | +| `optimizer.slow_update_samples` | int | `20` | Longitudinal evaluation samples | +| `optimizer.slow_update_gate_with_selection` | bool | `false` | Gate slow-update guidance on the selection split | +| `optimizer.longitudinal_pair_policy` | str | `mixed` | `mixed`, `changed`, or `unchanged` | +| `optimizer.use_meta_skill` | bool | `true` | Cross-epoch optimizer memory | +| `optimizer.use_skill_aware_reflection` | bool | `false` | Enable skill-defect vs execution-lapse routing | +| `optimizer.skill_aware_appendix_source` | str | `both` | `both` or `failure_only` | +| `optimizer.skill_aware_consolidate_threshold` | int | `0` | Appendix compaction threshold; `0` disables it | ## Evaluation (`evaluation`) | Parameter | Type | Default | Description | |---|---|---|---| -| `evaluation.use_gate` | bool | `true` | Enable validation gating (accept/reject updates) | -| `evaluation.eval_test` | bool | `true` | Run test evaluation after training | +| `evaluation.use_gate` | bool | `true` | Accept only improvements when enabled; `false` records validation but force-accepts each candidate | +| `evaluation.gate_metric` | str | `hard` | `hard`, `soft`, or `mixed` | +| `evaluation.gate_mixed_weight` | float | `0.5` | Soft-score weight for `mixed` | +| `evaluation.use_semantic_density` | bool | `false` | Add the optional instruction-density bonus | +| `evaluation.semantic_density_weight` | float | `0.05` | Density bonus weight | +| `evaluation.leading_words` | list/str | built in | Optional custom high-influence words | +| `evaluation.sel_env_num` | int | `0` | Selection size; `0` uses the full split | +| `evaluation.test_env_num` | int | `0` | Test size; `0` uses the full split | +| `evaluation.eval_test` | bool | `true` | Run final test evaluation | ## Environment (`env`) | Parameter | Type | Default | Description | |---|---|---|---| -| `env.name` | str | — | Benchmark name (e.g., `searchqa`, `docvqa`) | -| `env.data_path` | str | — | Path to dataset | -| `env.skill_init` | str | — | Path to initial seed skill (optional) | +| `env.name` | str | empty | Benchmark name | +| `env.skill_init` | str | empty | Initial skill document | | `env.split_mode` | str | `ratio` | `ratio` or `split_dir` | -| `env.split_ratio` | str | `2:1:7` | Train:val:test ratio | -| `env.exec_timeout` | int | 120 | Per-task timeout in seconds | -| `env.out_root` | str | — | Output directory | +| `env.split_ratio` | str | benchmark/default | Train:validation:test ratio | +| `env.split_seed` | int | `42` | Deterministic split seed | +| `env.split_dir` | str | empty | Materialized train/val/test directory | +| `env.data_path` | str | empty | Raw data path for ratio mode | +| `env.split_output_dir` | str | empty | Optional materialized split output | +| `env.exec_timeout` | int | `120` | Per-task timeout in seconds | +| `env.out_root` | str | generated by the train/eval CLIs | Output directory | -## Azure OpenAI Credentials +Benchmark-specific `env` keys are passed through to the adapter. + +## Credential Environment Variables + +### Azure-family backend | Variable | Description | |---|---| -| `AZURE_OPENAI_ENDPOINT` / `model.azure_openai_endpoint` | Azure resource endpoint | -| `AZURE_OPENAI_API_KEY` / `model.azure_openai_api_key` | Azure API key | -| `OPENAI_API_KEY` | OpenAI API key (for `openai_chat` backend) | -| `ANTHROPIC_API_KEY` | Anthropic API key (for `claude_code_exec` backend) | -| `QWEN_CHAT_BASE_URL` | Shared local vLLM endpoint for `qwen_chat` | -| `QWEN_CHAT_MODEL` | Shared served model name for `qwen_chat` | -| `QWEN_CHAT_API_KEY` | Optional API key for the shared Qwen endpoint | -| `OPTIMIZER_QWEN_CHAT_BASE_URL` | Optimizer-specific local vLLM endpoint | -| `OPTIMIZER_QWEN_CHAT_MODEL` | Optimizer-specific served model name | -| `TARGET_QWEN_CHAT_BASE_URL` | Target-specific local vLLM endpoint | -| `TARGET_QWEN_CHAT_MODEL` | Target-specific served model name | +| `AZURE_OPENAI_ENDPOINT` | Shared Azure endpoint or compatibility base URL | +| `AZURE_OPENAI_API_VERSION` | Azure API version | +| `AZURE_OPENAI_AUTH_MODE` | `api_key`, `azure_cli`, `managed_identity`, or `openai_compatible` | +| `AZURE_OPENAI_API_KEY` | Key for `api_key` or `openai_compatible` mode | +| `AZURE_OPENAI_AD_SCOPE` | Optional AAD scope | +| `AZURE_OPENAI_MANAGED_IDENTITY_CLIENT_ID` | Optional managed-identity client ID | + +Use `OPTIMIZER_AZURE_OPENAI_*` and `TARGET_AZURE_OPENAI_*` for role-specific +overrides. + +### Generic OpenAI-compatible backend + +| Variable suffix | Shared / per-role forms | +|---|---| +| `BASE_URL` | `OPENAI_COMPATIBLE_BASE_URL`, `OPTIMIZER_OPENAI_COMPATIBLE_BASE_URL`, `TARGET_OPENAI_COMPATIBLE_BASE_URL` | +| `API_KEY` | Corresponding shared/optimizer/target `*_API_KEY` names | +| `MODEL` | Corresponding shared/optimizer/target `*_MODEL` names | +| `TEMPERATURE` | Corresponding shared/optimizer/target `*_TEMPERATURE` names | +| `MAX_TOKENS` | Corresponding shared/optimizer/target `*_MAX_TOKENS` names | +| `TIMEOUT_SECONDS` | Corresponding shared/optimizer/target `*_TIMEOUT_SECONDS` names | + +The train/eval entry points set deployments from YAML `model.optimizer` and +`model.target` after backend initialization. For selected OpenAI-compatible or +Qwen roles, those values override the corresponding `*_MODEL` environment +variables; the environment model names mainly seed direct library use. + +Other backend families use the authenticated Claude CLI (`CLAUDE_CLI_BIN`; +optionally `ANTHROPIC_API_KEY`), `QWEN_CHAT_*`, and `MINIMAX_*`. +SkillOpt-Sleep's compatible endpoint uses `AZURE_OPENAI_*`, not the research +backend's `OPENAI_COMPATIBLE_*`; see +[the Sleep endpoint guide](../sleep/openai-compatible-endpoints.md). diff --git a/docs/sleep/README.md b/docs/sleep/README.md index b4fd45b..2e36eb1 100644 --- a/docs/sleep/README.md +++ b/docs/sleep/README.md @@ -4,11 +4,11 @@ local coding agent a nightly **sleep cycle** that reviews your past sessions, replays your recurring tasks on your own API budget, and consolidates what it learns into **validated** long-term memory and skills — behind a held-out gate, staged for your -review. The agent gets better the more you use it, with **no weight training** and -**zero inference-time overhead**. +review. It requires **no weight training** and adds no separate optimization loop to +normal agent requests. > **Preview.** This is an early preview we are actively iterating on; interfaces and -> defaults may change. The engine lives in the top-level [`skillopt_sleep/`](../../skillopt_sleep) +> defaults may change. The engine lives in the top-level [`skillopt_sleep/`](https://github.com/microsoft/SkillOpt/tree/main/skillopt_sleep) > package with **zero dependency** on the paper's `skillopt/` code (the validation gate > is vendored). @@ -26,29 +26,49 @@ It synthesizes **SkillOpt** (validation-gated bounded text edits), **Claude Drea (offline consolidation; review-then-adopt), and the **agent-sleep** idea (short-term experience → long-term competence). +> **Data boundary.** Harvesting is local and read-only. The `mock` backend makes no +> provider calls. A real backend, however, sends truncated excerpts from harvested +> sessions and derived tasks to the provider you select for mining, replay, judging, +> and reflection. Outbound prompts are not currently guaranteed to be secret-free; +> review your transcript source and provider policy before running on sensitive +> projects. For a reviewable workflow, harvest to a task file, inspect/redact it, mark +> it `"reviewed": true`, and then replay that file with the real backend. + ## How to use it ### Quickest path: the `skillopt-sleep` CLI (pip) ```bash pip install skillopt # installs the engine + the `skillopt-sleep` command -skillopt-sleep dry-run # harvest + mine + replay, report only (changes nothing) +skillopt-sleep dry-run # harvest + mine + replay, report only; stages nothing skillopt-sleep run # a full nightly cycle; the proposal is staged for review skillopt-sleep status # show state + the latest staged proposal skillopt-sleep adopt # apply the latest staged proposal skillopt-sleep schedule # install a nightly cron entry for this project ``` -The per-agent plugin shells below (Claude Code / Codex / Copilot) still come from the -repo; the CLI above is the standalone, pip-only way to run a cycle. +> **Version note.** This page tracks `main`. PyPI 0.2.0 provides the base +> commands above. Sleep handoff, non-Azure OpenAI-compatible endpoints, and +> `--preferences` landed later and require a source install from `main` until +> the next release. -One engine, thin per-agent shells (see [`plugins/`](../../plugins)): +The per-agent integrations below still come from the repo; the CLI above is the +standalone, pip-only way to run a cycle. Claude Code, Codex, Copilot, and Devin wrap +the shared engine. OpenClaw is a separate reference adaptation and has its own setup. + +One engine, thin per-agent shells (see [`plugins/`](https://github.com/microsoft/SkillOpt/tree/main/plugins)): | Platform | Folder | Install | |---|---|---| -| **Claude Code** | [`plugins/claude-code`](../../plugins/claude-code) | `/plugin marketplace add ./plugins/claude-code` → `/skillopt-sleep` | -| **Codex** | [`plugins/codex`](../../plugins/codex) | `bash plugins/codex/install.sh` → `skillopt-sleep` skill | -| **Copilot** | [`plugins/copilot`](../../plugins/copilot) | register `plugins/copilot/mcp_server.py` as an MCP server | +| **Claude Code** | [`plugins/claude-code`](https://github.com/microsoft/SkillOpt/tree/main/plugins/claude-code) | `/plugin marketplace add ./plugins/claude-code` → `/skillopt-sleep` | +| **Codex** | [`plugins/codex`](https://github.com/microsoft/SkillOpt/tree/main/plugins/codex) | `bash plugins/codex/install.sh` → `skillopt-sleep` skill | +| **Copilot** | [`plugins/copilot`](https://github.com/microsoft/SkillOpt/tree/main/plugins/copilot) | register `plugins/copilot/mcp_server.py` as an MCP server | +| **Devin** | [`plugins/devin`](https://github.com/microsoft/SkillOpt/tree/main/plugins/devin) | register `plugins/devin/mcp_server.py` as an MCP server | +| **OpenClaw** | [`plugins/openclaw`](https://github.com/microsoft/SkillOpt/tree/main/plugins/openclaw) | adapt the reference wrapper and paths for your installation | + +To use DeepSeek, vLLM, Ollama, or another Chat Completions server, see +**[OpenAI-compatible endpoints](openai-compatible-endpoints.md)**. That guide also +documents the separate HTTPS-only boundary for Azure managed-identity credentials. Deterministic proof (no API key): `python -m skillopt_sleep.experiments.run_experiment --persona researcher --assert-improves`. @@ -71,11 +91,12 @@ correctness signal; the validation gate still governs what ships. > scaling, and the dream-diversity ablation — are in > [`docs/sleep/RESULTS.md`](RESULTS.md).** The highlights: -**Protocol (identical for every row below).** 5 nights × 10 new real "today" tasks -per night; the full held-out **test** split is scored before night 1 (baseline) and -after night 5 (after); optimizer = GPT-5.5; single seed (42); run through the exact -shipped engine (`skillopt_sleep.dream.dream_consolidate`). Numbers are absolute -held-out accuracy; **Δ** = `after − baseline` in percentage points. +**Controlled experiment recipe (not the shipping CLI defaults).** 5 nights × 10 new +real "today" tasks per night; the full held-out **test** split is scored before night +1 (baseline) and after night 5 (after); optimizer = GPT-5.5; single seed (42). The +experiments use the shipped consolidation and gate components, while the nightly CLI +and benchmark harnesses remain separate entry points. Numbers are absolute held-out +accuracy; **Δ** = `after − baseline` in percentage points. **(a) End-to-end on real agents — [gbrain-evals](https://github.com/garrytan/gbrain-evals) `skillopt-v1`.** Deficient seed skills go **0.00 → 1.00** on the held-out set with **both Claude Code @@ -106,5 +127,6 @@ gate keeps the worst case bounded; keep it **on** by default. ## Learn more -Full reference (pipeline, the three plugins, the experience-replay knobs) is in the -**[Documentation & Reproduction Guide](https://microsoft.github.io/SkillOpt/docs/guideline.html#sleep)**. +See the [SkillOpt documentation index](../index.md), the +[CLI reference](../reference/cli.md), and the integration-specific READMEs under +[`plugins/`](https://github.com/microsoft/SkillOpt/tree/main/plugins). diff --git a/docs/sleep/RESULTS.md b/docs/sleep/RESULTS.md index c43d686..441ffa2 100644 --- a/docs/sleep/RESULTS.md +++ b/docs/sleep/RESULTS.md @@ -2,8 +2,9 @@ This is the evidence behind SkillOpt-Sleep: does a nightly, offline sleep cycle actually make a *deployed* agent better, and is it safe to run unattended? We -answer with a controlled deployment-scale study — the same protocol the plugin -runs in production, scored on full held-out test sets. +answer with a controlled deployment-scale study built from the same shipped +consolidation and gate components. Its multi-night benchmark recipe is an +experiment configuration, not the default configuration of the nightly CLI. ## Setup @@ -11,9 +12,10 @@ runs in production, scored on full held-out test sets. **10 new real "today" tasks**; the skill carries over and is refined night to night. The full held-out **test** split is scored before night 1 (*baseline*) and after night 5 (*after*); **Δ = after − baseline** in percentage points. Optimizer -model = **GPT-5.5**; single seed (42); every number is produced by the exact -shipped engine `skillopt_sleep.dream.dream_consolidate` (the experiment harness and -the plugin cycle call the same function). +model = **GPT-5.5**; single seed (42). The measurements use the shipped replay, +consolidation, and gate implementations. The nightly CLI and the checked-in +benchmark convenience harnesses are separate entry points and do not all call one +shared wrapper function. **Benchmarks** (real evaluators, not format heuristics): @@ -106,27 +108,31 @@ Replay-policy ablation (SearchQA, GPT-5.5): | Replay policy | Gate-free Δ | Gated Δ | |---|---|---| | none (tonight's tasks only) | +3.9 | +2.0 | -| **recall k=10 (shipped default-able)** | +5.1 | +4.4 | +| **recall k=10 (opt-in experiment)** | +5.1 | +4.4 | | cumulative (full history) | +4.8 | +6.0 | Recall captures most of cumulative's benefit at a fraction of the per-night cost. --- -## 4. Default hyperparameters are the sweet spot +## 4. Sensitivity around the experiment recipe We swept `dream_factor`, `rollouts`, `per_night`, and `nights` on the nano cell -(SearchQA, gated) to verify the shipped defaults are well-tuned: +(SearchQA, gated) around the study recipe: `dream_factor=2`, `rollouts=5`, +`per_night=10`, and `nights=5`. These are **experiment values**, not the shipping +defaults (`dream_factor=0`, `dream_rollouts=1`, and `recall_k=0`): -| Variant | Δ | vs default (+11.9) | +| Variant | Δ | vs experiment baseline (+11.9) | |---|---|---| -| dream_factor=4 (default 2) | +8.8 | −3.1 | -| rollouts=10 (default 5) | +9.5 | −2.4 | -| per_night=15 (default 10) | +2.7 | −9.2 | -| nights=8 (default 5) | +9.5 | −2.4 | +| dream_factor=4 (baseline 2) | +8.8 | −3.1 | +| rollouts=10 (baseline 5) | +9.5 | −2.4 | +| per_night=15 (baseline 10) | +2.7 | −9.2 | +| nights=8 (baseline 5) | +9.5 | −2.4 | -Every direction away from the default hurts. This means users get the best result -**out of the box** without tuning — the recipe is robust by design. +Every tested direction away from that baseline reduced the measured gain in this +cell. The result supports that particular study recipe; it does not establish a +universal optimum. Shipping stays conservative, and users must opt in to additional +dream rollouts or recall after considering task quality and provider cost. --- @@ -143,7 +149,7 @@ gains in Sections 1–2. Measured across an 18-cell deployment sweep (3 benchmar |---|---|---|---|---| | single-sample reflection (degraded) | −2.66 | **−52.8** | 7 / 18 | 5 / 18 | | diverse rollouts (K=5), no recall | +0.24 | −4.0 | 6 / 18 | 7 / 18 | -| **diverse rollouts + recall (shipped)** | **+0.53** | **−2.4** | 7 / 18 | 7 / 18 | +| **diverse rollouts + recall (experiment recipe)** | **+0.53** | **−2.4** | 7 / 18 | 7 / 18 | The catastrophic −52.8 is removed **at its source** by diverse rollouts: the same gate-free nano-SearchQA cell goes 0.554 → **0.586 (+2.7)** with no gate at all once @@ -182,4 +188,4 @@ cross-verify each other's consolidated skills. --- Back to the module overview: [`docs/sleep/README.md`](README.md) · -full reference: [Documentation & Reproduction Guide](https://microsoft.github.io/SkillOpt/docs/guideline.html#sleep). +documentation index: [SkillOpt documentation](../index.md). diff --git a/docs/sleep/openai-compatible-endpoints.md b/docs/sleep/openai-compatible-endpoints.md index 831a2ed..f925a18 100644 --- a/docs/sleep/openai-compatible-endpoints.md +++ b/docs/sleep/openai-compatible-endpoints.md @@ -1,11 +1,16 @@ # OpenAI-compatible endpoints for SkillOpt-Sleep (DeepSeek, local vLLM, …) -This document describes an enhancement to the `azure_openai` backend in -`skillopt_sleep/backend.py` that lets SkillOpt-Sleep drive **any -OpenAI-compatible chat-completions endpoint** — for example DeepSeek's hosted +This document describes the `azure_openai` backend in +`skillopt_sleep/backend.py`, which can drive servers that implement the expected +OpenAI-compatible Chat Completions request shape — for example DeepSeek's hosted API or a self-hosted vLLM/Ollama server — in addition to native Azure OpenAI -deployments. It also documents a concrete end-to-end integration: running the -nightly sleep cycle inside the Antigravity IDE against DeepSeek. +deployments. The included runner is a sanitized unattended-launch example that +was originally used alongside Antigravity; it is not an Antigravity transcript +integration. + +> **Version requirement.** This capability landed after v0.2.0. Until the next +> release, install SkillOpt from the latest `main`; the current PyPI 0.2.0 +> package does not provide this compatible-endpoint path. ## What changed @@ -32,15 +37,17 @@ is unchanged: every rollout `0.0` with no diagnostic.) 4. **Managed-identity credential guard.** The managed-identity path attaches an - Azure AD bearer token to every request. If a custom endpoint outside - `*.openai.azure.com` / `*.cognitiveservices.azure.com` is configured without - explicit compat auth, the backend now raises a clear `ValueError` instead of - sending Azure credentials to an arbitrary host. + Azure AD bearer token to every request. It therefore accepts only an **HTTPS** + endpoint whose hostname ends in `*.openai.azure.com` or + `*.cognitiveservices.azure.com`. An HTTP endpoint — even one with an + Azure-looking hostname — and any host outside those suffixes are rejected + before a credential-bearing client is created. 5. **Provider-neutral request shape.** In compat mode the backend sends only the standard OpenAI-compatible contract (`model`, `messages`, `max_tokens`). Provider-specific request fields are **opt-in** via environment variables - (below) — nothing is inferred from model-name substrings. + (below) and are attached only in compat mode — nothing is inferred from + model-name substrings, and the native Azure request remains unchanged. 6. **Reliable error state.** `_call()` records the last exception in `self.last_call_error` (surfaced in `diagnostics.json`), clears it when a @@ -57,16 +64,34 @@ sleep cycle): | Variable | Meaning | |---|---| | `AZURE_OPENAI_AUTH_MODE` | `openai_compatible` (or `compat`/`openai`) selects the plain OpenAI client. Unset/other = Azure managed identity (default). | -| `AZURE_OPENAI_ENDPOINT` | Base URL of the server, e.g. `https://api.deepseek.com`. | -| `AZURE_OPENAI_API_KEY` | API key sent by the compat client. | +| `AZURE_OPENAI_ENDPOINT` | Base URL of the server, e.g. `https://api.deepseek.com`. Azure managed identity requires HTTPS plus an approved Azure hostname. | +| `AZURE_OPENAI_API_KEY` | API key sent by the compat client to the configured base URL. | | `SKILLOPT_SLEEP_COMPAT_MAX_TOKENS` | Optional int (default `8192`): `max_tokens` sent in compat mode. | -| `SKILLOPT_SLEEP_CHAT_EXTRA_BODY` | Optional JSON object passed as `extra_body` for provider-specific fields. | +| `SKILLOPT_SLEEP_CHAT_EXTRA_BODY` | Optional JSON object passed as `extra_body` for provider-specific fields in compat mode only. It is ignored in native Azure mode. | + +## Data and transport boundaries + +- Harvesting reads local transcripts without modifying them, and the `mock` + backend makes no provider calls. A real backend sends **truncated transcript + excerpts and derived task content** to the selected provider for mining, + replay, judging, and reflection. +- Outbound prompts are not currently guaranteed to be free of secrets. Review + the provider's data policy and avoid a third-party endpoint for sensitive + transcripts unless you have first inspected and redacted the task material. + One reviewable path is `skillopt-sleep harvest --output tasks.json`, followed + by a reviewed `--tasks-file` run. +- Use HTTPS for every remote compatible provider. Plain HTTP is appropriate only + for an explicitly trusted loopback development server such as + `http://127.0.0.1:8000/v1`; the compat client sends its API key to the configured + URL. +- Azure managed-identity credentials have the stricter invariant described + above: HTTPS **and** an approved Azure hostname are both mandatory. ## How to use it ```bash export AZURE_OPENAI_AUTH_MODE=openai_compatible -export AZURE_OPENAI_ENDPOINT=https://api.deepseek.com # no /v1, no trailing path +export AZURE_OPENAI_ENDPOINT=https://api.deepseek.com # DeepSeek base URL export AZURE_OPENAI_API_KEY=sk-... # your provider key # DeepSeek reasoning models: enable the thinking channel (opt-in, not inferred) @@ -79,37 +104,49 @@ skillopt-sleep run \ --project /path/to/your/project ``` -The same pattern works for any OpenAI-compatible server — point -`AZURE_OPENAI_ENDPOINT` at it, set a matching `--model`, and omit -`SKILLOPT_SLEEP_CHAT_EXTRA_BODY` unless your provider needs extra request -fields. +The same pattern works for a server that implements this Chat Completions +contract: point `AZURE_OPENAI_ENDPOINT` at the provider-specific base URL, set a +matching `--model`, and omit `SKILLOPT_SLEEP_CHAT_EXTRA_BODY` unless the provider +needs extra request fields. Self-hosted vLLM and Ollama commonly use a `/v1` base +path, for example `http://127.0.0.1:8000/v1` or +`http://127.0.0.1:11434/v1`. -## End-to-end integration: Antigravity + DeepSeek +`--project` selects the project/transcript scope and the project `CLAUDE.md`; it +does **not** by itself select an arbitrary project `SKILL.md`. Pass +`--target-skill-path path/to/SKILL.md` when a specific skill is the optimization +target. Without that flag, SkillOpt-Sleep uses its configured managed skill. -The [`examples/`](examples/) directory contains a sanitized reference of how this -was wired into the [Antigravity](https://antigravity.google/) agent IDE so the -sleep cycle runs unattended: +## Unattended runner example (originally used with Antigravity) + +The [`examples/`](https://github.com/microsoft/SkillOpt/tree/main/docs/sleep/examples) directory contains a sanitized reference for running +the compatible backend unattended: - **`examples/runner.py`** — a thin launcher that loads a provider key from an `.env` file, exports the variables above, invokes `skillopt-sleep run` with the DeepSeek backend, and **exits with the child's return code** so - supervisors see failures as failures. It also implements a `session-end` hook - that appends task-outcome metadata to a rollout-evidence log (wired to - Antigravity's `Stop` hook) so future nights have richer sessions to mine. + supervisors see failures as failures. Its `session-end` action writes a small + local rollout-evidence event as an example hook target. - **`examples/watchdog.py`** — a minimal supervisor loop that invokes the runner on a fixed interval (e.g. every 4 hours) and logs non-zero exits as failures. On Windows this is registered as a Scheduled Task so it survives logout; on Linux/macOS a `systemd` timer or cron entry serves the same role. -### Verified result +The current engine does **not** read `brain/rollout-evidence.jsonl`, and it does +not harvest Antigravity transcripts. That hook output is illustrative metadata, +not additional training evidence. A real run must use a supported Claude +Code/Codex transcript source or a reviewed task file converted by the operator. -On a Windows 11 host, driving the cycle against `deepseek-v4-pro` in -`openai_compatible` mode: +### Contributor-reported validation + +The contributor reported the following results from a private Windows 11 setup +driving the cycle against `deepseek-v4-pro` in `openai_compatible` mode. They are +useful integration evidence, but the private session set is not a reproducible +benchmark bundled with this repository: - A direct backend smoke test returns a live completion (no `404`, `last_call_error` empty, client type `OpenAI`). -- A full nightly cycle mined tasks from real IDE sessions and the held-out - validation gate moved from `0.250 → 1.000`, **accepting** a DeepSeek-authored +- A full nightly cycle using the configured session source moved the held-out + validation gate from `0.250 → 1.000`, **accepting** a DeepSeek-authored skill edit (`accept_new_best`). `diagnostics.json` for that night reports `"backend": "azure_openai"` with a non-empty token count and an empty `call_error` — i.e. a genuine optimization night, versus the prior all-`0.0` @@ -124,13 +161,13 @@ Deterministic no-network coverage for the new behavior lives in endpoint/auth guard, request kwargs, retry error-state, empty-response diagnostics, and runner exit-code propagation). -## A note on Gemini (optional, unverified fallback) +## Unsupported Gemini proxy branch in the example -`examples/runner.py` also contains a fallback branch that, when only a Gemini key -is present, routes the **`claude` CLI backend** through a local -Anthropic-compatible proxy (e.g. [LiteLLM](https://github.com/BerriAI/litellm) on -`http://127.0.0.1:4000`) by setting `ANTHROPIC_BASE_URL`/`ANTHROPIC_API_KEY`. -There is **no native Gemini backend** in SkillOpt, and this proxy path was not -independently validated in this work — it is included only as a configuration -example. The verified, supported path in this document is DeepSeek via -`openai_compatible` mode. Treat the Gemini branch as illustrative, not tested. +`examples/runner.py` still contains an illustrative branch that routes the +**`claude` CLI backend** through a loopback Anthropic-compatible proxy such as +[LiteLLM](https://github.com/BerriAI/litellm). It is not a native Gemini backend, +has no validated model mapping in this example, and is not part of the supported +path documented here. The sample currently enters that branch whenever no +DeepSeek key is found, so a production adaptation should remove it or replace it +with an explicit opt-in, a separately configured model, and a trusted isolated +loopback proxy. Do not treat this branch as tested Gemini support. diff --git a/docs/superpowers/specs/2026-06-07-skillopt-sleep-claude-code-plugin-design.md b/docs/superpowers/specs/2026-06-07-skillopt-sleep-claude-code-plugin-design.md index e38d529..06c526d 100644 --- a/docs/superpowers/specs/2026-06-07-skillopt-sleep-claude-code-plugin-design.md +++ b/docs/superpowers/specs/2026-06-07-skillopt-sleep-claude-code-plugin-design.md @@ -1,5 +1,13 @@ # SkillOpt Sleep — Claude Code self-evolving plugin (design) +> **Historical design proposal.** This document records the June 2026 design +> target and includes planned controls that are not part of the current nightly +> CLI. It is not an installation or configuration reference. For implemented +> behavior, flags, defaults, and data boundaries, use +> [`docs/sleep/README.md`](../../sleep/README.md), +> [`docs/reference/cli.md`](../../reference/cli.md), and +> [`plugins/README.md`](https://github.com/microsoft/SkillOpt/blob/main/plugins/README.md). + **Status:** approved-for-build (autonomous offline session, 2026-06-07) **Author:** generated for Yifan Yang, executed autonomously while user is asleep **Branch:** `feat/claude-code-sleep-plugin` (worktree `my_repo/SkillOpt-sleep`) @@ -234,4 +242,3 @@ This session targets **Phase 0 + Phase 1 fully**, **Phase 2 scaffolded**, and th 3. **Real-API demo:** want me to spend live `ANTHROPIC_API_KEY` budget on the persona demo, or keep everything mock until you say go? 4. **Skill target:** evolve a *new* dedicated `skillopt-sleep`-managed skill, or also edit your existing hand-written skills in `~/.claude/skills`? 5. **Paper:** should this become a section/figure in the SkillOpt arXiv (Dream+Sleep framing as "deployment-time continual skill optimization")? -``` diff --git a/index.html b/index.html index 2be9a01..f543f8e 100644 --- a/index.html +++ b/index.html @@ -1778,6 +1778,7 @@ Evolution Transfer Citation + Docs Code
@@ -1915,7 +1916,7 @@

A skill is external state for an agent.

Instead of fine-tuning a model or hand-maintaining prompts, SkillOpt runs - the frozen agent on scored batches, asks a separate optimizer model to + the frozen agent on scored batches, asks an optimizer model to propose structured edits, and accepts a candidate only when validation performance improves.

@@ -2427,7 +2428,7 @@