url/SkillOpt
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: sync documentation with post-v0.2 changes

Browse Source
This commit is contained in:
Yif-Yang
2026-07-14 17:11:40 +00:00
parent efb30b4bcc
commit f31bf8c06b
44 changed files with 2285 additions and 2025 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/guide/configuration.md
+85 -12
View File
@@ -20,15 +20,53 @@ Benchmark configs inherit from `_base_/default.yaml` and override specific value
## Key Parameters
### Model
### Model Backends
`optimizer_backend` controls reflection and skill editing;
`target_backend` controls task rollout. The legacy `backend` field remains for
backward compatibility, but explicit role fields are the clearest configuration.
```yaml
model:
backend: azure_openai # azure_openai | openai_chat | claude_code_exec | qwen
optimizer: gpt-5.5 # Optimizer model (for reflection)
target: gpt-5.5 # Target model (for rollout)
backend: azure_openai # High-level compatibility label
optimizer_backend: openai_chat
target_backend: openai_chat
optimizer: gpt-5.5 # Optimizer deployment/model
target: gpt-5.5 # Target deployment/model
azure_openai_auth_mode: api_key
```
| Backend | Optimizer | Target | Configuration |
|---|:---:|:---:|---|
| `openai_chat` | ✓ | ✓ | Azure OpenAI, or its explicit compatibility auth mode |
| `openai_compatible` | ✓ | ✓ | Generic OpenAI Chat Completions endpoint |
| `claude_chat` | ✓ | ✓ | Claude Code CLI (`claude -p`) |
| `qwen_chat` | ✓ | ✓ | Qwen served through an OpenAI-compatible local endpoint |
| `minimax_chat` | ✓ | ✓ | MiniMax API |
| `codex_exec` | — | ✓ | Codex CLI execution harness |
| `claude_code_exec` | — | ✓ | Claude Code CLI execution harness |
The current MiniMax adapter has one shared deployment. Set
`model.minimax_model` when MiniMax is the target; a mixed-backend run cannot
independently select a MiniMax optimizer model and a different target model.
For a generic compatible provider, select the role backends explicitly rather
than relying on a high-level shorthand:
```yaml
model:
optimizer_backend: openai_compatible
target_backend: openai_compatible
optimizer: deepseek-chat
target: deepseek-chat
```
The train/eval entry points apply `model.optimizer` and `model.target` after
backend initialization. For the selected roles, these YAML values override
`OPENAI_COMPATIBLE_MODEL`, `QWEN_CHAT_MODEL`, and their per-role environment
forms. The environment model variables mainly seed direct library use; always
set the role models in a training or evaluation config.
### Training
```yaml
@@ -96,9 +134,16 @@ Notes:
```yaml
evaluation:
use_gate: true # Validation gating (accept/reject updates)
gate_metric: hard # hard | soft | mixed
gate_mixed_weight: 0.5 # Soft-score weight when metric=mixed
use_semantic_density: false # Optional instruction-density bonus
eval_test: true # Run test evaluation after training
```
The default and paper-style setting is `use_gate: true`. Setting it to `false`
still records selection scores but force-accepts every candidate, so it changes
the optimization semantics and should be reported explicitly.
### Environment (Data)
```yaml
@@ -117,9 +162,10 @@ Override any config value from the command line:
```bash
python scripts/train.py \
--config configs/searchqa/default.yaml \
optimizer.learning_rate=16 \
optimizer.lr_scheduler=linear \
gradient.analyst_workers=8
--cfg-options \
optimizer.learning_rate=16 \
optimizer.lr_scheduler=linear \
gradient.analyst_workers=8
```
## Environment Variables
@@ -128,11 +174,38 @@ Model credentials are loaded from environment variables:
| Variable | Backend | Description |
|---|---|---|
| `AZURE_OPENAI_ENDPOINT` | azure_openai | Azure resource endpoint |
| `AZURE_OPENAI_API_KEY` | azure_openai | Azure API key |
| `OPENAI_API_KEY` | openai | OpenAI API key |
| `ANTHROPIC_API_KEY` | claude | Anthropic API key |
| `QWEN_API_BASE` | qwen | Local Qwen vLLM endpoint |
| `AZURE_OPENAI_ENDPOINT` | `openai_chat` | Azure resource URL, or compatibility-mode base URL |
| `AZURE_OPENAI_API_VERSION` | `openai_chat` | Azure API version |
| `AZURE_OPENAI_AUTH_MODE` | `openai_chat` | `api_key`, `azure_cli`, `managed_identity`, or `openai_compatible` |
| `AZURE_OPENAI_API_KEY` | `openai_chat` | Required when auth mode is `api_key` or `openai_compatible` |
| `OPENAI_COMPATIBLE_BASE_URL` | `openai_compatible` | Generic Chat Completions base URL |
| `OPENAI_COMPATIBLE_API_KEY` | `openai_compatible` | Provider API key; optional for local servers |
| `OPENAI_COMPATIBLE_MODEL` | `openai_compatible` | Shared provider model ID for direct library use; train/eval YAML role models take precedence |
| `CLAUDE_CLI_BIN` | `claude_chat` | Optional path to the `claude` executable; defaults to `claude` |
| `ANTHROPIC_API_KEY` | `claude_chat` | Optional authentication method understood by the Claude CLI, not a direct SkillOpt API client |
| `QWEN_CHAT_BASE_URL` | `qwen_chat` | Local Qwen/vLLM endpoint |
| `QWEN_CHAT_MODEL` | `qwen_chat` | Served model name for direct library use; train/eval YAML role models take precedence |
| `MINIMAX_BASE_URL` | `minimax_chat` | MiniMax-compatible base URL |
| `MINIMAX_API_KEY` | `minimax_chat` | MiniMax API key |
`OPTIMIZER_` and `TARGET_` prefixes provide per-role overrides for the
Azure, OpenAI-compatible, and Qwen variable families. See the
[Configuration Reference](../reference/config.md) for exact names.
`claude_chat` launches the installed Claude Code CLI with `claude -p`; install
and authenticate that CLI before use. Setting `ANTHROPIC_API_KEY` is one way
the CLI may authenticate, but SkillOpt does not call the Anthropic API
directly through this backend.
### Three OpenAI-compatible paths
- Research, generic provider: select `openai_compatible` and use
`OPENAI_COMPATIBLE_*`.
- Research, Azure-family compatibility mode: keep `openai_chat`, set
`AZURE_OPENAI_AUTH_MODE=openai_compatible`, and use `AZURE_OPENAI_*`.
- SkillOpt-Sleep: run with `--backend azure_openai` and use the same
compatibility-mode `AZURE_OPENAI_*` variables. Sleep does not read the
research backend's role-specific variables.
## Full Reference
docs/guide/dl-analogy.md
+5 -3
View File
@@ -14,10 +14,9 @@ SkillOpt is designed around a core insight: **optimizing natural-language prompt
| **Gradient aggregation** | Patch aggregation | Merge similar edits |
| **Gradient clipping** | Edit selection | Cap max edits per step |
| **Learning rate** | `learning_rate` | Max number of edits applied per step |
| **LR scheduler** | `lr_scheduler` | Decay schedule: cosine, linear, constant |
| **LR scheduler** | `lr_scheduler` | Edit-budget schedule: cosine, linear, constant, or autonomous |
| **SGD step** | Skill update | Apply selected patches to document |
| **Validation set** | Selection split | Gate checks improvement before accepting |
| **Early stopping** | Gate patience | Reject updates that don't improve |
| **Training step** | Step | One rollout → reflect → update cycle |
| **Epoch** | Epoch | Full pass with slow update + meta memory |
| **Momentum** | Slow update | Longitudinal comparison at epoch boundary |
@@ -34,7 +33,10 @@ SkillOpt is designed around a core insight: **optimizing natural-language prompt
1. **Familiar mental model**: ML practitioners immediately understand how to tune SkillOpt
2. **Principled hyperparameter search**: Grid search over `learning_rate` × `lr_scheduler` works just like in DL
3. **Proven mechanisms**: Gating ≈ validation-based selection, patience ≈ early stopping, slow update ≈ momentum — all with strong theoretical motivation
3. **Reusable mechanisms**: Gating provides validation-based model selection, while slow update plays a momentum-like role across epochs
The gate is a per-candidate accept/reject decision. SkillOpt does not implement
a gate-patience counter or stop training after a run of rejected candidates.
## Hyperparameter Transfer Rules
docs/guide/first-experiment.md
+68 -41
View File
@@ -4,17 +4,43 @@ This guide walks through running a complete SkillOpt training on SearchQA.
## 1. Choose a Benchmark
SkillOpt includes ready-to-use configs for several benchmarks:
SkillOpt includes ready-to-use configs for several benchmarks. End-to-end
runtime depends on the chosen models, provider latency, worker limits, and
dataset size, so the project does not promise fixed wall-clock estimates.
| Benchmark | Difficulty | Typical Runtime |
| Benchmark | Modality | Additional setup |
|---|---|---|
| SearchQA | ⭐ Easy | ~30 min |
| DocVQA | ⭐⭐ Medium | ~2 hours |
| ALFWorld | ⭐⭐⭐ Hard | ~3 hours |
| SearchQA | Text QA | Materialize the released ID manifest |
| DocVQA | Document/image QA | Obtain and materialize images and examples |
| ALFWorld | Embodied agent | Install ALFWorld and download its assets |
We'll use **SearchQA** as it's the fastest to complete.
We'll use **SearchQA** because it is the simplest text-only walkthrough.
## 2. Configure
## 2. Install and Materialize SearchQA
The repository contains a stable SearchQA ID manifest, not the full runnable
examples. From a source checkout, install the data extra and materialize the
split once:
```bash
python -m pip install -e ".[searchqa]"
python scripts/materialize_searchqa.py
```
By default, the materializer reads `data/searchqa_id_split/` and writes the
train/validation/test payloads expected by the config to
`data/searchqa_split/`; both paths have command-line overrides.
## 3. Configure
Configure and export one model backend as described in
[Installation](installation.md#environment-variables). For example:
```bash
cp .env.example .env
# Edit .env, choose one authentication mode, then export it:
set -a; source .env; set +a
```
Review the config file:
@@ -42,56 +68,55 @@ evaluation:
use_gate: true # (validation gating)
```
## 3. Train
## 4. Train
```bash
python scripts/train.py --config configs/searchqa/default.yaml
python scripts/train.py \
--config configs/searchqa/default.yaml \
--out_root outputs/searchqa_first_run
```
You'll see output like:
The command prints the resolved backend/data configuration, per-step rollout
and gate progress, and the generated output directory.
## 5. Monitor
The explicit `--out_root` above creates this run directory:
```
[Step 1/8] Rollout: 20 items, 4 workers...
[Step 1/8] Score: 0.65 → Reflect...
[Step 1/8] 6 edit patches generated
[Step 1/8] Selected 4 edits (lr=8, cosine → 7.7)
[Step 1/8] Gate: val score 0.68 > 0.65 ✓ ACCEPT
[Step 2/8] ...
```
## 4. Monitor
Training outputs are saved to `outputs/<benchmark>/<run_id>/`:
```
outputs/searchqa/2024-01-15_10-30-00/
├── steps/
│ ├── step_0001/
│ │ ├── candidate_skill.md
│ │ ├── step_record.json
│ │ └── trajectory_digest.json
│ └── step_0002/
├── slow_update/
│ └── epoch_02/
├── meta_skill/
│ └── epoch_02/
├── skills/
│ └── step_0001.md
├── best_skill.md
outputs/searchqa_first_run/
├── config.json
├── runtime_state.json
├── history.json
└── config.yaml
├── best_skill.md
├── skills/
│ └── skill_vXXXX.md
├── steps/
│ └── step_XXXX/
│ ├── candidate_skill.md
│ ├── step_record.json
│ └── trajectory_digest.json
├── slow_update/
│ └── epoch_XX/
└── meta_skill/
└── epoch_XX/
```
## 5. Evaluate
## 6. Evaluate
Evaluate the best skill on the test split:
```bash
python scripts/eval_only.py \
--config configs/searchqa/default.yaml \
--skill outputs/searchqa/<run_id>/skills/best_skill.md
--skill outputs/searchqa_first_run/best_skill.md \
--split valid_unseen
```
The `--skill` path above is the training artifact. Evaluation writes
`eval_summary.json` to its own timestamped `outputs/eval_.../` directory unless
you pass an explicit `--out_root`; it does not overwrite the training run.
## WebUI
Prefer a graphical interface? Launch the WebUI:
@@ -101,7 +126,9 @@ pip install -e ".[webui]"
python -m skillopt_webui.app
```
Then open `http://localhost:7860` in your browser to configure parameters and launch training.
Then open `http://localhost:7860` in your browser to configure parameters and
launch training. The default host is `0.0.0.0`; pass `--host 127.0.0.1` for a
local-only dashboard.
## Next Steps
docs/guide/installation.md
+98 -23
View File
@@ -3,16 +3,44 @@
## Requirements
- Python ≥ 3.10
- At least one model API key (Azure OpenAI, OpenAI, Anthropic, or local Qwen)
- For research training/evaluation, access to at least one configured model
backend (hosted API, local server, or an installed execution CLI)
- The SkillOpt-Sleep `mock` backend needs no credentials
## Quick Install
## Choose an Install
### PyPI
Use PyPI for the Python packages and installed commands:
```bash
python -m pip install skillopt
skillopt-sleep --help
```
This installs `skillopt-train`, `skillopt-eval`, and `skillopt-sleep`. The wheel
does not include the repository's benchmark configs, data materializers,
agent-integration shells/MCP servers, or development tests; use a source
checkout for those files.
!!! important "PyPI versus `main`"
These docs track the latest `main`. The current PyPI release is `0.2.0`.
The generic research `openai_compatible` backend, SkillOpt-Sleep handoff,
Sleep support for non-Azure OpenAI-compatible endpoints, and the Sleep
`--preferences` flag landed after that release and require a source install
from `main` until the next release.
### Source checkout
```bash
git clone https://github.com/microsoft/SkillOpt.git
cd SkillOpt
pip install -e .
python -m pip install -e .
```
Use the source checkout for paper reproduction, built-in benchmark configs,
and contributions.
## Optional Dependencies
Install extras for specific benchmarks or backends:
@@ -20,68 +48,115 @@ Install extras for specific benchmarks or backends:
=== "ALFWorld"
```bash
pip install -e ".[alfworld]"
python -m pip install -e ".[alfworld]"
```
=== "Claude Backend"
=== "Claude agent SDK (optional)"
```bash
pip install -e ".[claude]"
python -m pip install -e ".[claude]"
```
This extra does not install the `claude` executable. The research
`claude_chat` backend launches `claude -p`, so install and authenticate the
Claude Code CLI separately. The SDK extra is only needed when selecting an
SDK-backed Claude Code exec path.
=== "Qwen (Local)"
```bash
pip install -e ".[qwen]"
python -m pip install -e ".[qwen]"
```
=== "SearchQA data"
```bash
python -m pip install -e ".[searchqa]"
```
=== "WebUI"
```bash
pip install -e ".[webui]"
python -m pip install -e ".[webui]"
```
=== "Development"
```bash
pip install -e ".[dev]"
python -m pip install -e ".[dev]"
```
=== "All"
```bash
pip install -e ".[alfworld,claude,qwen,webui,dev]"
python -m pip install -e ".[alfworld,claude,qwen,searchqa,webui,docs,dev]"
```
## Environment Variables
Copy the example `.env` file and fill in your credentials:
From a source checkout, copy the template and fill in only the backend you
will use:
```bash
cp .env.example .env
```
Edit `.env` with your API keys:
SkillOpt does not automatically load `.env`; export it into the current shell
before running commands:
```ini
# Azure OpenAI (default backend)
AZURE_OPENAI_ENDPOINT=https://your-resource.openai.azure.com/
AZURE_OPENAI_API_KEY=your-key
# Or use OpenAI directly
OPENAI_API_KEY=sk-...
# Or Anthropic Claude
ANTHROPIC_API_KEY=sk-ant-...
```bash
set -a
source .env
set +a
```
For Azure OpenAI with API-key authentication, the minimum settings are:
```ini
AZURE_OPENAI_ENDPOINT=https://your-resource.openai.azure.com/
AZURE_OPENAI_API_VERSION=2024-12-01-preview
AZURE_OPENAI_API_KEY=your-key
AZURE_OPENAI_AUTH_MODE=api_key
```
Use `AZURE_OPENAI_AUTH_MODE=azure_cli` for Azure CLI credentials, or
`managed_identity` with an optional
`AZURE_OPENAI_MANAGED_IDENTITY_CLIENT_ID`.
The research `claude_chat` backend is a Claude Code CLI adapter, not a direct
Anthropic API client. Install and authenticate `claude`, and set
`CLAUDE_CLI_BIN` only if the executable is not available as `claude` on
`PATH`. `ANTHROPIC_API_KEY` is one authentication option the CLI may consume.
OpenAI-compatible servers have three distinct entry points:
1. The research engine's generic `openai_compatible` backend uses
`OPENAI_COMPATIBLE_BASE_URL`, `OPENAI_COMPATIBLE_API_KEY`, and
`OPENAI_COMPATIBLE_MODEL`.
2. The research `openai_chat` backend can use
`AZURE_OPENAI_AUTH_MODE=openai_compatible` with
`AZURE_OPENAI_ENDPOINT` and `AZURE_OPENAI_API_KEY`.
3. SkillOpt-Sleep uses the same Azure-family variables as item 2 with
`skillopt-sleep run --backend azure_openai`.
For research train/eval commands, `model.optimizer` and `model.target` in the
YAML config are applied after backend initialization. They override model-name
environment variables such as `OPENAI_COMPATIBLE_MODEL` and
`QWEN_CHAT_MODEL`; set both role models explicitly when selecting those
backends.
!!! tip
You only need credentials for the backend you plan to use. Azure OpenAI is the default.
You only need to configure the backend you plan to use. See
[Configuration](configuration.md#model-backends) for exact backend names
and role-specific overrides.
## Verify Installation
```bash
python -c "import skillopt; print('SkillOpt ready!')"
skillopt-train --help
skillopt-eval --help
skillopt-sleep --help
```
## Next Steps
docs/guide/local-env-smoke.md
+15 -9
View File
@@ -35,9 +35,14 @@ Use the split names your adapter maps to SkillOpt phases:
- `val` or `valid_seen` for selection/gating
- `test` or `valid_unseen` for final evaluation
## 2. Support an offline mock mode
## 2. Support a genuinely offline mock mode
Add a configuration flag such as `mock: true` to your adapter. In mock mode, `rollout()` should return deterministic responses without calling external model APIs.
Add a configuration flag such as `mock: true` to your adapter. In mock mode,
`rollout()` should return deterministic responses without calling external
model APIs. The inherited `EnvAdapter.reflect()` does call the configured
optimizer backend, so a no-credential smoke test must also override
`reflect()` in mock mode to return a small, schema-valid deterministic patch
(and delegate to `super().reflect(...)` otherwise).
This lets you verify the SkillOpt loop with a fast command such as:
@@ -46,13 +51,14 @@ python scripts/train.py \
--config configs/myenv/tiny_mock.yaml
```
Mock mode should still write the same artifacts as a real run, for example:
Mock mode should still exercise the trainer's normal artifact paths, including:
- `responses.json`
- `rollout_results.json`
- `ranked_edits.json`
- `candidate_skill.md`
- `summary.json`
- `config.json`, `runtime_state.json`, and `history.json`
- `skills/skill_vXXXX.md`
- `steps/step_XXXX/ranked_edits.json`
- `steps/step_XXXX/candidate_skill.md`
- `steps/step_XXXX/step_record.json`
- the final `summary.json`
## 3. Keep the smoke config tiny
@@ -127,7 +133,7 @@ For the real tiny run, verify that:
- the run completes
- `summary.json` is written
- `ranked_edits.json` contains the expected ranking metadata
- the step directory's `ranked_edits.json` contains the expected ranking metadata
- any optimizer bridge log marks the response schema as valid
- no generated files are written outside `out_root`
docs/guide/new-backend.md
+146 -129
View File
@@ -1,11 +1,17 @@
# Add a New Model Backend
SkillOpt supports multiple LLM backends. This guide shows how to add your own.
SkillOpt's model layer is function-based: each chat backend is a Python module
that exposes the call, token-tracking, and deployment-setting functions used by
`skillopt.model`. There is no backend base class or registry object to subclass.
## Built-in: the generic OpenAI-compatible backend
!!! note "Version requirement"
This backend landed after v0.2.0. Install from the latest `main` until it is
included in the next release.
Before writing a new backend, check whether your provider already speaks the
OpenAI Chat Completions protocol. Most do — in which case you can use the
OpenAI Chat Completions protocol. Most do, in which case you can use the
built-in **`openai_compatible`** backend
(`skillopt/model/openai_compatible_backend.py`) with no code changes.
@@ -21,15 +27,16 @@ A single `base_url` + `api_key` pair lets you point SkillOpt at, for example:
| LiteLLM proxy | `http://localhost:4000` | any proxied model |
| OpenRouter / Fireworks / xAI / … | provider base URL | provider model id |
Select it as the optimizer and/or target backend:
### Python API
Select and configure the backend directly when embedding SkillOpt as a Python
library:
```python
import skillopt.model as model
# Shorthand: use it for both optimizer and target.
# Use the generic backend for both optimizer and target calls.
model.set_backend("openai_compatible")
# Point it at a provider (shared, or per-role with optimizer_*/target_*).
model.configure_openai_compatible(
base_url="https://api.deepseek.com/v1",
api_key="sk-...",
@@ -37,147 +44,157 @@ model.configure_openai_compatible(
)
```
Or configure it entirely through environment variables (role-specific
`OPTIMIZER_*` / `TARGET_*` variants override the shared ones):
`configure_openai_compatible()` also accepts `optimizer_*` and `target_*`
arguments when the two roles use different endpoints or models.
### Environment variables
The shared variables below configure both roles. Role-specific
`OPTIMIZER_OPENAI_COMPATIBLE_*` and `TARGET_OPENAI_COMPATIBLE_*` variables take
precedence:
```bash
export TARGET_BACKEND=openai_compatible
export OPENAI_COMPATIBLE_BASE_URL="https://api.groq.com/openai/v1"
export OPENAI_COMPATIBLE_API_KEY="gsk_..."
export OPENAI_COMPATIBLE_MODEL="llama-3.3-70b-versatile"
# Optional: OPENAI_COMPATIBLE_TEMPERATURE, _MAX_TOKENS, _TIMEOUT_SECONDS
```
The backend uses the official `openai` SDK, records token usage through the
shared tracker, supports tool/function calling via
`chat_target_messages(..., tools=...)`, and exposes
`count_tokens()` (tiktoken with a character-based fallback for non-OpenAI
models). Only write a brand-new backend if your provider is *not*
OpenAI-compatible.
## Backend Architecture
```
skillopt/model/
├── base.py # Abstract base class
├── azure_openai.py # Azure OpenAI backend
├── openai_model.py # Direct OpenAI backend
├── claude.py # Anthropic Claude backend
├── qwen.py # Local Qwen (vLLM) backend
└── your_backend.py # Your new backend
```
## Step 1: Create the Backend
Create `skillopt/model/your_backend.py`:
```python
from skillopt.model.base import ModelBackend, ModelResponse
class YourBackend(ModelBackend):
"""Your custom model backend."""
def __init__(self, cfg: dict):
super().__init__(cfg)
self.model_name = cfg.get('model_name', 'your-default-model')
self.api_key = os.environ.get('YOUR_API_KEY', '')
self.client = self._init_client()
def _init_client(self):
"""Initialize API client."""
# TODO: Set up your API client
pass
async def generate(
self,
messages: list[dict],
temperature: float = 0.7,
max_tokens: int = 4096,
**kwargs
) -> ModelResponse:
"""
Generate a completion.
Args:
messages: Chat messages [{"role": "...", "content": "..."}]
temperature: Sampling temperature
max_tokens: Maximum tokens in response
Returns:
ModelResponse with content, usage, and metadata
"""
response = await self.client.chat(
model=self.model_name,
messages=messages,
temperature=temperature,
max_tokens=max_tokens,
)
return ModelResponse(
content=response.text,
usage={
'prompt_tokens': response.usage.input,
'completion_tokens': response.usage.output,
},
model=self.model_name,
)
async def generate_with_tools(
self,
messages: list[dict],
tools: list[dict],
**kwargs
) -> ModelResponse:
"""Generate with tool/function calling support."""
# Optional: implement if your model supports tool use
raise NotImplementedError("Tool use not supported")
```
## Step 2: Register the Backend
Add to `skillopt/model/__init__.py`:
```python
from .your_backend import YourBackend
BACKEND_REGISTRY = {
# ... existing backends ...
'your_backend': YourBackend,
}
```
## Step 3: Configure
Use your backend in any config:
For direct library use, `OPTIMIZER_BACKEND=openai_compatible` and/or
`TARGET_BACKEND=openai_compatible` select the role. The training and evaluation
scripts resolve backend selection from their config, so set the split fields
explicitly there:
```yaml
model:
backend: your_backend
model_name: your-model-id
temperature: 0.7
max_tokens: 4096
optimizer_backend: openai_compatible
target_backend: openai_compatible
optimizer: llama-3.3-70b-versatile
target: llama-3.3-70b-versatile
```
Set credentials via environment variable:
Equivalently, override those fields on the command line:
```bash
export YOUR_API_KEY="your-key"
python scripts/train.py --config configs/searchqa/default.yaml \
--cfg-options \
model.optimizer_backend=openai_compatible \
model.target_backend=openai_compatible \
model.optimizer=llama-3.3-70b-versatile \
model.target=llama-3.3-70b-versatile
```
## Required Interface
Do not rely on the legacy high-level `model.backend` label to replace the two
role-specific fields in a structured config.
Your backend must implement these methods:
The generic backend uses the official `openai` SDK and the Chat Completions
API. It records token usage through the shared tracker, supports provider tool
calling through `chat_*_messages(..., tools=...)`, and exposes `count_tokens()`
(tiktoken when available, with a character-based fallback). Provider-specific
Responses API features are outside this backend's contract.
| Method | Required | Description |
|---|---|---|
| `generate()` | ✅ | Basic text generation |
| `generate_with_tools()` | Optional | Tool/function calling |
| `count_tokens()` | Optional | Token counting for context management |
Only write a new backend when the provider is not compatible with this surface
or requires behavior that cannot be expressed by its configuration.
## Tips
## Backend architecture
!!! tip
- Test your backend with `python -c "from skillopt.model.your_backend import YourBackend"` first
- Use `async` methods for all API calls — SkillOpt uses asyncio throughout
- Implement retry logic with exponential backoff for production use
- Add your API key to `.env.example` when submitting a PR
The active split optimizer/target dispatcher is the public
`skillopt/model/__init__.py` module:
```text
skillopt/model/
├── common.py # aliases, default models, token/response helpers
├── backend_config.py # optimizer/target whitelists and runtime selection
├── __init__.py # public API and split-role dispatch
├── openai_compatible_backend.py # generic Chat Completions example
├── qwen_backend.py # raw-HTTP chat example with per-role config
├── minimax_backend.py # compact raw-HTTP chat example
├── codex_harness.py # target-only exec harnesses
└── router.py # legacy single-backend compatibility surface
```
`router.py` is not the dispatcher used by the current training loop. Update it
only if the new backend must also be exposed through that legacy single-backend
API.
## Step 1: implement the module contract
Create a module such as `skillopt/model/your_backend.py`. Copy the signatures
from `openai_compatible_backend.py` or `qwen_backend.py`; model calls in the
current framework are synchronous.
For a chat backend that supports both roles, the public module surface is:
| Function | Purpose |
|---|---|
| `chat_optimizer(...)` | Optimizer system/user call; returns `(text, usage)` |
| `chat_target(...)` | Target system/user call; returns `(text, usage)` |
| `chat_optimizer_messages(...)` | Optimizer message-list call, including optional tools |
| `chat_target_messages(...)` | Target message-list call, including optional tools |
| `get_token_summary()` | Return per-stage counters plus `_total` |
| `reset_token_tracker()` | Clear this backend's counters |
| `set_optimizer_deployment(name)` | Change the optimizer model at runtime |
| `set_target_deployment(name)` | Change the target model at runtime |
| `set_reasoning_effort(effort)` | Apply or safely ignore the shared reasoning setting |
Every call returns a usage dict with `prompt_tokens`, `completion_tokens`, and
`total_tokens`. Use `TokenTracker` from `skillopt.model.common` and record each
call exactly once. Message-list calls that accept tools should return the
compatibility message objects from `common.py` when `return_message=True`.
Provider-specific configuration helpers and `count_tokens()` are optional, but
their state must be safe to update while calls may run concurrently. Keep
credentials out of logs and persisted artifacts.
Exec-style targets do not implement this chat contract. They are target-only
and are integrated through `codex_harness.py` plus environment-specific rollout
code.
## Step 2: register and route the backend
A new backend normally requires all of the following:
1. Add its canonical name, aliases, and default model to
`skillopt/model/common.py`.
2. Add the canonical name to the appropriate optimizer and/or target whitelist
in `skillopt/model/backend_config.py`. Do not advertise a role the module
cannot execute.
3. Import the module in `skillopt/model/__init__.py` and add dispatch branches
for every supported call surface.
4. Include its counters in `get_token_summary()` / `reset_token_tracker()` and
forward the shared deployment/reasoning setters where applicable.
5. If it has YAML settings, add structured-to-flat mappings in
`skillopt/config.py`, wire them through `scripts/train.py` and
`scripts/eval_only.py`, and document their precedence over environment
variables.
6. Update `router.py` only when legacy single-backend compatibility is part of
the intended feature.
Backend selection in `scripts/train.py` must use
`model.optimizer_backend` and `model.target_backend`. A high-level
`model.backend` alias alone is not a substitute for this explicit split.
## Step 3: test the integration
Add focused tests under `tests/` that do not call a live provider. At minimum,
cover:
- optimizer and target whitelist validation;
- routing for text and message-list calls;
- role-specific configuration precedence;
- tool-call compatibility, if supported;
- deployment/reasoning setters;
- token accounting, including a single correct `_total`;
- actionable errors for missing credentials or invalid responses.
Then run the focused test, the full suite, and the documentation build:
```bash
python -m pytest tests/test_your_backend.py -q
python -m pytest tests/ -q
mkdocs build --strict
```
Also update `.env.example`, the configuration reference, and the backend table
in the API reference. Add an optional dependency extra only when the backend
requires a package that is not already a core dependency.
docs/guide/new-benchmark.md
+60 -23
View File
@@ -14,16 +14,18 @@ To add a benchmark you implement four things:
1. **A `SplitDataLoader` subclass** — knows how to load train / val / test
item dicts from disk.
2. **A rollout helper** — runs the target model on a batch of items
under the current skill and scores each prediction.
2. **A rollout helper** — runs the target model on a batch of items, scores
each prediction, and persists the per-item conversation consumed by the
shared reflection stage.
3. **An `EnvAdapter` subclass** — wires the loader + rollout helper into
SkillOpt's lifecycle (`build_*_env`, `rollout`, `reflect`,
`get_task_types`).
SkillOpt's lifecycle (`build_*_env`, `rollout`, and `get_task_types`).
The shared `reflect()` implementation is inherited unless the benchmark
needs custom reflection logic.
4. **A YAML config** — references your env name plus the standard
train / optimizer / gradient knobs.
Then one line in `scripts/train.py`'s `_register_builtins()` makes it
discoverable.
Then lazy registration in the training and evaluation scripts makes it
discoverable without importing optional dependencies at startup.
---
@@ -99,8 +101,8 @@ def _score(prediction: str, ground_truth: str) -> tuple[int, float]:
return hard, soft
def _rollout_one(item: dict, skill_content: str,
*, max_completion_tokens: int) -> dict:
def _rollout_one(item: dict, skill_content: str, *, prediction_dir: Path,
max_completion_tokens: int) -> dict:
system = skill_content
user = (
f"Question: {item['question']}\n\n"
@@ -113,14 +115,33 @@ def _rollout_one(item: dict, skill_content: str,
max_completion_tokens=max_completion_tokens,
)
hard, soft = _score(prediction, item.get("ground_truth", ""))
# EnvAdapter.reflect() reads this exact trajectory path. Keep item IDs
# unique and filesystem-safe.
task_dir = prediction_dir / str(item["id"])
task_dir.mkdir(parents=True, exist_ok=True)
conversation = [
{"role": "system", "content": system},
{"role": "user", "content": user},
{"role": "assistant", "content": prediction},
]
(task_dir / "conversation.json").write_text(
json.dumps(conversation, ensure_ascii=False, indent=2),
encoding="utf-8",
)
return {
"id": str(item["id"]),
"hard": hard,
"soft": soft,
"predicted_answer": prediction,
"task_description": item.get("question", ""),
"question": item.get("question", ""),
"reference_text": item.get("reference_text", ""),
"task_type": item.get("task_type", "docfaithful"),
"target_system_prompt": system,
"target_user_prompt": user,
"n_turns": 1,
}
@@ -128,15 +149,18 @@ def run_batch(*, items: list[dict], skill_content: str, out_root: str,
workers: int = 4, max_completion_tokens: int = 4096) -> list[dict]:
"""Run a batch of episodes sequentially or with a thread pool."""
os.makedirs(out_root, exist_ok=True)
prediction_dir = Path(out_root, "predictions")
# For brevity we go sequentially — swap in concurrent.futures.ThreadPoolExecutor
# when network / model latency dominates.
results = [
_rollout_one(item, skill_content,
prediction_dir=prediction_dir,
max_completion_tokens=max_completion_tokens)
for item in items
]
Path(out_root, "rollouts.json").write_text(
json.dumps(results, ensure_ascii=False, indent=2)
json.dumps(results, ensure_ascii=False, indent=2),
encoding="utf-8",
)
return results
```
@@ -150,9 +174,17 @@ Two design points worth flagging:
- **Use `skillopt.model.chat_target`**, not raw OpenAI/Claude calls.
That routes through whichever **chat** target backend the user
configured (`openai_chat` / `claude_chat` / `qwen_chat` /
`minimax_chat`) without your adapter caring. Exec-style backends
(`codex_exec`, `claude_code_exec`) need env-specific rollout code —
see `skillopt/envs/swebench/` for an example.
`minimax_chat` / `openai_compatible`) without your adapter caring.
Exec-style backends (`codex_exec`, `claude_code_exec`) need
environment-specific rollout code —
see `skillopt/model/codex_harness.py` together with the rollout modules in
`skillopt/envs/searchqa/`, `skillopt/envs/docvqa/`, or
`skillopt/envs/officeqa/` for working examples.
- **Persist a conversation for reflection.** The shared `EnvAdapter.reflect()`
looks under `<rollout_dir>/predictions/<result-id>/conversation.json` and
skips results whose trajectory is absent or empty. Returning `hard`/`soft`
scores alone is sufficient for evaluation, but it cannot produce learning
patches.
## Step 4 — Implement the environment adapter
@@ -277,9 +309,11 @@ the answer against `item["ground_truth"]`, and returns a list of dicts:
]
```
The trainer only requires `id`, `hard`, `soft`. The rest is preserved on
`RolloutResult.extras` (see `skillopt/types.py`) and is what your
`reflect()` consumes via `run_minibatch_reflect`.
The trainer requires `id`, `hard`, and `soft` for scoring. The remaining fields
are preserved on `RolloutResult.extras` (see `skillopt/types.py`). The shared
reflection implementation combines those fields with each persisted
`predictions/<id>/conversation.json`; without that file the result is omitted
from reflection.
## Step 5 — Register the adapter
@@ -294,9 +328,11 @@ and add to `_register_builtins()`:
pass # docfaithful deps not installed — skip
```
There is **no `BENCHMARK_REGISTRY` dict in `skillopt/envs/__init__.py`** —
the registry lives in `scripts/train.py` and is populated lazily so that
optional deps don't break `--help`.
Mirror the same lazy registration in
[`scripts/eval_only.py`](https://github.com/microsoft/SkillOpt/blob/main/scripts/eval_only.py)
so standalone evaluation can resolve the environment too. There is **no
`BENCHMARK_REGISTRY` dict in `skillopt/envs/__init__.py`**; both entry points
keep a small lazy registry so optional dependencies do not break `--help`.
## Step 6 — Create the YAML config
@@ -322,9 +358,7 @@ optimizer:
env:
name: docfaithful
# Optional: a seed skill document. Create this file (or any markdown
# file) yourself before the first run, or omit the key to let SkillOpt
# start from an empty skill.
# Point to an existing Markdown file. Use an empty file to start blank.
skill_init: skillopt/envs/docfaithful/skills/initial.md
split_mode: split_dir
split_dir: data/docfaithful_split
@@ -341,7 +375,7 @@ env:
## Step 7 — Run
```bash
# If you set skill_init above, create the seed skill first:
# Create the file referenced by env.skill_init before the first run:
# mkdir -p skillopt/envs/docfaithful/skills
# echo "# DocFaithful initial skill" > skillopt/envs/docfaithful/skills/initial.md
@@ -364,8 +398,11 @@ you forgot to implement one of the four abstract methods on `EnvAdapter`:
`hard` / `soft`.
- Noisy scoring kills the optimizer. Spend time on `run_batch`'s scoring
before you spend time on prompts.
- If training repeatedly reports `skip_no_patches`, first verify that every
rollout result has a non-empty
`rollout/predictions/<id>/conversation.json` using the same `id` string.
- If your benchmark needs heavy optional deps (selenium, vllm, ...),
wrap the registration block with `try / except ImportError` (Step 5)
wrap both registration blocks with `try / except ImportError` (Step 5)
so people without those deps can still `--help`.
- Copy `skillopt/envs/_template/` as a starting skeleton — it now
implements the real abstract methods.
docs/guide/skill-document.md
+30 -7
View File
@@ -38,23 +38,45 @@ During training, the skill document is modified by **edit patches**:
2. **Modifications**: Refining existing rules that are partially correct
3. **Deletions**: Removing rules that consistently lead to errors
Each edit is validated through the **gate** mechanism before being permanently accepted.
Selected edits are applied together to produce a candidate skill. With the
validation gate enabled, that candidate replaces the current skill only when
its score on the selection split strictly improves.
SkillOpt may maintain two protected, machine-managed regions:
```markdown
<!-- SLOW_UPDATE_START -->
... epoch-level longitudinal guidance ...
<!-- SLOW_UPDATE_END -->
<!-- APPENDIX_START -->
... skill-aware execution reminders ...
<!-- APPENDIX_END -->
```
Normal edit patches cannot modify either region. Slow update owns the first;
optional skill-aware reflection owns the second. Preserve these markers when
copying or manually inspecting a trained skill.
## Initial Skill
You can start training with:
- **Empty skill**: The system learns everything from scratch
- **Empty skill**: Point `env.skill_init` to an empty Markdown file
- **Seed skill**: Provide initial instructions to bootstrap training
- **Pre-trained skill**: Transfer a skill from a related benchmark
Configure the initial skill in your YAML:
```yaml
train:
init_skill: "path/to/initial_skill.md" # or omit for empty
env:
skill_init: path/to/initial_skill.md
```
To start from scratch, create an empty Markdown file and use its path. A missing
path currently also starts blank, so using an explicit file avoids silently
treating a typo as an empty skill.
## Skill Quality Metrics
Track your skill's evolution through:
@@ -62,15 +84,16 @@ Track your skill's evolution through:
- **Validation score**: Primary metric on the selection split
- **Test score**: Final metric on held-out test data
- **Skill length**: Total tokens in the document
- **Edit acceptance rate**: Fraction of proposed edits that pass gating
- **Candidate acceptance rate**: Fraction of candidate skill updates that pass
gating; multiple proposed edits can be combined into one candidate
## Best Practices
!!! tip "Tips for better skills"
1. **Start with a seed skill** (`env.skill_init`) if you have domain knowledge — it converges faster
2. **Use cosine LR schedule** — aggressive early exploration + careful late refinement
3. **Enable slow update** (`use_slow_update: true`) to prevent forgetting across epochs
4. **Enable meta skill** (`use_meta_skill: true`) so the optimizer accumulates strategy memory
3. **Enable slow update** (`optimizer.use_slow_update: true`) to counter forgetting across epochs
4. **Enable meta skill** (`optimizer.use_meta_skill: true`) so the optimizer accumulates strategy memory
## Next Steps
docs/guide/training-loop.md
+22 -9
View File
@@ -37,12 +37,11 @@ scores = evaluate(predictions, ground_truth)
### 2. Reflect (Backward Pass)
The **optimizer** model analyzes failed trajectories and produces **edit patches** — structured suggestions for improving the skill document.
Two modes:
- **Shallow**: Analyze each trajectory independently
- **Deep**: Cross-reference multiple failures to find systemic issues
The **optimizer** model analyzes trajectory minibatches and produces **edit
patches** — structured suggestions for improving the skill document. Failure
minibatches are always eligible for analysis; successful trajectories are also
analyzed unless `gradient.failure_only` is enabled. Independent minibatches can
run concurrently according to `gradient.analyst_workers`.
```python
# Analogy: computing gradients
@@ -74,17 +73,31 @@ Selected edits are applied to the skill document, producing a new version.
### 6. Gate (Validation)
The updated skill is evaluated on a **selection split** (analogous to a validation set). The update is only accepted if performance improves.
The updated skill is evaluated on a **selection split** (analogous to a
validation set). With the gate enabled, the candidate is accepted only when its
configured gate score (`hard`, `soft`, or `mixed`) is strictly higher than the
current skill's score. With `evaluation.use_gate: false`, validation is still
recorded but candidates are force-accepted.
## Epoch Boundary Mechanisms
### Slow Update
At the end of each epoch (starting from epoch 2), the system performs a **longitudinal comparison**: it rolls out both the previous epoch's skill and the current skill on the same samples, categorizes items as improved/regressed/persistent_fail/stable_success, then generates high-level **guidance** that is injected into the skill document. This prevents catastrophic forgetting of earlier improvements.
At the end of each epoch (starting from epoch 2), the system performs a
**longitudinal comparison**: it rolls out both the previous epoch's skill and
the current skill on the same samples, categorizes items as
improved/regressed/persistent-fail/stable-success, then generates high-level
**guidance** for the skill document. Depending on
`optimizer.slow_update_gate_with_selection`, that guidance is either checked on
the selection split or applied unconditionally. Its purpose is to counter
cross-epoch forgetting.
### Meta Skill
A **meta-skill memory** accumulates high-level strategy notes across the entire training run. At the end of each epoch, the optimizer reflects on what changed between epochs and produces a compact memory that is provided as additional context during future reflection steps.
A **meta-skill memory** accumulates high-level strategy notes across the training
run. Starting at the end of epoch 2, the optimizer compares the previous and
current epoch, writes a compact memory, and provides the prior epoch's memory as
additional context during later reflection and update stages.
## Next Steps