SkillOpt v0.1.0: initial release

- Skill optimization framework with training loop analogy - 11 benchmarks, 4 model backends (Azure OpenAI, Claude, Codex, Qwen) - WebUI for browser-based training control - Pluggable architecture for extending benchmarks and backends
2026-05-21 17:22:04 +00:00
commit 244e346b83
237 changed files with 30248 additions and 0 deletions
@@ -0,0 +1,81 @@
+# API Reference
+
+## Core Classes
+
+### `EnvAdapter`
+
+Abstract base class for benchmark environments.
+
+```python
+class EnvAdapter(ABC):
+    async def execute(self, item, skill, model) -> TaskResult
+    def evaluate(self, prediction, ground_truth) -> float
+    def build_prompt(self, item, skill) -> str
+```
+
+### `DataLoader`
+
+Abstract base class for data loading and splitting.
+
+```python
+class DataLoader(ABC):
+    def setup(self, cfg: dict) -> None
+    def get_split_items(self, split: str) -> list[DataItem]
+```
+
+### `ModelBackend`
+
+Abstract base class for LLM backends.
+
+```python
+class ModelBackend(ABC):
+    async def generate(self, messages, **kwargs) -> ModelResponse
+    async def generate_with_tools(self, messages, tools, **kwargs) -> ModelResponse
+```
+
+### `Trainer`
+
+Main training loop orchestrator.
+
+```python
+class Trainer:
+    def __init__(self, cfg: dict)
+    async def train(self) -> TrainResult
+    async def evaluate(self, skill: str, split: str) -> EvalResult
+```
+
+## Data Classes
+
+### `DataItem`
+
+```python
+@dataclass
+class DataItem:
+    id: str
+    input: str
+    ground_truth: str
+    metadata: dict = field(default_factory=dict)
+```
+
+### `TaskResult`
+
+```python
+@dataclass
+class TaskResult:
+    item_id: str
+    prediction: str
+    score: float
+    trajectory: list[dict]
+```
+
+### `ModelResponse`
+
+```python
+@dataclass
+class ModelResponse:
+    content: str
+    usage: dict
+    model: str
+```
+
+For detailed source code, see the [`skillopt/`](https://github.com/microsoft/SkillOpt/tree/main/skillopt) directory.
@@ -0,0 +1,71 @@
+# CLI Reference
+
+## Training
+
+```bash
+python scripts/train.py --config <config.yaml> [overrides...]
+```
+
+### Arguments
+
+| Argument | Description |
+|---|---|
+| `--config` | Path to YAML config file (required) |
+| `key=value` | Override any config parameter |
+
+### Examples
+
+```bash
+# Basic training
+python scripts/train.py --config configs/searchqa/default.yaml
+
+# With overrides
+python scripts/train.py \
+  --config configs/searchqa/default.yaml \
+  --cfg-options optimizer.learning_rate=16 optimizer.lr_scheduler=linear
+
+# With custom initial skill
+python scripts/train.py \
+  --config configs/searchqa/default.yaml \
+  --cfg-options env.skill_init=skills/my_seed.md
+```
+
+## Evaluation
+
+```bash
+python scripts/eval_only.py --config <config.yaml> --skill <skill.md>
+```
+
+### Arguments
+
+| Argument | Description |
+|---|---|
+| `--config` | Path to YAML config file (required) |
+| `--skill` | Path to skill document to evaluate (required) |
+| `--split` | Evaluation split: `test` (default), `valid`, `train` |
+
+### Examples
+
+```bash
+# Evaluate best skill on test set
+python scripts/eval_only.py \
+  --config configs/searchqa/default.yaml \
+  --skill outputs/searchqa/run_001/skills/best_skill.md
+
+# Evaluate on validation set
+python scripts/eval_only.py \
+  --config configs/searchqa/default.yaml \
+  --skill outputs/searchqa/run_001/skills/best_skill.md \
+  --split valid
+```
+
+## WebUI
+
+```bash
+python -m skillopt_webui.app [--port PORT] [--share]
+```
+
+| Argument | Default | Description |
+|---|---|---|
+| `--port` | 7860 | Port number |
+| `--share` | false | Create public Gradio link |
@@ -0,0 +1,72 @@
+# Configuration Reference
+
+Complete reference for all SkillOpt configuration parameters.
+
+## Model
+
+| Parameter | Type | Default | Description |
+|---|---|---|---|
+| `model.backend` | str | `azure_openai` | Backend: `azure_openai` / `openai_chat` / `claude_code_exec` / `qwen` |
+| `model.teacher` | str | `gpt-5.5` | Teacher model (for reflection & slow update) |
+| `model.student` | str | `gpt-5.5` | Student model (for rollout execution) |
+| `model.reasoning_effort` | str | `medium` | Reasoning effort level |
+
+## Training (`train`)
+
+| Parameter | Type | Default | DL Analogy | Description |
+|---|---|---|---|---|
+| `train.num_epochs` | int | 4 | Epochs | Number of training epochs |
+| `train.batch_size` | int | 40 | Batch size | Tasks sampled per step |
+| `train.accumulation` | int | 1 | Gradient accumulation | Accumulation rounds per step |
+| `train.seed` | int | 42 | Random seed | Reproducibility seed |
+
+## Gradient / Reflection (`gradient`)
+
+| Parameter | Type | Default | Description |
+|---|---|---|---|
+| `gradient.minibatch_size` | int | 8 | Reflect minibatch size |
+| `gradient.merge_batch_size` | int | 8 | Patch merge batch size |
+| `gradient.analyst_workers` | int | 16 | Parallel reflection workers |
+| `gradient.max_analyst_rounds` | int | 3 | Max rounds of analyst reflection |
+| `gradient.failure_only` | bool | `false` | Only reflect on failures |
+
+## Optimizer (`optimizer`)
+
+| Parameter | Type | Default | DL Analogy | Description |
+|---|---|---|---|---|
+| `optimizer.learning_rate` | int | 4 | Learning rate | Max edit patches per step (edit budget) |
+| `optimizer.min_learning_rate` | int | 2 | Min LR | Min edits for decay schedulers |
+| `optimizer.lr_scheduler` | str | `cosine` | LR schedule | `constant` / `linear` / `cosine` / `autonomous` |
+| `optimizer.skill_update_mode` | str | `patch` | — | `patch` / `rewrite_from_suggestions` / `full_rewrite_minibatch` |
+| `optimizer.use_slow_update` | bool | `true` | Momentum | Epoch-boundary longitudinal comparison & guidance |
+| `optimizer.slow_update_samples` | int | 20 | — | Samples for slow update evaluation |
+| `optimizer.use_meta_skill` | bool | `true` | Meta-learning | Cross-epoch teacher-side strategy memory |
+| `optimizer.longitudinal_pair_policy` | str | `mixed` | — | `mixed` / `changed` / `unchanged` |
+
+## Evaluation (`evaluation`)
+
+| Parameter | Type | Default | Description |
+|---|---|---|---|
+| `evaluation.use_gate` | bool | `true` | Enable validation gating (accept/reject updates) |
+| `evaluation.eval_test` | bool | `true` | Run test evaluation after training |
+
+## Environment (`env`)
+
+| Parameter | Type | Default | Description |
+|---|---|---|---|
+| `env.name` | str | — | Benchmark name (e.g., `searchqa`, `docvqa`) |
+| `env.data_path` | str | — | Path to dataset |
+| `env.skill_init` | str | — | Path to initial seed skill (optional) |
+| `env.split_mode` | str | `ratio` | `ratio` or `split_dir` |
+| `env.split_ratio` | str | `2:1:7` | Train:val:test ratio |
+| `env.exec_timeout` | int | 120 | Per-task timeout in seconds |
+| `env.out_root` | str | — | Output directory |
+
+## Azure OpenAI Credentials
+
+| Variable | Description |
+|---|---|
+| `AZURE_OPENAI_ENDPOINT` / `model.azure_openai_endpoint` | Azure resource endpoint |
+| `AZURE_OPENAI_API_KEY` / `model.azure_openai_api_key` | Azure API key |
+| `OPENAI_API_KEY` | OpenAI API key (for `openai_chat` backend) |
+| `ANTHROPIC_API_KEY` | Anthropic API key (for `claude_code_exec` backend) |