Add docs/guideline.html, a single self-contained documentation guide
(left-nav + content + on-this-page TOC) covering installation, data
preparation, training/eval, full configuration reference, framework
internals, and an API reference. Link it from the README with local,
htmlpreview, and GitHub Pages access instructions.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-31 09:01:25 +00:00
2 changed files with 934 additions and 0 deletions
<title>SkillOpt — Documentation & Reproduction Guide</title>
<metaname="description"content="Complete documentation and reproduction guide for SkillOpt: installation, data preparation, training, configuration reference, framework internals, and API reference.">
<divclass="glabel"><spanclass="num">8</span> API Reference</div>
<ahref="#module-map">Module map</a>
<ahref="#functions">Core functions</a>
<ahref="#cli">CLI scripts</a>
<ahref="#webui">WebUI</a>
</div>
</nav>
<!-- ───────────── MAIN CONTENT ───────────── -->
<mainclass="content">
<spanclass="eyebrow">Microsoft Research</span>
<h1>SkillOpt Documentation & Reproduction Guide</h1>
<pclass="lead">Train agent skills like you train neural networks — with epochs, (mini-)batch size, learning rates, and validation gates — but without touching any model weights.</p>
<p>This guide walks you from a clean checkout to a reproduced result and a full reference for every configuration knob and core function. It is generated from, and kept consistent with, the current state of the codebase.</p>
<h2>1.1 What is SkillOpt <aclass="anchor"href="#what-is">#</a></h2>
<p><strong>SkillOpt</strong> is a text-space optimizer that improves a <em>frozen</em> language agent by iteratively editing a natural-language <strong>skill document</strong> — never the model weights. The skill document is a Markdown file that conditions a target model as it executes tasks. SkillOpt treats this document as the "weights" and runs a training loop that mirrors deep-learning training: rollout (forward pass), reflect (backward pass / gradients), select & apply edits (optimizer step), and a validation gate (accept/reject).</p>
<p>Two roles split every model call:</p>
<ul>
<li><strong>Target</strong> — executes tasks using the current skill document (the agent being improved).</li>
<li><strong>Optimizer</strong> — analyzes the target's trajectories and proposes edits to the skill document.</li>
</ul>
<p>The same loop drives six benchmarks out of the box (QA, document QA, embodied agents, math, spreadsheet code generation, and tool-augmented QA).</p>
<tr><td>Epoch</td><td>Epoch (+ slow update & meta skill)</td><td><code>engine/trainer.py</code></td></tr>
</tbody>
</table>
</div>
<divclass="note tip"><spanclass="nh">What transfers from DL</span>
<p>Cosine schedule tends to beat constant; moderate learning rates (≈4–16 edits/step) beat very high/low; slow update curbs cross-epoch forgetting; meta-skill memory improves reflection quality. Conversely, bigger rollout batches and many epochs show diminishing returns — skills converge in ~2–4 epochs.</p>
</div>
</section>
<sectionid="features">
<h2>1.3 Key Features <aclass="anchor"href="#features">#</a></h2>
<divclass="cards">
<divclass="card"><h4>Validation gating</h4><p>Every candidate skill is scored on a held-out selection split and only accepted if it beats the current/best skill.</p></div>
<divclass="card"><h4>Slow update</h4><p>Epoch-boundary longitudinal comparison writes guidance into a protected region — momentum against forgetting. Force-injected or selection-gated.</p></div>
<divclass="card"><h4>Meta skill</h4><p>Optimizer-side memory that reflects on what worked across epochs and feeds back into reflection.</p></div>
<divclass="card"><h4>Pluggable backends</h4><p>OpenAI / Azure OpenAI, Anthropic Claude, local Qwen (vLLM), plus Codex/Claude-Code exec backends for the target.</p></div>
<divclass="card"><h4>Six benchmarks</h4><p>SearchQA, DocVQA, ALFWorld, LiveMathematicianBench, SpreadsheetBench, OfficeQA — each a self-contained env module.</p></div>
<divclass="card"><h4>Auto-resume</h4><p>Every run is checkpointed step-by-step; re-running the same command continues from the last completed step.</p></div>
<divclass="note info"><spanclass="nh">One env-var family for all OpenAI modes</span>
<p>SkillOpt reuses the <code>AZURE_OPENAI_*</code> variable names even for plain OpenAI — there is no separate <code>OPENAI_API_KEY</code> knob. <code>AZURE_OPENAI_ENDPOINT</code> is required for every OpenAI auth mode.</p>
<!-- ===================== 3. DATA ===================== -->
<sectionid="split-dir">
<h2>3.1 Split Directory Format <aclass="anchor"href="#split-dir">#</a></h2>
<p>With <code>env.split_mode: split_dir</code> (the recommended, deterministic mode), SkillOpt reads a directory containing <code>train/</code>, <code>val/</code>, and <code>test/</code> subfolders, each holding a JSON array of task items:</p>
<pre><code>data/my_split/
├─ train/items.json <spanclass="tok-c"># used for rollout (the "train split")</span>
<p>Internally the splits are referred to as <code>train</code>, <code>valid_seen</code> (validation/selection), and <code>valid_unseen</code> (test). The <code>--split</code> flag of <code>eval_only.py</code> uses these names.</p>
<p>Required fields depend on the benchmark; consult <code>skillopt/envs/<benchmark>/dataloader.py</code> for the exact contract. A SearchQA item, for example:</p>
<tr><td><code>split_dir</code></td><td>Use a pre-built directory with explicit <code>train/val/test</code> folders (set <code>env.split_dir</code>). Deterministic and reproducible.</td></tr>
<tr><td><code>ratio</code></td><td>Build a deterministic split on the fly from a single <code>env.data_path</code>, using <code>split_seed</code> (and a train:val:test ratio). Convenient for quick experiments.</td></tr>
<p>Swap the config for another benchmark (e.g. <code>configs/livemathematicianbench/default.yaml</code>, <code>configs/alfworld/default.yaml</code>). Common CLI arguments:</p>
<p>Each completed step persists its state to <code>runtime_state.json</code> and a <code>steps/step_XXXX/</code> directory. Re-running the <em>same command</em> against the same <code>out_root</code> detects finished work and continues from the last completed step — including epoch-boundary slow-update and meta-skill stages.</p>
</section>
<!-- ===================== 5. HOW IT WORKS ===================== -->
<sectionid="loop">
<h2>5.1 The Training Loop <aclass="anchor"href="#loop">#</a></h2>
<p>The loop lives in <code>ReflACTTrainer</code> (<code>skillopt/engine/trainer.py</code>). Each epoch runs a series of optimization steps over rollout batches, then performs two epoch-boundary stages.</p>
<tr><td><strong>1. Rollout</strong></td><td>The target model runs each task in the batch with the current skill as context, producing trajectories and scores.</td><td><code>envs/<b>/rollout.py</code></td></tr>
<tr><td><strong>2. Reflect</strong></td><td>The optimizer runs an error analyst (and optional success analyst) over minibatches of trajectories, emitting structured edit patches. Runs in parallel across <code>analyst_workers</code>.</td><td><code>gradient/reflect.py</code></td></tr>
<tr><td><strong>3. Aggregate</strong></td><td>Semantically similar patches are merged hierarchically to remove redundancy.</td><td><code>gradient/aggregate.py</code> → <code>merge_patches</code></td></tr>
<tr><td><strong>4. Select</strong></td><td>Patches are ranked and clipped to the current learning rate (max edits this step), set by the scheduler.</td><td><code>optimizer/clip.py</code> → <code>rank_and_select</code></td></tr>
<tr><td><strong>5. Update</strong></td><td>Selected edits are applied to the skill document, producing a candidate skill (patch / rewrite modes).</td><td><code>optimizer/skill.py</code>, <code>update_modes.py</code></td></tr>
<tr><td><strong>6. Gate</strong></td><td>The candidate is scored on the selection split and accepted only if it improves (see §5.3).</td><td><code>evaluation/gate.py</code> → <code>evaluate_gate</code></td></tr>
<p><code>evaluate_gate</code> is a pure decision function. It compares the candidate's selection-set score against the <em>current</em> and <em>best</em> skills:</p>
<ul>
<li><strong>accept_new_best</strong> — candidate > current <em>and</em> candidate > best → becomes both current and best.</li>
<li><strong>accept</strong> — candidate > current but ≤ best → becomes current only.</li>
<li><strong>reject</strong> — candidate ≤ current → discarded; current/best unchanged.</li>
</ul>
<p>The comparison metric is configurable via <code>evaluation.gate_metric</code>:</p>
<tr><td><code>mixed</code></td><td>Weighted blend, controlled by <code>gate_mixed_weight</code></td></tr>
</tbody>
</table></div>
<divclass="note info"><spanclass="nh">When to use soft/mixed</span>
<p>The <code>soft</code>/<code>mixed</code> metrics (contributed config <code>configs/examples/soft_gate.yaml</code>) help when the selection split is small and rewards are continuous, where a discrete hard gate may reject every candidate and stall training. Paper numbers use the default <code>hard</code> gate.</p>
<p>At each epoch boundary (from epoch 2), the slow update rolls out both the <em>previous</em> epoch's skill and the <em>current</em> skill on the same sampled tasks, categorizes items (improved / regressed / persistent-fail / stable-success), and asks the optimizer to write a free-form <strong>guidance</strong> block. This guidance lands in a <strong>protected region</strong> of the skill that step-level edits cannot touch — only the slow update overwrites it. It is SkillOpt's analogue of momentum, countering cross-epoch forgetting.</p>
<p>Acceptance has two modes, selected by <code>optimizer.slow_update_gate_with_selection</code>:</p>
<tr><td><code>false</code><spanclass="pill def">default</span> — force-injected</td><td>Guidance is injected into both current and best skills unconditionally. The longitudinal guidance always persists; it is not gated by step-level selection scores.</td></tr>
<tr><td><code>true</code> — gated</td><td>The slow-update candidate is scored on the selection split and accepted/rejected through the same validation gate as step-level updates.</td></tr>
</tbody>
</table></div>
</section>
<sectionid="meta-skill">
<h2>5.5 Meta Skill (Optimizer Memory) <aclass="anchor"href="#meta-skill">#</a></h2>
<p>The meta skill is <strong>optimizer-side memory</strong> — it never modifies the target skill document. At the end of each epoch (skipped for epoch 1), the optimizer compares the previous and current epoch's last-step skills on the same sampled tasks and writes a compact, evidence-based reflection on what kind of edits helped or hurt. That memory is then injected as extra context into the next epoch's reflect / merge / learning-rate / ranking stages, so the optimizer accumulates strategy across the run.</p>
<p>A skill document is plain Markdown. Initial skills can be empty (learn from scratch) or seeded with domain knowledge via <code>env.skill_init</code>. During training the document accrues rules, patterns, and edge-case handling through accepted edit patches. A dedicated protected region holds the slow-update guidance, delimited by HTML-comment markers:</p>
<p>Helpers in <code>optimizer/slow_update.py</code> manage this region: <code>inject_empty_slow_update_field</code> (placeholder at epoch 1), <code>extract_slow_update_field</code> (read), and <code>replace_slow_update_field</code> (overwrite). Step-level edits are blocked from modifying anything inside the markers.</p>
<h2>6.1 Configuration System <aclass="anchor"href="#config-system">#</a></h2>
<p>Configs are <strong>structured YAML</strong> with section blocks (<code>model</code>, <code>train</code>, <code>gradient</code>, <code>optimizer</code>, <code>evaluation</code>, <code>env</code>) and <code>_base_</code> inheritance. A benchmark config inherits the shared defaults and overrides only what differs:</p>
<divclass="note info"><spanclass="nh">Reading the tables below</span>
<p>Each section lists the key (relative to its YAML block), type, default (from <code>configs/_base_/default.yaml</code>), allowed values, and meaning. Defaults shown are the shipped base defaults.</p>
<tr><td><code>backend</code></td><td>str</td><tdclass="def">azure_openai</td><td>High-level backend label for the run.</td></tr>
<tr><td><code>optimizer</code></td><td>str</td><tdclass="def">gpt-5.5</td><td>Optimizer model deployment (writes skill edits).</td></tr>
<tr><td><code>target</code></td><td>str</td><tdclass="def">gpt-5.5</td><td>Target model deployment (executes tasks).</td></tr>
<tr><td><code>optimizer_backend</code></td><td>str</td><tdclass="def">openai_chat</td><td>Client path for the optimizer: <code>openai_chat</code> or <code>claude_chat</code>.</td></tr>
<tr><td><code>target_backend</code></td><td>str</td><tdclass="def">openai_chat</td><td>Client path for the target: <code>openai_chat</code> / <code>claude_chat</code> / <code>qwen_chat</code> / <code>codex_exec</code> / <code>claude_code_exec</code>.</td></tr>
<p>Every <code>azure_openai_*</code> key also has <code>optimizer_azure_openai_*</code> and <code>target_azure_openai_*</code> variants, letting you point the optimizer and target at different Azure resources. Exec backends (<code>codex_exec</code>, <code>claude_code_exec</code>) add their own <code>codex_exec_*</code> / <code>claude_code_exec_*</code> knobs (sandbox, reasoning effort, SDK mode, etc.).</p>
<tr><td><code>num_epochs</code></td><td>int</td><tdclass="def">4</td><td>Epochs</td><td>Number of training epochs.</td></tr>
<tr><td><code>train_size</code></td><td>int</td><tdclass="def">0</td><td>Train-set size</td><td>0 = derive from the dataset split. (Fixed by split size when using <code>split_dir</code>.)</td></tr>
<tr><td><code>batch_size</code></td><td>int</td><tdclass="def">40</td><td>Batch size</td><td>Tasks rolled out per optimization step.</td></tr>
<tr><td><code>accumulation</code></td><td>int</td><tdclass="def">1</td><td>Grad accumulation</td><td>Accumulation rounds per step.</td></tr>
<tr><td><code>learning_rate</code></td><td>int</td><tdclass="def">4</td><td>Learning rate</td><td>Max edit patches applied per step (the "edit budget").</td></tr>
<tr><td><code>min_learning_rate</code></td><td>int</td><tdclass="def">2</td><td>Min LR</td><td>Floor edit budget for decaying schedulers.</td></tr>
<tr><td><code>slow_update_samples</code></td><td>int</td><tdclass="def">20</td><td>—</td><td>Tasks sampled for the longitudinal comparison.</td></tr>
<tr><td><code>slow_update_gate_with_selection</code></td><td>bool</td><tdclass="def">false</td><td>—</td><td><code>false</code> = force-inject guidance; <code>true</code> = gate it on the selection split (see §5.4).</td></tr>
<tr><td><code>longitudinal_pair_policy</code></td><td>str</td><tdclass="def">mixed</td><td>—</td><td><code>mixed</code> / <code>changed</code> / <code>unchanged</code> — which comparison pairs to keep.</td></tr>
<tr><td><code>use_gate</code></td><td>bool</td><tdclass="def">true</td><td>Validation gating is mandatory in this branch (must remain <code>true</code>).</td></tr>
<tr><td><code>gate_metric</code></td><td>str</td><tdclass="def">hard</td><td><code>hard</code> / <code>soft</code> / <code>mixed</code> — score used by the gate (see §5.3).</td></tr>
<tr><td><code>gate_mixed_weight</code></td><td>float</td><tdclass="def">0.5</td><td>Weight on the soft score when <code>gate_metric = mixed</code>.</td></tr>
<tr><td><code>sel_env_num</code></td><td>int</td><tdclass="def">0</td><td>Selection-split eval size (0 = use full split).</td></tr>
<tr><td><code>test_env_num</code></td><td>int</td><tdclass="def">0</td><td>Test-split eval size (0 = use full split).</td></tr>
<tr><td><code>eval_test</code></td><td>bool</td><tdclass="def">true</td><td>Run a final test evaluation after training.</td></tr>
</tbody>
</table></div>
<divclass="note warn"><spanclass="nh">Gate is required</span>
<p>Setting <code>evaluation.use_gate: false</code> raises an error — validation gating cannot be disabled in this branch.</p>
<tr><td><code>name</code></td><td>str</td><tdclass="def">""</td><td>Benchmark name (<code>searchqa</code>, <code>docvqa</code>, <code>alfworld</code>, …). Selects the env module.</td></tr>
<tr><td><code>skill_init</code></td><td>str</td><tdclass="def">""</td><td>Path to a seed skill (empty = start from scratch).</td></tr>
<tr><td><code>split_mode</code></td><td>str</td><tdclass="def">ratio</td><td><code>ratio</code> or <code>split_dir</code> (see §3.3).</td></tr>
<p>Env blocks may carry extra benchmark-specific keys (e.g. <code>max_turns</code>, <code>workers</code>, <code>max_completion_tokens</code>, <code>limit</code>). Unmapped env keys are passed straight through to the benchmark adapter — check the relevant <code>configs/<benchmark>/default.yaml</code>.</p>
<p>Each benchmark is a self-contained module under <code>skillopt/envs/<benchmark>/</code> with an <code>adapter.py</code>, <code>dataloader.py</code>, <code>rollout.py</code>, and <code>evaluator.py</code> (some add a custom <code>reflect.py</code>). Packaged reference skills live in <code>ckpt/<benchmark>/</code>.</p>
</section>
<sectionid="bench-new">
<h2>7.2 Add a New Benchmark <aclass="anchor"href="#bench-new">#</a></h2>
<p>Use <code>skillopt/envs/_template/</code> as a starting point. At minimum, implement:</p>
<ol>
<li><strong>Dataloader</strong> — read your item JSON into the framework's item dicts (<code>dataloader.py</code>).</li>
<li><strong>Rollout</strong> — run the target on one item with the current skill and return a trajectory + score (<code>rollout.py</code>).</li>
<li><strong>Evaluator</strong> — score predictions against ground truth (<code>evaluator.py</code>).</li>
<li><strong>Adapter</strong> — wire the above into the trainer's expected interface and register the env name (<code>adapter.py</code>).</li>
</ol>
<p>Then add a <code>configs/<name>/default.yaml</code> inheriting <code>_base_/default.yaml</code> and set <code>env.name</code> to your new benchmark.</p>
</section>
<!-- ===================== 8. API REFERENCE ===================== -->
<tr><td><code>skillopt/config.py</code></td><td>Load structured YAML, resolve <code>_base_</code> inheritance, flatten to the trainer's flat dict, apply CLI overrides.</td></tr>
<tr><td><code>skillopt/engine/trainer.py</code></td><td><code>ReflACTTrainer</code> — orchestrates the whole loop, gating, slow update, meta skill, resume, and artifact writing.</td></tr>
<tr><td><code>run_minibatch_reflect</code></td><td><code>gradient/reflect.py</code></td><td>Run error/success analysts over trajectory minibatches → edit patches.</td></tr>
<tr><td><code>merge_patches</code></td><td><code>gradient/aggregate.py</code></td><td>Hierarchically merge semantically similar patches.</td></tr>
<tr><td><code>rank_and_select</code></td><td><code>optimizer/clip.py</code></td><td>Rank edits and clip to the learning-rate budget.</td></tr>
<tr><td><code>build_scheduler</code></td><td><code>optimizer/scheduler.py</code></td><td>Construct the LR (edit-budget) scheduler: constant/linear/cosine/autonomous.</td></tr>
<tr><td><code>decide_autonomous_learning_rate</code></td><td><code>optimizer/lr_autonomous.py</code></td><td>Let the optimizer pick the next learning rate (autonomous mode).</td></tr>
<tr><td><code>apply_patch</code> / <code>apply_edit</code></td><td><code>optimizer/skill.py</code></td><td>Apply edits to the skill document (respecting the protected region).</td></tr>
<tr><td><code>rewrite_skill_from_suggestions</code></td><td><code>optimizer/rewrite.py</code></td><td>Full-rewrite update mode from accumulated suggestions.</td></tr>
<tr><td><code>replace_slow_update_field</code> / <code>extract_slow_update_field</code></td><td><code>optimizer/slow_update.py</code></td><td>Read/overwrite the protected guidance region.</td></tr>
<tr><td><code>run_meta_skill</code> / <code>format_meta_skill_context</code></td><td><code>optimizer/meta_skill.py</code></td><td>Generate cross-epoch optimizer memory and render it into reflection context.</td></tr>
<p>Runs a full training loop. Required: <code>--config</code>. Override config via <code>--cfg-options section.key=value …</code> or legacy flat flags (<code>--num_epochs</code>, <code>--batch_size</code>, <code>--optimizer_model</code>, <code>--target_model</code>, <code>--lr_scheduler</code>, <code>--edit_budget</code>, <code>--split_dir</code>, …).</p>
<h4>scripts/eval_only.py</h4>
<p>Evaluates a skill document without training. Required: <code>--config</code> and <code>--skill</code>. Use <code>--split</code> to choose <code>train</code> / <code>valid_seen</code> / <code>valid_unseen</code> / <code>all</code>.</p>
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Cuzyoung