url/SkillOpt
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: sync documentation with post-v0.2 changes

Browse Source
This commit is contained in:
Yif-Yang
2026-07-14 17:11:40 +00:00
parent efb30b4bcc
commit f31bf8c06b
44 changed files with 2285 additions and 2025 deletions
Download Patch File Download Diff File Expand all files Collapse all files
plugins/copilot/skillopt/README.md
+14 -5
View File
@@ -66,11 +66,20 @@ skill"*, or *"evaluate this skill on the dataset"*. Copilot calls the MCP tools:
| `skillopt_train` | `config` | Runs a reflective optimization loop. Long-running; spends budget. |
| `skillopt_eval` | `config`, `skill` | Evaluates one skill markdown file; no training. |
Common optional args (both train and eval): `env`, `backend`,
`optimizer_model`, `target_model`, `out_root`, `cfg_options` (space-separated
`KEY=VALUE` YAML overrides), and `extra_args` (raw passthrough flags for the
underlying script). `skillopt_train` also accepts `num_epochs`, `batch_size`,
`seed`, and `use_gate`.
Common optional args (both train and eval): `env`, `backend`,
`optimizer_model`, `target_model`, `out_root`, `cfg_options` (space-separated
`section.key=value` YAML overrides), and `extra_args` (raw passthrough flags
for the underlying script). `skillopt_train` also accepts `num_epochs`,
`batch_size`, `seed`, and `use_gate`. `use_gate` defaults to `true`; setting it
to `false` still records validation scores but force-accepts every candidate,
which changes the optimization semantics.
The MCP schema's `backend` argument follows the underlying script's
`--backend` choices. Role-specific and generic OpenAI-compatible backends can
be selected through config or `extra_args` (for example,
`--optimizer_backend openai_compatible --target_backend openai_compatible`);
see the repository's [backend guide](../../../docs/guide/new-backend.md) for
the required environment variables.
Runs can be very long. The server's subprocess timeout defaults to 6 hours;
override it with the `SKILLOPT_RUN_TIMEOUT` environment variable (seconds).
plugins/copilot/skillopt/copilot-instructions.snippet.md
+34 -33
View File
@@ -1,33 +1,34 @@
<!--
Copy this block into your repo's .github/copilot-instructions.md so Copilot
knows the SkillOpt research-engine tools exist. (Copilot reads
copilot-instructions.md automatically as ambient guidance.)
-->
## SkillOpt (research skill-optimization engine)
This repo exposes the core **SkillOpt** training/eval engine via an MCP server
(`skillopt`). SkillOpt is validation-gated, text-space skill optimization: it
reflects on rollouts, makes bounded edits to a skill, and keeps a change only
if it improves a held-out validation set.
When the user asks to "optimize a skill", "train on <benchmark>", "run
SkillOpt", "evaluate this skill", or "what configs can I run", use the MCP
tools:
- `skillopt_list_configs` — list the benchmark YAML configs you can pass as `config`
- `skillopt_train` — run a reflective skill-optimization loop on a config (long-running; spends API/compute budget)
- `skillopt_eval` — evaluate a single skill markdown file on a dataset (no training)
Guidance:
- Always run `skillopt_list_configs` first if you don't already know a valid `config` path.
- `skillopt_train` and `skillopt_eval` are long-running and consume the user's
model backend/budget — confirm the `config`, `backend`, and model choices
with the user before launching, and surface the held-out gate result when the
run finishes.
- For one-off YAML overrides use `cfg_options` (e.g. `seed=123 batch_size=40`);
for any other underlying flag use `extra_args`.
This is distinct from the **SkillOpt-Sleep** MCP server (`skillopt-sleep`,
`sleep_*` tools), which evolves a local coding agent from past sessions rather
than running the research benchmarks.
<!--
Copy this block into your repo's .github/copilot-instructions.md so Copilot
knows the SkillOpt research-engine tools exist. (Copilot reads
copilot-instructions.md automatically as ambient guidance.)
-->
## SkillOpt (research skill-optimization engine)
This repo exposes the core **SkillOpt** training/eval engine via an MCP server
(`skillopt`). SkillOpt is validation-gated, text-space skill optimization: it
reflects on rollouts, makes bounded edits to a skill, and keeps a change only
if it improves a held-out validation set.
When the user asks to "optimize a skill", "train on <benchmark>", "run
SkillOpt", "evaluate this skill", or "what configs can I run", use the MCP
tools:
- `skillopt_list_configs` — list the benchmark YAML configs you can pass as `config`
- `skillopt_train` — run a reflective skill-optimization loop on a config (long-running; spends API/compute budget)
- `skillopt_eval` — evaluate a single skill markdown file on a dataset (no training)
Guidance:
- Always run `skillopt_list_configs` first if you don't already know a valid `config` path.
- `skillopt_train` and `skillopt_eval` are long-running and consume the user's
model backend/budget — confirm the `config`, `backend`, and model choices
with the user before launching. Surface the held-out gate result for training,
or the evaluation score and output directory for eval-only runs.
- For one-off YAML overrides use dotted `cfg_options` for structured configs
(e.g. `train.seed=123 train.batch_size=40`);
for any other underlying flag use `extra_args`.
This is distinct from the **SkillOpt-Sleep** MCP server (`skillopt-sleep`,
`sleep_*` tools), which evolves a local coding agent from past sessions rather
than running the research benchmarks.