url/SkillOpt
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: sync documentation with post-v0.2 changes

Browse Source
This commit is contained in:
Yif-Yang
2026-07-14 17:11:40 +00:00
parent efb30b4bcc
commit f31bf8c06b
44 changed files with 2285 additions and 2025 deletions
Download Patch File Download Diff File Expand all files Collapse all files
plugins/copilot/README.md
+17 -9
View File
@@ -27,8 +27,8 @@ Requires Python ≥ 3.10. No third-party packages — the server is pure stdlib.
"mcpServers": {
"skillopt-sleep": {
"command": "python3",
"args": ["/abs/path/SkillOpt-Sleep/plugins/copilot/mcp_server.py"],
"env": { "SKILLOPT_SLEEP_REPO": "/abs/path/SkillOpt-Sleep" }
"args": ["/abs/path/SkillOpt/plugins/copilot/mcp_server.py"],
"env": { "SKILLOPT_SLEEP_REPO": "/abs/path/SkillOpt" }
}
}
}
@@ -42,14 +42,21 @@ Requires Python ≥ 3.10. No third-party packages — the server is pure stdlib.
## Use
Ask Copilot things like *"run the sleep cycle"*, *"what did the last sleep
propose?"*, *"adopt the staged sleep proposal"*. Copilot calls the MCP tools:
`sleep_status`, `sleep_dry_run`, `sleep_run`, `sleep_adopt`, `sleep_harvest`.
propose?"*, *"adopt the staged sleep proposal"*. The server exposes seven MCP
tools: `sleep_status`, `sleep_dry_run`, `sleep_run`, `sleep_adopt`,
`sleep_harvest`, `sleep_schedule`, and `sleep_unschedule`.
Each tool takes optional `project`, `backend` (`mock`/`claude`/`codex`/`copilot`), and
`scope` arguments. Default backend is `mock` (no API spend). The `copilot`
backend drives the GitHub Copilot CLI (`copilot -p ... --output-format json`)
and requires the `copilot` CLI to be installed and authenticated.
Harvesting is local and read-only, and the default `mock` backend makes no
provider calls. A real backend sends truncated transcript excerpts and derived
tasks to the selected provider. Outbound prompts are not currently guaranteed
to be secret-free; review sensitive data and provider policy first. See the
[shared data-boundary guidance](../README.md#data-boundary).
For speed, the `copilot` backend runs each call against an isolated
`COPILOT_HOME` with built-in MCP servers and custom instructions disabled, so
your user MCP servers (including this project's own) are not spawned per call
@@ -65,12 +72,13 @@ printf '%s\n' \
'{"jsonrpc":"2.0","id":2,"method":"tools/list"}' \
| SKILLOPT_SLEEP_REPO="$(pwd)" python3 plugins/copilot/mcp_server.py
```
You should see the server info and the five `sleep_*` tools.
You should see the server info and all seven `sleep_*` tools.
## Notes / status
- MCP is the stable, official Copilot extension surface, so this is the most
portable of the three integrations (one server → CLI + IDE).
- The engine and all its controls (gate on/off, multi-rollout, budget,
preferences, optimizer/target split) are identical across platforms — see
[the SkillOpt-Sleep guide section](https://microsoft.github.io/SkillOpt/docs/guideline.html#sleep).
portable shared-engine integration (one server → CLI + IDE).
- The MCP schema exposes the main CLI's implemented controls, including task and
session caps, target-skill selection, scheduling, and staged adoption. It does
not add experiment-only gate, rollout, token/time-budget, or optimizer/target
split flags. See the [shared CLI reference](../README.md#supported-cli-surface).
plugins/copilot/copilot-instructions.snippet.md
+22 -9
View File
@@ -8,26 +8,29 @@ automatically as ambient guidance.)
This project has SkillOpt-Sleep available via an MCP server (`skillopt-sleep`).
It gives the agent a nightly "sleep cycle": it reviews past sessions, replays
recurring tasks offline, and consolidates validated memory + skills behind a
held-out gate.
recurring tasks through a selected backend, and stages validation-gated changes
to project `CLAUDE.md` and a configured `SKILL.md`.
When the user asks to "run the sleep cycle", "review my past sessions", "learn
my preferences", or "make the agent improve from past usage", use the MCP tools:
- `sleep_status` — what's happened + the latest staged proposal
- `sleep_dry_run` — safe preview, stages nothing
- `sleep_run` — full cycle, stages a reviewed proposal (nothing live changes)
- `sleep_adopt` — apply the staged proposal (backs up first)
- `sleep_dry_run` — no-staging preview; a real backend still makes provider calls
- `sleep_run` — full cycle, stages a validation-gated proposal by default;
explicit `auto_adopt` may update live files
- `sleep_adopt` — apply the staged proposal (backs up an existing live file first)
- `sleep_harvest` — list mined recurring tasks
- `sleep_schedule` — install a nightly cron entry (set `hour`/`minute`)
- `sleep_unschedule` — remove the nightly cron entry
### Key parameters (pass as MCP tool arguments)
- `backend` — `mock` (default, free), `claude`, `codex`, or `copilot`
- `backend` — `mock` (default, no provider calls), `claude`, `codex`, or `copilot`
- `source` — `claude`, `codex`, or `auto` (where to read transcripts)
- `target_skill_path` — explicit SKILL.md to evolve
- `tasks_file` — pre-built TaskRecord JSON (skip harvest)
- `target_skill_path` — explicit SKILL.md to evolve; use this for a skill that
the current agent actually loads
- `tasks_file` — reviewed TaskRecord JSON (skip harvest); real backends require
its metadata to contain `"reviewed": true`
- `max_tasks` / `max_sessions` — cap workload
- `auto_adopt` — auto-adopt if the gate passes
- `json` — machine-readable output for programmatic use
@@ -40,4 +43,14 @@ my preferences", or "make the agent improve from past usage", use the MCP tools:
Always show the user the held-out baseline → candidate score and the proposed
edits before suggesting `sleep_adopt`. Never hand-edit the user's memory/skill
files; only `sleep_adopt` does that, with a backup.
files; use `sleep_adopt` (or an explicitly requested `auto_adopt`) so the engine
applies its staging manifest and backup behavior.
Harvesting is local and read-only, and `backend: "mock"` makes no provider
calls. A real backend sends truncated transcript excerpts and derived tasks to
the selected provider; outbound prompts are not guaranteed to be secret-free.
Review sensitive data and provider policy before selecting a real backend.
`sleep_schedule` persists only the project, backend, time, and optional
auto-adopt setting. Put a non-default transcript source or target skill in
`~/.skillopt-sleep/config.json` before scheduling it.
plugins/copilot/skillopt/README.md
+14 -5
View File
@@ -66,11 +66,20 @@ skill"*, or *"evaluate this skill on the dataset"*. Copilot calls the MCP tools:
| `skillopt_train` | `config` | Runs a reflective optimization loop. Long-running; spends budget. |
| `skillopt_eval` | `config`, `skill` | Evaluates one skill markdown file; no training. |
Common optional args (both train and eval): `env`, `backend`,
`optimizer_model`, `target_model`, `out_root`, `cfg_options` (space-separated
`KEY=VALUE` YAML overrides), and `extra_args` (raw passthrough flags for the
underlying script). `skillopt_train` also accepts `num_epochs`, `batch_size`,
`seed`, and `use_gate`.
Common optional args (both train and eval): `env`, `backend`,
`optimizer_model`, `target_model`, `out_root`, `cfg_options` (space-separated
`section.key=value` YAML overrides), and `extra_args` (raw passthrough flags
for the underlying script). `skillopt_train` also accepts `num_epochs`,
`batch_size`, `seed`, and `use_gate`. `use_gate` defaults to `true`; setting it
to `false` still records validation scores but force-accepts every candidate,
which changes the optimization semantics.
The MCP schema's `backend` argument follows the underlying script's
`--backend` choices. Role-specific and generic OpenAI-compatible backends can
be selected through config or `extra_args` (for example,
`--optimizer_backend openai_compatible --target_backend openai_compatible`);
see the repository's [backend guide](../../../docs/guide/new-backend.md) for
the required environment variables.
Runs can be very long. The server's subprocess timeout defaults to 6 hours;
override it with the `SKILLOPT_RUN_TIMEOUT` environment variable (seconds).
plugins/copilot/skillopt/copilot-instructions.snippet.md
+34 -33
View File
@@ -1,33 +1,34 @@
<!--
Copy this block into your repo's .github/copilot-instructions.md so Copilot
knows the SkillOpt research-engine tools exist. (Copilot reads
copilot-instructions.md automatically as ambient guidance.)
-->
## SkillOpt (research skill-optimization engine)
This repo exposes the core **SkillOpt** training/eval engine via an MCP server
(`skillopt`). SkillOpt is validation-gated, text-space skill optimization: it
reflects on rollouts, makes bounded edits to a skill, and keeps a change only
if it improves a held-out validation set.
When the user asks to "optimize a skill", "train on <benchmark>", "run
SkillOpt", "evaluate this skill", or "what configs can I run", use the MCP
tools:
- `skillopt_list_configs` — list the benchmark YAML configs you can pass as `config`
- `skillopt_train` — run a reflective skill-optimization loop on a config (long-running; spends API/compute budget)
- `skillopt_eval` — evaluate a single skill markdown file on a dataset (no training)
Guidance:
- Always run `skillopt_list_configs` first if you don't already know a valid `config` path.
- `skillopt_train` and `skillopt_eval` are long-running and consume the user's
model backend/budget — confirm the `config`, `backend`, and model choices
with the user before launching, and surface the held-out gate result when the
run finishes.
- For one-off YAML overrides use `cfg_options` (e.g. `seed=123 batch_size=40`);
for any other underlying flag use `extra_args`.
This is distinct from the **SkillOpt-Sleep** MCP server (`skillopt-sleep`,
`sleep_*` tools), which evolves a local coding agent from past sessions rather
than running the research benchmarks.
<!--
Copy this block into your repo's .github/copilot-instructions.md so Copilot
knows the SkillOpt research-engine tools exist. (Copilot reads
copilot-instructions.md automatically as ambient guidance.)
-->
## SkillOpt (research skill-optimization engine)
This repo exposes the core **SkillOpt** training/eval engine via an MCP server
(`skillopt`). SkillOpt is validation-gated, text-space skill optimization: it
reflects on rollouts, makes bounded edits to a skill, and keeps a change only
if it improves a held-out validation set.
When the user asks to "optimize a skill", "train on <benchmark>", "run
SkillOpt", "evaluate this skill", or "what configs can I run", use the MCP
tools:
- `skillopt_list_configs` — list the benchmark YAML configs you can pass as `config`
- `skillopt_train` — run a reflective skill-optimization loop on a config (long-running; spends API/compute budget)
- `skillopt_eval` — evaluate a single skill markdown file on a dataset (no training)
Guidance:
- Always run `skillopt_list_configs` first if you don't already know a valid `config` path.
- `skillopt_train` and `skillopt_eval` are long-running and consume the user's
model backend/budget — confirm the `config`, `backend`, and model choices
with the user before launching. Surface the held-out gate result for training,
or the evaluation score and output directory for eval-only runs.
- For one-off YAML overrides use dotted `cfg_options` for structured configs
(e.g. `train.seed=123 train.batch_size=40`);
for any other underlying flag use `extra_args`.
This is distinct from the **SkillOpt-Sleep** MCP server (`skillopt-sleep`,
`sleep_*` tools), which evolves a local coding agent from past sessions rather
than running the research benchmarks.