docs: sync documentation with post-v0.2 changes
This commit is contained in:
Yif-Yang
+86
-83
@@ -1,112 +1,115 @@
|
||||
# OpenClaw Plugin for SkillOpt-Sleep
|
||||
# OpenClaw reference adaptation for SkillOpt-Sleep
|
||||
|
||||
Thin shell for running [SkillOpt-Sleep](https://github.com/microsoft/SkillOpt) on [OpenClaw](https://github.com/openclaw/openclaw).
|
||||
This directory is a contributed reference for connecting
|
||||
[SkillOpt-Sleep](https://github.com/microsoft/SkillOpt) to
|
||||
[OpenClaw](https://github.com/openclaw/openclaw) with a custom DeepSeek/Ollama
|
||||
backend.
|
||||
|
||||
## What it does
|
||||
> **Reference status.** This is not one of the shared, plug-and-play
|
||||
> `skillopt_sleep` wrappers. Several scripts and the sample config contain
|
||||
> environment-specific absolute paths and assumptions from the original setup,
|
||||
> and the contributed wrapper has unresolved Python 3.10 syntax and backend
|
||||
> factory-signature gaps. The current checkout is not directly runnable; treat
|
||||
> it as porting source material, not an installation.
|
||||
|
||||
Adds a nightly "sleep cycle" to any OpenClaw agent. The cycle:
|
||||
## Included components
|
||||
|
||||
1. **Harvests** recent session transcripts from `~/.openclaw/agents/<name>/sessions/*.jsonl`
|
||||
2. **Mines** recurring task patterns using the optimizer LLM
|
||||
3. **Replays** each pattern with the current `SKILL.md` (baseline) and a candidate `SKILL.md` (with proposed edits)
|
||||
4. **Gates** the candidate against the held-out score (rejects regressions)
|
||||
5. **Stages** the accepted proposal in `~/.skillopt-sleep/staging/<night>/`
|
||||
6. Leaves adoption to the operator (Ethan)
|
||||
| File | Purpose |
|
||||
|---|---|
|
||||
| `run_sleep.py` | custom cycle entry point |
|
||||
| `skillopt_sleep_openclaw.py` | DeepSeek Chat Completions backend plus local Ollama embeddings |
|
||||
| `run_sleep_cron.sh` | category-oriented cron wrapper |
|
||||
| `slash_sleep.py` | experimental `/sleep` command helper |
|
||||
| `config.json` | example engine configuration |
|
||||
| `SKILL.md` | OpenClaw skill manifest |
|
||||
| `tests/*.json` | example task sets for research, DevOps, and wiki workflows |
|
||||
|
||||
Nothing live changes until you adopt. Every adopt backs up first.
|
||||
The adaptation imports the shared engine but registers its own backend and
|
||||
maintains its own wrapper behavior. Changes to the shared CLI documentation do
|
||||
not automatically make every option available through these custom scripts.
|
||||
|
||||
## Install
|
||||
## Intended cycle
|
||||
|
||||
The plugin is a thin wrapper around the engine at `~/.openclaw/workspace/SkillOpt/skillopt_sleep/`:
|
||||
|
||||
```bash
|
||||
# 1. Clone the engine (one-time)
|
||||
cd ~/.openclaw/workspace
|
||||
git clone https://github.com/microsoft/SkillOpt.git
|
||||
|
||||
# 2. Install the OpenClaw skill (this folder)
|
||||
ln -s /path/to/openclaw ~/.openclaw/workspace/skills/skillopt-sleep
|
||||
|
||||
# 3. Configure
|
||||
cp ~/.openclaw/workspace/skills/skillopt-sleep/config.json ~/.skillopt-sleep/config.json
|
||||
$EDITOR ~/.skillopt-sleep/config.json
|
||||
# Set backend = "openclaw-deepseek"
|
||||
# Set model = "deepseek-v4-pro" (or "deepseek-v4-flash" for budget)
|
||||
|
||||
# 4. Set API key
|
||||
echo 'export DEEPSEEK_API_KEY="sk-..."' >> ~/.openclaw/.env
|
||||
|
||||
# 5. Add the nightly cron
|
||||
(crontab -l 2>/dev/null; echo "0 3 * * * cd ~/.openclaw/workspace/skills/skillopt-sleep && bash run_sleep_cron.sh >> ~/.skillopt-sleep/nightly.log 2>&1") | crontab -
|
||||
```text
|
||||
harvest supported session data or load a task file
|
||||
→ replay with the current skill
|
||||
→ propose bounded edits
|
||||
→ validate the candidate on held-out tasks
|
||||
→ stage a proposal for operator review
|
||||
```
|
||||
|
||||
## Use
|
||||
The intended safety boundary is manual adoption: review the generated report and
|
||||
staged files before changing a live skill.
|
||||
|
||||
### Manual trigger
|
||||
## Adapt before use
|
||||
|
||||
```bash
|
||||
# Run one cycle now
|
||||
python3 ~/.openclaw/workspace/skills/skillopt-sleep/run_sleep.py
|
||||
1. Clone SkillOpt into a location you control:
|
||||
|
||||
# Dry run (report only)
|
||||
python3 ~/.openclaw/workspace/skills/skillopt-sleep/run_sleep.py --dry-run
|
||||
```bash
|
||||
git clone https://github.com/microsoft/SkillOpt.git
|
||||
cd SkillOpt/plugins/openclaw
|
||||
```
|
||||
|
||||
# One category only
|
||||
python3 ~/.openclaw/workspace/skills/skillopt-sleep/run_sleep.py --tasks tests/research-cron-tasks.json
|
||||
```
|
||||
2. Inspect and replace the sample absolute paths in `run_sleep.py`,
|
||||
`slash_sleep.py`, `run_sleep_cron.sh`, and `config.json`. Confirm the engine
|
||||
checkout, OpenClaw workspace, state directory, skill directory, and task-file
|
||||
paths all point to isolated test locations.
|
||||
|
||||
### Slash command
|
||||
3. Review `config.json`. In particular, do not assume that values such as
|
||||
`max_tokens_per_night` or `replay_mode` are enforced by this custom wrapper
|
||||
merely because they appear in the example config.
|
||||
|
||||
```bash
|
||||
# In any OpenClaw session
|
||||
/sleep status
|
||||
/sleep run
|
||||
/sleep run research-cron
|
||||
/sleep dry-run
|
||||
/sleep adopt # adopt most recent accepted proposal
|
||||
/sleep reject # discard most recent
|
||||
/sleep cost
|
||||
```
|
||||
4. Supply credentials through your normal secret-management mechanism. Do not
|
||||
commit a DeepSeek key or place it in a world-readable file.
|
||||
|
||||
## Architecture
|
||||
5. Resolve every known porting gap listed in [`SKILL.md`](SKILL.md), add
|
||||
isolated tests for your adapted backend, and verify that `--help` imports
|
||||
cleanly on Python 3.10+. Only then start with a dry run and one reviewed
|
||||
task file. The target command should be shaped like:
|
||||
|
||||
```
|
||||
plugins/openclaw/
|
||||
├── README.md # this file
|
||||
├── run_sleep_cron.sh # wrapper for cron invocation
|
||||
├── run_sleep.py # main entry point
|
||||
├── slash_sleep.py # /sleep command implementation
|
||||
├── skillopt_sleep_openclaw.py # DeepSeek + Ollama backend
|
||||
├── config.json # engine config
|
||||
├── SKILL.md # OpenClaw skill manifest
|
||||
└── tests/ # held-out test sets
|
||||
├── research-cron-tasks.json
|
||||
├── devops-tasks.json
|
||||
└── wiki-tasks.json
|
||||
```
|
||||
```bash
|
||||
cd /path/to/SkillOpt/plugins/openclaw
|
||||
python3 run_sleep.py --config /path/to/reviewed-config.json \
|
||||
--tasks tests/research-cron-tasks.json --dry-run
|
||||
```
|
||||
|
||||
The OpenClaw shell is one engine (skillopt_sleep/) + one backend (DeepSeek/Ollama) + four thin wrappers (cron, slash, skill, tests).
|
||||
6. Inspect the report, paths, network destinations, and proposed edits before
|
||||
considering a non-dry run or scheduling.
|
||||
|
||||
## Why this matters for OpenClaw
|
||||
## Data boundary
|
||||
|
||||
OpenClaw currently has no built-in "self-evolving skills" mechanism. The community has:
|
||||
The custom `openclaw-deepseek` backend sends task, skill, response, rubric, and
|
||||
reflection content to the configured DeepSeek endpoint. Its embedding helper can
|
||||
send truncated text to the configured local Ollama service. Do not assume these
|
||||
outbound prompts have been fully redacted; inspect transcript/task inputs and the
|
||||
provider's retention policy before using real data.
|
||||
|
||||
- **Manual skills** — Ethan writes them
|
||||
- **LLM-generated skills** — one-shot, no validation
|
||||
- **Self-revision** — unbounded, no quality bar
|
||||
Use HTTPS for a remote DeepSeek-compatible endpoint. Keep any plaintext Ollama
|
||||
endpoint on a trusted loopback interface. For a network-free engine smoke test,
|
||||
use the shared SkillOpt-Sleep CLI with `--backend mock` rather than assuming this
|
||||
custom wrapper is isolated.
|
||||
|
||||
SkillOpt-Sleep adds a 4th option: **validated self-evolution**. The skill is the training target, the engine is the optimizer, the gate is the quality bar, the operator is the human-in-the-loop.
|
||||
## Scheduling
|
||||
|
||||
## Validation
|
||||
`run_sleep_cron.sh` and the scheduling helpers are examples, not portable
|
||||
installers. Adapt their paths, create log directories, verify their environment,
|
||||
and run the exact command manually before adding a cron entry. Scheduled runs
|
||||
must preserve the same manual-adoption and credential boundaries as interactive
|
||||
runs.
|
||||
|
||||
Validated on the public [gbrain-evals](https://github.com/garrytan/gbrain-evals) `skillopt-v1` benchmark with real Claude and Codex (deficient skills 0.00 → 1.00 on held-out, all 4 seeds).
|
||||
## Validation scope
|
||||
|
||||
End-to-end test on our own 14-task held-out set: pipeline runs, gate correctly rejects non-improvements, staging artifacts land in `~/.skillopt-sleep/staging/<night>/`.
|
||||
The bundled JSON files are example held-out task sets, not a universal OpenClaw
|
||||
benchmark. Provider cost and quality depend on the selected model, task content,
|
||||
number of calls, and pricing at run time; this reference does not promise a fixed
|
||||
nightly cost. Validate the adapted workflow in an isolated workspace before
|
||||
using it on live skills.
|
||||
|
||||
## Cost
|
||||
|
||||
Measured: ~$0.02/night with `deepseek-v4-pro` at 12 tasks/night. ~$0.59/month, $7.18/year.
|
||||
For the supported shared-engine CLI and its current flags, see the
|
||||
[integration reference](../README.md#supported-cli-surface). For measured
|
||||
SkillOpt-Sleep results and limitations, see
|
||||
[`docs/sleep/RESULTS.md`](../../docs/sleep/RESULTS.md).
|
||||
|
||||
## License
|
||||
|
||||
MIT (same as SkillOpt core).
|
||||
MIT, consistent with SkillOpt core.
|
||||
|
||||
+81
-107
@@ -1,129 +1,103 @@
|
||||
---
|
||||
name: skillopt-sleep
|
||||
description: Validate and refine agent skills through nightly sleep cycles with held-out gates. Wraps Microsoft's SkillOpt-Sleep engine for the OpenClaw/DeepSeek stack.
|
||||
description: Reference-only OpenClaw adaptation of SkillOpt-Sleep. Use it to study or port the contributed DeepSeek wrapper, not as a ready-to-run installation.
|
||||
---
|
||||
|
||||
# skillopt-sleep — OpenClaw Adaptation of Microsoft SkillOpt-Sleep
|
||||
# SkillOpt-Sleep OpenClaw reference adaptation
|
||||
|
||||
A nightly self-improvement loop that reads our session transcripts, mines recurring workflow patterns, replays them with proposed skill edits, and gates the proposals against a held-out test set. Only improvements that beat baseline are staged for human adoption.
|
||||
This directory is a contributed **reference**, not a supported, plug-and-play
|
||||
OpenClaw integration. It illustrates one way to connect the shared
|
||||
`skillopt_sleep` cycle to a custom DeepSeek Chat Completions backend and a set of
|
||||
environment-specific task fixtures.
|
||||
|
||||
## When To Use
|
||||
Do not run or schedule the files unchanged. Several scripts and the sample
|
||||
configuration preserve assumptions from the contributor's original machine,
|
||||
and parts of the wrapper have not yet been ported to the current shared-engine
|
||||
interfaces. Start with the directory's [README.md](README.md), which is the
|
||||
authoritative status and adaptation guide.
|
||||
|
||||
- After Hermes's Weekly Skill Review (or as its replacement)
|
||||
- When a skill is being used 10+ times/week and could be tighter
|
||||
- Before promoting a new skill from `skill-proposals/` to `skills/`
|
||||
- When a skill regresses in observed quality
|
||||
## What is included
|
||||
|
||||
## What It Does (One Cycle)
|
||||
- `skillopt_sleep_openclaw.py` — a contributed DeepSeek backend prototype. It
|
||||
also contains an Ollama embedding helper, but that helper is not wired into
|
||||
the current shared sleep cycle.
|
||||
- `run_sleep.py` — a custom cycle wrapper with environment-specific paths and a
|
||||
backend-registration shim.
|
||||
- `slash_sleep.py` — an experimental command helper written for an older
|
||||
staging-manifest shape.
|
||||
- `run_sleep_cron.sh` — a machine-specific category runner, not a portable cron
|
||||
installer.
|
||||
- `config.json` — a sample configuration, not a set of guaranteed or enforced
|
||||
runtime limits.
|
||||
- `tests/*.json` — example task fixtures from one environment, not a universal
|
||||
OpenClaw benchmark.
|
||||
|
||||
```
|
||||
harvest session transcripts -> mine recurring task patterns
|
||||
-> replay each pattern (current skill vs proposed)
|
||||
-> GATE: must improve held-out score
|
||||
-> stage proposal
|
||||
-> Ethan adopts (manual)
|
||||
```
|
||||
## Known porting gaps
|
||||
|
||||
Nothing live changes until Ethan adopts. Every adopt backs up first.
|
||||
Before treating this as an integration, a maintainer must at least:
|
||||
|
||||
## Architecture
|
||||
1. Replace every absolute workspace, repository, state, skill, log, and task
|
||||
path with explicit user configuration.
|
||||
2. Update the custom backend factory to the current `get_backend` call contract,
|
||||
including the project directory, and update its backend methods and edit
|
||||
records to the current protocol.
|
||||
3. Replace the experimental adoption logic with the current staging manifest
|
||||
and `skillopt_sleep.staging.adopt` behavior. Current staging artifacts use
|
||||
`proposed_SKILL.md` / `proposed_CLAUDE.md`, `manifest.json`, and report files;
|
||||
they do not expose the old `manifest.proposed_skill` field.
|
||||
4. Decide how real OpenClaw transcripts are converted into a supported session
|
||||
format. Pointing `claude_home` at an arbitrary agent directory does not by
|
||||
itself make its files Claude Code-compatible JSONL.
|
||||
5. Build scheduling around the adapted wrapper. The shared scheduler launches
|
||||
the shared CLI; it does not automatically preserve this custom backend or
|
||||
its category task-file flow.
|
||||
6. Add isolated end-to-end tests for dry-run, accepted/rejected gates, staging,
|
||||
adoption and backup, credential failure, and scheduled execution.
|
||||
|
||||
```
|
||||
skills/skillopt-sleep/
|
||||
├── SKILL.md # this file
|
||||
├── config.json # engine config (backend, budgets, etc.)
|
||||
├── run_sleep.py # entry point
|
||||
└── skillopt_sleep_openclaw.py # DeepSeek/Ollama backend
|
||||
```
|
||||
Until those gaps are resolved, use the supported shared
|
||||
`python -m skillopt_sleep` CLI with `--backend mock` to test SkillOpt-Sleep itself,
|
||||
and treat this directory only as source material for a future OpenClaw port.
|
||||
|
||||
The engine itself is at `~/.openclaw/workspace/SkillOpt/skillopt_sleep/` (cloned from microsoft/SkillOpt).
|
||||
## Shared-engine features are not wrapper features
|
||||
|
||||
## Usage
|
||||
At this revision the supported shared CLI backends are `mock`, `claude`,
|
||||
`codex`, `copilot`, `handoff`, and `azure_openai`; the
|
||||
[plugin integration reference](../README.md#supported-cli-surface) is the
|
||||
authoritative list. The shared engine can consolidate a selected skill and
|
||||
project `CLAUDE.md` memory (controlled by `evolve_skill` and `evolve_memory`),
|
||||
and its `schedule` / `unschedule` actions manage shared-engine cron entries.
|
||||
Those capabilities do **not** make the custom OpenClaw wrapper portable: the
|
||||
shared scheduler will not invoke the prototype backend or its category
|
||||
fixtures. Use the shared documentation for those features, not this reference
|
||||
SKILL.
|
||||
|
||||
```bash
|
||||
# Run one cycle with current config
|
||||
cd ~/.openclaw/workspace/skills/skillopt-sleep
|
||||
python3 run_sleep.py
|
||||
## Data and credential boundary
|
||||
|
||||
# Dry run (report only, no staging)
|
||||
python3 run_sleep.py --dry-run
|
||||
The prototype DeepSeek backend sends task, skill, memory, response, rubric, and
|
||||
reflection content to its configured Chat Completions endpoint. Its source also
|
||||
contains a helper that can send text to an Ollama service if a future port wires
|
||||
that helper into the cycle. Neither path should be assumed to remove every
|
||||
secret or private detail.
|
||||
|
||||
# Use a pre-built task set (recommended for testing)
|
||||
python3 run_sleep.py --tasks tests/research-cron-tasks.json
|
||||
```
|
||||
Before any port is tested with real data:
|
||||
|
||||
## Scheduling
|
||||
- use isolated, synthetic or explicitly reviewed task files;
|
||||
- replace sample business names, personal references, URLs, and machine paths;
|
||||
- load credentials through the operator's secret-management mechanism;
|
||||
- verify TLS and retention policy for every remote endpoint; and
|
||||
- inspect all staged artifacts before adoption.
|
||||
|
||||
```bash
|
||||
python3 slash_sleep.py schedule --hour 3 --minute 17
|
||||
python3 slash_sleep.py unschedule
|
||||
python3 slash_sleep.py unschedule --all
|
||||
```
|
||||
The bundled fixtures are examples only. Their scores and any old cost estimates
|
||||
do not establish effectiveness, safety, or a stable nightly price for another
|
||||
OpenClaw deployment.
|
||||
|
||||
Installs a nightly cron entry using the shared SkillOpt-Sleep scheduler. This is an alternative to the external `run_sleep_cron.sh` script.
|
||||
## Further information
|
||||
|
||||
## Alternative backends
|
||||
- [OpenClaw README](README.md) — current reference status and adaptation checklist
|
||||
- [plugin integration reference](../README.md) — supported shared-engine CLI
|
||||
surface and data boundary
|
||||
- [SkillOpt-Sleep documentation](../../docs/sleep/README.md) — concepts,
|
||||
results, and limitations
|
||||
|
||||
While OpenClaw defaults to `openclaw-deepseek` (DeepSeek V4 Pro + Ollama), the shared engine also supports:
|
||||
- `--backend mock` — deterministic, no API spend (for testing)
|
||||
- `--backend claude` — uses the Claude CLI
|
||||
- `--backend codex` — uses the Codex CLI
|
||||
- `--backend copilot` — uses the GitHub Copilot CLI
|
||||
|
||||
These can be used via the engine directly (`python -m skillopt_sleep`).
|
||||
|
||||
## Shared-engine flags
|
||||
|
||||
When invoking the engine directly, all standard flags are available:
|
||||
- `--source codex` / `--source auto` — harvest from Codex Desktop sessions
|
||||
- `--tasks-file PATH` — use a pre-built task set
|
||||
- `--target-skill-path PATH` — explicit SKILL.md target
|
||||
- `--max-tasks N` / `--max-sessions N` — cap workload
|
||||
- `--progress` — print phase progress
|
||||
- `--json` — machine-readable output
|
||||
- `--auto-adopt` — auto-adopt if gate passes
|
||||
|
||||
Config keys: `preferences`, `gate_mode`, `gate_metric`, `dream_rollouts`, `recall_k`, `evolve_memory`, `evolve_skill`.
|
||||
|
||||
## Config (config.json)
|
||||
|
||||
Key knobs:
|
||||
- `backend: "openclaw-deepseek"` — our custom backend
|
||||
- `model: "deepseek-v4-pro"` — optimizer model
|
||||
- `edit_budget: 3` — max bounded edits per night
|
||||
- `gate_mode: "on"` — validation-gated (rejects regressions)
|
||||
- `auto_adopt: false` — require Ethan to adopt manually
|
||||
- `max_tasks_per_night: 12` — cap to control cost
|
||||
|
||||
## Cost Estimate
|
||||
|
||||
Per night: 12 tasks × (1 attempt + 1 judge + 1 reflect) × ~$0.005/1K tokens × ~3K tokens/call ≈ **$0.50-2.00/night**.
|
||||
|
||||
## Outputs
|
||||
|
||||
- Report: `~/.skillopt-sleep/state.json` (running totals)
|
||||
- Staging: `~/.skillopt-sleep/staging/<night>/`
|
||||
- `report.md` — readable summary
|
||||
- `best_skill.md` — proposed skill
|
||||
- `edits.json` — bounded edit list
|
||||
- `before.md` / `after.md` — diffs
|
||||
|
||||
## Held-Out Test Sets (Phase 2)
|
||||
|
||||
Located at `tests/<category>-tasks.json`. Each task has:
|
||||
- `prompt` — the recurring task
|
||||
- `reference` — exact-match gold answer
|
||||
- `rubric` — soft score rubric (0-1)
|
||||
- `domain` — research/devops/wiki/etc.
|
||||
|
||||
Currently building for 3 categories:
|
||||
- research-cron-output
|
||||
- devops-infrastructure-check
|
||||
- wiki-canonical-guide
|
||||
|
||||
## When NOT To Use
|
||||
|
||||
- For a one-off workflow (not a recurring pattern)
|
||||
- During a crisis/incident (humans must lead)
|
||||
- When session transcripts are < 24h old (not enough signal)
|
||||
- For skills < 300 tokens (over-optimization risk)
|
||||
Contributions that turn this reference into a portable integration should add
|
||||
tests and update all three documents together.
|
||||
|
||||
Reference in New Issue
Block a user