feat(sleep): add handoff backend — session-executed model calls, no API subprocess (#125)
Adds --backend handoff: the engine runs all deterministic stages and outsources attempt/judge/reflect to prompt/answer files an interactive agent session fills between runs (exit 3 = pending batch, re-run to resume). Deterministic replay + the prompt-hash answer cache make resume stateless; sentinel detection aborts any call built from unanswered output so placeholders never reach scores or staging. Session digests and mined tasks are pinned per night (secret-redacted) so the sessions answering prompts cannot shift the task set, and LLM mining is routed through the same handoff files. Ships a /skillopt-sleep-handoff Claude Code command that answers each prompt in a fresh-context subagent to protect the held-out gate. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -60,6 +60,9 @@ they shell out to the CLIs you already have.
|
||||
/skillopt-sleep run # full cycle: stages a reviewed proposal (still no live edits)
|
||||
/skillopt-sleep status # see history + the latest staged proposal
|
||||
/skillopt-sleep adopt # apply the staged proposal to CLAUDE.md / SKILL.md (with backup)
|
||||
|
||||
/skillopt-sleep-handoff run # same cycle, but THIS session answers the model calls
|
||||
# (no claude -p subprocess, no API key — subscription-friendly)
|
||||
```
|
||||
|
||||
Or call the engine directly (Python ≥ 3.10):
|
||||
@@ -74,6 +77,26 @@ Default backend is **`mock`** — deterministic, no API spend — so you can try
|
||||
plumbing for free. Switch to `--backend claude` or `--backend codex` for genuine
|
||||
improvement on your own budget.
|
||||
|
||||
### Handoff mode (session answers the model calls)
|
||||
|
||||
`--backend handoff` runs the cycle without any model subprocess: the engine
|
||||
executes the deterministic stages and writes every model call it needs to
|
||||
`.skillopt-sleep-handoff/PROMPTS.md` + `pending.json` (exit code 3). You (or
|
||||
the `/skillopt-sleep-handoff` command, which automates the loop with isolated
|
||||
fresh-context subagents) write each raw answer to `answers/<id>.md` and re-run
|
||||
the same command; it resumes from the answers and either finishes or stages
|
||||
the next batch. Typically 3–6 rounds per night.
|
||||
|
||||
```bash
|
||||
python -m skillopt_sleep run --backend handoff --project "$(pwd)"
|
||||
# ... answer .skillopt-sleep-handoff/PROMPTS.md into answers/<id>.md ...
|
||||
python -m skillopt_sleep run --backend handoff --project "$(pwd)" # resume
|
||||
```
|
||||
|
||||
Answer every prompt in a **fresh context** — a session that has already seen
|
||||
the mined tasks and their references would contaminate the held-out gate.
|
||||
Details: [the plugins README](../README.md#--backend-handoff--session-executed-calls-no-api-subprocess).
|
||||
|
||||
## Does it actually improve? (real models, public benchmark)
|
||||
|
||||
SkillOpt-Sleep is validated against [gbrain-evals](https://github.com/garrytan/gbrain-evals)'
|
||||
|
||||
@@ -0,0 +1,67 @@
|
||||
---
|
||||
description: Run the SkillOpt-Sleep cycle with the handoff backend — no API subprocess; this session answers the engine's model calls via prompt/answer files, in isolated fresh-context subagents
|
||||
argument-hint: "[run | dry-run] [--preferences \"...\"] (default: run)"
|
||||
allowed-tools: Bash, Read, Write, Task
|
||||
---
|
||||
|
||||
# /skillopt-sleep-handoff — session-executed sleep cycle
|
||||
|
||||
You are driving **SkillOpt-Sleep in handoff mode**: the Python engine runs
|
||||
every deterministic stage (harvest → mine → replay scoring → gate → stage)
|
||||
and outsources each model call (attempt / judge / reflect) to YOU via
|
||||
prompt files. No `claude -p` subprocess, no API key — the model work runs
|
||||
on this session's budget, but each prompt MUST be answered in a fresh,
|
||||
isolated context so the validation gate stays honest.
|
||||
|
||||
## Requested action: $ARGUMENTS
|
||||
|
||||
(If `$ARGUMENTS` is empty, treat it as `run`.)
|
||||
|
||||
## The loop
|
||||
|
||||
Repeat until the engine exits 0 (done) — at most 8 rounds:
|
||||
|
||||
1. **Run the engine** via the bundled runner:
|
||||
|
||||
```bash
|
||||
"${CLAUDE_PLUGIN_ROOT}/scripts/sleep.sh" <action> --backend handoff --project "$(pwd)" --scope invoked
|
||||
```
|
||||
|
||||
- exit 0 → the night is complete; go to "Finish" below.
|
||||
- exit 3 → pending model calls; continue with step 2.
|
||||
- anything else → stop and show the user the error output.
|
||||
|
||||
2. **Read the batch**: `Read` `.skillopt-sleep-handoff/pending.json` in the
|
||||
project. Each entry has `id`, `prompt`, `max_tokens`, `answer_file`.
|
||||
|
||||
3. **Answer each prompt in ISOLATION** — this is the integrity rule:
|
||||
- For each entry, launch a subagent (Task tool) whose ENTIRE input is
|
||||
the `prompt` text verbatim. Add nothing: no summary of this session,
|
||||
no mention of SkillOpt, no other prompts from the batch.
|
||||
- Take the subagent's reply and `Write` the raw answer text (no
|
||||
commentary, no code fences) to the entry's `answer_file`.
|
||||
- NEVER answer from this session's own context — you have seen the
|
||||
mined tasks and their references, so inline answers would contaminate
|
||||
the held-out gate and fake the improvement score.
|
||||
|
||||
4. **Re-run the same engine command** — it resumes from the answers
|
||||
directory and either finishes or stages the next batch.
|
||||
|
||||
## Finish
|
||||
|
||||
- `Read` the `report.md` in the staging dir the engine printed and show
|
||||
the user: held-out baseline → candidate score, the gate decision, the
|
||||
proposed edits, and where the proposal is staged.
|
||||
- Tell the user nothing live changed; offer `/skillopt-sleep adopt`.
|
||||
- The engine archives `.skillopt-sleep-handoff/` on a completed real run;
|
||||
do not delete it yourself.
|
||||
|
||||
## Safety reminders
|
||||
|
||||
- **Never** edit `CLAUDE.md` or `SKILL.md` yourself — only `adopt` does
|
||||
that, with a backup.
|
||||
- Mined tasks are pinned to `.skillopt-sleep-handoff/tasks.json` on round
|
||||
one, so sessions created while answering prompts cannot shift the task
|
||||
set. Do not edit that file.
|
||||
- If a batch looks like it contains secrets or content the user would not
|
||||
want re-processed, stop and ask before answering.
|
||||
Reference in New Issue
Block a user