feat(sleep): add handoff backend — session-executed model calls, no API subprocess (#125)

Adds --backend handoff: the engine runs all deterministic stages and
outsources attempt/judge/reflect to prompt/answer files an interactive
agent session fills between runs (exit 3 = pending batch, re-run to
resume). Deterministic replay + the prompt-hash answer cache make resume
stateless; sentinel detection aborts any call built from unanswered
output so placeholders never reach scores or staging. Session digests
and mined tasks are pinned per night (secret-redacted) so the sessions
answering prompts cannot shift the task set, and LLM mining is routed
through the same handoff files. Ships a /skillopt-sleep-handoff Claude
Code command that answers each prompt in a fresh-context subagent to
protect the held-out gate.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
dimitarvdenev
2026-07-12 18:23:53 +02:00
committed by GitHub
parent 7df49656d1
commit b309723baa
10 changed files with 745 additions and 6 deletions
+23
View File
@@ -60,6 +60,9 @@ they shell out to the CLIs you already have.
/skillopt-sleep run # full cycle: stages a reviewed proposal (still no live edits)
/skillopt-sleep status # see history + the latest staged proposal
/skillopt-sleep adopt # apply the staged proposal to CLAUDE.md / SKILL.md (with backup)
/skillopt-sleep-handoff run # same cycle, but THIS session answers the model calls
# (no claude -p subprocess, no API key — subscription-friendly)
```
Or call the engine directly (Python ≥ 3.10):
@@ -74,6 +77,26 @@ Default backend is **`mock`** — deterministic, no API spend — so you can try
plumbing for free. Switch to `--backend claude` or `--backend codex` for genuine
improvement on your own budget.
### Handoff mode (session answers the model calls)
`--backend handoff` runs the cycle without any model subprocess: the engine
executes the deterministic stages and writes every model call it needs to
`.skillopt-sleep-handoff/PROMPTS.md` + `pending.json` (exit code 3). You (or
the `/skillopt-sleep-handoff` command, which automates the loop with isolated
fresh-context subagents) write each raw answer to `answers/<id>.md` and re-run
the same command; it resumes from the answers and either finishes or stages
the next batch. Typically 36 rounds per night.
```bash
python -m skillopt_sleep run --backend handoff --project "$(pwd)"
# ... answer .skillopt-sleep-handoff/PROMPTS.md into answers/<id>.md ...
python -m skillopt_sleep run --backend handoff --project "$(pwd)" # resume
```
Answer every prompt in a **fresh context** — a session that has already seen
the mined tasks and their references would contaminate the held-out gate.
Details: [the plugins README](../README.md#--backend-handoff--session-executed-calls-no-api-subprocess).
## Does it actually improve? (real models, public benchmark)
SkillOpt-Sleep is validated against [gbrain-evals](https://github.com/garrytan/gbrain-evals)'
@@ -0,0 +1,67 @@
---
description: Run the SkillOpt-Sleep cycle with the handoff backend — no API subprocess; this session answers the engine's model calls via prompt/answer files, in isolated fresh-context subagents
argument-hint: "[run | dry-run] [--preferences \"...\"] (default: run)"
allowed-tools: Bash, Read, Write, Task
---
# /skillopt-sleep-handoff — session-executed sleep cycle
You are driving **SkillOpt-Sleep in handoff mode**: the Python engine runs
every deterministic stage (harvest → mine → replay scoring → gate → stage)
and outsources each model call (attempt / judge / reflect) to YOU via
prompt files. No `claude -p` subprocess, no API key — the model work runs
on this session's budget, but each prompt MUST be answered in a fresh,
isolated context so the validation gate stays honest.
## Requested action: $ARGUMENTS
(If `$ARGUMENTS` is empty, treat it as `run`.)
## The loop
Repeat until the engine exits 0 (done) — at most 8 rounds:
1. **Run the engine** via the bundled runner:
```bash
"${CLAUDE_PLUGIN_ROOT}/scripts/sleep.sh" <action> --backend handoff --project "$(pwd)" --scope invoked
```
- exit 0 → the night is complete; go to "Finish" below.
- exit 3 → pending model calls; continue with step 2.
- anything else → stop and show the user the error output.
2. **Read the batch**: `Read` `.skillopt-sleep-handoff/pending.json` in the
project. Each entry has `id`, `prompt`, `max_tokens`, `answer_file`.
3. **Answer each prompt in ISOLATION** — this is the integrity rule:
- For each entry, launch a subagent (Task tool) whose ENTIRE input is
the `prompt` text verbatim. Add nothing: no summary of this session,
no mention of SkillOpt, no other prompts from the batch.
- Take the subagent's reply and `Write` the raw answer text (no
commentary, no code fences) to the entry's `answer_file`.
- NEVER answer from this session's own context — you have seen the
mined tasks and their references, so inline answers would contaminate
the held-out gate and fake the improvement score.
4. **Re-run the same engine command** — it resumes from the answers
directory and either finishes or stages the next batch.
## Finish
- `Read` the `report.md` in the staging dir the engine printed and show
the user: held-out baseline → candidate score, the gate decision, the
proposed edits, and where the proposal is staged.
- Tell the user nothing live changed; offer `/skillopt-sleep adopt`.
- The engine archives `.skillopt-sleep-handoff/` on a completed real run;
do not delete it yourself.
## Safety reminders
- **Never** edit `CLAUDE.md` or `SKILL.md` yourself — only `adopt` does
that, with a backup.
- Mined tasks are pinned to `.skillopt-sleep-handoff/tasks.json` on round
one, so sessions created while answering prompts cannot shift the task
set. Do not edit that file.
- If a batch looks like it contains secrets or content the user would not
want re-processed, stop and ask before answering.