feat(sleep): add handoff backend — session-executed model calls, no API subprocess (#125)

Adds --backend handoff: the engine runs all deterministic stages and outsources attempt/judge/reflect to prompt/answer files an interactive agent session fills between runs (exit 3 = pending batch, re-run to resume). Deterministic replay + the prompt-hash answer cache make resume stateless; sentinel detection aborts any call built from unanswered output so placeholders never reach scores or staging. Session digests and mined tasks are pinned per night (secret-redacted) so the sessions answering prompts cannot shift the task set, and LLM mining is routed through the same handoff files. Ships a /skillopt-sleep-handoff Claude Code command that answers each prompt in a fresh-context subagent to protect the held-out gate. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 18:23:53 +02:00
parent 7df49656d1
commit b309723baa
10 changed files with 745 additions and 6 deletions
@@ -22,7 +22,7 @@ sleep** idea (short-term experience → long-term competence).

 | Platform | Folder | Mechanism | Status |
 |---|---|---|---|
-| **Claude Code** | [`claude-code/`](claude-code) | `.claude-plugin` + `/skillopt-sleep` command + skill + hooks | full, installable |
+| **Claude Code** | [`claude-code/`](claude-code) | `.claude-plugin` + `/skillopt-sleep` + `/skillopt-sleep-handoff` commands + skill + hooks | full, installable |
 | **Codex** | [`codex/`](codex) | user-level `skillopt-sleep` skill + shared runner | full |
 | **Copilot** | [`copilot/`](copilot) | MCP server (`sleep_*` tools) + `copilot-instructions` | full (MCP) |
 | **Devin** | [`devin/`](devin) | MCP server (`sleep_*` tools) + Devin ATIF-v1.7 harvest + `.devin/rules` | full (MCP) |
@@ -149,6 +149,47 @@ The reward can weight not just correctness but **cost and speed**, so a skill ca
 learn to be cheaper and faster, not only more accurate. *What it does for you:*
 "answer directly instead of opening five files" becomes a learned habit.

+### `--backend handoff` — session-executed calls (no API subprocess)
+
+For subscription seats and environments where the engine shouldn't spawn
+`claude -p` / API calls itself. The engine still runs every deterministic
+stage (harvest → mine → replay scoring → gate → stage), but each model call
+(attempt / judge / reflect) is written to a prompt file that **your own agent
+session answers between engine runs**:
+
+```bash
+python -m skillopt_sleep run --backend handoff --project "$(pwd)"
+# exit 3 => .skillopt-sleep-handoff/PROMPTS.md + pending.json were written
+#   answer each prompt (each in a FRESH context) into answers/<id>.md
+# re-run the same command => it resumes from the answers and either
+#   finishes (exit 0) or stages the next prompt batch (exit 3)
+```
+
+A typical night converges in 3–6 rounds: baseline attempts → reflect →
+candidate re-scoring per accepted edit. Resume is stateless — replay is
+deterministic and answers are cached by prompt hash, so re-running skips
+everything already answered. Mined tasks are pinned to
+`.skillopt-sleep-handoff/tasks.json` on the first round, so the sessions that
+answer the prompts can't shift the task set and invalidate earlier answers.
+On a completed real run the handoff directory is archived to
+`.skillopt-sleep-handoff.night<N>.done`.
+
+On Claude Code, `/skillopt-sleep-handoff run` drives the whole loop for you,
+answering each prompt in an isolated fresh-context subagent.
+
+**Integrity rule:** answer every prompt in a fresh context (a subagent with no
+conversation history). Answering from a session that has already seen the
+mined tasks and their references contaminates the held-out gate and fakes the
+improvement score.
+
+*What it does for you:* the sleep cycle runs entirely on your interactive
+session's subscription budget — no API key, no headless subprocess — while the
+gate, splits, and staging discipline stay in the engine.
+
+Limitations: `--rollouts-k > 1` gives no contrastive spread (identical prompt
+→ identical answer file), and tool-loop tasks fall back to the single-shot
+`TOOL_CALL:` marker convention.
+
 ### `schedule` / `unschedule` — set it and forget it

 Built-in nightly scheduling (no manual cron):
@@ -176,7 +217,7 @@ schedule, if you trust it).

 | Flag | Default | Meaning |
 |---|---|---|
-| `--backend mock\|claude\|codex\|copilot` | `mock` | who runs/optimizes (mock = free) |
+| `--backend mock\|claude\|codex\|copilot\|handoff` | `mock` | who runs/optimizes (mock = free; handoff = your own session answers) |
 | `--preferences "..."` | – | your house rules, as a prior |
 | `--gate on\|off` | `on` | strict held-out gate vs. greedy |
 | `--rollouts-k K` | `1` | multi-rollout contrastive reflection |
@@ -60,6 +60,9 @@ they shell out to the CLIs you already have.
 /skillopt-sleep run         # full cycle: stages a reviewed proposal (still no live edits)
 /skillopt-sleep status      # see history + the latest staged proposal
 /skillopt-sleep adopt       # apply the staged proposal to CLAUDE.md / SKILL.md (with backup)
+
+/skillopt-sleep-handoff run # same cycle, but THIS session answers the model calls
+                            # (no claude -p subprocess, no API key — subscription-friendly)
 ```

 Or call the engine directly (Python ≥ 3.10):
@@ -74,6 +77,26 @@ Default backend is **`mock`** — deterministic, no API spend — so you can try
 plumbing for free. Switch to `--backend claude` or `--backend codex` for genuine
 improvement on your own budget.

+### Handoff mode (session answers the model calls)
+
+`--backend handoff` runs the cycle without any model subprocess: the engine
+executes the deterministic stages and writes every model call it needs to
+`.skillopt-sleep-handoff/PROMPTS.md` + `pending.json` (exit code 3). You (or
+the `/skillopt-sleep-handoff` command, which automates the loop with isolated
+fresh-context subagents) write each raw answer to `answers/<id>.md` and re-run
+the same command; it resumes from the answers and either finishes or stages
+the next batch. Typically 3–6 rounds per night.
+
+```bash
+python -m skillopt_sleep run --backend handoff --project "$(pwd)"
+# ... answer .skillopt-sleep-handoff/PROMPTS.md into answers/<id>.md ...
+python -m skillopt_sleep run --backend handoff --project "$(pwd)"   # resume
+```
+
+Answer every prompt in a **fresh context** — a session that has already seen
+the mined tasks and their references would contaminate the held-out gate.
+Details: [the plugins README](../README.md#--backend-handoff--session-executed-calls-no-api-subprocess).
+
 ## Does it actually improve? (real models, public benchmark)

 SkillOpt-Sleep is validated against [gbrain-evals](https://github.com/garrytan/gbrain-evals)'
@@ -0,0 +1,67 @@
+---
+description: Run the SkillOpt-Sleep cycle with the handoff backend — no API subprocess; this session answers the engine's model calls via prompt/answer files, in isolated fresh-context subagents
+argument-hint: "[run | dry-run] [--preferences \"...\"] (default: run)"
+allowed-tools: Bash, Read, Write, Task
+---
+
+# /skillopt-sleep-handoff — session-executed sleep cycle
+
+You are driving **SkillOpt-Sleep in handoff mode**: the Python engine runs
+every deterministic stage (harvest → mine → replay scoring → gate → stage)
+and outsources each model call (attempt / judge / reflect) to YOU via
+prompt files. No `claude -p` subprocess, no API key — the model work runs
+on this session's budget, but each prompt MUST be answered in a fresh,
+isolated context so the validation gate stays honest.
+
+## Requested action: $ARGUMENTS
+
+(If `$ARGUMENTS` is empty, treat it as `run`.)
+
+## The loop
+
+Repeat until the engine exits 0 (done) — at most 8 rounds:
+
+1. **Run the engine** via the bundled runner:
+
+   ```bash
+   "${CLAUDE_PLUGIN_ROOT}/scripts/sleep.sh" <action> --backend handoff --project "$(pwd)" --scope invoked
+   ```
+
+   - exit 0 → the night is complete; go to "Finish" below.
+   - exit 3 → pending model calls; continue with step 2.
+   - anything else → stop and show the user the error output.
+
+2. **Read the batch**: `Read` `.skillopt-sleep-handoff/pending.json` in the
+   project. Each entry has `id`, `prompt`, `max_tokens`, `answer_file`.
+
+3. **Answer each prompt in ISOLATION** — this is the integrity rule:
+   - For each entry, launch a subagent (Task tool) whose ENTIRE input is
+     the `prompt` text verbatim. Add nothing: no summary of this session,
+     no mention of SkillOpt, no other prompts from the batch.
+   - Take the subagent's reply and `Write` the raw answer text (no
+     commentary, no code fences) to the entry's `answer_file`.
+   - NEVER answer from this session's own context — you have seen the
+     mined tasks and their references, so inline answers would contaminate
+     the held-out gate and fake the improvement score.
+
+4. **Re-run the same engine command** — it resumes from the answers
+   directory and either finishes or stages the next batch.
+
+## Finish
+
+- `Read` the `report.md` in the staging dir the engine printed and show
+  the user: held-out baseline → candidate score, the gate decision, the
+  proposed edits, and where the proposal is staged.
+- Tell the user nothing live changed; offer `/skillopt-sleep adopt`.
+- The engine archives `.skillopt-sleep-handoff/` on a completed real run;
+  do not delete it yourself.
+
+## Safety reminders
+
+- **Never** edit `CLAUDE.md` or `SKILL.md` yourself — only `adopt` does
+  that, with a backup.
+- Mined tasks are pinned to `.skillopt-sleep-handoff/tasks.json` on round
+  one, so sessions created while answering prompts cannot shift the task
+  set. Do not edit that file.
+- If a batch looks like it contains secrets or content the user would not
+  want re-processed, stop and ask before answering.