Yifan Yang
dae974a5e3
Remove every non-ASCII/CJK character for a professional open-source repo:
- harvest.py: drop hardcoded Chinese feedback phrases; add an env-based
extensibility hook (SKILLOPT_SLEEP_NEG_FEEDBACK / _POS_FEEDBACK) so any
locale can be added without baking one in. Verified with a German example.
- rollout.py / consolidate.py: English comments.
- README.md section heading + anchor, CONTROLLABLE_DREAMING.md, plugin.json,
marketplace.json (also fixed stale path skillopt-sleep-plugin ->
plugins/claude-code), SKILL.md: English only.
- Remove the internal WAKE_UP_SUMMARY.md note (not user-facing, not referenced).
Verified: zero CJK chars remain anywhere; 29 tests pass.
Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
118 lines
5.9 KiB
Markdown
118 lines
5.9 KiB
Markdown
# SkillOpt-Sleep — controllable dreaming architecture
|
||
|
||
The sleep engine is no longer a single fixed pipeline. It is a controllable
|
||
offline "dream / imagination" loop the user steers. This documents the knobs
|
||
added in the four-stage refactor and how they map to the user's design.
|
||
|
||
## The mental model
|
||
|
||
> Sleep = an offline imagination rollout. Re-run the user's real
|
||
> tasks (and dream-augmented variants) many times, look at what went well vs
|
||
> badly, distil durable rules, and keep only what survives a real-task check —
|
||
> unless the user opts out of that check.
|
||
|
||
## 1. Data splits — train (dream) / val (real) / test (real)
|
||
|
||
The anti-overfitting foundation:
|
||
|
||
| Split | Source | Role |
|
||
|---|---|---|
|
||
| **train** | real tasks **+ dream-augmented** variants | drives reflection (the imagination pool — over-dreaming is fine) |
|
||
| **val** | **real only**, disjoint from test | gates updates (prevents overfitting) |
|
||
| **test** | **real only**, disjoint from val | the final held-out measure, kept close to real usage |
|
||
|
||
Hard guarantee (unit-tested): a task with `origin='dream'` **never** lands in
|
||
val or test. `assign_splits(val_fraction, test_fraction)` does the deterministic
|
||
3-way split; gbrain's own held-out maps to our `test`.
|
||
|
||
## 2. The validation gate is optional
|
||
|
||
`--gate on` (default): an edit is accepted only if it strictly improves the
|
||
**val** score — the SkillOpt discipline that blocks regressions and reward
|
||
hacking.
|
||
|
||
`--gate off`: greedy. Edits are kept without the hard val-improvement
|
||
requirement (the user decides they don't want hard filtering), but val/test
|
||
movement is still reported (`greedy_improved` / `greedy_regressed` /
|
||
`greedy_flat`) so nothing is hidden.
|
||
|
||
## 3. Slow-update — long-term memory, gate-independent
|
||
|
||
Even with the gate off, the engine runs a **slow-update** at the end of the
|
||
nights: it compares behaviour under the first-night vs final skill across the
|
||
val tasks and distils durable longitudinal guidance into a **protected field**
|
||
(`<!-- SLOW_UPDATE_START --> … <!-- SLOW_UPDATE_END -->`, the same markers as
|
||
the main SkillOpt repo). Step-level edits never touch this field. This is the
|
||
"short-term experience → long-term memory" consolidation; turning the gate off
|
||
does not cost you long-term memory.
|
||
|
||
## 4. Budget — the user picks the spend
|
||
|
||
`--budget-tokens N` / `--budget-minutes M`: the engine auto-plans depth
|
||
(`nights × rollouts_per_task`) to fit the budget (`plan_depth`). Stops cleanly
|
||
when exhausted and logs what it skipped — no silent truncation. The whole thing
|
||
is offline imagination on the user's own quota.
|
||
|
||
## 5. Multi-rollout contrastive reflection — the imagination core
|
||
|
||
`--rollouts-k K` (K>1): each train task is rolled out K times. The optimizer is
|
||
shown the **high-scoring vs low-scoring** attempts of the same task and asked
|
||
what the good ones did that the bad ones didn't, distilling a general rule. This
|
||
is a far stronger signal than a single failure, and it is exactly the user's
|
||
"run it many times, learn from the contrast" idea. Tasks with the highest score
|
||
*spread* (some passed, some failed) are the most informative and are prioritised.
|
||
|
||
## 6. Multi-objective reward — accuracy ↑, tokens ↓, latency ↓
|
||
|
||
Every rollout records its `tokens` and `latency_ms`.
|
||
`multi_objective_reward(w_acc, w_tokens, w_latency)` is a weighted reward so a
|
||
skill can be optimised to be **cheaper and faster**, not only more accurate
|
||
(cost terms normalised against a reference; default weights = accuracy-only, so
|
||
existing behaviour is unchanged). This turns "gets better the more you use it"
|
||
into "more accurate, cheaper, and faster the more you use it".
|
||
|
||
## 7. User preferences as a prior
|
||
|
||
`--preferences "<free text>"`: injected into the optimizer's reflect prompt as a
|
||
prior (set on the optimizer model for dual backends), so the user's stated
|
||
preferences steer what rules get written.
|
||
|
||
## How the knobs compose (one command)
|
||
|
||
```bash
|
||
python -m skillopt.sleep.experiments.run_gbrain \
|
||
--optimizer-backend claude --optimizer-model sonnet \ # strong optimizer
|
||
--target-backend claude --target-model haiku \ # cheap target (transfer)
|
||
--seeds thorough-analyst \
|
||
--gate on \ # or off for greedy
|
||
--rollouts-k 2 \ # contrastive imagination
|
||
--budget-tokens 60000 \ # auto-plan depth
|
||
--preferences "Prefer concise, British English." \ # prior
|
||
--nights 3
|
||
```
|
||
|
||
All of this is exercised by the deterministic test suite (29 tests) and
|
||
validated on real Claude + Codex (see `real_api_results.md` / `FINAL_REPORT.md`).
|
||
|
||
## Real cross-validation of the new features (Claude ⟷ Codex)
|
||
|
||
Three live runs exercised the new code paths on both runtimes (raw logs under
|
||
`docs/sleep/raw/crosscheck_*.txt`):
|
||
|
||
| # | Config | What it proves | Result |
|
||
|---|---|---|---|
|
||
| **A** | Claude Sonnet→Haiku, **gate=off**, **rollouts_k=2** | greedy mode + multi-rollout + 3-way split (val & test both reported) | brief-writer **test 0→1.00**, action `greedy_improved`, val=1.0 test=1.0 |
|
||
| **B** | **Codex**, gate=on, **rollouts_k=2** | new paths on the other runtime | brief-writer **test 0→1.00**, 2-night `accept_new_best`, val+test reported |
|
||
| **C** | Claude Sonnet→Haiku, thorough-analyst, 3 nights | **slow-update** long-term memory fires | test 0→0.33 (val gate holds nights 2–3) and the slow-update distilled a durable meta-rule |
|
||
|
||
The slow-update guidance C produced is the kind of cross-night lesson the field
|
||
is for — note it is general, not task-specific:
|
||
|
||
> *"On character-constrained tasks (≤1200 chars), plan structure before writing:
|
||
> allocate space per point explicitly and cut until the outline fits, then fill —
|
||
> never draft freely and trim after."*
|
||
|
||
Takeaways confirmed live: the **gate-off greedy path**, the **3-way val/test
|
||
split**, **multi-rollout** on both runtimes, and the **gate-independent
|
||
slow-update** all work with real models on both Claude and Codex.
|