url/SkillOpt
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(sleep): experience replay + dream rollouts in the cycle (opt-in)

Browse Source 
Wires two consolidation mechanisms into the shipped nightly cycle, both default
OFF so existing behavior is unchanged:
  - dream_rollouts (>1): multi-rollout contrastive reflection per task
  - recall_k (>0): associative recall of the K most-similar past tasks (from a
    capped task_archive persisted in state.json) into tonight's dream
  - dream_factor (>0): synthetic task variants

New shared engine module skillopt_sleep/dream.py (recall_similar, dream_augment,
dream_consolidate) is called by both the plugin cycle and the experiment harness,
so reported numbers exercise the exact shipped code. Built on the existing
rollouts_k/sample_id support already in consolidate.py/rollout.py.

Validated (5 nights x 10 real tasks/night, full held-out test, GPT-5.5, gated):
the gain scales with recall depth on a clean signal —
SearchQA recall_k=10 +3.1, recall_k=20 +4.5, full-history reference +5.6;
SpreadsheetBench (nano, gate-free) +3.6. Flat within noise on saturated/noisy
cells. See docs/sleep/EXPERIENCE_REPLAY.md (+ raw runs under blog_runs/v2_port/).

Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
This commit is contained in:
Yifan Yang
2026-06-15 15:58:27 +00:00
parent 576f2f8bad
commit 722ce646d4
11 changed files with 800 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/sleep/EXPERIENCE_REPLAY.md
+64
View File
@@ -0,0 +1,64 @@
# SkillOpt-Sleep — experience replay & dream rollouts (opt-in)
Two opt-in mechanisms that strengthen the nightly consolidation when your tasks
have a clean correctness signal. Both default **off**, so enabling them is the
only way they change behavior.
## What they do
| Config knob | Default | Effect |
|---|---|---|
| `dream_rollouts` | `1` | Run each task **K** times and learn from the *contrast* between the good and bad attempts (contrastive reflection) instead of a single failure. |
| `recall_k` | `0` | **Associative recall** — each night, pull the `K` past tasks most similar to tonight's new ones (from a persisted task archive) into the dream, so related experience is revisited without replaying the whole history. |
| `dream_factor` | `0` | Add `N` lightweight synthetic variants of each task to the training pool. |
The validation gate still governs what ships, so these only ever *enlarge the
signal the optimizer reflects on* — the held-out gate decides what is kept.
## How to enable
```jsonc
// ~/.skillopt-sleep/config.json (or pass via the plugin's config)
{
"dream_rollouts": 5, // contrastive dreaming
"recall_k": 20, // recall ~20 similar past tasks each night
"gate_mode": "on" // keep the gate on (recommended)
}
```
`recall_k` draws from a capped `task_archive` that the cycle persists in
`state.json`, so recall becomes useful from the second night onward (once there
is history to recall from).
## Measured effect
Deployment protocol (5 nights × 10 new real tasks/night, full held-out test
sets, GPT-5.5 optimizer), run through the **same engine the plugin executes**
(`skillopt_sleep.dream.dream_consolidate`):
**SearchQA (GPT-5.5, full 1,400-item test, gated) — the gain scales with recall depth:**
| Config | Δ vs baseline |
|---|---|
| `recall_k=10, dream_rollouts=5` | +3.1 |
| `dream_rollouts=8` | +3.7 |
| **`recall_k=20, dream_rollouts=5`** | **+4.5** |
| full-history replay (reference) | +5.6 |
**Second-benchmark confirmation** (SpreadsheetBench, GPT-5.4-nano, gate-free,
shipped path): 0.279 → **0.314 (+3.6)**.
## When it helps — and when it doesn't
- **Helps** when tasks recur and have a checkable correctness signal (the
optimizer has something real to learn and the gate can verify it).
- **Roughly flat** on saturated or noisy tasks (e.g. a strong model already near
ceiling) — within run-to-run noise (±1–2 points, single seed).
- The validation gate keeps the downside bounded; keep it on by default.
Trade-off: `dream_rollouts > 1` multiplies the per-night rollout cost (K×), and
`recall_k > 0` adds the recalled tasks to each night's replay. Since the cycle
runs offline on idle quota this is usually acceptable, but budget accordingly
(`budget_tokens` / `budget_seconds`).
Raw per-run results for the table above: `docs/sleep/blog_runs/v2_port/`.
docs/sleep/blog_runs/v2_port/conf_ss_nano_free.json
+94
View File
@@ -0,0 +1,94 @@
{
"experiment": "skillopt-sleep/nightly",
"model": "gpt-5.4-nano",
"results": [
{
"benchmark": "spreadsheet",
"gate": "off",
"replay_mode": "retrieval",
"retrieve_k": 10,
"nights": 5,
"per_night": 10,
"rollouts": 5,
"n_val": 40,
"n_test": 280,
"test_baseline": 0.2786,
"test_final": 0.3143,
"delta": 0.0357,
"progression": [
0.2786,
0.3036,
0.3143,
0.3107,
0.3179,
0.3143
],
"nights_log": [
{
"night": 0,
"n_train": 0,
"test_hard": 0.2786,
"action": "baseline",
"accepted": false
},
{
"night": 1,
"n_train": 10,
"n_replayed": 0,
"n_dream": 20,
"val_hard": 0.0,
"test_hard": 0.3036,
"action": "greedy_applied",
"accepted": true,
"n_edits": 4
},
{
"night": 2,
"n_train": 20,
"n_replayed": 10,
"n_dream": 40,
"val_hard": 0.0,
"test_hard": 0.3143,
"action": "greedy_applied",
"accepted": true,
"n_edits": 4
},
{
"night": 3,
"n_train": 30,
"n_replayed": 10,
"n_dream": 40,
"val_hard": 0.0,
"test_hard": 0.3107,
"action": "greedy_applied",
"accepted": true,
"n_edits": 4
},
{
"night": 4,
"n_train": 40,
"n_replayed": 10,
"n_dream": 40,
"val_hard": 0.0,
"test_hard": 0.3179,
"action": "greedy_applied",
"accepted": true,
"n_edits": 4
},
{
"night": 5,
"n_train": 50,
"n_replayed": 10,
"n_dream": 40,
"val_hard": 0.0,
"test_hard": 0.3143,
"action": "greedy_applied",
"accepted": true,
"n_edits": 4
}
],
"tokens": 13587597,
"final_skill_tail": "t/headers rather than hardcoding specific cell coordinates or values.\n- When searching for specific text, use an exact match check on the cell string, e.g. `if cell_value == \"Georgia Its Tax\": ...` (not partial regex, not truncated comparisons).\n- If a cell contains multiple tokens separated by semicolons, split and normalize before comparing: `parts = [p.strip() for p in str(cell_value).split(';') if p.strip()]` and then test membership/lookup using `parts`.\n<!-- SKILLOPT-SLEEP:LEARNED END -->\n"
}
]
}
docs/sleep/blog_runs/v2_port/imp_cumulative_gate.json
+94
View File
@@ -0,0 +1,94 @@
{
"experiment": "skillopt-sleep/nightly",
"model": "gpt-5.5",
"results": [
{
"benchmark": "searchqa",
"gate": "on",
"replay_mode": "cumulative",
"retrieve_k": 0,
"nights": 5,
"per_night": 10,
"rollouts": 5,
"n_val": 60,
"n_test": 1400,
"test_baseline": 0.7957,
"test_final": 0.8514,
"delta": 0.0557,
"progression": [
0.7957,
0.8336,
0.8514,
0.8514,
0.8514,
0.8514
],
"nights_log": [
{
"night": 0,
"n_train": 0,
"test_hard": 0.7957,
"action": "baseline",
"accepted": false
},
{
"night": 1,
"n_train": 10,
"n_replayed": 0,
"n_dream": 20,
"val_hard": 0.85,
"test_hard": 0.8336,
"action": "accept_new_best",
"accepted": true,
"n_edits": 2
},
{
"night": 2,
"n_train": 20,
"n_replayed": 10,
"n_dream": 40,
"val_hard": 0.9,
"test_hard": 0.8514,
"action": "accept_new_best",
"accepted": true,
"n_edits": 3
},
{
"night": 3,
"n_train": 30,
"n_replayed": 20,
"n_dream": 60,
"val_hard": 0.9,
"test_hard": 0.8514,
"action": "reject",
"accepted": false,
"n_edits": 0
},
{
"night": 4,
"n_train": 40,
"n_replayed": 30,
"n_dream": 80,
"val_hard": 0.9,
"test_hard": 0.8514,
"action": "reject",
"accepted": false,
"n_edits": 0
},
{
"night": 5,
"n_train": 50,
"n_replayed": 40,
"n_dream": 100,
"val_hard": 0.9,
"test_hard": 0.8514,
"action": "reject",
"accepted": false,
"n_edits": 0
}
],
"tokens": 15132599,
"final_skill_tail": " the title or key sentence over a county, institution, or category.\n- Return the shortest exact answer span that satisfies the question, inside <answer>...</answer>; prefer a single-word entity when sufficient.\n- Do not expand a context-supported short name into a fuller name unless the question specifically requires the full name.\n- Match the requested answer type exactly: for a country/nation answer, output only the country name, not a title or role phrase.\n<!-- SKILLOPT-SLEEP:LEARNED END -->\n"
}
]
}
docs/sleep/blog_runs/v2_port/imp_recall20_gate.json
+94
View File
@@ -0,0 +1,94 @@
{
"experiment": "skillopt-sleep/nightly",
"model": "gpt-5.5",
"results": [
{
"benchmark": "searchqa",
"gate": "on",
"replay_mode": "retrieval",
"retrieve_k": 20,
"nights": 5,
"per_night": 10,
"rollouts": 5,
"n_val": 60,
"n_test": 1400,
"test_baseline": 0.8029,
"test_final": 0.8479,
"delta": 0.045,
"progression": [
0.8029,
0.8236,
0.8236,
0.8479,
0.8479,
0.8479
],
"nights_log": [
{
"night": 0,
"n_train": 0,
"test_hard": 0.8029,
"action": "baseline",
"accepted": false
},
{
"night": 1,
"n_train": 10,
"n_replayed": 0,
"n_dream": 20,
"val_hard": 0.8667,
"test_hard": 0.8236,
"action": "accept_new_best",
"accepted": true,
"n_edits": 2
},
{
"night": 2,
"n_train": 20,
"n_replayed": 10,
"n_dream": 40,
"val_hard": 0.8667,
"test_hard": 0.8236,
"action": "reject",
"accepted": false,
"n_edits": 0
},
{
"night": 3,
"n_train": 30,
"n_replayed": 20,
"n_dream": 60,
"val_hard": 0.8833,
"test_hard": 0.8479,
"action": "accept_new_best",
"accepted": true,
"n_edits": 3
},
{
"night": 4,
"n_train": 40,
"n_replayed": 20,
"n_dream": 60,
"val_hard": 0.8833,
"test_hard": 0.8479,
"action": "reject",
"accepted": false,
"n_edits": 0
},
{
"night": 5,
"n_train": 50,
"n_replayed": 20,
"n_dream": 60,
"val_hard": 0.8833,
"test_hard": 0.8479,
"action": "reject",
"accepted": false,
"n_edits": 0
}
],
"tokens": 15596999,
"final_skill_tail": " Put only the shortest exact answer span in the final '<answer>...</answer>' tags; remove extra descriptors, categories, titles, and surrounding words.\n- If the question asks for a country/place from a phrase like 'King of Spain' or a title like 'Ferdinand VII of Spain', answer only the place name, e.g. 'Spain'.\n- For person answers, use the minimal unambiguous name supported by the clue; do not expand a surname to a full name unless the question requires it.\n<!-- SKILLOPT-SLEEP:LEARNED END -->\n"
}
]
}
docs/sleep/blog_runs/v2_port/imp_rollouts8_gate.json
+94
View File
@@ -0,0 +1,94 @@
{
"experiment": "skillopt-sleep/nightly",
"model": "gpt-5.5",
"results": [
{
"benchmark": "searchqa",
"gate": "on",
"replay_mode": "retrieval",
"retrieve_k": 10,
"nights": 5,
"per_night": 10,
"rollouts": 8,
"n_val": 60,
"n_test": 1400,
"test_baseline": 0.7979,
"test_final": 0.835,
"delta": 0.0371,
"progression": [
0.7979,
0.8179,
0.835,
0.835,
0.835,
0.835
],
"nights_log": [
{
"night": 0,
"n_train": 0,
"test_hard": 0.7979,
"action": "baseline",
"accepted": false
},
{
"night": 1,
"n_train": 10,
"n_replayed": 0,
"n_dream": 20,
"val_hard": 0.8667,
"test_hard": 0.8179,
"action": "accept_new_best",
"accepted": true,
"n_edits": 2
},
{
"night": 2,
"n_train": 20,
"n_replayed": 10,
"n_dream": 40,
"val_hard": 0.8833,
"test_hard": 0.835,
"action": "accept_new_best",
"accepted": true,
"n_edits": 3
},
{
"night": 3,
"n_train": 30,
"n_replayed": 10,
"n_dream": 40,
"val_hard": 0.8833,
"test_hard": 0.835,
"action": "reject",
"accepted": false,
"n_edits": 0
},
{
"night": 4,
"n_train": 40,
"n_replayed": 10,
"n_dream": 40,
"val_hard": 0.8833,
"test_hard": 0.835,
"action": "reject",
"accepted": false,
"n_edits": 0
},
{
"night": 5,
"n_train": 50,
"n_replayed": 10,
"n_dream": 40,
"val_hard": 0.8833,
"test_hard": 0.835,
"action": "reject",
"accepted": false,
"n_edits": 0
}
],
"tokens": 16846499,
"final_skill_tail": "less the question asks for the title itself.\n- Always put only the final answer in \"<answer>...</answer>\" and keep it \"concise -- typically a few words or a short phrase\".\n- Use the shortest sufficient answer span; do not add first names, modifiers, counties, countries, or parent locations unless explicitly required.\n- Match the question’s granularity exactly: if it asks for a state, give only the state; if it asks for a term’s meaning, give only the meaning.\n<!-- SKILLOPT-SLEEP:LEARNED END -->\n"
}
]
}
docs/sleep/blog_runs/v2_port/parity_sq_g55_free.json
+94
View File
@@ -0,0 +1,94 @@
{
"experiment": "skillopt-sleep/nightly",
"model": "gpt-5.5",
"results": [
{
"benchmark": "searchqa",
"gate": "off",
"replay_mode": "retrieval",
"retrieve_k": 10,
"nights": 5,
"per_night": 10,
"rollouts": 5,
"n_val": 60,
"n_test": 1400,
"test_baseline": 0.8079,
"test_final": 0.8393,
"delta": 0.0314,
"progression": [
0.8079,
0.8321,
0.84,
0.8436,
0.84,
0.8393
],
"nights_log": [
{
"night": 0,
"n_train": 0,
"test_hard": 0.8079,
"action": "baseline",
"accepted": false
},
{
"night": 1,
"n_train": 10,
"n_replayed": 0,
"n_dream": 20,
"val_hard": 0.0,
"test_hard": 0.8321,
"action": "greedy_applied",
"accepted": true,
"n_edits": 3
},
{
"night": 2,
"n_train": 20,
"n_replayed": 10,
"n_dream": 40,
"val_hard": 0.0,
"test_hard": 0.84,
"action": "greedy_applied",
"accepted": true,
"n_edits": 1
},
{
"night": 3,
"n_train": 30,
"n_replayed": 10,
"n_dream": 40,
"val_hard": 0.0,
"test_hard": 0.8436,
"action": "greedy_applied",
"accepted": true,
"n_edits": 2
},
{
"night": 4,
"n_train": 40,
"n_replayed": 10,
"n_dream": 40,
"val_hard": 0.0,
"test_hard": 0.84,
"action": "greedy_applied",
"accepted": true,
"n_edits": 3
},
{
"night": 5,
"n_train": 50,
"n_replayed": 10,
"n_dream": 40,
"val_hard": 0.0,
"test_hard": 0.8393,
"action": "greedy_applied",
"accepted": true,
"n_edits": 2
}
],
"tokens": 27990836,
"final_skill_tail": "Sultan of Brunei\".\n- For author/creator questions from titles like \"Trees by Joyce Kilmer\", output only the creator name, e.g. \"Joyce Kilmer\", not the work title.\n- Do not introduce diacritics or alternate spellings not present in the context/title; prefer the ASCII surface form such as \"Vaclav Havel\" over \"Václav Havel\".\n- Return the full canonical entity name from the context/title, including hyphens, e.g. \"Winnie-the-Pooh\" rather than the shortened \"Pooh\".\n<!-- SKILLOPT-SLEEP:LEARNED END -->\n"
}
]
}
docs/sleep/blog_runs/v2_port/parity_sq_g55_gate.json
+94
View File
@@ -0,0 +1,94 @@
{
"experiment": "skillopt-sleep/nightly",
"model": "gpt-5.5",
"results": [
{
"benchmark": "searchqa",
"gate": "on",
"replay_mode": "retrieval",
"retrieve_k": 10,
"nights": 5,
"per_night": 10,
"rollouts": 5,
"n_val": 60,
"n_test": 1400,
"test_baseline": 0.8021,
"test_final": 0.8336,
"delta": 0.0315,
"progression": [
0.8021,
0.83,
0.8336,
0.8336,
0.8336,
0.8336
],
"nights_log": [
{
"night": 0,
"n_train": 0,
"test_hard": 0.8021,
"action": "baseline",
"accepted": false
},
{
"night": 1,
"n_train": 10,
"n_replayed": 0,
"n_dream": 20,
"val_hard": 0.8667,
"test_hard": 0.83,
"action": "accept_new_best",
"accepted": true,
"n_edits": 4
},
{
"night": 2,
"n_train": 20,
"n_replayed": 10,
"n_dream": 40,
"val_hard": 0.9,
"test_hard": 0.8336,
"action": "accept_new_best",
"accepted": true,
"n_edits": 4
},
{
"night": 3,
"n_train": 30,
"n_replayed": 10,
"n_dream": 40,
"val_hard": 0.9,
"test_hard": 0.8336,
"action": "reject",
"accepted": false,
"n_edits": 0
},
{
"night": 4,
"n_train": 40,
"n_replayed": 10,
"n_dream": 40,
"val_hard": 0.9,
"test_hard": 0.8336,
"action": "reject",
"accepted": false,
"n_edits": 0
},
{
"night": 5,
"n_train": 50,
"n_replayed": 10,
"n_dream": 40,
"val_hard": 0.9,
"test_hard": 0.8336,
"action": "reject",
"accepted": false,
"n_edits": 0
}
],
"tokens": 15946118,
"final_skill_tail": "roperty; do not substitute a broader category or page title.\n- For location questions asking for a state/country, output only that level, e.g. \"Maryland\", not the full hierarchy \"Baltimore County, Maryland, United States\".\n- For name-part questions such as surname/last name, output only that part, e.g. \"Genet\", not the full name \"Jean Genet\".\n- Put only the concise final answer inside \"<answer>...</answer>\"; avoid extra modifiers, lists, or explanatory words.\n<!-- SKILLOPT-SLEEP:LEARNED END -->\n"
}
]
}