url/SkillOpt
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(sleep): real Claude<->Codex cross-validation of the new features

Browse Source 
Three live runs exercise the new code paths on both runtimes:
  A) Claude Sonnet->Haiku, gate=OFF + rollouts_k=2: brief-writer test 0->1.00,
     action 'greedy_improved', val & test both reported (3-way split works).
  B) Codex, gate=ON + rollouts_k=2: brief-writer test 0->1.00 in 2 nights.
  C) Claude Sonnet->Haiku, thorough-analyst, 3 nights: slow-update fires and
     distils a durable cross-night meta-rule (general, not task-specific).

Confirms gate-off greedy path, 3-way val/test split, multi-rollout, and the
gate-independent slow-update all work with real models on Claude AND Codex.
Raw logs under docs/sleep/raw/crosscheck_*.txt.

Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
This commit is contained in:
Yifan Yang
2026-06-08 14:31:51 +00:00
parent 9379e494bf
commit e2de84d36f
4 changed files with 162 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/sleep/CONTROLLABLE_DREAMING.md
+22
View File
@@ -92,3 +92,25 @@ python -m skillopt.sleep.experiments.run_gbrain \
All of this is exercised by the deterministic test suite (29 tests) and
validated on real Claude + Codex (see `real_api_results.md` / `FINAL_REPORT.md`).
## Real cross-validation of the new features (Claude ⟷ Codex)
Three live runs exercised the new code paths on both runtimes (raw logs under
`docs/sleep/raw/crosscheck_*.txt`):
| # | Config | What it proves | Result |
|---|---|---|---|
| **A** | Claude Sonnet→Haiku, **gate=off**, **rollouts_k=2** | greedy mode + multi-rollout + 3-way split (val & test both reported) | brief-writer **test 0→1.00**, action `greedy_improved`, val=1.0 test=1.0 |
| **B** | **Codex**, gate=on, **rollouts_k=2** | new paths on the other runtime | brief-writer **test 0→1.00**, 2-night `accept_new_best`, val+test reported |
| **C** | Claude Sonnet→Haiku, thorough-analyst, 3 nights | **slow-update** long-term memory fires | test 0→0.33 (val gate holds nights 2–3) and the slow-update distilled a durable meta-rule |
The slow-update guidance C produced is the kind of cross-night lesson the field
is for — note it is general, not task-specific:
> *"On character-constrained tasks (≤1200 chars), plan structure before writing:
> allocate space per point explicitly and cut until the outline fits, then fill —
> never draft freely and trim after."*
Takeaways confirmed live: the **gate-off greedy path**, the **3-way val/test
split**, **multi-rollout** on both runtimes, and the **gate-independent
slow-update** all work with real models on both Claude and Codex.
docs/sleep/raw/crosscheck_A_claude_gateoff_rollouts.txt
+38
View File
@@ -0,0 +1,38 @@
=== REAL cross-check A: Sonnet->Haiku, gate=OFF, rollouts_k=2, brief-writer (exercises new paths) ===
{
"benchmark": "gbrain-evals/skillopt-v1",
"backend": "target=claude/optimizer=claude",
"model": "(default)",
"n_seeds": 1,
"n_improved": 1,
"tokens_used": 11271,
"results": [
{
"seed": "brief-writer",
"held_out_before": 0.0,
"held_out_after": 1.0,
"improved": true,
"nights": 1,
"trace": [
{
"night": 0,
"test_hard": 0.0,
"action": "baseline"
},
{
"night": 1,
"val_hard": 1.0,
"test_hard": 1.0,
"action": "greedy_improved",
"accepted": true,
"edits": [
"Every brief MUST include a section with the exact heading '## Key Risks' that lists the primary risks relevant to the recommendation. This section is required in every output regardless of topic.",
"Every brief MUST include a 'Confidence:' label (satisfying /[Cc]onfidence\\s*[:=]/) that states the confidence level in the recommendation (e.g., 'Confidence: Medium'). Place it near the answer/recommendation line or at the end of the brief."
]
}
],
"slow_update": null,
"final_skill_tail": "at lists the primary risks relevant to the recommendation. This section is required in every output regardless of topic.\n- Every brief MUST include a 'Confidence:' label (satisfying /[Cc]onfidence\\s*[:=]/) that states the confidence level in the recommendation (e.g., 'Confidence: Medium'). Place it near the answer/recommendation line or at the end of the brief.\n<!-- SKILLOPT-SLEEP:LEARNED END -->\n"
}
]
}
docs/sleep/raw/crosscheck_B_codex_rollouts.txt
+48
View File
@@ -0,0 +1,48 @@
=== REAL cross-check B: Codex, gate=ON (default), rollouts_k=2, brief-writer ===
{
"benchmark": "gbrain-evals/skillopt-v1",
"backend": "codex",
"model": "(default)",
"n_seeds": 1,
"n_improved": 1,
"tokens_used": 17251,
"results": [
{
"seed": "brief-writer",
"held_out_before": 0.0,
"held_out_after": 1.0,
"improved": true,
"nights": 2,
"trace": [
{
"night": 0,
"test_hard": 0.0,
"action": "baseline"
},
{
"night": 1,
"val_hard": 0.667,
"test_hard": 0.333,
"action": "accept_new_best",
"accepted": true,
"edits": [
"Every brief must include a section/heading titled exactly 'Key Risks'.",
"Every brief must include a confidence line labeled exactly 'Confidence:' so the response matches /[Cc]onfidence\\s*[:=]/."
]
},
{
"night": 2,
"val_hard": 1.0,
"test_hard": 1.0,
"action": "accept_new_best",
"accepted": true,
"edits": [
"OVERRIDE any brevity guidance: every brief must include a standalone Markdown heading line exactly '## Key Risks' to satisfy section_present=Key Risks, even when the brief is very short."
]
}
],
"slow_update": null,
"final_skill_tail": "clude a section/heading titled exactly 'Key Risks'.\n- Every brief must include a confidence line labeled exactly 'Confidence:' so the response matches /[Cc]onfidence\\s*[:=]/.\n- OVERRIDE any brevity guidance: every brief must include a standalone Markdown heading line exactly '## Key Risks' to satisfy section_present=Key Risks, even when the brief is very short.\n<!-- SKILLOPT-SLEEP:LEARNED END -->\n"
}
]
}
docs/sleep/raw/crosscheck_C_claude_slowupdate.txt
+54
View File
@@ -0,0 +1,54 @@
=== cross-check C: Sonnet->Haiku thorough-analyst (2 nights, slow-update should fire) ===
{
"benchmark": "gbrain-evals/skillopt-v1",
"backend": "target=claude/optimizer=claude",
"model": "(default)",
"n_seeds": 1,
"n_improved": 1,
"tokens_used": 26010,
"results": [
{
"seed": "thorough-analyst",
"held_out_before": 0.0,
"held_out_after": 0.333,
"improved": true,
"nights": 3,
"trace": [
{
"night": 0,
"test_hard": 0.0,
"action": "baseline"
},
{
"night": 1,
"val_hard": 0.667,
"test_hard": 0.667,
"action": "accept_new_best",
"accepted": true,
"edits": [
"OVERRIDE (supersedes 'be exhaustive and detailed', 'Explore every angle', 'consider many scenarios', and 'Write multiple paragraphs'): the ENTIRE response must be at most 1200 characters long, counting every character including spaces, newlines, and punctuation. This hard character limit takes priority over all instructions to be thorough, exhaustive, or multi-paragraph.",
"To stay within 1200 characters while still being useful: lead with the single most critical trade-off, then list 2-3 key considerations as tight bullet points. Omit headers, preamble, and restating the question."
]
},
{
"night": 2,
"val_hard": 0.667,
"test_hard": 0.667,
"action": "reject",
"accepted": false,
"edits": []
},
{
"night": 3,
"val_hard": 0.667,
"test_hard": 0.667,
"action": "reject",
"accepted": false,
"edits": []
}
],
"slow_update": "• On character-constrained tasks (≤1200 chars), plan structure before writing: allocate space per point explicitly and cut until the outline fits, then fill — never draft freely and trim after.\n• Multi-variable business/strategy analyses are high-risk for overrun; default to covering only the 2–3 most decisive factors rather than attempting exhaustive coverage.\n• Lead with the conclusion or recommendation first; eliminate all introductory restatement of the question, hedging preamble, and transitional filler under tight limits.\n• Persistent failures on the same task signal a structural habit, not a one-off error — treat repeated length violations as a signal to change the drafting approach entirely, not just edit more aggressively.",
"final_skill_tail": "ead with the conclusion or recommendation first; eliminate all introductory restatement of the question, hedging preamble, and transitional filler under tight limits.\n• Persistent failures on the same task signal a structural habit, not a one-off error — treat repeated length violations as a signal to change the drafting approach entirely, not just edit more aggressively.\n<!-- SLOW_UPDATE_END -->\n"
}
]
}