diff --git a/docs/sleep/CONTROLLABLE_DREAMING.md b/docs/sleep/CONTROLLABLE_DREAMING.md index 10ab835..0286097 100644 --- a/docs/sleep/CONTROLLABLE_DREAMING.md +++ b/docs/sleep/CONTROLLABLE_DREAMING.md @@ -92,3 +92,25 @@ python -m skillopt.sleep.experiments.run_gbrain \ All of this is exercised by the deterministic test suite (29 tests) and validated on real Claude + Codex (see `real_api_results.md` / `FINAL_REPORT.md`). + +## Real cross-validation of the new features (Claude ⟷ Codex) + +Three live runs exercised the new code paths on both runtimes (raw logs under +`docs/sleep/raw/crosscheck_*.txt`): + +| # | Config | What it proves | Result | +|---|---|---|---| +| **A** | Claude Sonnet→Haiku, **gate=off**, **rollouts_k=2** | greedy mode + multi-rollout + 3-way split (val & test both reported) | brief-writer **test 0→1.00**, action `greedy_improved`, val=1.0 test=1.0 | +| **B** | **Codex**, gate=on, **rollouts_k=2** | new paths on the other runtime | brief-writer **test 0→1.00**, 2-night `accept_new_best`, val+test reported | +| **C** | Claude Sonnet→Haiku, thorough-analyst, 3 nights | **slow-update** long-term memory fires | test 0→0.33 (val gate holds nights 2–3) and the slow-update distilled a durable meta-rule | + +The slow-update guidance C produced is the kind of cross-night lesson the field +is for — note it is general, not task-specific: + +> *"On character-constrained tasks (≤1200 chars), plan structure before writing: +> allocate space per point explicitly and cut until the outline fits, then fill — +> never draft freely and trim after."* + +Takeaways confirmed live: the **gate-off greedy path**, the **3-way val/test +split**, **multi-rollout** on both runtimes, and the **gate-independent +slow-update** all work with real models on both Claude and Codex. diff --git a/docs/sleep/raw/crosscheck_A_claude_gateoff_rollouts.txt b/docs/sleep/raw/crosscheck_A_claude_gateoff_rollouts.txt new file mode 100644 index 0000000..b6c7766 --- /dev/null +++ b/docs/sleep/raw/crosscheck_A_claude_gateoff_rollouts.txt @@ -0,0 +1,38 @@ +=== REAL cross-check A: Sonnet->Haiku, gate=OFF, rollouts_k=2, brief-writer (exercises new paths) === +{ + "benchmark": "gbrain-evals/skillopt-v1", + "backend": "target=claude/optimizer=claude", + "model": "(default)", + "n_seeds": 1, + "n_improved": 1, + "tokens_used": 11271, + "results": [ + { + "seed": "brief-writer", + "held_out_before": 0.0, + "held_out_after": 1.0, + "improved": true, + "nights": 1, + "trace": [ + { + "night": 0, + "test_hard": 0.0, + "action": "baseline" + }, + { + "night": 1, + "val_hard": 1.0, + "test_hard": 1.0, + "action": "greedy_improved", + "accepted": true, + "edits": [ + "Every brief MUST include a section with the exact heading '## Key Risks' that lists the primary risks relevant to the recommendation. This section is required in every output regardless of topic.", + "Every brief MUST include a 'Confidence:' label (satisfying /[Cc]onfidence\\s*[:=]/) that states the confidence level in the recommendation (e.g., 'Confidence: Medium'). Place it near the answer/recommendation line or at the end of the brief." + ] + } + ], + "slow_update": null, + "final_skill_tail": "at lists the primary risks relevant to the recommendation. This section is required in every output regardless of topic.\n- Every brief MUST include a 'Confidence:' label (satisfying /[Cc]onfidence\\s*[:=]/) that states the confidence level in the recommendation (e.g., 'Confidence: Medium'). Place it near the answer/recommendation line or at the end of the brief.\n\n" + } + ] +} diff --git a/docs/sleep/raw/crosscheck_B_codex_rollouts.txt b/docs/sleep/raw/crosscheck_B_codex_rollouts.txt new file mode 100644 index 0000000..d20b3cc --- /dev/null +++ b/docs/sleep/raw/crosscheck_B_codex_rollouts.txt @@ -0,0 +1,48 @@ +=== REAL cross-check B: Codex, gate=ON (default), rollouts_k=2, brief-writer === +{ + "benchmark": "gbrain-evals/skillopt-v1", + "backend": "codex", + "model": "(default)", + "n_seeds": 1, + "n_improved": 1, + "tokens_used": 17251, + "results": [ + { + "seed": "brief-writer", + "held_out_before": 0.0, + "held_out_after": 1.0, + "improved": true, + "nights": 2, + "trace": [ + { + "night": 0, + "test_hard": 0.0, + "action": "baseline" + }, + { + "night": 1, + "val_hard": 0.667, + "test_hard": 0.333, + "action": "accept_new_best", + "accepted": true, + "edits": [ + "Every brief must include a section/heading titled exactly 'Key Risks'.", + "Every brief must include a confidence line labeled exactly 'Confidence:' so the response matches /[Cc]onfidence\\s*[:=]/." + ] + }, + { + "night": 2, + "val_hard": 1.0, + "test_hard": 1.0, + "action": "accept_new_best", + "accepted": true, + "edits": [ + "OVERRIDE any brevity guidance: every brief must include a standalone Markdown heading line exactly '## Key Risks' to satisfy section_present=Key Risks, even when the brief is very short." + ] + } + ], + "slow_update": null, + "final_skill_tail": "clude a section/heading titled exactly 'Key Risks'.\n- Every brief must include a confidence line labeled exactly 'Confidence:' so the response matches /[Cc]onfidence\\s*[:=]/.\n- OVERRIDE any brevity guidance: every brief must include a standalone Markdown heading line exactly '## Key Risks' to satisfy section_present=Key Risks, even when the brief is very short.\n\n" + } + ] +} diff --git a/docs/sleep/raw/crosscheck_C_claude_slowupdate.txt b/docs/sleep/raw/crosscheck_C_claude_slowupdate.txt new file mode 100644 index 0000000..88bc2c4 --- /dev/null +++ b/docs/sleep/raw/crosscheck_C_claude_slowupdate.txt @@ -0,0 +1,54 @@ +=== cross-check C: Sonnet->Haiku thorough-analyst (2 nights, slow-update should fire) === +{ + "benchmark": "gbrain-evals/skillopt-v1", + "backend": "target=claude/optimizer=claude", + "model": "(default)", + "n_seeds": 1, + "n_improved": 1, + "tokens_used": 26010, + "results": [ + { + "seed": "thorough-analyst", + "held_out_before": 0.0, + "held_out_after": 0.333, + "improved": true, + "nights": 3, + "trace": [ + { + "night": 0, + "test_hard": 0.0, + "action": "baseline" + }, + { + "night": 1, + "val_hard": 0.667, + "test_hard": 0.667, + "action": "accept_new_best", + "accepted": true, + "edits": [ + "OVERRIDE (supersedes 'be exhaustive and detailed', 'Explore every angle', 'consider many scenarios', and 'Write multiple paragraphs'): the ENTIRE response must be at most 1200 characters long, counting every character including spaces, newlines, and punctuation. This hard character limit takes priority over all instructions to be thorough, exhaustive, or multi-paragraph.", + "To stay within 1200 characters while still being useful: lead with the single most critical trade-off, then list 2-3 key considerations as tight bullet points. Omit headers, preamble, and restating the question." + ] + }, + { + "night": 2, + "val_hard": 0.667, + "test_hard": 0.667, + "action": "reject", + "accepted": false, + "edits": [] + }, + { + "night": 3, + "val_hard": 0.667, + "test_hard": 0.667, + "action": "reject", + "accepted": false, + "edits": [] + } + ], + "slow_update": "• On character-constrained tasks (≤1200 chars), plan structure before writing: allocate space per point explicitly and cut until the outline fits, then fill — never draft freely and trim after.\n• Multi-variable business/strategy analyses are high-risk for overrun; default to covering only the 2–3 most decisive factors rather than attempting exhaustive coverage.\n• Lead with the conclusion or recommendation first; eliminate all introductory restatement of the question, hedging preamble, and transitional filler under tight limits.\n• Persistent failures on the same task signal a structural habit, not a one-off error — treat repeated length violations as a signal to change the drafting approach entirely, not just edit more aggressively.", + "final_skill_tail": "ead with the conclusion or recommendation first; eliminate all introductory restatement of the question, hedging preamble, and transitional filler under tight limits.\n• Persistent failures on the same task signal a structural habit, not a one-off error — treat repeated length violations as a signal to change the drafting approach entirely, not just edit more aggressively.\n\n" + } + ] +}