From e2de84d36f9909c7f94e0bb1d9a17eae4b884467 Mon Sep 17 00:00:00 2001 From: Yifan Yang Date: Mon, 8 Jun 2026 14:31:51 +0000 Subject: [PATCH] docs(sleep): real Claude<->Codex cross-validation of the new features Three live runs exercise the new code paths on both runtimes: A) Claude Sonnet->Haiku, gate=OFF + rollouts_k=2: brief-writer test 0->1.00, action 'greedy_improved', val & test both reported (3-way split works). B) Codex, gate=ON + rollouts_k=2: brief-writer test 0->1.00 in 2 nights. C) Claude Sonnet->Haiku, thorough-analyst, 3 nights: slow-update fires and distils a durable cross-night meta-rule (general, not task-specific). Confirms gate-off greedy path, 3-way val/test split, multi-rollout, and the gate-independent slow-update all work with real models on Claude AND Codex. Raw logs under docs/sleep/raw/crosscheck_*.txt. Co-Authored-By: Claude Opus 4 --- docs/sleep/CONTROLLABLE_DREAMING.md | 22 ++++++++ .../crosscheck_A_claude_gateoff_rollouts.txt | 38 +++++++++++++ .../sleep/raw/crosscheck_B_codex_rollouts.txt | 48 +++++++++++++++++ .../raw/crosscheck_C_claude_slowupdate.txt | 54 +++++++++++++++++++ 4 files changed, 162 insertions(+) create mode 100644 docs/sleep/raw/crosscheck_A_claude_gateoff_rollouts.txt create mode 100644 docs/sleep/raw/crosscheck_B_codex_rollouts.txt create mode 100644 docs/sleep/raw/crosscheck_C_claude_slowupdate.txt diff --git a/docs/sleep/CONTROLLABLE_DREAMING.md b/docs/sleep/CONTROLLABLE_DREAMING.md index 10ab835..0286097 100644 --- a/docs/sleep/CONTROLLABLE_DREAMING.md +++ b/docs/sleep/CONTROLLABLE_DREAMING.md @@ -92,3 +92,25 @@ python -m skillopt.sleep.experiments.run_gbrain \ All of this is exercised by the deterministic test suite (29 tests) and validated on real Claude + Codex (see `real_api_results.md` / `FINAL_REPORT.md`). + +## Real cross-validation of the new features (Claude ⟷ Codex) + +Three live runs exercised the new code paths on both runtimes (raw logs under +`docs/sleep/raw/crosscheck_*.txt`): + +| # | Config | What it proves | Result | +|---|---|---|---| +| **A** | Claude Sonnet→Haiku, **gate=off**, **rollouts_k=2** | greedy mode + multi-rollout + 3-way split (val & test both reported) | brief-writer **test 0→1.00**, action `greedy_improved`, val=1.0 test=1.0 | +| **B** | **Codex**, gate=on, **rollouts_k=2** | new paths on the other runtime | brief-writer **test 0→1.00**, 2-night `accept_new_best`, val+test reported | +| **C** | Claude Sonnet→Haiku, thorough-analyst, 3 nights | **slow-update** long-term memory fires | test 0→0.33 (val gate holds nights 2–3) and the slow-update distilled a durable meta-rule | + +The slow-update guidance C produced is the kind of cross-night lesson the field +is for — note it is general, not task-specific: + +> *"On character-constrained tasks (≤1200 chars), plan structure before writing: +> allocate space per point explicitly and cut until the outline fits, then fill — +> never draft freely and trim after."* + +Takeaways confirmed live: the **gate-off greedy path**, the **3-way val/test +split**, **multi-rollout** on both runtimes, and the **gate-independent +slow-update** all work with real models on both Claude and Codex. diff --git a/docs/sleep/raw/crosscheck_A_claude_gateoff_rollouts.txt b/docs/sleep/raw/crosscheck_A_claude_gateoff_rollouts.txt new file mode 100644 index 0000000..b6c7766 --- /dev/null +++ b/docs/sleep/raw/crosscheck_A_claude_gateoff_rollouts.txt @@ -0,0 +1,38 @@ +=== REAL cross-check A: Sonnet->Haiku, gate=OFF, rollouts_k=2, brief-writer (exercises new paths) === +{ + "benchmark": "gbrain-evals/skillopt-v1", + "backend": "target=claude/optimizer=claude", + "model": "(default)", + "n_seeds": 1, + "n_improved": 1, + "tokens_used": 11271, + "results": [ + { + "seed": "brief-writer", + "held_out_before": 0.0, + "held_out_after": 1.0, + "improved": true, + "nights": 1, + "trace": [ + { + "night": 0, + "test_hard": 0.0, + "action": "baseline" + }, + { + "night": 1, + "val_hard": 1.0, + "test_hard": 1.0, + "action": "greedy_improved", + "accepted": true, + "edits": [ + "Every brief MUST include a section with the exact heading '## Key Risks' that lists the primary risks relevant to the recommendation. This section is required in every output regardless of topic.", + "Every brief MUST include a 'Confidence:' label (satisfying /[Cc]onfidence\\s*[:=]/) that states the confidence level in the recommendation (e.g., 'Confidence: Medium'). Place it near the answer/recommendation line or at the end of the brief." + ] + } + ], + "slow_update": null, + "final_skill_tail": "at lists the primary risks relevant to the recommendation. This section is required in every output regardless of topic.\n- Every brief MUST include a 'Confidence:' label (satisfying /[Cc]onfidence\\s*[:=]/) that states the confidence level in the recommendation (e.g., 'Confidence: Medium'). Place it near the answer/recommendation line or at the end of the brief.\n\n" + } + ] +} diff --git a/docs/sleep/raw/crosscheck_B_codex_rollouts.txt b/docs/sleep/raw/crosscheck_B_codex_rollouts.txt new file mode 100644 index 0000000..d20b3cc --- /dev/null +++ b/docs/sleep/raw/crosscheck_B_codex_rollouts.txt @@ -0,0 +1,48 @@ +=== REAL cross-check B: Codex, gate=ON (default), rollouts_k=2, brief-writer === +{ + "benchmark": "gbrain-evals/skillopt-v1", + "backend": "codex", + "model": "(default)", + "n_seeds": 1, + "n_improved": 1, + "tokens_used": 17251, + "results": [ + { + "seed": "brief-writer", + "held_out_before": 0.0, + "held_out_after": 1.0, + "improved": true, + "nights": 2, + "trace": [ + { + "night": 0, + "test_hard": 0.0, + "action": "baseline" + }, + { + "night": 1, + "val_hard": 0.667, + "test_hard": 0.333, + "action": "accept_new_best", + "accepted": true, + "edits": [ + "Every brief must include a section/heading titled exactly 'Key Risks'.", + "Every brief must include a confidence line labeled exactly 'Confidence:' so the response matches /[Cc]onfidence\\s*[:=]/." + ] + }, + { + "night": 2, + "val_hard": 1.0, + "test_hard": 1.0, + "action": "accept_new_best", + "accepted": true, + "edits": [ + "OVERRIDE any brevity guidance: every brief must include a standalone Markdown heading line exactly '## Key Risks' to satisfy section_present=Key Risks, even when the brief is very short." + ] + } + ], + "slow_update": null, + "final_skill_tail": "clude a section/heading titled exactly 'Key Risks'.\n- Every brief must include a confidence line labeled exactly 'Confidence:' so the response matches /[Cc]onfidence\\s*[:=]/.\n- OVERRIDE any brevity guidance: every brief must include a standalone Markdown heading line exactly '## Key Risks' to satisfy section_present=Key Risks, even when the brief is very short.\n\n" + } + ] +} diff --git a/docs/sleep/raw/crosscheck_C_claude_slowupdate.txt b/docs/sleep/raw/crosscheck_C_claude_slowupdate.txt new file mode 100644 index 0000000..88bc2c4 --- /dev/null +++ b/docs/sleep/raw/crosscheck_C_claude_slowupdate.txt @@ -0,0 +1,54 @@ +=== cross-check C: Sonnet->Haiku thorough-analyst (2 nights, slow-update should fire) === +{ + "benchmark": "gbrain-evals/skillopt-v1", + "backend": "target=claude/optimizer=claude", + "model": "(default)", + "n_seeds": 1, + "n_improved": 1, + "tokens_used": 26010, + "results": [ + { + "seed": "thorough-analyst", + "held_out_before": 0.0, + "held_out_after": 0.333, + "improved": true, + "nights": 3, + "trace": [ + { + "night": 0, + "test_hard": 0.0, + "action": "baseline" + }, + { + "night": 1, + "val_hard": 0.667, + "test_hard": 0.667, + "action": "accept_new_best", + "accepted": true, + "edits": [ + "OVERRIDE (supersedes 'be exhaustive and detailed', 'Explore every angle', 'consider many scenarios', and 'Write multiple paragraphs'): the ENTIRE response must be at most 1200 characters long, counting every character including spaces, newlines, and punctuation. This hard character limit takes priority over all instructions to be thorough, exhaustive, or multi-paragraph.", + "To stay within 1200 characters while still being useful: lead with the single most critical trade-off, then list 2-3 key considerations as tight bullet points. Omit headers, preamble, and restating the question." + ] + }, + { + "night": 2, + "val_hard": 0.667, + "test_hard": 0.667, + "action": "reject", + "accepted": false, + "edits": [] + }, + { + "night": 3, + "val_hard": 0.667, + "test_hard": 0.667, + "action": "reject", + "accepted": false, + "edits": [] + } + ], + "slow_update": "• On character-constrained tasks (≤1200 chars), plan structure before writing: allocate space per point explicitly and cut until the outline fits, then fill — never draft freely and trim after.\n• Multi-variable business/strategy analyses are high-risk for overrun; default to covering only the 2–3 most decisive factors rather than attempting exhaustive coverage.\n• Lead with the conclusion or recommendation first; eliminate all introductory restatement of the question, hedging preamble, and transitional filler under tight limits.\n• Persistent failures on the same task signal a structural habit, not a one-off error — treat repeated length violations as a signal to change the drafting approach entirely, not just edit more aggressively.", + "final_skill_tail": "ead with the conclusion or recommendation first; eliminate all introductory restatement of the question, hedging preamble, and transitional filler under tight limits.\n• Persistent failures on the same task signal a structural habit, not a one-off error — treat repeated length violations as a signal to change the drafting approach entirely, not just edit more aggressively.\n\n" + } + ] +}