url/SkillOpt
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(sleep): definitive clean results — Sonnet->Haiku 3/3 seeds 0->1.00

Browse Source 
Strong-optimizer/weak-target (Sonnet -> Haiku), fully isolated:
  brief-writer, advisor, thorough-analyst all 0.00 -> 1.00 on held-out.
thorough-analyst shows 2-night convergence (0.33 -> 1.00). Codex self-optimized
brief-writer also 0 -> 1.00.

Key finding answering the optimizer/target-split request: the OPTIMIZER MODEL is
decisive — weak Haiku-as-optimizer is flaky (0 or 1.0 across runs), strong
Sonnet-as-optimizer reliably hits 1.0 on every seed. Raw logs under docs/sleep/raw/.

Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
This commit is contained in:
Yifan Yang
2026-06-08 14:31:51 +00:00
parent 023950a291
commit 4186e5bb73
4 changed files with 337 additions and 85 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/sleep/raw/codex_brief_writer.txt
+45
View File
@@ -0,0 +1,45 @@
=== gbrain brief-writer CODEX, improved prompt, 2 nights, 3+3 tasks ===
{
"benchmark": "gbrain-evals/skillopt-v1",
"backend": "codex",
"model": "(default)",
"n_seeds": 1,
"n_improved": 1,
"tokens_used": 9990,
"results": [
{
"seed": "brief-writer",
"held_out_before": 0.0,
"held_out_after": 1.0,
"improved": true,
"nights": 2,
"trace": [
{
"night": 0,
"held_out_hard": 0.0,
"action": "baseline"
},
{
"night": 1,
"held_out_hard": 0.0,
"action": "accept_new_best",
"accepted": true,
"edits": [
"Every brief must include a clearly labeled section exactly titled `Key Risks`.",
"Every brief must include a line beginning `Confidence:` followed by a concise confidence level or rationale."
]
},
{
"night": 2,
"held_out_hard": 1.0,
"action": "accept_new_best",
"accepted": true,
"edits": [
"- Preserve required sections even when keeping the brief short; shorten the analysis before omitting `## Key Risks` or `Confidence:`."
]
}
],
"final_skill_tail": "tside this block are never touched._\n\n- Every brief must include a clearly labeled section exactly titled `Key Risks`.\n- Every brief must include a line beginning `Confidence:` followed by a concise confidence level or rationale.\n- Preserve required sections even when keeping the brief short; shorten the analysis before omitting `## Key Risks` or `Confidence:`.\n<!-- SKILLOPT-SLEEP:LEARNED END -->\n"
}
]
}