Yifan Yang
4186e5bb73
Strong-optimizer/weak-target (Sonnet -> Haiku), fully isolated: brief-writer, advisor, thorough-analyst all 0.00 -> 1.00 on held-out. thorough-analyst shows 2-night convergence (0.33 -> 1.00). Codex self-optimized brief-writer also 0 -> 1.00. Key finding answering the optimizer/target-split request: the OPTIMIZER MODEL is decisive — weak Haiku-as-optimizer is flaky (0 or 1.0 across runs), strong Sonnet-as-optimizer reliably hits 1.0 on every seed. Raw logs under docs/sleep/raw/. Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
46 lines
1.6 KiB
Plaintext
46 lines
1.6 KiB
Plaintext
=== gbrain brief-writer CODEX, improved prompt, 2 nights, 3+3 tasks ===
|
|
{
|
|
"benchmark": "gbrain-evals/skillopt-v1",
|
|
"backend": "codex",
|
|
"model": "(default)",
|
|
"n_seeds": 1,
|
|
"n_improved": 1,
|
|
"tokens_used": 9990,
|
|
"results": [
|
|
{
|
|
"seed": "brief-writer",
|
|
"held_out_before": 0.0,
|
|
"held_out_after": 1.0,
|
|
"improved": true,
|
|
"nights": 2,
|
|
"trace": [
|
|
{
|
|
"night": 0,
|
|
"held_out_hard": 0.0,
|
|
"action": "baseline"
|
|
},
|
|
{
|
|
"night": 1,
|
|
"held_out_hard": 0.0,
|
|
"action": "accept_new_best",
|
|
"accepted": true,
|
|
"edits": [
|
|
"Every brief must include a clearly labeled section exactly titled `Key Risks`.",
|
|
"Every brief must include a line beginning `Confidence:` followed by a concise confidence level or rationale."
|
|
]
|
|
},
|
|
{
|
|
"night": 2,
|
|
"held_out_hard": 1.0,
|
|
"action": "accept_new_best",
|
|
"accepted": true,
|
|
"edits": [
|
|
"- Preserve required sections even when keeping the brief short; shorten the analysis before omitting `## Key Risks` or `Confidence:`."
|
|
]
|
|
}
|
|
],
|
|
"final_skill_tail": "tside this block are never touched._\n\n- Every brief must include a clearly labeled section exactly titled `Key Risks`.\n- Every brief must include a line beginning `Confidence:` followed by a concise confidence level or rationale.\n- Preserve required sections even when keeping the brief short; shorten the analysis before omitting `## Key Risks` or `Confidence:`.\n<!-- SKILLOPT-SLEEP:LEARNED END -->\n"
|
|
}
|
|
]
|
|
}
|