url/SkillOpt
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(sleep): real Claude<->Codex cross-validation of the new features

Browse Source 
Three live runs exercise the new code paths on both runtimes:
  A) Claude Sonnet->Haiku, gate=OFF + rollouts_k=2: brief-writer test 0->1.00,
     action 'greedy_improved', val & test both reported (3-way split works).
  B) Codex, gate=ON + rollouts_k=2: brief-writer test 0->1.00 in 2 nights.
  C) Claude Sonnet->Haiku, thorough-analyst, 3 nights: slow-update fires and
     distils a durable cross-night meta-rule (general, not task-specific).

Confirms gate-off greedy path, 3-way val/test split, multi-rollout, and the
gate-independent slow-update all work with real models on Claude AND Codex.
Raw logs under docs/sleep/raw/crosscheck_*.txt.

Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
This commit is contained in:
Yifan Yang
2026-06-08 14:31:51 +00:00
parent 9379e494bf
commit e2de84d36f
4 changed files with 162 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/sleep/raw/crosscheck_B_codex_rollouts.txt
+48
View File
@@ -0,0 +1,48 @@
=== REAL cross-check B: Codex, gate=ON (default), rollouts_k=2, brief-writer ===
{
"benchmark": "gbrain-evals/skillopt-v1",
"backend": "codex",
"model": "(default)",
"n_seeds": 1,
"n_improved": 1,
"tokens_used": 17251,
"results": [
{
"seed": "brief-writer",
"held_out_before": 0.0,
"held_out_after": 1.0,
"improved": true,
"nights": 2,
"trace": [
{
"night": 0,
"test_hard": 0.0,
"action": "baseline"
},
{
"night": 1,
"val_hard": 0.667,
"test_hard": 0.333,
"action": "accept_new_best",
"accepted": true,
"edits": [
"Every brief must include a section/heading titled exactly 'Key Risks'.",
"Every brief must include a confidence line labeled exactly 'Confidence:' so the response matches /[Cc]onfidence\\s*[:=]/."
]
},
{
"night": 2,
"val_hard": 1.0,
"test_hard": 1.0,
"action": "accept_new_best",
"accepted": true,
"edits": [
"OVERRIDE any brevity guidance: every brief must include a standalone Markdown heading line exactly '## Key Risks' to satisfy section_present=Key Risks, even when the brief is very short."
]
}
],
"slow_update": null,
"final_skill_tail": "clude a section/heading titled exactly 'Key Risks'.\n- Every brief must include a confidence line labeled exactly 'Confidence:' so the response matches /[Cc]onfidence\\s*[:=]/.\n- OVERRIDE any brevity guidance: every brief must include a standalone Markdown heading line exactly '## Key Risks' to satisfy section_present=Key Risks, even when the brief is very short.\n<!-- SKILLOPT-SLEEP:LEARNED END -->\n"
}
]
}