fix(skillopt-sleep): surface codex auth/model/version failures instead of silently scoring 0
A nightly sleep cycle could run for weeks emitting held-out 0.0 -> 0.0 (gate reject, zero edits), indistinguishable from "nothing to learn", when the real cause was the codex backend returning an error (expired auth / model unsupported on the account / outdated CLI) that got scored as a failed rollout. backend (CodexCliBackend): - split _call into _call_once + a retry wrapper: transient empties/timeouts are retried instead of silently returning "" (mirrors AzureOpenAIBackend's guard); - on a non-zero exit, surface the reason via last_call_error and return "" rather than leaking the CLI error text as if it were a model response; - fail fast (no retries) on fatal auth/model/version errors (401, refresh_token_reused, token_expired, "not supported when using Codex with a ChatGPT account", "requires a newer version of Codex"). backend (CliBackend.reflect): retain last_reflect_raw so a no-edits night is diagnosable. consolidate: ConsolidationResult now carries per-task held-out detail (response, hard/soft, fail_reason) + reflect_raw + call_error. cycle: write diagnostics.json per cycle so a 0.0 night self-explains instead of being a black box. tests: 4 new (retry-not-silent-zero, auth-error-surfaced-not-scored, holdout-detail, reflect-raw). Also gitignore the .skillopt-sleep/ runtime dir. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -276,6 +276,28 @@ def run_sleep_cycle(
|
||||
live_memory_path=live_memory_path,
|
||||
report_md=report_md,
|
||||
)
|
||||
# Observability: persist per-task held-out evidence + optimizer/codex errors so a
|
||||
# 0.0->0.0 night self-explains (empty responses vs failing checks vs no edits) — the
|
||||
# cycle previously captured none of this, making the gate a black box (#learning-stall).
|
||||
try:
|
||||
import json as _json
|
||||
with open(os.path.join(staging_dir, "diagnostics.json"), "w", encoding="utf-8") as _fh:
|
||||
_json.dump({
|
||||
"night": night,
|
||||
"backend": cfg.get("backend"),
|
||||
"gate_mode": cfg.get("gate_mode"),
|
||||
"n_tasks": len(tasks),
|
||||
"baseline_score": result.baseline_score,
|
||||
"candidate_score": result.candidate_score,
|
||||
"accepted": result.accepted,
|
||||
"n_applied_edits": len(result.applied_edits),
|
||||
"n_rejected_edits": len(result.rejected_edits),
|
||||
"call_error": getattr(result, "call_error", ""),
|
||||
"reflect_raw_head": (getattr(result, "reflect_raw", "") or "")[:1200],
|
||||
"holdout_detail": getattr(result, "holdout_detail", []),
|
||||
}, _fh, indent=2)
|
||||
except Exception:
|
||||
pass
|
||||
state.set_last_harvest(project, started)
|
||||
state.record_night({
|
||||
"night": night, "accepted": result.accepted,
|
||||
|
||||
Reference in New Issue
Block a user