feat(sleep): 3-way train/val/test split + gate_mode on|off
Data-split refactor (the anti-overfitting foundation the user asked for):
- TaskRecord gains split∈{train,val,test} and origin∈{real,dream}.
- assign_splits: real tasks deterministically split into val/test (disjoint);
DREAM-augmented tasks (origin='dream') NEVER enter val/test — they only go to
train. val gates updates; test is the final held-out measure.
- gbrain loader maps its held-out.jsonl -> test, benchmark.jsonl -> train/val,
so the gbrain held-out stays the true final score.
- consolidate(): train drives reflect, val gates; adds gate_mode='off' (greedy,
no hard filter) reporting val movement (greedy_improved/regressed/flat).
- run_gbrain/transfer/experiment score on test (val fallback); run_gbrain gains
--gate on|off. Legacy replay/holdout names normalized.
New test proves dream tasks never land in val/test. 21 tests pass; mock
experiment + gate=off both green.
Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
This commit is contained in:
@@ -30,10 +30,13 @@ DEFAULTS: Dict[str, Any] = {
|
||||
# ── budgets ────────────────────────────────────────────────────────────
|
||||
"max_tasks_per_night": 40,
|
||||
"max_tokens_per_night": 400_000,
|
||||
"holdout_fraction": 0.34, # fraction of mined tasks reserved for the gate
|
||||
"holdout_fraction": 0.34, # legacy alias for val_fraction
|
||||
"val_fraction": 0.34, # real tasks reserved to gate updates
|
||||
"test_fraction": 0.0, # real tasks reserved as the final held-out measure
|
||||
# ── optimizer ──────────────────────────────────────────────────────────
|
||||
"backend": "mock", # "mock" | "claude" | "codex"
|
||||
"model": "", # backend-specific; "" => backend default
|
||||
"gate_mode": "on", # "on" (validation-gated) | "off" (greedy, no hard filter)
|
||||
"codex_path": "", # "" => auto-detect the real @openai/codex binary
|
||||
"edit_budget": 4, # textual learning rate (max edits/night)
|
||||
"gate_metric": "mixed", # hard | soft | mixed (mixed best for tiny holdouts)
|
||||
|
||||
Reference in New Issue
Block a user