feat(sleep): 3-way train/val/test split + gate_mode on|off
Data-split refactor (the anti-overfitting foundation the user asked for):
- TaskRecord gains split∈{train,val,test} and origin∈{real,dream}.
- assign_splits: real tasks deterministically split into val/test (disjoint);
DREAM-augmented tasks (origin='dream') NEVER enter val/test — they only go to
train. val gates updates; test is the final held-out measure.
- gbrain loader maps its held-out.jsonl -> test, benchmark.jsonl -> train/val,
so the gbrain held-out stays the true final score.
- consolidate(): train drives reflect, val gates; adds gate_mode='off' (greedy,
no hard filter) reporting val movement (greedy_improved/regressed/flat).
- run_gbrain/transfer/experiment score on test (val fallback); run_gbrain gains
--gate on|off. Legacy replay/holdout names normalized.
New test proves dream tasks never land in val/test. 21 tests pass; mock
experiment + gate=off both green.
Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
This commit is contained in:
@@ -37,7 +37,10 @@ from skillopt.sleep.replay import aggregate_scores, replay_batch
|
||||
|
||||
|
||||
def _holdout_hard(backend, tasks, skill, memory="") -> float:
|
||||
ho = [t for t in tasks if t.split == "holdout"] or tasks
|
||||
# transfer is measured on the true held-out TEST split
|
||||
ho = [t for t in tasks if t.split == "test"]
|
||||
if not ho:
|
||||
ho = [t for t in tasks if t.split in ("val", "holdout")] or tasks
|
||||
pairs = replay_batch(backend, ho, skill, memory)
|
||||
h, _s = aggregate_scores(pairs)
|
||||
return h
|
||||
@@ -59,13 +62,15 @@ def _optimize(backend, skill, tasks, *, nights, edit_budget) -> str:
|
||||
def run_seed(seed, skill, tasks, *, source, target, nights, edit_budget,
|
||||
limit_replay, limit_holdout, do_direct=True) -> dict:
|
||||
if limit_replay or limit_holdout:
|
||||
replay = [t for t in tasks if t.split == "replay"]
|
||||
holdout = [t for t in tasks if t.split == "holdout"]
|
||||
train = [t for t in tasks if t.split == "train"]
|
||||
val = [t for t in tasks if t.split == "val"]
|
||||
test = [t for t in tasks if t.split == "test"]
|
||||
if limit_replay:
|
||||
replay = replay[:limit_replay]
|
||||
train = train[:limit_replay]
|
||||
if limit_holdout:
|
||||
holdout = holdout[:limit_holdout]
|
||||
tasks = replay + holdout
|
||||
val = val[:limit_holdout]
|
||||
test = test[:limit_holdout]
|
||||
tasks = train + val + test
|
||||
|
||||
baseline_target = _holdout_hard(target, tasks, skill)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user