feat(sleep): 3-way train/val/test split + gate_mode on|off
Data-split refactor (the anti-overfitting foundation the user asked for):
- TaskRecord gains split∈{train,val,test} and origin∈{real,dream}.
- assign_splits: real tasks deterministically split into val/test (disjoint);
DREAM-augmented tasks (origin='dream') NEVER enter val/test — they only go to
train. val gates updates; test is the final held-out measure.
- gbrain loader maps its held-out.jsonl -> test, benchmark.jsonl -> train/val,
so the gbrain held-out stays the true final score.
- consolidate(): train drives reflect, val gates; adds gate_mode='off' (greedy,
no hard filter) reporting val movement (greedy_improved/regressed/flat).
- run_gbrain/transfer/experiment score on test (val fallback); run_gbrain gains
--gate on|off. Legacy replay/holdout names normalized.
New test proves dream tasks never land in val/test. 21 tests pass; mock
experiment + gate=off both green.
Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
This commit is contained in:
+10
-1
@@ -61,7 +61,16 @@ class TaskRecord:
|
||||
judge: Dict[str, Any] = field(default_factory=dict) # gbrain-style rule judge
|
||||
tags: List[str] = field(default_factory=list)
|
||||
source_sessions: List[str] = field(default_factory=list)
|
||||
split: str = "replay" # replay (train) | holdout (test)
|
||||
# split ∈ {train, val, test}. val + test come ONLY from real mined tasks and
|
||||
# never overlap (val gates updates, test is the final held-out measure). train
|
||||
# may be dream-augmented (see origin). Legacy values replay->train,
|
||||
# holdout->val are normalized on load.
|
||||
split: str = "train"
|
||||
# origin ∈ {real, dream}. 'real' = mined from the user's actual sessions;
|
||||
# 'dream' = synthetic/augmented for the training pool. Dream tasks are NEVER
|
||||
# allowed into val/test, which is the anti-overfitting guarantee.
|
||||
origin: str = "real"
|
||||
derived_from: str = "" # for dream tasks: the real task id it varies
|
||||
|
||||
def to_dict(self) -> Dict[str, Any]:
|
||||
return asdict(self)
|
||||
|
||||
Reference in New Issue
Block a user