url/SkillOpt
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(sleep): 3-way train/val/test split + gate_mode on|off

Browse Source 
Data-split refactor (the anti-overfitting foundation the user asked for):
  - TaskRecord gains split∈{train,val,test} and origin∈{real,dream}.
  - assign_splits: real tasks deterministically split into val/test (disjoint);
    DREAM-augmented tasks (origin='dream') NEVER enter val/test — they only go to
    train. val gates updates; test is the final held-out measure.
  - gbrain loader maps its held-out.jsonl -> test, benchmark.jsonl -> train/val,
    so the gbrain held-out stays the true final score.
  - consolidate(): train drives reflect, val gates; adds gate_mode='off' (greedy,
    no hard filter) reporting val movement (greedy_improved/regressed/flat).
  - run_gbrain/transfer/experiment score on test (val fallback); run_gbrain gains
    --gate on|off. Legacy replay/holdout names normalized.

New test proves dream tasks never land in val/test. 21 tests pass; mock
experiment + gate=off both green.

Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
This commit is contained in:
Yifan Yang
2026-06-08 14:31:51 +00:00
parent 99ec2caf6b
commit 6f1351edb9
10 changed files with 220 additions and 82 deletions
Download Patch File Download Diff File Expand all files Collapse all files
skillopt/sleep/experiments/gbrain_bench.py
+24 -4
View File
@@ -63,8 +63,17 @@ def _to_task(rec: dict, *, seed: str, split: str) -> TaskRecord:
)
def load_seed(data_root: str, seed: str) -> Tuple[str, List[TaskRecord]]:
"""Return (deficient_skill_md, tasks) for one gbrain seed."""
def load_seed(data_root: str, seed: str, *, val_fraction: float = 0.34,
split_seed: int = 42) -> Tuple[str, List[TaskRecord]]:
"""Return (deficient_skill_md, tasks) for one gbrain seed.
Faithful split mapping:
* gbrain held-out.jsonl -> our ``test`` (the true final measure)
* gbrain benchmark.jsonl -> split deterministically into ``train`` + ``val``
(val gates updates; train drives reflect)
All tasks are origin='real' (gbrain provides no synthetic tasks).
"""
import hashlib
sub = SEED_DIRS.get(seed, seed)
seed_dir = os.path.join(data_root, sub)
skill_path = os.path.join(seed_dir, "SKILL.md")
@@ -73,10 +82,21 @@ def load_seed(data_root: str, seed: str) -> Tuple[str, List[TaskRecord]]:
with open(skill_path, encoding="utf-8") as f:
skill = f.read()
tasks: List[TaskRecord] = []
# benchmark pool -> train/val
val_cut = int(round(val_fraction * 100))
for rec in _load_jsonl(os.path.join(seed_dir, "benchmark.jsonl")):
tasks.append(_to_task(rec, seed=seed, split="replay"))
t = _to_task(rec, seed=seed, split="train")
bucket = int(hashlib.sha256((str(split_seed) + t.id).encode()).hexdigest(), 16) % 100
t.split = "val" if bucket < val_cut else "train"
tasks.append(t)
# held-out -> test
for rec in _load_jsonl(os.path.join(seed_dir, "held-out.jsonl")):
tasks.append(_to_task(rec, seed=seed, split="holdout"))
tasks.append(_to_task(rec, seed=seed, split="test"))
# guarantee a non-empty val
if not any(t.split == "val" for t in tasks):
train_only = [t for t in tasks if t.split == "train"]
if train_only:
train_only[0].split = "val"
return skill, tasks