feat(sleep): optimizer/target model split, transfer experiment, LLM miner

Three additions driven by the goal of price-aware, model-flexible sleep:

1. DualBackend + build_backend(): route attempt->TARGET model and
   reflect/judge->OPTIMIZER model (SkillOpt's target-vs-optimizer split).
   gbrain runner gains --optimizer-backend/-model + --target-backend/-model.

2. run_transfer.py: sleep-scenario cross-model transfer. Optimize a skill on a
   SOURCE model (e.g. cheap haiku), freeze it, evaluate held-out on a TARGET
   model (e.g. expensive sonnet) with no further optimization — plus a direct
   reference. Mirrors the SkillOpt paper's transfer table; quantifies the
   "optimize cheap overnight, deploy anywhere" value prop.

3. llm_miner.py: turn real harvested transcripts into TaskRecords WITH checkable
   rule/rubric judges, wired into the cycle for non-mock backends, so real-data
   lift becomes measurable (heuristic miner remains the no-API fallback).
   Fixed a str.format brace bug the new unit test caught.

19 tests pass.

Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
This commit is contained in:
Yifan Yang
2026-06-08 14:31:51 +00:00
parent 63c79b3602
commit 7d9900b6af
7 changed files with 409 additions and 2 deletions
+11 -2
View File
@@ -24,7 +24,7 @@ import json
import sys
from typing import Dict, List, Optional
from skillopt.sleep.backend import get_backend
from skillopt.sleep.backend import build_backend, get_backend
from skillopt.sleep.consolidate import consolidate, select_gate_score
from skillopt.sleep.experiments.gbrain_bench import (
available_seeds,
@@ -90,6 +90,10 @@ def main(argv=None) -> int:
ap = argparse.ArgumentParser(description="Run gbrain-evals skillopt-v1 with SkillOpt-Sleep")
ap.add_argument("--backend", default="mock", choices=["mock", "claude", "codex"])
ap.add_argument("--model", default="")
ap.add_argument("--optimizer-backend", default="", help="route reflect/judge here (dual)")
ap.add_argument("--optimizer-model", default="")
ap.add_argument("--target-backend", default="", help="route attempt here (dual)")
ap.add_argument("--target-model", default="")
ap.add_argument("--codex-path", default="")
ap.add_argument("--data-root", default="", help="path to eval/data/skillopt-v1")
ap.add_argument("--seeds", default="", help="comma list; default = all available")
@@ -107,7 +111,12 @@ def main(argv=None) -> int:
return 2
seeds = [s.strip() for s in args.seeds.split(",") if s.strip()] or available_seeds(data_root)
backend = get_backend(args.backend, model=args.model, codex_path=args.codex_path)
backend = build_backend(
backend=args.backend, model=args.model,
optimizer_backend=args.optimizer_backend, optimizer_model=args.optimizer_model,
target_backend=args.target_backend, target_model=args.target_model,
codex_path=args.codex_path,
)
results = []
for seed in seeds: