feat(sleep): optimizer/target model split, transfer experiment, LLM miner
Three additions driven by the goal of price-aware, model-flexible sleep: 1. DualBackend + build_backend(): route attempt->TARGET model and reflect/judge->OPTIMIZER model (SkillOpt's target-vs-optimizer split). gbrain runner gains --optimizer-backend/-model + --target-backend/-model. 2. run_transfer.py: sleep-scenario cross-model transfer. Optimize a skill on a SOURCE model (e.g. cheap haiku), freeze it, evaluate held-out on a TARGET model (e.g. expensive sonnet) with no further optimization — plus a direct reference. Mirrors the SkillOpt paper's transfer table; quantifies the "optimize cheap overnight, deploy anywhere" value prop. 3. llm_miner.py: turn real harvested transcripts into TaskRecords WITH checkable rule/rubric judges, wired into the cycle for non-mock backends, so real-data lift becomes measurable (heuristic miner remains the no-API fallback). Fixed a str.format brace bug the new unit test caught. 19 tests pass. Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
This commit is contained in:
@@ -24,7 +24,7 @@ import json
|
||||
import sys
|
||||
from typing import Dict, List, Optional
|
||||
|
||||
from skillopt.sleep.backend import get_backend
|
||||
from skillopt.sleep.backend import build_backend, get_backend
|
||||
from skillopt.sleep.consolidate import consolidate, select_gate_score
|
||||
from skillopt.sleep.experiments.gbrain_bench import (
|
||||
available_seeds,
|
||||
@@ -90,6 +90,10 @@ def main(argv=None) -> int:
|
||||
ap = argparse.ArgumentParser(description="Run gbrain-evals skillopt-v1 with SkillOpt-Sleep")
|
||||
ap.add_argument("--backend", default="mock", choices=["mock", "claude", "codex"])
|
||||
ap.add_argument("--model", default="")
|
||||
ap.add_argument("--optimizer-backend", default="", help="route reflect/judge here (dual)")
|
||||
ap.add_argument("--optimizer-model", default="")
|
||||
ap.add_argument("--target-backend", default="", help="route attempt here (dual)")
|
||||
ap.add_argument("--target-model", default="")
|
||||
ap.add_argument("--codex-path", default="")
|
||||
ap.add_argument("--data-root", default="", help="path to eval/data/skillopt-v1")
|
||||
ap.add_argument("--seeds", default="", help="comma list; default = all available")
|
||||
@@ -107,7 +111,12 @@ def main(argv=None) -> int:
|
||||
return 2
|
||||
|
||||
seeds = [s.strip() for s in args.seeds.split(",") if s.strip()] or available_seeds(data_root)
|
||||
backend = get_backend(args.backend, model=args.model, codex_path=args.codex_path)
|
||||
backend = build_backend(
|
||||
backend=args.backend, model=args.model,
|
||||
optimizer_backend=args.optimizer_backend, optimizer_model=args.optimizer_model,
|
||||
target_backend=args.target_backend, target_model=args.target_model,
|
||||
codex_path=args.codex_path,
|
||||
)
|
||||
|
||||
results = []
|
||||
for seed in seeds:
|
||||
|
||||
Reference in New Issue
Block a user