feat(sleep): optimizer/target model split, transfer experiment, LLM miner
Three additions driven by the goal of price-aware, model-flexible sleep: 1. DualBackend + build_backend(): route attempt->TARGET model and reflect/judge->OPTIMIZER model (SkillOpt's target-vs-optimizer split). gbrain runner gains --optimizer-backend/-model + --target-backend/-model. 2. run_transfer.py: sleep-scenario cross-model transfer. Optimize a skill on a SOURCE model (e.g. cheap haiku), freeze it, evaluate held-out on a TARGET model (e.g. expensive sonnet) with no further optimization — plus a direct reference. Mirrors the SkillOpt paper's transfer table; quantifies the "optimize cheap overnight, deploy anywhere" value prop. 3. llm_miner.py: turn real harvested transcripts into TaskRecords WITH checkable rule/rubric judges, wired into the cycle for non-mock backends, so real-data lift becomes measurable (heuristic miner remains the no-API fallback). Fixed a str.format brace bug the new unit test caught. 19 tests pass. Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
This commit is contained in:
@@ -125,11 +125,22 @@ def run_sleep_cycle(
|
||||
limit=cfg.get("max_tasks_per_night", 40) * 3,
|
||||
)
|
||||
n_sessions = len(digests)
|
||||
# When a real backend is configured, use it to mine checkable tasks from
|
||||
# the transcripts (rubric/rule judges); otherwise fall back to the
|
||||
# heuristic miner (no API, no checkable reference).
|
||||
llm_miner = None
|
||||
if cfg.get("backend", "mock") != "mock" and cfg.get("llm_mine", True):
|
||||
try:
|
||||
from skillopt.sleep.llm_miner import make_llm_miner
|
||||
llm_miner = make_llm_miner(backend, max_tasks=cfg.get("max_tasks_per_night", 40))
|
||||
except Exception:
|
||||
llm_miner = None
|
||||
tasks = mine(
|
||||
digests,
|
||||
max_tasks=cfg.get("max_tasks_per_night", 40),
|
||||
holdout_fraction=cfg.get("holdout_fraction", 0.34),
|
||||
seed=cfg.get("seed", 42),
|
||||
llm_miner=llm_miner,
|
||||
)
|
||||
|
||||
# ── live skill/memory docs ───────────────────────────────────────────
|
||||
|
||||
Reference in New Issue
Block a user