feat(sleep): optimizer/target model split, transfer experiment, LLM miner
Three additions driven by the goal of price-aware, model-flexible sleep: 1. DualBackend + build_backend(): route attempt->TARGET model and reflect/judge->OPTIMIZER model (SkillOpt's target-vs-optimizer split). gbrain runner gains --optimizer-backend/-model + --target-backend/-model. 2. run_transfer.py: sleep-scenario cross-model transfer. Optimize a skill on a SOURCE model (e.g. cheap haiku), freeze it, evaluate held-out on a TARGET model (e.g. expensive sonnet) with no further optimization — plus a direct reference. Mirrors the SkillOpt paper's transfer table; quantifies the "optimize cheap overnight, deploy anywhere" value prop. 3. llm_miner.py: turn real harvested transcripts into TaskRecords WITH checkable rule/rubric judges, wired into the cycle for non-mock backends, so real-data lift becomes measurable (heuristic miner remains the no-API fallback). Fixed a str.format brace bug the new unit test caught. 19 tests pass. Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
This commit is contained in:
@@ -177,6 +177,42 @@ class TestGbrainLoader(unittest.TestCase):
|
||||
self.assertEqual(score_rule_judge(ho.judge, skill)[0], 0.0)
|
||||
|
||||
|
||||
class TestLlmMiner(unittest.TestCase):
|
||||
def test_miner_emits_checkable_tasks(self):
|
||||
# a stub backend whose _call returns canned miner JSON => deterministic
|
||||
from skillopt.sleep.backend import Backend
|
||||
from skillopt.sleep.llm_miner import make_llm_miner
|
||||
|
||||
class StubBackend(Backend):
|
||||
name = "stub"
|
||||
def _call(self, prompt, *, max_tokens=1024):
|
||||
return ('[{"intent":"write a research brief",'
|
||||
'"checks":[{"op":"section_present","arg":"Key Risks"}],'
|
||||
'"rubric":"has a risks section","satisfied":false}]')
|
||||
|
||||
digest = SessionDigest(session_id="s1", project="/p",
|
||||
user_prompts=["write a brief on X"],
|
||||
assistant_finals=["a brief"], n_user_turns=1)
|
||||
miner = make_llm_miner(StubBackend())
|
||||
tasks = miner([digest])
|
||||
self.assertEqual(len(tasks), 1)
|
||||
self.assertEqual(tasks[0].reference_kind, "rule")
|
||||
self.assertEqual(tasks[0].judge["checks"][0]["op"], "section_present")
|
||||
|
||||
def test_miner_drops_uncheckable(self):
|
||||
from skillopt.sleep.backend import Backend
|
||||
from skillopt.sleep.llm_miner import make_llm_miner
|
||||
|
||||
class EmptyBackend(Backend):
|
||||
name = "stub"
|
||||
def _call(self, prompt, *, max_tokens=1024):
|
||||
return "[]"
|
||||
|
||||
digest = SessionDigest(session_id="s1", project="/p",
|
||||
user_prompts=["chat"], n_user_turns=1)
|
||||
self.assertEqual(make_llm_miner(EmptyBackend())([digest]), [])
|
||||
|
||||
|
||||
class TestFullCycleAndAdopt(unittest.TestCase):
|
||||
def test_cycle_stage_then_adopt_with_backup(self):
|
||||
with tempfile.TemporaryDirectory() as proj, tempfile.TemporaryDirectory() as home:
|
||||
|
||||
Reference in New Issue
Block a user