feat(sleep): SkillOpt-Sleep plugin update (preview) — engine robustness + scheduling
Updates the SkillOpt-Sleep plugin on top of the current main. User-facing and engine improvements since the initial drop: * Command renamed /sleep -> /skillopt-sleep across Claude Code + Codex shells; refreshed plugin READMEs and install scripts. * Built-in scheduling (skillopt_sleep/scheduler.py + __main__): schedule / unschedule the nightly cycle without external cron wiring. * Backend robustness: bounded retry with backoff (no more silent empty-string on transient 429/timeout), content-filter-safe rollout prompt, an output-contract guardrail that rejects edits violating the task's required format, and a per-sample cache key so repeated dream rollouts are independent samples (fixes degenerate single-sample reflection). * consolidate / rollout / replay: parallel multi-rollout dreaming, gate-mode controls, TaskRecord.system framing field. Scope: this commit ships only the plugin engine + shells. Research/benchmark harnesses and their data are intentionally not included; the public package has no dependency on them (the one research-evaluator import is now guarded). Marked as an early preview in the README; we'll keep iterating. 99/99 unit tests pass. Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
This commit is contained in:
@@ -58,12 +58,34 @@ def multi_rollout(
|
||||
memory: str,
|
||||
*,
|
||||
k: int = 3,
|
||||
workers: int = 0,
|
||||
) -> RolloutSet:
|
||||
"""Run ``task`` K times. replay_one is deterministic for mock; for real
|
||||
backends the model's own sampling yields variation across attempts."""
|
||||
backends the model's own sampling yields variation across attempts.
|
||||
|
||||
The K attempts are independent, so they run concurrently (this is the dream
|
||||
phase's dominant cost). ``workers`` defaults to the SKILLOPT_SLEEP_WORKERS
|
||||
env (capped at k); set to 1 to force serial (used by the mock tests).
|
||||
"""
|
||||
import os
|
||||
rs = RolloutSet(task=task)
|
||||
for _ in range(max(1, k)):
|
||||
rs.attempts.append(replay_one(backend, task, skill, memory))
|
||||
k = max(1, k)
|
||||
if workers <= 0:
|
||||
try:
|
||||
workers = int(os.environ.get("SKILLOPT_SLEEP_WORKERS", "1"))
|
||||
except ValueError:
|
||||
workers = 1
|
||||
workers = max(1, min(workers, k))
|
||||
if workers == 1:
|
||||
for i in range(k):
|
||||
rs.attempts.append(replay_one(backend, task, skill, memory, sample_id=i))
|
||||
return rs
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
with ThreadPoolExecutor(max_workers=workers) as ex:
|
||||
futs = [ex.submit(replay_one, backend, task, skill, memory, sample_id=i)
|
||||
for i in range(k)]
|
||||
for f in futs:
|
||||
rs.attempts.append(f.result())
|
||||
return rs
|
||||
|
||||
|
||||
@@ -97,6 +119,11 @@ def contrastive_reflect(
|
||||
f"- BAD attempt (score {rs.worst.hard:.1f}): {rs.worst.response[:200]}\n"
|
||||
f" (bad failed: {rs.worst.fail_reason[:100]})"
|
||||
)
|
||||
# the output contract the proposed rules must not violate (same guardrail the
|
||||
# single-shot reflect uses — prevents harness-violating rules like "return VBA"
|
||||
# or "ask the user for the range" on SpreadsheetBench).
|
||||
from skillopt_sleep.backend import _task_guardrail
|
||||
guard = _task_guardrail([(rs.task, rs.best) for rs in informative])
|
||||
prompt = (
|
||||
"You are SkillOpt's optimizer doing CONTRASTIVE reflection. For each task "
|
||||
"below the agent was run multiple times; some attempts succeeded and some "
|
||||
@@ -104,6 +131,10 @@ def contrastive_reflect(
|
||||
f"and propose at most {edit_budget} SHORT, GENERAL, reusable rules for the "
|
||||
f"{target} that would make the good behavior reliable every time. Quote "
|
||||
"concrete thresholds/formats verbatim; do not paraphrase vaguely. "
|
||||
"Every rule MUST obey the task output contract (if shown) — never propose "
|
||||
"a rule that changes the required output format/language or tells the agent "
|
||||
"to ask the user a question; such a rule scores ZERO.\n"
|
||||
f"{guard}"
|
||||
'Return ONLY a JSON array: '
|
||||
'[{"op":"add","content":"<rule>","rationale":"<what good did that bad didnt>"}].\n\n'
|
||||
+ "\n\n".join(blocks)
|
||||
|
||||
Reference in New Issue
Block a user