The LinearScheduler.known_decay_sequence test now reflects the new
t = (step-1)/(total_steps-1) contract where step 1 returns max_lr.
Updated docstrings for midpoint and early-step tests to match.
All 43 tests now pass against the updated formula.
LinearScheduler and CosineScheduler now guarantee step() returns max_lr
on the first call and min_lr on the total_steps-th call, using the
standard endpoint parameterisation t = (step-1)/(total_steps-1).
This resolves the inconsistent first-step semantics reported in review.
Three filters to stop machine-generated prompts polluting the mined task
pool (they dominated ~80% of tasks on this machine):
- _is_meta_prompt: drop expanded slash-command bodies (<command-message>
tags or '# /' headers) — plugin self-invocations are not user intents
- _AGENT_SESSION_MARKERS + _is_agent_session: skip sessions whose first
prompt is another tool's agent brief (claude-mem observers, CLAUDE.md
critic sub-agents, SkillOpt-Sleep's own command body)
- load walk: skip <session>/subagents/ dirs and agent-*.jsonl files —
Agent-tool sidechain transcripts are Claude-authored, not user tasks
Verified: harvest went from 120 sessions / mostly-noise tasks to 55
sessions / 38 real user tasks.
Co-authored-by: codeL1985 <l@cypherlab.tech>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Two missing pieces in the minimax_chat dispatch chain:
1. minimax_backend.py did not define chat_optimizer or chat_optimizer_messages;
only chat_target / chat_target_messages existed. Any caller using
optimizer_backend='minimax_chat' would hit AttributeError or fall through
to _openai (Azure).
2. skillopt/model/__init__.py's chat_optimizer and chat_optimizer_messages
dispatchers checked claude_chat and qwen_chat but not minimax_chat,
so minimax_chat callers would silently fall through to _openai.chat_optimizer
(the Azure path), which fails with 'Azure OpenAI endpoint is not configured'
on any setup without AZURE_OPENAI_* env vars.
Adds chat_optimizer to minimax_backend.py (mirrors chat_target via
_chat_messages_impl) and minimax_chat branches to both
chat_optimizer / chat_optimizer_messages dispatchers.
Verified locally: 1-epoch training on a 4-item SearchQA-format dataset
went from '[skip] no usable patches — skill unchanged' (baseline fallback)
to a successful accept_new_best with success_patches=1 per step.
Co-authored-by: Mavis (MiniMax) <Mavis@MiniMax.local>
Co-authored-by: jc <jc@users.noreply.github.com>
The qwen_chat backend (the generic OpenAI-compatible client) hardcoded
max_tokens and always sent temperature, so reasoning models behind
OpenAI-compatible gateways (GPT-5.x, Claude Opus 4.8 via Azure/LiteLLM)
would 400.
- Add opt-in QWEN_CHAT_USE_MAX_COMPLETION_TOKENS (+ role variants) that
swaps the payload key max_tokens -> max_completion_tokens.
- Treat an explicit empty / none / off temperature as "omit" instead of
collapsing to the 0.7 default (via _resolve_temperature).
- Thread both through configure_qwen_chat / _update_config.
- Defaults unchanged; fully backward compatible. Adds 6 tests.
Fixes#127
Co-authored-by: Chirag Singhal <chirag127@users.noreply.github.com>
Adds --backend handoff: the engine runs all deterministic stages and
outsources attempt/judge/reflect to prompt/answer files an interactive
agent session fills between runs (exit 3 = pending batch, re-run to
resume). Deterministic replay + the prompt-hash answer cache make resume
stateless; sentinel detection aborts any call built from unanswered
output so placeholders never reach scores or staging. Session digests
and mined tasks are pinned per night (secret-redacted) so the sessions
answering prompts cannot shift the task set, and LLM mining is routed
through the same handoff files. Ships a /skillopt-sleep-handoff Claude
Code command that answers each prompt in a fresh-context subagent to
protect the held-out gate.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(skillopt-sleep): surface Claude CLI spawn failures instead of silent zero scores
In _call and attempt_with_tools, the bare 'except Exception: return ""'
swallows FileNotFoundError (e.g. bare 'claude' on Windows with npm .cmd
shim) and any other spawn failure, returning an empty string that the
trainer treats as a legitimate model response that scores 0.0 everywhere.
Now the exception is caught explicitly: last_call_error is set, a
warning is logged, and the empty-string return is preserved for
backward compatibility of the control flow. This mirrors the pattern
from #92 (codex backend) which fixed the same class of 'dead CLI
masquerades as nothing to learn' bug.
Issue: #121
* test(skillopt-sleep): add tests verifying Claude CLI spawn failures are surfaced
Add two tests to TestClaudeCliBackendBare:
- test_spawn_failure_sets_last_call_error: _call sets last_call_error
and returns '' when subprocess.run raises FileNotFoundError.
- test_attempt_tools_spawn_failure_sets_last_call_error: same for
attempt_with_tools.
These prove the fix from the parent commit (surface spawn failures
instead of silently scoring 0) and guard against regressions.
* fix(backend): make ClaudeCliBackend work on Windows (.cmd shim + argv limit)
Every claude call failed with WinError 2 and was swallowed by the bare
except -> return '', so real-backend cycles silently scored 0.000/0.000
and produced zero edits.
- resolve claude_path via shutil.which: the npm-installed claude is a
.cmd shim that CreateProcess cannot resolve by bare name
- pass the prompt via stdin instead of argv: the .cmd shim routes
through cmd.exe whose command line caps at ~8K chars, which
reflect/judge prompts exceed
Verified: reflect() on a synthetic max_chars failure now returns a
concrete bounded edit via the real CLI (was [] before).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(backend): suppress console window flashes on Windows (CREATE_NO_WINDOW)
Every CLI subprocess (claude/codex/copilot) spawned from a console-less
parent allocates a visible console window on Windows. A cycle making
hundreds of calls strobes cmd windows and steals focus, making the
machine unusable while the engine runs. Pass CREATE_NO_WINDOW on all
CLI call sites; no-op on POSIX.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: codeL1985 <l@cypherlab.tech>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Add explicit assertions for held-out scores and gate actions to the verifier discipline test suite to strengthen its guarantees.
- Assert the concrete held-out baseline and candidate scores in test_gate_rejects_reward_hacking_edit.
- Add test_gate_accepts_beneficial_edit using MockBeneficialBackend to provide a paired case where an edit genuinely improves the held-out slice, expecting accepted=True and gate_action='accept_new_best'.
_load_yaml opened config files with the platform default (locale)
encoding instead of UTF-8. The shipped configs (e.g.
configs/_base_/default.yaml) contain UTF-8 non-ASCII characters
(em-dash, arrows, box-drawing), so load_config() raises
UnicodeDecodeError on any non-UTF-8 locale (e.g. Windows cp936/cp1252),
breaking scripts/train.py and scripts/eval_only.py before startup.
YAML is UTF-8 by spec, and the rest of the codebase already opens text
files with encoding="utf-8". Pass encoding="utf-8" here for parity.
The Claude Code and Codex plugin shells' runner (run-sleep.sh) only
resolved the engine by searching for a skillopt_sleep/ source directory
on disk. After v0.2.0 shipped the engine on PyPI (skillopt-sleep CLI),
the plugins still errored for users who installed via pip/uv without
cloning the repo — even though the error message said "pip install
skillopt" would work.
Add two fallbacks before the error:
1. skillopt-sleep CLI on PATH (covers uv tool install, pipx, pip install)
2. python -m skillopt_sleep when importable (covers pip install into
the active Python)
Fallback 1 is checked before fallback 2 because uv tool install / pipx
isolate the package from the system Python's import path, so the import
check would fail even though the CLI is available.
The existing source-checkout resolution path is unchanged — repo-clone
users keep working as before. The fix lands in both plugins/run-sleep.sh
(shared, used by Codex) and plugins/claude-code/scripts/run-sleep.sh
(bundled copy, byte-identical, used by the Claude Code marketplace
install which fetches only the plugins/claude-code/ subtree).
Runnable SearchQA splits should remain disjoint. A duplicate manifest id across train, val, or test previously collapsed into the wanted-id set and reused the same row in multiple output splits without warning.
Constraint: Preserve manifest order and output schema for valid manifests.
Rejected: Deduplicate automatically | hiding split overlap would make evaluation contamination harder to notice.
Confidence: high
Scope-risk: narrow
Reversibility: clean
Directive: Fail fast on split-manifest integrity problems instead of repairing them silently.
Tested: uv run --with pytest pytest tests/test_materialize_searchqa.py -q
Tested: uv run --with ruff ruff check scripts/materialize_searchqa.py tests/test_materialize_searchqa.py
Not-tested: Loading the live Hugging Face dataset.
Rollout hard scores can be continuous when smoothed rewards are used. Converting the field through int() turned values like 0.75 into 0, which loses signal before scoring and serialization.
Constraint: Keep the existing RolloutResult dictionary shape unchanged.
Rejected: Clamp hard scores to 0 or 1 | contradicts existing continuous-score support in compute_score and sleep replay types.
Confidence: high
Scope-risk: narrow
Reversibility: clean
Directive: Treat hard as numeric reward data, not only a binary label.
Tested: uv run --with pytest pytest tests/test_types.py tests/test_scoring.py -q
Tested: uv run --with ruff ruff check skillopt/types.py tests/test_types.py
Not-tested: Full benchmark rollouts.
PR #92 added a per-cycle diagnostics.json that surfaces backend stderr,
optimizer replies, and task responses so a 0.0 night is self-diagnosing.
Those free-text fields can carry credentials (e.g. a codex 401 stderr dump
containing an auth token), so persisting them verbatim was a new on-disk
leak surface.
- Add a shared redact_secrets() in staging.py and route diagnostics.json's
call_error / reflect_raw_head / holdout_detail through it before writing.
- Redact the codex and Claude auth-error log lines too (a secondary sink
when a file log handler is attached); last_call_error stays raw in memory
so _AUTH_MARKERS matching is unaffected.
- Centralize _SECRET_PATTERNS in staging.py (harvest_codex now reuses them)
and extend coverage to AWS / GitHub / Slack / Google / JWT token shapes.
- Tests: secret-shape coverage, private-key blocks, recursive/scalar
passthrough, no over-redaction of plain prose, fail-fast auth-error log
redaction, and an end-to-end check that diagnostics.json has no secret.
Observability-only; the gate and learning algorithm are unchanged.
Co-Authored-By: Claude <noreply@anthropic.com>
Splits CodexCliBackend._call into _call_once + a retry wrapper so transient empties/timeouts are retried instead of silently scored 0, and fails fast on fatal auth/model/version errors (401, refresh_token_reused, token_expired, ChatGPT-account-unsupported, newer-Codex-required). On non-zero exit the CLI error text is surfaced via last_call_error instead of being returned as a model response. Adds per-cycle diagnostics.json (observability only; gate and learning algorithm unchanged) so a 0.0 night self-explains.
Adds a reward-hacking stress test ensuring the consolidation gate rejects skill edits that game train/replay tasks while degrading held-out behavior. Also wires the minimax_chat backend into scripts/eval_only.py (coexisting with the qwen wiring from #85). Closes#67.
Add a regression test to ensure the validation gate correctly rejects
reward-hacking skill edits. It has been observed that optimizers
sometimes propose shortcuts that improve train/replay metrics but fail
to improve held-out behavior. This test codifies that the gate blocks
such artifacts.
Add TestVerifierDiscipline to the test_sleep_engine.py suite:
- Create MockRewardHackingBackend that simulates a reward-hacking rule
which passes the train set but degrades the held-out tasks.
- Assert that the proposed edit is rejected by the gate.
Add missing configuration setup in scripts/eval_only.py to properly
support the minimax_chat backend, which was entirely omitted.
Fix the following coverage gaps in eval_only.py:
- Add minimax CLI arguments
- Include the minimax config mappings in _MAP
- Update the backend parsing logic
- Call configure_minimax_chat
Follow-up from a fresh-context review of the prior commit: CodexCliBackend.attempt_with_tools
(the rollout path for tool-requiring tasks) ran codex exec inline, swallowed all exceptions,
and never set last_call_error — so an auth/model/version failure on the tool path still
produced a silent empty->0 with no diagnostic signal, the exact failure class the prior commit
fixed for the _call path. Now it surfaces timeout/exception/non-zero-exit via last_call_error
(response stays empty; never leaks the CLI error text), so a failed tool rollout shows up in
diagnostics.json. Adds a regression test.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A nightly sleep cycle could run for weeks emitting held-out 0.0 -> 0.0 (gate reject, zero
edits), indistinguishable from "nothing to learn", when the real cause was the codex backend
returning an error (expired auth / model unsupported on the account / outdated CLI) that got
scored as a failed rollout.
backend (CodexCliBackend):
- split _call into _call_once + a retry wrapper: transient empties/timeouts are retried
instead of silently returning "" (mirrors AzureOpenAIBackend's guard);
- on a non-zero exit, surface the reason via last_call_error and return "" rather than
leaking the CLI error text as if it were a model response;
- fail fast (no retries) on fatal auth/model/version errors (401, refresh_token_reused,
token_expired, "not supported when using Codex with a ChatGPT account",
"requires a newer version of Codex").
backend (CliBackend.reflect): retain last_reflect_raw so a no-edits night is diagnosable.
consolidate: ConsolidationResult now carries per-task held-out detail (response, hard/soft,
fail_reason) + reflect_raw + call_error.
cycle: write diagnostics.json per cycle so a 0.0 night self-explains instead of being a black box.
tests: 4 new (retry-not-silent-zero, auth-error-surfaced-not-scored, holdout-detail, reflect-raw).
Also gitignore the .skillopt-sleep/ runtime dir.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Wires skillopt_sleep into Devin via a stdlib-only MCP server and an ATIF-v1.7 harvester, following the plugins/copilot thin-shell pattern. Includes path-expansion fix, tests + ATIF fixture, schema/tool parity with copilot, and a harvest fix so single-turn sessions aren't dropped by the <3s replay filter.
A harvested single-turn Devin session spanned only 1s (reply written 1000ms
after the prompt), which the engine's harvest filter conservatively classifies
as a <3s headless replay (skillopt_sleep Issue #62) and skips — so a real
single-turn session mined 0 tasks. Widen the prompt->reply gap to 5s. With this,
an end-to-end dry-run mines the task: "night 1: 1 sessions -> 1 tasks".
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Mirror the copilot MCP server: same rich _TOOL_SCHEMA (source, model,
tasks_file, target_skill_path, max_sessions, max_tasks, lookback_hours,
auto_adopt, json, edit_budget, hour, minute) and generic flag forwarding, plus
sleep_schedule / sleep_unschedule. Devin specifics retained: the ATIF-v1.7
harvest step (run before data-reading actions, engine pointed at it via
--claude-home, default --source claude) and post-adopt sync into .devin/skills/.
Tests + README + rules snippet updated for the 7-tool interface.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Review fixes:
- Path bug: SKILLOPT_DEVIN_CLAUDE_HOME (and SKILLOPT_SLEEP_REPO) read from the
env are now wrapped in os.path.expanduser, so the documented "~/..." config
no longer passes a literal ~ to --claude-home (which yielded zero mined
sessions). expanduser on an absolute default is a no-op.
- tests/test_devin_plugin.py: tool-schema completeness, action→subcommand map,
backend enum, the CLAUDE_HOME expansion regression, and an ATIF-v1.7 harvest
shape test against a bundled fixture.
- plugins/devin/fixtures/devin_sample.json: sample ATIF-v1.7 transcript.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replicates the trainer's backend setup in scripts/eval_only.py so eval-only no longer silently falls back to an unconfigured local endpoint. Closes#84.
Wires the skillopt_sleep engine into Devin (Cognition) via an MCP server,
following the same thin-shell pattern as plugins/copilot.
- mcp_server.py: stdlib-only stdio MCP server exposing the standard sleep_*
tools (status, dry-run, run, adopt, harvest). REPO_ROOT defaults to ../.. so
it finds skillopt_sleep automatically when run from plugins/devin/.
- harvest_devin.py: converts Devin ATIF-v1.7 transcripts, agentmemory, and
.devin/skills/*/SKILL.md into the Claude Code-compatible JSONL the engine
consumes; enriches with taskKey + outcome envelopes (hard test/build signal
or judge rubric). Workspace auto-detection; cross-platform paths.
- judge.py, mcp-config.example.json, devin-rules.snippet.md, README.md.
- plugins/README.md: add Devin to the platform + install tables.
No changes to skillopt_sleep; shells out to `python -m skillopt_sleep` like the
other plugins. Pure stdlib; default backend mock (no API spend).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The eval-only tool skipped configuring some of the backend types, that
the training did configure. Because of this, the eval is silently
fell back to a local endpoint that wasn't actually configured, and
all evaluations runs failed.
Replicate the backend setup based on the trainer's code, and eval-only
can run with the qwen_chat backends.
Co-authored-by: Qwen-Coder <noreply@qwen.ai>
Follow-up to the string-aware brace scan: that change only skipped
double-quoted prose, so brace-shaped text in single quotes, backticks, or
bare prose (e.g. `{op: delete}`, '{x: 1}') still reached json_repair and was
fabricated into a bogus dict — strictly worse than None, since extract_json
feeds the optimizer's skill edits.
Add a _looks_json_like() guard before repair: a genuine JSON object's first
non-space char after `{` is `"` (a key) or `}` (empty). Prose pseudo-objects
start with a bare word and are rejected, while legitimate repair targets
(trailing commas, unescaped quotes inside string values) all begin with `"`
and pass — including objects whose string VALUES contain single quotes or
backticks, which must not be rejected.
Found by an independent GPT-5.5 re-review of the merged #79 code. Adds
regression tests for single-quoted / backticked / bare prose (-> None) and
for legitimate objects with quote/backtick string values (still repaired).
Tests: 30 pass (+3 skip) without json_repair, 33 pass with it, both clean
under -W error::RuntimeWarning.
Co-authored-by: Claude <noreply@anthropic.com>
Yifan Yang