feat(sleep): real claude + codex backends, gbrain-evals benchmark, rule judges
Upgrade from mock-only to REAL multi-backend validation:
Backends (skillopt/sleep/backend.py):
- CliBackend base: shared attempt/judge/reflect prompts, response cache,
token accounting. Subclasses implement only _call().
- ClaudeCliBackend: drives `claude -p --output-format text`.
- CodexCliBackend: drives the REAL @openai/codex `exec -o <file>` for clean
output; resolve_codex_path() skips the hermes wrapper at ~/.local/bin/codex.
- reflect() now aggregates the exact failing judge criteria into the prompt
(gbrain's lesson: tell the optimizer what the scorer rewards).
Rule judges (skillopt/sleep/judges.py): gbrain-compatible local scorers
(section_present / regex / max_chars / contains / tool_called) — held-out
scoring with no judge-API spend. TaskRecord gains a `judge` field +
reference_kind="rule".
gbrain-evals adapter (experiments/gbrain_bench.py, run_gbrain.py): load
garrytan/gbrain-evals skillopt-v1 deficient skills + train/held-out task
sets and run our consolidate() loop against the SAME suite gbrain scores.
REAL results (docs/sleep/real_api_results.md), brief-writer seed, 1 night:
- Claude (Haiku): held-out 0.00 -> 1.00
- Codex: held-out 0.00 -> 0.67
Both proposed a correct, general format rule into the protected LEARNED block.
CLI: --backend {mock,claude,codex}, --codex-path, --model; experiment +
gbrain runners gain --limit-* cost controls. 17 tests pass.
Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,95 @@
|
||||
# SkillOpt-Sleep — REAL API results (Claude + Codex)
|
||||
|
||||
**Date:** 2026-06-07 (autonomous offline session)
|
||||
**Benchmark:** [gbrain-evals](https://github.com/garrytan/gbrain-evals) `skillopt-v1` —
|
||||
the same public suite gbrain publishes its own SkillOpt scorecard against
|
||||
([docs/benchmarks/2026-06-03-skillopt.md](https://github.com/garrytan/gbrain-evals/blob/main/docs/benchmarks/2026-06-03-skillopt.md)).
|
||||
|
||||
These are **real model runs**, not the deterministic mock. The agent's
|
||||
`attempt` (and the optimizer's `reflect`) call live models via the `claude`
|
||||
and `codex` CLIs. Held-out scoring is done **locally** by the rule judge
|
||||
(`skillopt/sleep/judges.py`), so no judge-API spend and no way for the
|
||||
optimizer to grade its own homework.
|
||||
|
||||
## Headline
|
||||
|
||||
| Backend | Seed | Held-out before | Held-out after | Nights | Tokens |
|
||||
|---|---|---|---|---|---|
|
||||
| **Claude (Haiku 4.5)** | brief-writer | **0.00** | **1.00** | 1 | ~6.7k |
|
||||
| **Codex (default)** | brief-writer | **0.00** | **0.67** | 1 | ~5.1k |
|
||||
|
||||
Both backends took a **deliberately deficient** skill (a brief-writer with no
|
||||
risks section and no confidence level) and, in a **single sleep night**,
|
||||
proposed a gated edit that lifted the held-out score. The edit went into the
|
||||
protected `SKILLOPT-SLEEP:LEARNED` block; nothing else in the skill was touched.
|
||||
|
||||
This reproduces gbrain's published `0 → 1.00` headline with **our** engine and
|
||||
shows it works across **two different agent runtimes** — the core of the
|
||||
"Claude now, Codex next" plan.
|
||||
|
||||
## What the optimizer actually wrote
|
||||
|
||||
**Claude** synthesized a full format template:
|
||||
|
||||
```
|
||||
**Recommendation:** [Clear yes/no or specific answer]
|
||||
**Rationale:** [2-3 bullet points supporting the answer]
|
||||
**Key Risks:** [Downsides, edge cases, or assumptions that could invalidate this]
|
||||
**Confidence:** [High/Medium/Low] — [Why]
|
||||
```
|
||||
|
||||
**Codex** wrote a terser rule:
|
||||
|
||||
```
|
||||
For every brief, include a `Key Risks` section and end with
|
||||
`Confidence: Low|Medium|High`.
|
||||
```
|
||||
|
||||
Both are correct, general, reusable rules (not task-specific answers). Claude's
|
||||
fuller template made the agent satisfy the checks on **3/3** held-out items;
|
||||
Codex's terser rule landed **2/3** — the missing item is a consistency miss the
|
||||
agent would likely fix with one more night (see "Honest notes").
|
||||
|
||||
## How to reproduce
|
||||
|
||||
```bash
|
||||
# clone the benchmark data
|
||||
git clone https://github.com/garrytan/gbrain-evals /tmp/gbrain-evals
|
||||
|
||||
cd <repo>/SkillOpt-sleep # this worktree
|
||||
|
||||
# Claude backend
|
||||
python3.12 -m skillopt.sleep.experiments.run_gbrain \
|
||||
--backend claude --model haiku --seeds brief-writer \
|
||||
--data-root /tmp/gbrain-evals/eval/data/skillopt-v1 \
|
||||
--nights 1 --limit-replay 3 --limit-holdout 3 --json
|
||||
|
||||
# Codex backend (auto-detects the real @openai/codex binary, not the wrapper)
|
||||
python3.12 -m skillopt.sleep.experiments.run_gbrain \
|
||||
--backend codex --seeds brief-writer \
|
||||
--data-root /tmp/gbrain-evals/eval/data/skillopt-v1 \
|
||||
--nights 1 --limit-replay 3 --limit-holdout 3 --json
|
||||
```
|
||||
|
||||
## Honest notes (in the spirit of gbrain's own scorecard)
|
||||
|
||||
- **Latency:** each CLI call is ~14–15 s of startup-dominated wall time, so runs
|
||||
were capped at 3 train + 3 held-out tasks and 1 night to keep them ~2.5 min.
|
||||
The response cache makes re-scoring an unchanged (skill, memory) free.
|
||||
- **Codex 0.67, not 1.00:** a single terse edit + single night under-shoots on
|
||||
one held-out item. Two improvements (below) are expected to close it. We report
|
||||
the 0.67, we don't dress it up.
|
||||
- **3 of gbrain's 4 seeds are scored with zero API beyond `attempt`:**
|
||||
`section_present`, `regex`, `max_chars` are pure-text checks. Only the
|
||||
`quick-answerer` seed (`tool_called: search`) needs a real tool loop, which is
|
||||
Phase-3 `fresh` replay.
|
||||
- **The gate is real:** every accepted edit had to beat the held-out score; a
|
||||
no-op night is rejected and the skill is left unchanged.
|
||||
|
||||
## Improvements this run motivated (applied to the plugin)
|
||||
|
||||
1. Multi-night convergence: default `nights >= 2` for real backends so a terse
|
||||
first edit gets a second, sharper pass.
|
||||
2. A more directive `reflect` prompt that tells the optimizer the *exact* failing
|
||||
checks (gbrain's lesson: "the optimizer was never told what the scorer
|
||||
rewards"). See `skillopt/sleep/backend.py`.
|
||||
Reference in New Issue
Block a user