Yifan Yang
b1f41a7506
Machine-generated benchmark_report.md from a 9-config sweep: - Direct (Sonnet->Haiku): brief-writer/advisor/thorough-analyst 0->1.00 - Direct (Codex): brief-writer/advisor 0->1.00 - Transfer (4/4 positive, incl. cross-runtime Codex<->Claude): all 0->1.00 Cross-model transfer confirms the price-difference value prop: a skill optimized on a cheap model deploys for free on an expensive one, and skills move between Codex and Claude. sweep.jsonl is the committed source data. Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
1.9 KiB
1.9 KiB
SkillOpt-Sleep — benchmark report
Auto-generated from sweep.jsonl. Benchmark: gbrain-evals skillopt-v1 (deficient skills, train/held-out split, local rule judge — no judge-API).
Held-out scores are computed by the harness, not the optimizer.
Direct improvement (optimize, then deploy)
| Optimizer → Target | Seed | Held-out before | Held-out after | Nights | Tokens |
|---|---|---|---|---|---|
| claude:sonnet → claude:haiku | brief-writer | 0.00 | 1.00 | 2 | 6657 |
| claude:sonnet → claude:haiku | advisor | 0.00 | 1.00 | 2 | 7891 |
| claude:sonnet → claude:haiku | thorough-analyst | 0.00 | 1.00 | 2 | 17960 |
| codex:default → codex:default | brief-writer | 0.00 | 1.00 | 2 | 9969 |
| codex:default → codex:default | advisor | 0.00 | 1.00 | 2 | 6210 |
5/5 configurations improved on held-out.
Cross-model transfer (optimize on SOURCE, deploy frozen on TARGET)
The price-difference story: spend cheap tokens optimizing overnight, then deploy the frozen skill on any model with no further optimization.
| Source (optimizer) | Target (deploy) | Seed | Target baseline | Transferred | Gain |
|---|---|---|---|---|---|
| claude:haiku | claude:sonnet | brief-writer | 0.00 | 1.00 | +1.00 |
| claude:sonnet | claude:haiku | brief-writer | 0.00 | 1.00 | +1.00 |
| codex:default | claude:haiku | brief-writer | 0.00 | 1.00 | +1.00 |
| claude:haiku | codex:default | brief-writer | 0.00 | 1.00 | +1.00 |
4/4 transfers were positive (frozen skill helped a different model than it was optimized on).
How to reproduce
git clone https://github.com/garrytan/gbrain-evals /tmp/gbrain-evals
python -m skillopt.sleep.experiments.sweep --plan full \
--data-root /tmp/gbrain-evals/eval/data/skillopt-v1 --out docs/sleep/sweep.jsonl
python -m skillopt.sleep.experiments.report \
--in docs/sleep/sweep.jsonl --out docs/sleep/benchmark_report.md