The shipped env_template.py and loader_template.py described the same
fictional async execute / evaluate / build_prompt API documented in
docs/reference/api.md. As a result TemplateBenchmarkEnv(cfg) raised
'TypeError: Can't instantiate abstract class' for every copy-and-paste
user who followed the in-tree scaffold.
Rewrite the template so it's a working starting point:
- env_template.py: TemplateBenchmarkEnv(EnvAdapter) now implements all
five real abstract methods (build_train_env, build_eval_env, rollout,
reflect, get_task_types) with no-op defaults documented as TODO.
Instantiable today; pytest 60/60 still passes.
- loader_template.py: TemplateBenchmarkLoader(SplitDataLoader)
implements load_split_items for .json / .jsonl input and explains the
optional load_raw_items override for split_mode="ratio".
- README.md: usage steps now point at scripts/train.py's _ENV_REGISTRY
(the real registry) instead of a non-existent BENCHMARK_REGISTRY in
skillopt/envs/__init__.py, and link to the rewritten new-benchmark
guide.
- config_template.yaml: _base_ is a string path (not a list, which the
loader rejects); skill_init is commented out with a note so the
template config doesn't reference a file the user hasn't created.
Verified locally: 'from skillopt.envs._template.env_template import
TemplateBenchmarkEnv; TemplateBenchmarkEnv()' succeeds. Refs
microsoft/SkillOpt#30.
Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
Remove sealqa, babyvision, mathverse, mmrb, swebench envs and configs.
Remove deep_probe, deep_reflect, meta_reflect modules and prompts.
Remove download_babyvision script.
These are not part of the core released benchmarks.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Skill optimization framework with training loop analogy
- 11 benchmarks, 4 model backends (Azure OpenAI, Claude, Codex, Qwen)
- WebUI for browser-based training control
- Pluggable architecture for extending benchmarks and backends
Yifan Yang