envs/_template: make template instantiable against real EnvAdapter ABC
The shipped env_template.py and loader_template.py described the same fictional async execute / evaluate / build_prompt API documented in docs/reference/api.md. As a result TemplateBenchmarkEnv(cfg) raised 'TypeError: Can't instantiate abstract class' for every copy-and-paste user who followed the in-tree scaffold. Rewrite the template so it's a working starting point: - env_template.py: TemplateBenchmarkEnv(EnvAdapter) now implements all five real abstract methods (build_train_env, build_eval_env, rollout, reflect, get_task_types) with no-op defaults documented as TODO. Instantiable today; pytest 60/60 still passes. - loader_template.py: TemplateBenchmarkLoader(SplitDataLoader) implements load_split_items for .json / .jsonl input and explains the optional load_raw_items override for split_mode="ratio". - README.md: usage steps now point at scripts/train.py's _ENV_REGISTRY (the real registry) instead of a non-existent BENCHMARK_REGISTRY in skillopt/envs/__init__.py, and link to the rewritten new-benchmark guide. - config_template.yaml: _base_ is a string path (not a list, which the loader rejects); skill_init is commented out with a note so the template config doesn't reference a file the user hasn't created. Verified locally: 'from skillopt.envs._template.env_template import TemplateBenchmarkEnv; TemplateBenchmarkEnv()' succeeds. Refs microsoft/SkillOpt#30. Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
This commit is contained in:
@@ -4,16 +4,40 @@ This directory provides scaffold files for adding a new benchmark to SkillOpt.
|
||||
|
||||
## Files
|
||||
|
||||
- `env_template.py` — Environment adapter template
|
||||
- `loader_template.py` — Data loader template
|
||||
- `config_template.yaml` — Config file template
|
||||
- `env_template.py` — Environment adapter template (subclasses
|
||||
`EnvAdapter`; implements the 5 abstract methods so the file is
|
||||
instantiable out of the box).
|
||||
- `loader_template.py` — Data loader template (subclasses
|
||||
`SplitDataLoader`; implements `load_split_items` for `.json`/`.jsonl`).
|
||||
- `config_template.yaml` — Config file template.
|
||||
|
||||
## Usage
|
||||
|
||||
1. Copy this directory: `cp -r skillopt/envs/_template skillopt/envs/your_benchmark`
|
||||
2. Rename files: remove `_template` suffix
|
||||
3. Implement the `TODO` sections
|
||||
4. Register in `skillopt/envs/__init__.py`
|
||||
5. Create config at `configs/your_benchmark/default.yaml`
|
||||
1. **Copy the directory:**
|
||||
```bash
|
||||
cp -r skillopt/envs/_template skillopt/envs/your_benchmark
|
||||
```
|
||||
2. **Rename the files** (drop the `_template` suffix):
|
||||
```bash
|
||||
cd skillopt/envs/your_benchmark
|
||||
mv env_template.py adapter.py
|
||||
mv loader_template.py loader.py
|
||||
```
|
||||
…and inside each file rename the classes
|
||||
(`TemplateBenchmarkEnv → YourBenchmarkAdapter`,
|
||||
`TemplateBenchmarkLoader → YourBenchmarkLoader`)
|
||||
and fix the cross-import in `adapter.py`.
|
||||
3. **Implement the TODO blocks** inside `adapter.py:rollout` and the
|
||||
`_normalize_item` helper in `loader.py`. If you want real reflection,
|
||||
uncomment the `run_minibatch_reflect` block in `adapter.py:reflect`.
|
||||
4. **Register** the adapter — add a `try / except ImportError` block in
|
||||
`scripts/train.py`'s `_register_builtins()` mapping the registry key
|
||||
to your `YourBenchmarkAdapter` class. There is no
|
||||
`BENCHMARK_REGISTRY` dict in `skillopt/envs/__init__.py`; the live
|
||||
registry is `_ENV_REGISTRY` in `scripts/train.py`.
|
||||
5. **Create the config** at `configs/your_benchmark/default.yaml`
|
||||
(start from `config_template.yaml`). `_base_` is a **string path**,
|
||||
not a list.
|
||||
|
||||
See the [documentation](../../docs/guide/new-benchmark.md) for the full guide.
|
||||
See the [Add a New Benchmark guide](../../../docs/guide/new-benchmark.md)
|
||||
for the full step-by-step with a worked `docfaithful` example.
|
||||
|
||||
Reference in New Issue
Block a user