copilot-swe-agent[bot]
131 lines
3.8 KiB
Markdown
131 lines
3.8 KiB
Markdown
# Add a New Benchmark
|
|
|
|
Extend SkillOpt with your own benchmark in ~100 lines of code.
|
|
|
|
## Overview
|
|
|
|
To add a benchmark, you need:
|
|
|
|
1. **Data Loader** — Subclass `SplitDataLoader` to load your split data
|
|
2. **Environment Adapter** — Subclass `EnvAdapter` and implement rollout/reflect hooks
|
|
3. **Config** — YAML configuration file
|
|
4. **Registration** — Add your adapter to the train script registry
|
|
|
|
## Step 1: Create the Benchmark Package
|
|
|
|
```bash
|
|
mkdir -p skillopt/envs/my_benchmark
|
|
touch skillopt/envs/my_benchmark/__init__.py
|
|
```
|
|
|
|
## Step 2: Implement the Data Loader
|
|
|
|
Create `skillopt/envs/my_benchmark/dataloader.py`:
|
|
|
|
```python
|
|
from skillopt.datasets.base import SplitDataLoader
|
|
|
|
|
|
class MyBenchmarkDataLoader(SplitDataLoader):
|
|
"""Load benchmark items from raw data and/or split directories."""
|
|
|
|
def load_raw_items(self, data_path: str) -> list[dict]:
|
|
# For ratio mode, parse your source dataset from data_path.
|
|
# Return list[dict] where each item has at least a unique, deterministic "id".
|
|
return super().load_raw_items(data_path)
|
|
|
|
def load_split_items(self, split_path: str) -> list[dict]:
|
|
# For split_dir mode, parse one split directory.
|
|
return super().load_split_items(split_path)
|
|
```
|
|
|
|
## Step 3: Implement the Environment Adapter
|
|
|
|
Create `skillopt/envs/my_benchmark/adapter.py`:
|
|
|
|
```python
|
|
from skillopt.envs.base import EnvAdapter
|
|
from skillopt.envs.my_benchmark.dataloader import MyBenchmarkDataLoader
|
|
|
|
class MyBenchmarkAdapter(EnvAdapter):
|
|
def __init__(self, split_dir: str = "", data_path: str = "", **kwargs):
|
|
self.dataloader = MyBenchmarkDataLoader(split_dir=split_dir, data_path=data_path, **kwargs)
|
|
|
|
def setup(self, cfg: dict) -> None:
|
|
super().setup(cfg)
|
|
self.dataloader.setup(cfg)
|
|
|
|
def get_dataloader(self):
|
|
return self.dataloader
|
|
|
|
def build_train_env(self, batch_size: int, seed: int, **kwargs):
|
|
return self.dataloader.build_train_batch(batch_size=batch_size, seed=seed, **kwargs).payload
|
|
|
|
def build_eval_env(self, env_num: int, split: str, seed: int, **kwargs):
|
|
return self.dataloader.build_eval_batch(env_num=env_num, split=split, seed=seed, **kwargs).payload
|
|
|
|
def rollout(self, env_manager, skill_content: str, out_dir: str, **kwargs) -> list[dict]:
|
|
# env_manager is the payload returned by build_train_env/build_eval_env
|
|
# (commonly list[dict] task items).
|
|
# Run target model on each item and return list[dict].
|
|
# Required keys per row: "id", "hard" (0/1), "soft" (0.0-1.0)
|
|
raise NotImplementedError
|
|
|
|
def reflect(self, results: list[dict], skill_content: str, out_dir: str, **kwargs) -> list[dict | None]:
|
|
# Convert failure/success analysis into RawPatch-like dicts.
|
|
raise NotImplementedError
|
|
|
|
def get_task_types(self) -> list[str]:
|
|
return ["my_benchmark"]
|
|
```
|
|
|
|
## Step 4: Register the Benchmark
|
|
|
|
Add your adapter to `_register_builtins()` in `scripts/train.py`:
|
|
|
|
```python
|
|
from skillopt.envs.my_benchmark.adapter import MyBenchmarkAdapter
|
|
|
|
_ENV_REGISTRY["my_benchmark"] = MyBenchmarkAdapter
|
|
```
|
|
|
|
## Step 5: Create Config
|
|
|
|
Create `configs/my_benchmark/default.yaml`:
|
|
|
|
```yaml
|
|
_base_: ../_base_/default.yaml
|
|
|
|
env:
|
|
name: my_benchmark
|
|
data_path: data/my_benchmark
|
|
split_mode: ratio
|
|
split_ratio: "2:1:7"
|
|
|
|
train:
|
|
num_epochs: 4
|
|
batch_size: 40
|
|
|
|
optimizer:
|
|
learning_rate: 4
|
|
lr_scheduler: cosine
|
|
use_slow_update: true
|
|
use_meta_skill: true
|
|
|
|
gradient:
|
|
analyst_workers: 16
|
|
```
|
|
|
|
## Step 6: Run
|
|
|
|
```bash
|
|
python scripts/train.py --config configs/my_benchmark/default.yaml
|
|
```
|
|
|
|
## Tips
|
|
|
|
!!! tip
|
|
- Use a small `batch_size` (10-20) for initial testing
|
|
- Start from `skillopt/envs/_template/` and adapt from there
|
|
- Use an existing adapter (for example `skillopt/envs/officeqa/adapter.py`) as a concrete reference
|