Yif-Yang
138 lines
3.7 KiB
Markdown
138 lines
3.7 KiB
Markdown
# Your First Experiment
|
|
|
|
This guide walks through running a complete SkillOpt training on SearchQA.
|
|
|
|
## 1. Choose a Benchmark
|
|
|
|
SkillOpt includes ready-to-use configs for several benchmarks. End-to-end
|
|
runtime depends on the chosen models, provider latency, worker limits, and
|
|
dataset size, so the project does not promise fixed wall-clock estimates.
|
|
|
|
| Benchmark | Modality | Additional setup |
|
|
|---|---|---|
|
|
| SearchQA | Text QA | Materialize the released ID manifest |
|
|
| DocVQA | Document/image QA | Obtain and materialize images and examples |
|
|
| ALFWorld | Embodied agent | Install ALFWorld and download its assets |
|
|
|
|
We'll use **SearchQA** because it is the simplest text-only walkthrough.
|
|
|
|
## 2. Install and Materialize SearchQA
|
|
|
|
The repository contains a stable SearchQA ID manifest, not the full runnable
|
|
examples. From a source checkout, install the data extra and materialize the
|
|
split once:
|
|
|
|
```bash
|
|
python -m pip install -e ".[searchqa]"
|
|
python scripts/materialize_searchqa.py
|
|
```
|
|
|
|
By default, the materializer reads `data/searchqa_id_split/` and writes the
|
|
train/validation/test payloads expected by the config to
|
|
`data/searchqa_split/`; both paths have command-line overrides.
|
|
|
|
## 3. Configure
|
|
|
|
Configure and export one model backend as described in
|
|
[Installation](installation.md#environment-variables). For example:
|
|
|
|
```bash
|
|
cp .env.example .env
|
|
# Edit .env, choose one authentication mode, then export it:
|
|
set -a; source .env; set +a
|
|
```
|
|
|
|
Review the config file:
|
|
|
|
```bash
|
|
cat configs/searchqa/default.yaml
|
|
```
|
|
|
|
Key parameters (deep learning analogy in parentheses):
|
|
|
|
```yaml
|
|
train:
|
|
num_epochs: 4 # (epochs)
|
|
batch_size: 40 # (batch size)
|
|
|
|
optimizer:
|
|
learning_rate: 4 # (max edits per step)
|
|
lr_scheduler: cosine # (learning rate schedule)
|
|
use_slow_update: true # (momentum at epoch boundary)
|
|
use_meta_skill: true # (cross-epoch optimizer memory)
|
|
|
|
gradient:
|
|
analyst_workers: 16 # (parallel reflection workers)
|
|
|
|
evaluation:
|
|
use_gate: true # (validation gating)
|
|
```
|
|
|
|
## 4. Train
|
|
|
|
```bash
|
|
python scripts/train.py \
|
|
--config configs/searchqa/default.yaml \
|
|
--out_root outputs/searchqa_first_run
|
|
```
|
|
|
|
The command prints the resolved backend/data configuration, per-step rollout
|
|
and gate progress, and the generated output directory.
|
|
|
|
## 5. Monitor
|
|
|
|
The explicit `--out_root` above creates this run directory:
|
|
|
|
```
|
|
outputs/searchqa_first_run/
|
|
├── config.json
|
|
├── runtime_state.json
|
|
├── history.json
|
|
├── best_skill.md
|
|
├── skills/
|
|
│ └── skill_vXXXX.md
|
|
├── steps/
|
|
│ └── step_XXXX/
|
|
│ ├── candidate_skill.md
|
|
│ ├── step_record.json
|
|
│ └── trajectory_digest.json
|
|
├── slow_update/
|
|
│ └── epoch_XX/
|
|
└── meta_skill/
|
|
└── epoch_XX/
|
|
```
|
|
|
|
## 6. Evaluate
|
|
|
|
Evaluate the best skill on the test split:
|
|
|
|
```bash
|
|
python scripts/eval_only.py \
|
|
--config configs/searchqa/default.yaml \
|
|
--skill outputs/searchqa_first_run/best_skill.md \
|
|
--split valid_unseen
|
|
```
|
|
|
|
The `--skill` path above is the training artifact. Evaluation writes
|
|
`eval_summary.json` to its own timestamped `outputs/eval_.../` directory unless
|
|
you pass an explicit `--out_root`; it does not overwrite the training run.
|
|
|
|
## WebUI
|
|
|
|
Prefer a graphical interface? Launch the WebUI:
|
|
|
|
```bash
|
|
pip install -e ".[webui]"
|
|
python -m skillopt_webui.app
|
|
```
|
|
|
|
Then open `http://localhost:7860` in your browser to configure parameters and
|
|
launch training. The default host is `0.0.0.0`; pass `--host 127.0.0.1` for a
|
|
local-only dashboard.
|
|
|
|
## Next Steps
|
|
|
|
- [Understand the training loop](training-loop.md)
|
|
- [Configuration reference](../reference/config.md)
|
|
- [Add a new benchmark](new-benchmark.md)
|