# Paper-aligned SkillOpt reference skills (GPT-5.5)

This folder provides a subset of the paper's main Table 1 GPT-5.5 optimized
skills as reference artifacts — one `gpt5.5_skill.md` per currently included
benchmark. You can plug them into `scripts/eval_only.py` to evaluate the
provided skills on a given split without re-running the training loop.

> These are checkpoints associated with the paper, not a general-purpose
> tool. They're here so you can verify the reported numbers and use the
> skills as portable artifacts. If you want to *train* your own skill,
> use `scripts/train.py` per the top-level README.
>
> This is the first optimized-skill artifact batch. We plan to continue
> uploading remaining paper artifacts as they are cleaned and verified. All
> six lightweight ID/path split manifests are already checked in under
> `data/`; most still require materializing their upstream benchmark payload.

## What's here

| Benchmark | Skill artifact | Matching config |
|---|---|---|
| SearchQA | `ckpt/searchqa/gpt5.5_skill.md` | `configs/searchqa/default.yaml` |
| ALFWorld | `ckpt/alfworld/gpt5.5_skill.md` | `configs/alfworld/default.yaml` |
| DocVQA | `ckpt/docvqa/gpt5.5_skill.md` | `configs/docvqa/default.yaml` |
| LiveMathematicianBench | `ckpt/livemath/gpt5.5_skill.md` | `configs/livemathematicianbench/default.yaml` |
| OfficeQA | `ckpt/officeqa/gpt5.5_skill.md` | `configs/officeqa/default.yaml` |
| SpreadsheetBench | `ckpt/spreadsheetbench/gpt5.5_skill.md` | `configs/spreadsheetbench/default.yaml` |

Each file is a plain Markdown skill document (~2k–13k chars). It contains a
protected `SLOW_UPDATE` section at the end that holds epoch-wise
longitudinal guidance — that's expected, not a formatting issue.

## How to evaluate a provided skill

`scripts/eval_only.py` runs a single skill against a data split without
invoking the optimizer. Example for SearchQA against the test split:

```bash
# The checked-in SearchQA split is ID-only; materialize full examples first.
python -m pip install -e ".[searchqa]"
python scripts/materialize_searchqa.py

python scripts/eval_only.py \
  --config configs/searchqa/default.yaml \
  --skill ckpt/searchqa/gpt5.5_skill.md \
  --split valid_unseen \
  --split_dir data/searchqa_split \
  --azure_openai_endpoint https://your-resource.openai.azure.com/ \
  --azure_openai_auth_mode api_key \
  --target_model gpt-5.5
```

Substitute the benchmark, config, skill path, and `--split_dir` to evaluate
any of the other five. `--split valid_unseen` is the test split, `valid_seen`
is the selection / validation split, `train` is the training split, and
`all` runs all three.

## On comparing to the paper numbers

To compare against the paper-reported cells, use the same dataset split and
scorer. SearchQA's ID manifest is checked in at `data/searchqa_id_split/` (400
train / 200 selection / 1400 test); the materializer writes the runnable
payload to `data/searchqa_split/`. All six lightweight split manifests are
checked in under `data/`. ALFWorld's manifest records game-file paths; the
other ID manifests still require you to materialize the corresponding
upstream benchmark payload into the documented `split_dir`. See
[`data/README.md`](../data/README.md) for the exact status of each benchmark.
When using `split_mode: ratio` instead, the loader is deterministic from
`split_seed` (default `42`) + `split_ratio` (default `2:1:7`), so a given
`data_path` + seed reproduces across machines.

## Why force-accept vs. gated slow-update matters

These `ckpt/` skills were produced with the gated slow-update semantics
described in paper Section 3.6:

```yaml
optimizer:
  slow_update_gate_with_selection: true
```

Current `main` defaults to `false` (force-accept mode), a newer
post-submission behavior where the slow-update guidance is written into
`current_skill` and `best_skill` unconditionally at the epoch boundary. If
you re-train with the current default, you may produce a *different*
`best_skill.md` than the one checked in here. Both modes are supported; see
the [configuration reference](../docs/reference/config.md).
