Merge pull request #32 from Yif-Yang/fix/issue-30-docs-and-template

Fix/issue 30 docs and template
envs/_template: make template instantiable against real EnvAdapter ABC
2026-06-02 10:12:48 +08:00 · 2026-06-01 20:15:12 +00:00 · 2026-06-01 20:14:54 +00:00 · 2026-06-02 03:27:50 +08:00 · 2026-06-02 03:27:26 +08:00 · 2026-06-01 16:02:14 +00:00
54 changed files with 15857 additions and 1432 deletions
@@ -27,3 +27,8 @@ export AZURE_OPENAI_API_KEY=
 # ── Qwen Local Model (for qwen_chat backend) ────────────────────────
 # export QWEN_CHAT_BASE_URL=http://localhost:8000/v1
 # export QWEN_CHAT_MODEL=Qwen/Qwen3.5-4B
+
+# ── MiniMax (for minimax_chat backend) ──────────────────────────────
+# export MINIMAX_BASE_URL=https://api.minimax.io/v1
+# export MINIMAX_API_KEY=...
+# export MINIMAX_MODEL=MiniMax-M2.7
@@ -5,7 +5,20 @@ build/
 dist/
 site/

-data/
+data/*
+!data/README.md
+!data/searchqa_id_split/
+!data/searchqa_id_split/**
+!data/livemathematicianbench_id_split/
+!data/livemathematicianbench_id_split/**
+!data/docvqa_id_split/
+!data/docvqa_id_split/**
+!data/officeqa_id_split/
+!data/officeqa_id_split/**
+!data/spreadsheetbench_id_split/
+!data/spreadsheetbench_id_split/**
+!data/alfworld_path_split/
+!data/alfworld_path_split/**
 outputs/
 logs/
 external/
@@ -40,3 +53,4 @@ docs/reflact_overview.html
 docs/render_ablation_paper_tables.py
 docs/让*
 .gradio/
+.venv
@@ -4,7 +4,37 @@

 [![Project Page](https://img.shields.io/badge/Project%20Page-SkillOpt-8dbb3c)](https://microsoft.github.io/SkillOpt/) [![Paper](https://img.shields.io/badge/Paper-arXiv-b31b1b)](https://arxiv.org/abs/2605.23904) [![Project Video](https://img.shields.io/badge/Project%20Video-Watch%20Demo-ff0000)](https://youtu.be/JUBMDTCiM0M) [![Python 3.10+](https://img.shields.io/badge/Python-3.10%2B-blue.svg)](https://www.python.org/) [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)

-## 🎬 SkillOpt Demo Video
+---
+
+## Overview
+
+Modern agent skills are usually hand-crafted, generated one-shot by a strong
+LLM, or evolved through loosely controlled self-revision — none of which
+behaves like a deep-learning optimizer for the skill itself, and none of
+which reliably improves over its starting point under feedback.
+
+**SkillOpt treats the skill document as the trainable state of a frozen
+agent**, and trains it with the discipline that makes weight-space
+optimization reproducible. A separate optimizer model turns scored rollouts
+into bounded add / delete / replace edits on a single skill document; a
+candidate edit is accepted only when it strictly improves a held-out
+validation score. A textual learning-rate budget, a rejected-edit buffer,
+and an epoch-wise slow / meta update make skill training stable while
+adding **zero inference-time model calls** at deployment.
+
+The deployed artifact is a compact `best_skill.md` (typically 300–2,000
+tokens) that runs against the unchanged target model. Across **six
+benchmarks, seven target models, and three execution harnesses** (direct
+chat, Codex CLI, Claude Code CLI), SkillOpt is best or tied-best on **all
+52 evaluated (model, benchmark, harness) cells** and on GPT-5.5 lifts the
+average no-skill accuracy by **+23.5 points in direct chat, +24.8 inside
+the Codex agentic loop, and +19.1 inside Claude Code**. Optimized skill
+artifacts transfer across model scales, between Codex and Claude Code
+harnesses, and to nearby benchmarks without further optimization.
+
+For the full method, ablations, and per-cell results see the [paper](https://arxiv.org/abs/2605.23904); for a visual walkthrough of the loop see the [project page](https://microsoft.github.io/SkillOpt/); for deeper API / backend / benchmark docs see [`docs/`](docs/).
+
+## 🎬 Demo Video

 https://github.com/user-attachments/assets/eb12d3bc-371c-467f-904d-91b61f339ed7

@@ -14,39 +44,18 @@ https://github.com/user-attachments/assets/eb12d3bc-371c-467f-904d-91b61f339ed7

 ---

-## Documentation
-
-A complete, self-contained **Documentation & Reproduction Guide** lives at
-[`docs/guideline.html`](docs/guideline.html). It covers installation, data
-preparation, training/eval commands, the full configuration reference, the
-framework internals (training loop, validation gate, slow update, meta skill),
-and an API/function reference — all in a single page with a left navigation
-sidebar.
-
-Because GitHub shows raw source for `.html` files instead of rendering them,
-open the guide one of these ways:
-
- **Locally** — clone the repo and open `docs/guideline.html` in any browser
-  (no build step required).
- **Rendered online (no setup)** — via the htmlpreview proxy:
-  [`htmlpreview.github.io/?…/docs/guideline.html`](https://htmlpreview.github.io/?https://github.com/microsoft/SkillOpt/blob/main/docs/guideline.html)
- **GitHub Pages** — the repository's GitHub Pages site already serves the
-  project homepage from the repo root, so the guide is reachable alongside it at
-  `https://microsoft.github.io/SkillOpt/docs/guideline.html` (the homepage at
-  `https://microsoft.github.io/SkillOpt/` is unaffected).
-
---
-
 ## Install

-**Requirements:** Python 3.10+
+### Requirements
+
+- Python 3.10+

 ```bash
 git clone https://github.com/microsoft/SkillOpt.git
 cd SkillOpt
 pip install -e .

-# For ALFWorld benchmark (optional):
+# For the ALFWorld benchmark (optional):
 pip install -e ".[alfworld]"
 alfworld-download
 ```
@@ -59,7 +68,8 @@ cp .env.example .env
 source .env
 ```

-**Azure OpenAI** (recommended):
+#### Azure OpenAI *(recommended)*
+
 ```bash
 export AZURE_OPENAI_ENDPOINT="https://your-resource.openai.azure.com/"
 # Option 1: API key auth
@@ -68,74 +78,56 @@ export AZURE_OPENAI_API_KEY="your-key"
 export AZURE_OPENAI_AUTH_MODE="azure_cli"
 ```

-> **Note:** `AZURE_OPENAI_ENDPOINT` is required for all three modes (`api_key`, `azure_cli`,
-> `openai_compatible`). Without it, all LLM calls will fail.
+> **Note:** `AZURE_OPENAI_ENDPOINT` is required for all three modes (`api_key`, `azure_cli`, `openai_compatible`). Without it, all LLM calls will fail.
+
+#### OpenAI-compatible endpoints

-**OpenAI-compatible endpoints**:
 ```bash
 export AZURE_OPENAI_ENDPOINT="https://api.openai.com/v1"
 export AZURE_OPENAI_API_KEY="sk-..."
 export AZURE_OPENAI_AUTH_MODE="openai_compatible"
 ```

-This routes all calls through the plain OpenAI Python client (no Azure auth, no `api-version`
-header).
+This routes all calls through the plain OpenAI Python client (no Azure auth, no `api-version` header).

-> **Note:** SkillOpt reuses the `AZURE_OPENAI_*` env var names even in this mode — there is no
-> separate `OPENAI_API_KEY` knob.
+> **Note:** SkillOpt reuses the `AZURE_OPENAI_*` env var names even in this mode — there is no separate `OPENAI_API_KEY` knob.
+
+#### Anthropic Claude

-**Anthropic Claude**:
 ```bash
 export ANTHROPIC_API_KEY="sk-ant-..."
 ```

-**Qwen (local vLLM)**:
+#### Qwen *(local vLLM)*
+
 ```bash
 export QWEN_CHAT_BASE_URL="http://localhost:8000/v1"
 export QWEN_CHAT_MODEL="Qwen/Qwen3.5-4B"
 ```

---
+`qwen_chat` can also be used as the optimizer backend. When optimizer and
+target should point to different local vLLM services, use the role-specific
+settings:

-## Data Preparation
-
-SkillOpt expects data in a **split directory** with `train/`, `val/`, `test/` subdirectories, each containing a JSON file (e.g., `items.json`).
-
-```
-data/my_split/
-├── train/items.json
-├── val/items.json
-└── test/items.json
+```bash
+python scripts/train.py \
+    --config configs/searchqa/default.yaml \
+    --optimizer_backend qwen_chat \
+    --target_backend qwen_chat \
+    --optimizer_model Qwen/Qwen3.5-4B \
+    --target_model Qwen/Qwen3.5-4B \
+    --optimizer_qwen_chat_base_url http://localhost:8001/v1 \
+    --target_qwen_chat_base_url http://localhost:8000/v1
 ```

-Each JSON file is an array of task items. The required fields depend on the benchmark. For example, SearchQA items look like:
+#### MiniMax

-```json
-[
-  {
-    "id": "unique_item_id",
-    "question": "Who wrote the novel ...",
-    "context": "[DOC] relevant passage text ...",
-    "answers": ["expected answer"]
-  }
-]
+```bash
+export MINIMAX_BASE_URL="https://api.minimax.io/v1"
+export MINIMAX_API_KEY="..."
+export MINIMAX_MODEL="MiniMax-M2.7"
 ```

-See `skillopt/envs/<benchmark>/dataloader.py` for the exact format each benchmark expects.
-
-> **Note:** Benchmark datasets are not included in this repository. Prepare your own data following the format above.
-
-### Supported Benchmarks
-
-| Benchmark | Type | Config |
-|---|---|---|
-| SearchQA | QA | `configs/searchqa/default.yaml` |
-| ALFWorld | Embodied agent | `configs/alfworld/default.yaml` |
-| DocVQA | Document QA | `configs/docvqa/default.yaml` |
-| LiveMathematicianBench | Math | `configs/livemathematicianbench/default.yaml` |
-| SpreadsheetBench | Code generation | `configs/spreadsheetbench/default.yaml` |
-| OfficeQA | Tool-augmented QA | `configs/officeqa/default.yaml` |
-
 ---

 ## Quick Start
@@ -162,7 +154,7 @@ python scripts/train.py \
 # Train on ALFWorld:
 python scripts/train.py \
    --config configs/alfworld/default.yaml \
-    --split_dir /path/to/your/alfworld_split \
+    --split_dir data/alfworld_path_split \
    --azure_openai_endpoint https://your-resource.openai.azure.com/ \
    --optimizer_model gpt-5.5 \
    --target_model gpt-5.5
@@ -204,8 +196,7 @@ python scripts/eval_only.py \
  --azure_openai_endpoint https://your-resource.openai.azure.com/
 ```

-To evaluate a skill produced by a training run, replace `--skill` with that
-run's best-skill path, for example `outputs/my_run/best_skill.md`.
+To evaluate a skill produced by your own training run, replace `--skill` with that run's best-skill path, for example `outputs/my_run/best_skill.md`.

 | Split | Description |
 |---|---|
@@ -216,7 +207,7 @@ run's best-skill path, for example `outputs/my_run/best_skill.md`.

 ### Output Structure

-Each run writes to a structured output directory:
+Each training run writes to a structured output directory:

 ```
 outputs/<run_name>/
@@ -232,26 +223,147 @@ outputs/<run_name>/

 Re-running the same command auto-resumes from the last completed step.

+### Pretrained Skill Artifacts
+
+We provide a subset of the paper's main Table 1 GPT-5.5 optimized skills in
+[`ckpt/`](ckpt/) as reference artifacts. Use them with `scripts/eval_only.py`
+to evaluate the provided skills on a matching data split without re-running
+training. See [`ckpt/README.md`](ckpt/README.md) for the full per-benchmark
+command. This is the first artifact batch; we plan to continue uploading
+the remaining optimized skills and benchmark split manifests as they are
+cleaned and verified.
+
 ---

-## Community-contributed configs
+## Data Preparation

-These are **not** default SkillOpt settings — they are reference configs
+### Directory layout
+
+SkillOpt expects data in a **split directory** with `train/`, `val/`, `test/` subdirectories, each containing a JSON file (e.g., `items.json`):
+
+```
+data/my_split/
+├── train/items.json
+├── val/items.json
+└── test/items.json
+```
+
+Each JSON file is an array of task items. The required fields depend on the benchmark. For example, SearchQA items look like:
+
+```json
+[
+  {
+    "id": "unique_item_id",
+    "question": "Who wrote the novel ...",
+    "context": "[DOC] relevant passage text ...",
+    "answers": ["expected answer"]
+  }
+]
+```
+
+See `skillopt/envs/<benchmark>/dataloader.py` for the exact format each benchmark expects.
+
+> **Note:** Most benchmark datasets are not included in this repository. Prepare your own data following the format above. The exact SearchQA split used in the paper is provided at [`data/searchqa_id_split/`](data/searchqa_id_split) (400 train / 200 val / 1400 test). We are preparing the remaining benchmark split manifests for upload.
+
+### Supported Benchmarks
+
+| Benchmark | Type | Config |
+|---|---|---|
+| SearchQA | QA | `configs/searchqa/default.yaml` |
+| ALFWorld | Embodied agent | `configs/alfworld/default.yaml` |
+| DocVQA | Document QA | `configs/docvqa/default.yaml` |
+| LiveMathematicianBench | Math | `configs/livemathematicianbench/default.yaml` |
+| SpreadsheetBench | Code generation | `configs/spreadsheetbench/default.yaml` |
+| OfficeQA | Tool-augmented QA | `configs/officeqa/default.yaml` |
+
+---
+
+## Configuration
+
+### Default settings and paper-reproduction knobs
+
+`configs/_base_/default.yaml` is the single source of truth for SkillOpt's
+runtime knobs. Out of the box, every included benchmark config inherits
+from it and keeps the paper protocol visible: 4 epochs, rollout batch 40,
+reflection minibatch 8, textual learning rate 4 with cosine decay, strict
+hard validation gating, and slow-update + meta-skill enabled. One detail to
+watch is slow-update acceptance: the current `main` default is the newer
+post-submission force-accept mode, while the paper protocol and the
+paper-aligned skills under `ckpt/` use the gated semantics described in
+paper Section 3.6.
+
+### Slow-update acceptance mode
+
+The epoch-boundary slow / meta update can be applied two ways, controlled
+by `optimizer.slow_update_gate_with_selection`:
+
+```yaml
+optimizer:
+  slow_update_gate_with_selection: false   # current main default
+```
+
+- **`false`** *(current `main` default)*: force-accept. The
+  slow-update guidance is injected into both `current_skill` and
+  `best_skill` unconditionally at the epoch boundary. This is the newer
+  post-submission behavior on `main`.
+- **`true`** *(paper / ckpt-skill reproduction)*: gated, matching paper
+  Section 3.6 verbatim. The slow-update candidate is evaluated on the
+  selection split and accepted only if it passes the same validation gate
+  as a step-level edit. Use this setting when re-running optimization to
+  match the paper protocol and the provenance of the provided `ckpt/` skills.
+
+The trainer prints which mode is active at startup
+(`[slow update] acceptance=...`). See issue #22 for the discussion that
+led to the flag.
+
+### Gate metric (`hard` / `soft` / `mixed`)
+
+The validation gate compares candidate vs. current skills on the selection
+split using `gate_metric`:
+
+- **`hard`** *(default, paper)*: exact-match accuracy, strictly greater
+  than the current score is required.
+- **`soft`**: per-item soft / partial-credit score. Useful when the
+  selection split is small (e.g. ≤10 items) and the reward is continuous,
+  where the discrete hard gate often rejects every candidate.
+- **`mixed`**: weighted average, `(1 - w) * hard + w * soft`, with `w`
+  set by `gate_mixed_weight` (default `0.5`).
+
+Default is `hard`. Use the optional feature config below to switch.
+
+### Optional feature configs
+
+These are **not** default SkillOpt settings — they are optional feature configs
 contributed by users for specific scenarios. The paper-reported numbers
 were obtained with the default settings, not these.

- **`configs/examples/soft_gate.yaml`** *(PR #25, contributed by
-  [@lvbaocheng](https://github.com/lvbaocheng))* — switches the
-  validation gate from exact-match (`hard`) to soft / partial-credit
-  (`soft` or `mixed`). Useful when the held-out **selection split is
-  small** (e.g. ≤ ~10 items) and the **reward is continuous**, where the
-  discrete hard gate often rejects every candidate and training stalls.
-  See the comment at the top of the file for details and when not to use
-  it.
+- **[`configs/features/soft_gate.yaml`](configs/features/soft_gate.yaml)**
+  *(PR #25, contributed by [@lvbaocheng](https://github.com/lvbaocheng))* —
+  switches `gate_metric` to `soft` (or `mixed`). See the comment at the
+  top of the file for when to use and when not to.

 ---

-## WebUI
+## Extensibility & WebUI
+
+### Adding a new backend
+
+A backend = a chat / exec target (e.g. `openai_chat`, `claude_chat`,
+`qwen_chat`, `minimax_chat`, `codex_exec`, `claude_code_exec`). See
+[`docs/guide/new-backend.md`](docs/guide/new-backend.md) for the full
+contract; in short you add a `skillopt/model/<name>_backend.py` module,
+register it in `skillopt/model/common.py` + `backend_config.py`, and wire
+it through the router in `skillopt/model/__init__.py`. `qwen_backend.py`
+and `minimax_backend.py` are good templates.
+
+### Adding a new benchmark
+
+A benchmark = a `skillopt/envs/<name>/` package with a `dataloader.py`, a
+`rollout.py`, and an `initial.md` seed skill. See
+[`docs/guide/new-benchmark.md`](docs/guide/new-benchmark.md) for the full
+contract; the simplest reference is `skillopt/envs/searchqa/`.
+
+### WebUI

 Launch the monitoring dashboard (optional):

@@ -266,11 +378,6 @@ python -m skillopt_webui.app
 | `--host` | `0.0.0.0` | Bind address |
 | `--share` | off | Create a public Gradio share link |

-```bash
-# With public share link (useful for remote servers)
-python -m skillopt_webui.app --share
-```
-
 ---

 ## Citation
@@ -0,0 +1,79 @@
+# Paper-aligned SkillOpt reference skills (GPT-5.5)
+
+This folder provides a subset of the paper's main Table 1 GPT-5.5 optimized
+skills as reference artifacts — one `gpt5.5_skill.md` per currently included
+benchmark. You can plug them into `scripts/eval_only.py` to evaluate the
+provided skills on a given split without re-running the training loop.
+
+> These are checkpoints associated with the paper, not a general-purpose
+> tool. They're here so you can verify the reported numbers and use the
+> skills as portable artifacts. If you want to *train* your own skill,
+> use `scripts/train.py` per the top-level README.
+>
+> This is the first artifact batch. We plan to continue uploading the
+> remaining optimized skills and benchmark split manifests as they are
+> cleaned and verified.
+
+## What's here
+
+| Benchmark | Skill artifact | Matching config |
+|---|---|---|
+| SearchQA | `ckpt/searchqa/gpt5.5_skill.md` | `configs/searchqa/default.yaml` |
+| ALFWorld | `ckpt/alfworld/gpt5.5_skill.md` | `configs/alfworld/default.yaml` |
+| DocVQA | `ckpt/docvqa/gpt5.5_skill.md` | `configs/docvqa/default.yaml` |
+| LiveMathematicianBench | `ckpt/livemath/gpt5.5_skill.md` | `configs/livemathematicianbench/default.yaml` |
+| OfficeQA | `ckpt/officeqa/gpt5.5_skill.md` | `configs/officeqa/default.yaml` |
+| SpreadsheetBench | `ckpt/spreadsheetbench/gpt5.5_skill.md` | `configs/spreadsheetbench/default.yaml` |
+
+Each file is a plain Markdown skill document (~2k–13k chars). It contains a
+protected `SLOW_UPDATE` section at the end that holds epoch-wise
+longitudinal guidance — that's expected, not a formatting issue.
+
+## How to evaluate a provided skill
+
+`scripts/eval_only.py` runs a single skill against a data split without
+invoking the optimizer. Example for SearchQA against the test split:
+
+```bash
+python scripts/eval_only.py \
+  --config configs/searchqa/default.yaml \
+  --skill ckpt/searchqa/gpt5.5_skill.md \
+  --split valid_unseen \
+  --split_dir data/searchqa_id_split \
+  --azure_openai_endpoint https://your-resource.openai.azure.com/ \
+  --target_model gpt-5.5
+```
+
+Substitute the benchmark, config, skill path, and `--split_dir` to evaluate
+any of the other five. `--split valid_unseen` is the test split, `valid_seen`
+is the selection / validation split, `train` is the training split, and
+`all` runs all three.
+
+## On comparing to the paper numbers
+
+To compare against the paper-reported cells, use the same dataset split and
+scorer. SearchQA's split is checked in at `data/searchqa_id_split/` (400
+train / 200 selection / 1400 test). For the other benchmarks, point
+`--split_dir` at your own materialized split; the loader is deterministic
+from `split_seed` (default `42`) + `split_ratio` (default `2:1:7`) when
+`split_mode: ratio` is used, so a given `data_path` + seed reproduces
+across machines. Explicit per-benchmark split manifests are being prepared
+for upload — see issues #14 and #21.
+
+## Why force-accept vs. gated slow-update matters
+
+These `ckpt/` skills were produced with the gated slow-update semantics
+described in paper Section 3.6:
+
+```yaml
+optimizer:
+  slow_update_gate_with_selection: true
+```
+
+Current `main` defaults to `false` (force-accept mode), a newer
+post-submission behavior where the slow-update guidance is written into
+`current_skill` and `best_skill` unconditionally at the epoch boundary. If
+you re-train with the current default, you may produce a *different*
+`best_skill.md` than the one checked in here. Both modes are supported;
+see the top-level README's "Configuration -> Slow-update acceptance mode"
+section.
@@ -44,6 +44,18 @@ model:
  target_azure_openai_ad_scope: "https://cognitiveservices.azure.com/.default"
  target_azure_openai_managed_identity_client_id: ""

+  # MiniMax backend settings (minimax_chat target)
+  minimax_base_url: ""          # https://api.minimax.io/v1 if blank
+  minimax_api_key: ""
+  minimax_model: "MiniMax-M2.7"
+  minimax_temperature: "0.7"
+  minimax_max_tokens: "8000"
+  minimax_enable_thinking: "false"
+  optimizer_minimax_base_url: ""    # per-role override
+  target_minimax_base_url: ""       # per-role override
+  optimizer_minimax_api_key: ""
+  target_minimax_api_key: ""
+
 train:
  num_epochs: 4
  train_size: 0          # 0 = derive from dataset split when available
@@ -19,7 +19,7 @@ env:
  name: alfworld
  skill_init: skillopt/envs/alfworld/skills/initial.md
  split_mode: split_dir
-  split_dir: data/ablation_splits/alfworld/2-1-7_seed42
+  split_dir: data/alfworld_path_split
  data_path: ""
  split_output_dir: ""
  max_steps: 50
@@ -1,5 +1,5 @@
 # ─────────────────────────────────────────────────────────────────────────────
-# Example: soft / mixed validation-gate metric (community-contributed, PR #25)
+# Feature: soft / mixed validation-gate metric (community-contributed, PR #25)
 # ─────────────────────────────────────────────────────────────────────────────
 #
 # This is NOT a default SkillOpt setting and was NOT used to produce the
@@ -28,7 +28,7 @@
 #     and matches the design described in the paper.
 #
 # To use: inherit your env config from this file, e.g.
-#   _base_: ../examples/soft_gate.yaml
+#   _base_: ../features/soft_gate.yaml
 # or copy the `evaluation:` block below into your config.
 # ─────────────────────────────────────────────────────────────────────────────

@@ -9,7 +9,7 @@ env:
  name: livemathematicianbench
  skill_init: skillopt/envs/livemathematicianbench/skills/initial.md
  split_mode: split_dir
-  split_dir: data/ablation_splits/livemathematicianbench/2-1-7_seed42
+  split_dir: data/livemathematicianbench_split
  data_path: ""
  split_output_dir: ""
  max_turns: 1
@@ -0,0 +1,223 @@
+# Data Manifests
+
+This directory releases lightweight split manifests for the SkillOpt paper
+splits. These manifests are not full runnable benchmark payloads. To evaluate a
+benchmark, first materialize the full examples from the raw data source when
+needed, then point `--split_dir` at the split directory listed below.
+
+In this README, "coverage" describes which part of the upstream benchmark the
+manifest references. It does not mean the released manifest directory contains
+the full runnable examples.
+
+## Layout
+
+Every released manifest directory uses the same file layout:
+
+```text
+data/<benchmark>_<manifest_type>/
+|-- split_manifest.json
+|-- train/items.json
+|-- val/items.json
+`-- test/items.json
+```
+
+`split_manifest.json` records source metadata, split counts, and item fields.
+Each `items.json` contains only stable IDs or source-path hints.
+
+## Released Splits
+
+| Manifest directory | Benchmark | Counts | Coverage | Raw data source | `split_dir` |
+|---|---|---:|---|---|---|
+| `searchqa_id_split/` | SearchQA | 400 / 200 / 1400 | Official HF dataset IDs | [lucadiliello/searchqa](https://huggingface.co/datasets/lucadiliello/searchqa) | `data/searchqa_split` |
+| `livemathematicianbench_id_split/` | LiveMathematicianBench | 35 / 18 / 124 | Four official monthly files | [LiveMathematicianBench/LiveMathematicianBench](https://huggingface.co/datasets/LiveMathematicianBench/LiveMathematicianBench) | `data/livemathematicianbench_split` |
+| `docvqa_id_split/` | DocVQA | 107 / 53 / 374 | 10% subset of validation | [lmms-lab/DocVQA](https://huggingface.co/datasets/lmms-lab/DocVQA) | `data/docvqa/splits` |
+| `officeqa_id_split/` | OfficeQA | 50 / 24 / 172 | OfficeQA Full | [databricks/officeqa](https://huggingface.co/datasets/databricks/officeqa) | `data/officeqa_split` |
+| `spreadsheetbench_id_split/` | SpreadsheetBench | 80 / 40 / 280 | SpreadsheetBench Verified 400 | [KAKA22/SpreadsheetBench](https://huggingface.co/datasets/KAKA22/SpreadsheetBench) | `data/spreadsheetbench_split` |
+| `alfworld_path_split/` | ALFWorld | 39 / 18 / 134 | ALFWorld `json_2.1.1` paths | [alfworld/alfworld](https://github.com/alfworld/alfworld) | `data/alfworld_path_split` |
+
+Counts are ordered as train / val / test.
+
+## Direct Use
+
+Only `alfworld_path_split/` can be used directly as `--split_dir` from this
+release, because the ALFWorld loader reads `gamefile` and `task_type` from the
+split items.
+
+This does not mean the ALFWorld raw data is included. You still need to
+download ALFWorld separately with `alfworld-download` and set `$ALFWORLD_DATA`
+to the data root containing `json_2.1.1`.
+
+The other manifest directories are lookup manifests. They intentionally omit
+full example fields such as questions, answers, contexts, images, or task
+instructions. Materialize those benchmarks into the `split_dir` paths listed
+above before running SkillOpt.
+
+## Lookup Keys
+
+The manifests are sufficient to locate the corresponding raw examples after
+the raw data has been downloaded or otherwise made available:
+
+| Benchmark | Manifest lookup key |
+|---|---|
+| SearchQA | Match `items.json[].id` to the `key` field in `lucadiliello/searchqa`. |
+| LiveMathematicianBench | Open `source_file`, then match `no`; the manifest `id` is `<month>:<no>`. |
+| DocVQA | Match `questionId` within the official DocVQA `validation` split; `image_path` records the expected local image path. |
+| OfficeQA | Match `uid` in `officeqa_full.csv`; `source_files` and `source_docs` identify the supporting document. |
+| SpreadsheetBench | Match `id`; `spreadsheet_path` identifies the referenced spreadsheet directory. |
+| ALFWorld | Resolve `gamefile` relative to `$ALFWORLD_DATA`. |
+
+## Manifest Item Examples
+
+SearchQA:
+
+```json
+{
+  "id": "221c83e6630f4e7983da48fa28da1882"
+}
+```
+
+LiveMathematicianBench:
+
+```json
+{
+  "id": "202602:22",
+  "month": "202602",
+  "no": 22,
+  "paper_link": "http://arxiv.org/abs/2602.10700v1",
+  "source_file": "data/202602/qa_202602_final.json"
+}
+```
+
+DocVQA:
+
+```json
+{
+  "id": "50877",
+  "questionId": "50877",
+  "docId": "14724",
+  "image_path": "data/docvqa_images/q50877_d14724.png",
+  "source_split": "validation"
+}
+```
+
+OfficeQA:
+
+```json
+{
+  "id": "UID0002",
+  "uid": "UID0002",
+  "category": "easy",
+  "source_files": "treasury_bulletin_1944_01.txt"
+}
+```
+
+SpreadsheetBench:
+
+```json
+{
+  "id": "32438",
+  "spreadsheet_path": "spreadsheet/32438",
+  "instruction_type": "Cell-Level Manipulation"
+}
+```
+
+ALFWorld:
+
+```json
+{
+  "id": "train:0000",
+  "gamefile": "json_2.1.1/train/.../game.tw-pddl",
+  "task_type": "look_at_obj_in_light"
+}
+```
+
+## Benchmark Notes
+
+### SearchQA
+
+`searchqa_id_split/` is an ID-only manifest. Each released `id` exactly matches
+the `key` field in `lucadiliello/searchqa`.
+
+Materialized examples must include the fields consumed by the SearchQA
+environment, including:
+
+```text
+question
+context
+answers
+```
+
+### LiveMathematicianBench
+
+`livemathematicianbench_id_split/` was generated from these raw files:
+
+```text
+data/202511/qa_202511_final.json
+data/202512/qa_202512_final.json
+data/202601/qa_202601_final.json
+data/202602/qa_202602_final.json
+```
+
+The manifest stores IDs in the loader format:
+
+```text
+<month>:<no>
+```
+
+Materialized examples must include:
+
+```text
+question
+choices
+correct_choice
+theorem_type
+theorem
+sketch
+paper_link
+```
+
+### DocVQA
+
+`docvqa_id_split/` records `docvqa_validation_10pct`: a 10% subset sampled from
+the official DocVQA `validation` split.
+
+```text
+source_split: validation
+docvqa_validation_10pct: train=107, val=53, test=374
+```
+
+Each manifest item contains question/document IDs plus image location metadata.
+Materialized examples must provide `question`, `answer` or `ground_truth`, and
+an `image_path` that resolves locally.
+
+### OfficeQA
+
+`officeqa_id_split/` records the split over OfficeQA Full
+(`officeqa_full.csv`). The official OfficeQA CSVs are gated on Hugging Face, so
+materialization requires authorized access.
+
+Each manifest item contains `uid`, `category`, `source_files`, and
+`source_docs` hints. Materialized examples must include `question` and
+`ground_truth` or `answer`.
+
+### SpreadsheetBench
+
+`spreadsheetbench_id_split/` records the split over SpreadsheetBench Verified
+400, from `spreadsheetbench_verified_400.tar.gz`.
+
+Each manifest item contains task identity metadata such as `id`,
+`spreadsheet_path`, and `instruction_type`. Materialization must also place the
+referenced spreadsheet directories at:
+
+```text
+data/spreadsheetbench_verified_400
+```
+
+### ALFWorld
+
+`alfworld_path_split/` records `gamefile` paths relative to `$ALFWORLD_DATA`.
+The source payload is `json_2.1.1`, which must be downloaded separately with
+`alfworld-download`.
+
+This manifest can be used directly as `--split_dir` after `$ALFWORLD_DATA`
+points to the local ALFWorld data root containing `json_2.1.1`.
@@ -0,0 +1,29 @@
+{
+  "benchmark": "ALFWorld",
+  "manifest_type": "path_split",
+  "source_repo": "alfworld/alfworld",
+  "source_repo_type": "repository",
+  "source_url": "https://github.com/alfworld/alfworld",
+  "source_file": "json_2.1.1",
+  "source_method": "generated by alfworld-download",
+  "source_split_files": [
+    "split_train.json",
+    "split_val.json",
+    "split_test.json"
+  ],
+  "counts": {
+    "train": 39,
+    "val": 18,
+    "test": 134
+  },
+  "item_fields": [
+    "id",
+    "gamefile",
+    "task_type"
+  ],
+  "path_root": "$ALFWORLD_DATA",
+  "notes": [
+    "This is a path manifest, not the ALFWorld game payload.",
+    "The gamefile field is relative to ALFWORLD_DATA and must be expanded before direct use as split_dir data."
+  ]
+}
@@ -0,0 +1,672 @@
+[
+  {
+    "id": "test:0000",
+    "gamefile": "json_2.1.1/valid_unseen/look_at_obj_in_light-AlarmClock-None-DeskLamp-308/trial_T20190908_222917_366542/game.tw-pddl",
+    "task_type": "look_at_obj_in_light"
+  },
+  {
+    "id": "test:0001",
+    "gamefile": "json_2.1.1/valid_unseen/look_at_obj_in_light-AlarmClock-None-DeskLamp-308/trial_T20190908_222933_607649/game.tw-pddl",
+    "task_type": "look_at_obj_in_light"
+  },
+  {
+    "id": "test:0002",
+    "gamefile": "json_2.1.1/valid_unseen/look_at_obj_in_light-AlarmClock-None-DeskLamp-308/trial_T20190908_222951_616606/game.tw-pddl",
+    "task_type": "look_at_obj_in_light"
+  },
+  {
+    "id": "test:0003",
+    "gamefile": "json_2.1.1/valid_unseen/look_at_obj_in_light-Book-None-DeskLamp-308/trial_T20190908_020029_636862/game.tw-pddl",
+    "task_type": "look_at_obj_in_light"
+  },
+  {
+    "id": "test:0004",
+    "gamefile": "json_2.1.1/valid_unseen/look_at_obj_in_light-Book-None-DeskLamp-308/trial_T20190908_020048_814402/game.tw-pddl",
+    "task_type": "look_at_obj_in_light"
+  },
+  {
+    "id": "test:0005",
+    "gamefile": "json_2.1.1/valid_unseen/look_at_obj_in_light-Book-None-DeskLamp-308/trial_T20190908_144951_587345/game.tw-pddl",
+    "task_type": "look_at_obj_in_light"
+  },
+  {
+    "id": "test:0006",
+    "gamefile": "json_2.1.1/valid_unseen/look_at_obj_in_light-Bowl-None-DeskLamp-308/trial_T20190907_133919_856963/game.tw-pddl",
+    "task_type": "look_at_obj_in_light"
+  },
+  {
+    "id": "test:0007",
+    "gamefile": "json_2.1.1/valid_unseen/look_at_obj_in_light-Bowl-None-DeskLamp-308/trial_T20190907_133935_066606/game.tw-pddl",
+    "task_type": "look_at_obj_in_light"
+  },
+  {
+    "id": "test:0008",
+    "gamefile": "json_2.1.1/valid_unseen/look_at_obj_in_light-Bowl-None-DeskLamp-308/trial_T20190907_133953_562557/game.tw-pddl",
+    "task_type": "look_at_obj_in_light"
+  },
+  {
+    "id": "test:0009",
+    "gamefile": "json_2.1.1/valid_unseen/look_at_obj_in_light-CD-None-DeskLamp-308/trial_T20190908_141942_810052/game.tw-pddl",
+    "task_type": "look_at_obj_in_light"
+  },
+  {
+    "id": "test:0010",
+    "gamefile": "json_2.1.1/valid_unseen/look_at_obj_in_light-CD-None-DeskLamp-308/trial_T20190908_141958_463362/game.tw-pddl",
+    "task_type": "look_at_obj_in_light"
+  },
+  {
+    "id": "test:0011",
+    "gamefile": "json_2.1.1/valid_unseen/look_at_obj_in_light-CD-None-DeskLamp-308/trial_T20190908_142046_281296/game.tw-pddl",
+    "task_type": "look_at_obj_in_light"
+  },
+  {
+    "id": "test:0012",
+    "gamefile": "json_2.1.1/valid_unseen/look_at_obj_in_light-Mug-None-DeskLamp-308/trial_T20190908_161733_213242/game.tw-pddl",
+    "task_type": "look_at_obj_in_light"
+  },
+  {
+    "id": "test:0013",
+    "gamefile": "json_2.1.1/valid_unseen/look_at_obj_in_light-Mug-None-DeskLamp-308/trial_T20190908_201421_021646/game.tw-pddl",
+    "task_type": "look_at_obj_in_light"
+  },
+  {
+    "id": "test:0014",
+    "gamefile": "json_2.1.1/valid_unseen/look_at_obj_in_light-Mug-None-DeskLamp-308/trial_T20190908_201444_037645/game.tw-pddl",
+    "task_type": "look_at_obj_in_light"
+  },
+  {
+    "id": "test:0015",
+    "gamefile": "json_2.1.1/valid_unseen/look_at_obj_in_light-Pencil-None-DeskLamp-308/trial_T20190908_220545_153480/game.tw-pddl",
+    "task_type": "look_at_obj_in_light"
+  },
+  {
+    "id": "test:0016",
+    "gamefile": "json_2.1.1/valid_unseen/look_at_obj_in_light-Pencil-None-DeskLamp-308/trial_T20190908_220604_010430/game.tw-pddl",
+    "task_type": "look_at_obj_in_light"
+  },
+  {
+    "id": "test:0017",
+    "gamefile": "json_2.1.1/valid_unseen/look_at_obj_in_light-Pencil-None-DeskLamp-308/trial_T20190908_220656_510400/game.tw-pddl",
+    "task_type": "look_at_obj_in_light"
+  },
+  {
+    "id": "test:0018",
+    "gamefile": "json_2.1.1/valid_unseen/pick_and_place_simple-Mug-None-Desk-308/trial_T20190908_125200_737896/game.tw-pddl",
+    "task_type": "pick_and_place_simple"
+  },
+  {
+    "id": "test:0019",
+    "gamefile": "json_2.1.1/valid_unseen/pick_and_place_simple-Mug-None-Desk-308/trial_T20190909_203041_433487/game.tw-pddl",
+    "task_type": "pick_and_place_simple"
+  },
+  {
+    "id": "test:0020",
+    "gamefile": "json_2.1.1/valid_unseen/pick_and_place_simple-Mug-None-Desk-308/trial_T20190909_210238_431966/game.tw-pddl",
+    "task_type": "pick_and_place_simple"
+  },
+  {
+    "id": "test:0021",
+    "gamefile": "json_2.1.1/valid_unseen/pick_and_place_simple-Pencil-None-Shelf-308/trial_T20190908_121952_610012/game.tw-pddl",
+    "task_type": "pick_and_place_simple"
+  },
+  {
+    "id": "test:0022",
+    "gamefile": "json_2.1.1/valid_unseen/pick_and_place_simple-Pencil-None-Shelf-308/trial_T20190908_122024_052056/game.tw-pddl",
+    "task_type": "pick_and_place_simple"
+  },
+  {
+    "id": "test:0023",
+    "gamefile": "json_2.1.1/valid_unseen/pick_and_place_simple-Pencil-None-Shelf-308/trial_T20190908_122154_042763/game.tw-pddl",
+    "task_type": "pick_and_place_simple"
+  },
+  {
+    "id": "test:0024",
+    "gamefile": "json_2.1.1/valid_unseen/pick_and_place_simple-PepperShaker-None-Drawer-10/trial_T20190906_184021_215264/game.tw-pddl",
+    "task_type": "pick_and_place_simple"
+  },
+  {
+    "id": "test:0025",
+    "gamefile": "json_2.1.1/valid_unseen/pick_and_place_simple-PepperShaker-None-Drawer-10/trial_T20190918_154326_823501/game.tw-pddl",
+    "task_type": "pick_and_place_simple"
+  },
+  {
+    "id": "test:0026",
+    "gamefile": "json_2.1.1/valid_unseen/pick_and_place_simple-PepperShaker-None-Drawer-10/trial_T20190918_154424_844749/game.tw-pddl",
+    "task_type": "pick_and_place_simple"
+  },
+  {
+    "id": "test:0027",
+    "gamefile": "json_2.1.1/valid_unseen/pick_and_place_simple-SaltShaker-None-Cabinet-10/trial_T20190906_191429_743650/game.tw-pddl",
+    "task_type": "pick_and_place_simple"
+  },
+  {
+    "id": "test:0028",
+    "gamefile": "json_2.1.1/valid_unseen/pick_and_place_simple-SaltShaker-None-Cabinet-10/trial_T20190906_191445_723170/game.tw-pddl",
+    "task_type": "pick_and_place_simple"
+  },
+  {
+    "id": "test:0029",
+    "gamefile": "json_2.1.1/valid_unseen/pick_and_place_simple-SaltShaker-None-Cabinet-10/trial_T20190906_191501_563086/game.tw-pddl",
+    "task_type": "pick_and_place_simple"
+  },
+  {
+    "id": "test:0030",
+    "gamefile": "json_2.1.1/valid_unseen/pick_and_place_simple-SaltShaker-None-Drawer-10/trial_T20190909_021613_077537/game.tw-pddl",
+    "task_type": "pick_and_place_simple"
+  },
+  {
+    "id": "test:0031",
+    "gamefile": "json_2.1.1/valid_unseen/pick_and_place_simple-SaltShaker-None-Drawer-10/trial_T20190909_021650_880235/game.tw-pddl",
+    "task_type": "pick_and_place_simple"
+  },
+  {
+    "id": "test:0032",
+    "gamefile": "json_2.1.1/valid_unseen/pick_and_place_simple-SaltShaker-None-Drawer-10/trial_T20190909_021728_339782/game.tw-pddl",
+    "task_type": "pick_and_place_simple"
+  },
+  {
+    "id": "test:0033",
+    "gamefile": "json_2.1.1/valid_unseen/pick_and_place_simple-SoapBottle-None-Toilet-424/trial_T20190907_004321_405868/game.tw-pddl",
+    "task_type": "pick_and_place_simple"
+  },
+  {
+    "id": "test:0034",
+    "gamefile": "json_2.1.1/valid_unseen/pick_and_place_simple-SoapBottle-None-Toilet-424/trial_T20190907_004351_281384/game.tw-pddl",
+    "task_type": "pick_and_place_simple"
+  },
+  {
+    "id": "test:0035",
+    "gamefile": "json_2.1.1/valid_unseen/pick_and_place_simple-SoapBottle-None-Toilet-424/trial_T20190907_004404_604165/game.tw-pddl",
+    "task_type": "pick_and_place_simple"
+  },
+  {
+    "id": "test:0036",
+    "gamefile": "json_2.1.1/valid_unseen/pick_and_place_simple-Vase-None-Safe-219/trial_T20190908_205204_244321/game.tw-pddl",
+    "task_type": "pick_and_place_simple"
+  },
+  {
+    "id": "test:0037",
+    "gamefile": "json_2.1.1/valid_unseen/pick_and_place_simple-Vase-None-Safe-219/trial_T20190908_205221_748352/game.tw-pddl",
+    "task_type": "pick_and_place_simple"
+  },
+  {
+    "id": "test:0038",
+    "gamefile": "json_2.1.1/valid_unseen/pick_and_place_simple-Vase-None-Safe-219/trial_T20190908_205246_776817/game.tw-pddl",
+    "task_type": "pick_and_place_simple"
+  },
+  {
+    "id": "test:0039",
+    "gamefile": "json_2.1.1/valid_unseen/pick_and_place_simple-Watch-None-Safe-219/trial_T20190907_074524_006355/game.tw-pddl",
+    "task_type": "pick_and_place_simple"
+  },
+  {
+    "id": "test:0040",
+    "gamefile": "json_2.1.1/valid_unseen/pick_and_place_simple-Watch-None-Safe-219/trial_T20190907_074556_124850/game.tw-pddl",
+    "task_type": "pick_and_place_simple"
+  },
+  {
+    "id": "test:0041",
+    "gamefile": "json_2.1.1/valid_unseen/pick_and_place_simple-Watch-None-Safe-219/trial_T20190907_074643_810052/game.tw-pddl",
+    "task_type": "pick_and_place_simple"
+  },
+  {
+    "id": "test:0042",
+    "gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-Bowl-None-Cabinet-10/trial_T20190909_061130_844814/game.tw-pddl",
+    "task_type": "pick_clean_then_place_in_recep"
+  },
+  {
+    "id": "test:0043",
+    "gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-Bowl-None-Cabinet-10/trial_T20190909_061158_110530/game.tw-pddl",
+    "task_type": "pick_clean_then_place_in_recep"
+  },
+  {
+    "id": "test:0044",
+    "gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-Bowl-None-Cabinet-10/trial_T20190909_061232_368489/game.tw-pddl",
+    "task_type": "pick_clean_then_place_in_recep"
+  },
+  {
+    "id": "test:0045",
+    "gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-Cloth-None-Cabinet-424/trial_T20190908_022321_380927/game.tw-pddl",
+    "task_type": "pick_clean_then_place_in_recep"
+  },
+  {
+    "id": "test:0046",
+    "gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-Cloth-None-Cabinet-424/trial_T20190908_022436_073995/game.tw-pddl",
+    "task_type": "pick_clean_then_place_in_recep"
+  },
+  {
+    "id": "test:0047",
+    "gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-Cloth-None-CounterTop-424/trial_T20190908_100632_546757/game.tw-pddl",
+    "task_type": "pick_clean_then_place_in_recep"
+  },
+  {
+    "id": "test:0048",
+    "gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-Cloth-None-CounterTop-424/trial_T20190908_114340_674467/game.tw-pddl",
+    "task_type": "pick_clean_then_place_in_recep"
+  },
+  {
+    "id": "test:0049",
+    "gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-Egg-None-Microwave-10/trial_T20190909_120554_888709/game.tw-pddl",
+    "task_type": "pick_clean_then_place_in_recep"
+  },
+  {
+    "id": "test:0050",
+    "gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-Egg-None-Microwave-10/trial_T20190909_120632_691361/game.tw-pddl",
+    "task_type": "pick_clean_then_place_in_recep"
+  },
+  {
+    "id": "test:0051",
+    "gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-Egg-None-Microwave-10/trial_T20190909_120712_273910/game.tw-pddl",
+    "task_type": "pick_clean_then_place_in_recep"
+  },
+  {
+    "id": "test:0052",
+    "gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-Knife-None-CounterTop-10/trial_T20190909_110347_624008/game.tw-pddl",
+    "task_type": "pick_clean_then_place_in_recep"
+  },
+  {
+    "id": "test:0053",
+    "gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-Knife-None-CounterTop-10/trial_T20190909_110445_675754/game.tw-pddl",
+    "task_type": "pick_clean_then_place_in_recep"
+  },
+  {
+    "id": "test:0054",
+    "gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-Knife-None-CounterTop-10/trial_T20190909_110531_148235/game.tw-pddl",
+    "task_type": "pick_clean_then_place_in_recep"
+  },
+  {
+    "id": "test:0055",
+    "gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-Mug-None-CoffeeMachine-10/trial_T20190907_221208_560499/game.tw-pddl",
+    "task_type": "pick_clean_then_place_in_recep"
+  },
+  {
+    "id": "test:0056",
+    "gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-Mug-None-CoffeeMachine-10/trial_T20190907_221300_362511/game.tw-pddl",
+    "task_type": "pick_clean_then_place_in_recep"
+  },
+  {
+    "id": "test:0057",
+    "gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-Mug-None-CoffeeMachine-10/trial_T20190907_221355_558505/game.tw-pddl",
+    "task_type": "pick_clean_then_place_in_recep"
+  },
+  {
+    "id": "test:0058",
+    "gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-Pan-None-CounterTop-10/trial_T20190908_032434_013084/game.tw-pddl",
+    "task_type": "pick_clean_then_place_in_recep"
+  },
+  {
+    "id": "test:0059",
+    "gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-Pan-None-CounterTop-10/trial_T20190908_032518_891433/game.tw-pddl",
+    "task_type": "pick_clean_then_place_in_recep"
+  },
+  {
+    "id": "test:0060",
+    "gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-Pan-None-CounterTop-10/trial_T20190908_032543_712058/game.tw-pddl",
+    "task_type": "pick_clean_then_place_in_recep"
+  },
+  {
+    "id": "test:0061",
+    "gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-Plate-None-CounterTop-10/trial_T20190908_213356_017769/game.tw-pddl",
+    "task_type": "pick_clean_then_place_in_recep"
+  },
+  {
+    "id": "test:0062",
+    "gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-Plate-None-CounterTop-10/trial_T20190908_213420_728917/game.tw-pddl",
+    "task_type": "pick_clean_then_place_in_recep"
+  },
+  {
+    "id": "test:0063",
+    "gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-Plate-None-CounterTop-10/trial_T20190908_213533_897289/game.tw-pddl",
+    "task_type": "pick_clean_then_place_in_recep"
+  },
+  {
+    "id": "test:0064",
+    "gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-SoapBar-None-Cabinet-424/trial_T20190908_214926_337906/game.tw-pddl",
+    "task_type": "pick_clean_then_place_in_recep"
+  },
+  {
+    "id": "test:0065",
+    "gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-SoapBar-None-Cabinet-424/trial_T20190908_214946_567644/game.tw-pddl",
+    "task_type": "pick_clean_then_place_in_recep"
+  },
+  {
+    "id": "test:0066",
+    "gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-SoapBar-None-Cabinet-424/trial_T20190908_215019_162873/game.tw-pddl",
+    "task_type": "pick_clean_then_place_in_recep"
+  },
+  {
+    "id": "test:0067",
+    "gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-SoapBar-None-CounterTop-424/trial_T20190907_074045_109439/game.tw-pddl",
+    "task_type": "pick_clean_then_place_in_recep"
+  },
+  {
+    "id": "test:0068",
+    "gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-SoapBar-None-CounterTop-424/trial_T20190907_074106_050405/game.tw-pddl",
+    "task_type": "pick_clean_then_place_in_recep"
+  },
+  {
+    "id": "test:0069",
+    "gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-SoapBar-None-CounterTop-424/trial_T20190907_074124_966890/game.tw-pddl",
+    "task_type": "pick_clean_then_place_in_recep"
+  },
+  {
+    "id": "test:0070",
+    "gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-Spatula-None-Drawer-10/trial_T20190907_080730_211959/game.tw-pddl",
+    "task_type": "pick_clean_then_place_in_recep"
+  },
+  {
+    "id": "test:0071",
+    "gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-Spatula-None-Drawer-10/trial_T20190907_080800_275989/game.tw-pddl",
+    "task_type": "pick_clean_then_place_in_recep"
+  },
+  {
+    "id": "test:0072",
+    "gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-Spatula-None-Drawer-10/trial_T20190907_080825_222432/game.tw-pddl",
+    "task_type": "pick_clean_then_place_in_recep"
+  },
+  {
+    "id": "test:0073",
+    "gamefile": "json_2.1.1/valid_unseen/pick_cool_then_place_in_recep-Bread-None-CounterTop-10/trial_T20190908_091747_866951/game.tw-pddl",
+    "task_type": "pick_cool_then_place_in_recep"
+  },
+  {
+    "id": "test:0074",
+    "gamefile": "json_2.1.1/valid_unseen/pick_cool_then_place_in_recep-Bread-None-CounterTop-10/trial_T20190908_091811_414150/game.tw-pddl",
+    "task_type": "pick_cool_then_place_in_recep"
+  },
+  {
+    "id": "test:0075",
+    "gamefile": "json_2.1.1/valid_unseen/pick_cool_then_place_in_recep-Bread-None-CounterTop-10/trial_T20190908_091835_825830/game.tw-pddl",
+    "task_type": "pick_cool_then_place_in_recep"
+  },
+  {
+    "id": "test:0076",
+    "gamefile": "json_2.1.1/valid_unseen/pick_cool_then_place_in_recep-Lettuce-None-CounterTop-10/trial_T20190909_123133_763972/game.tw-pddl",
+    "task_type": "pick_cool_then_place_in_recep"
+  },
+  {
+    "id": "test:0077",
+    "gamefile": "json_2.1.1/valid_unseen/pick_cool_then_place_in_recep-Lettuce-None-CounterTop-10/trial_T20190909_174807_646433/game.tw-pddl",
+    "task_type": "pick_cool_then_place_in_recep"
+  },
+  {
+    "id": "test:0078",
+    "gamefile": "json_2.1.1/valid_unseen/pick_cool_then_place_in_recep-Lettuce-None-CounterTop-10/trial_T20190909_174840_771703/game.tw-pddl",
+    "task_type": "pick_cool_then_place_in_recep"
+  },
+  {
+    "id": "test:0079",
+    "gamefile": "json_2.1.1/valid_unseen/pick_cool_then_place_in_recep-Mug-None-Cabinet-10/trial_T20190909_121559_082363/game.tw-pddl",
+    "task_type": "pick_cool_then_place_in_recep"
+  },
+  {
+    "id": "test:0080",
+    "gamefile": "json_2.1.1/valid_unseen/pick_cool_then_place_in_recep-Mug-None-Cabinet-10/trial_T20190909_121635_622676/game.tw-pddl",
+    "task_type": "pick_cool_then_place_in_recep"
+  },
+  {
+    "id": "test:0081",
+    "gamefile": "json_2.1.1/valid_unseen/pick_cool_then_place_in_recep-Mug-None-Cabinet-10/trial_T20190909_121710_650938/game.tw-pddl",
+    "task_type": "pick_cool_then_place_in_recep"
+  },
+  {
+    "id": "test:0082",
+    "gamefile": "json_2.1.1/valid_unseen/pick_cool_then_place_in_recep-Mug-None-CoffeeMachine-10/trial_T20190907_183715_299073/game.tw-pddl",
+    "task_type": "pick_cool_then_place_in_recep"
+  },
+  {
+    "id": "test:0083",
+    "gamefile": "json_2.1.1/valid_unseen/pick_cool_then_place_in_recep-Mug-None-CoffeeMachine-10/trial_T20190907_183807_477267/game.tw-pddl",
+    "task_type": "pick_cool_then_place_in_recep"
+  },
+  {
+    "id": "test:0084",
+    "gamefile": "json_2.1.1/valid_unseen/pick_cool_then_place_in_recep-Mug-None-CoffeeMachine-10/trial_T20190907_183853_958104/game.tw-pddl",
+    "task_type": "pick_cool_then_place_in_recep"
+  },
+  {
+    "id": "test:0085",
+    "gamefile": "json_2.1.1/valid_unseen/pick_cool_then_place_in_recep-Pan-None-CounterTop-10/trial_T20190908_114545_244903/game.tw-pddl",
+    "task_type": "pick_cool_then_place_in_recep"
+  },
+  {
+    "id": "test:0086",
+    "gamefile": "json_2.1.1/valid_unseen/pick_cool_then_place_in_recep-Pan-None-CounterTop-10/trial_T20190908_114622_738670/game.tw-pddl",
+    "task_type": "pick_cool_then_place_in_recep"
+  },
+  {
+    "id": "test:0087",
+    "gamefile": "json_2.1.1/valid_unseen/pick_cool_then_place_in_recep-Pan-None-CounterTop-10/trial_T20190908_114656_768805/game.tw-pddl",
+    "task_type": "pick_cool_then_place_in_recep"
+  },
+  {
+    "id": "test:0088",
+    "gamefile": "json_2.1.1/valid_unseen/pick_cool_then_place_in_recep-Potato-None-Microwave-10/trial_T20190907_033157_424297/game.tw-pddl",
+    "task_type": "pick_cool_then_place_in_recep"
+  },
+  {
+    "id": "test:0089",
+    "gamefile": "json_2.1.1/valid_unseen/pick_cool_then_place_in_recep-Potato-None-Microwave-10/trial_T20190907_033228_194678/game.tw-pddl",
+    "task_type": "pick_cool_then_place_in_recep"
+  },
+  {
+    "id": "test:0090",
+    "gamefile": "json_2.1.1/valid_unseen/pick_cool_then_place_in_recep-Potato-None-Microwave-10/trial_T20190907_033306_962974/game.tw-pddl",
+    "task_type": "pick_cool_then_place_in_recep"
+  },
+  {
+    "id": "test:0091",
+    "gamefile": "json_2.1.1/valid_unseen/pick_cool_then_place_in_recep-Tomato-None-Microwave-10/trial_T20190909_102608_318800/game.tw-pddl",
+    "task_type": "pick_cool_then_place_in_recep"
+  },
+  {
+    "id": "test:0092",
+    "gamefile": "json_2.1.1/valid_unseen/pick_cool_then_place_in_recep-Tomato-None-Microwave-10/trial_T20190909_102644_926781/game.tw-pddl",
+    "task_type": "pick_cool_then_place_in_recep"
+  },
+  {
+    "id": "test:0093",
+    "gamefile": "json_2.1.1/valid_unseen/pick_cool_then_place_in_recep-Tomato-None-Microwave-10/trial_T20190909_102710_795182/game.tw-pddl",
+    "task_type": "pick_cool_then_place_in_recep"
+  },
+  {
+    "id": "test:0094",
+    "gamefile": "json_2.1.1/valid_unseen/pick_heat_then_place_in_recep-Apple-None-Fridge-10/trial_T20190906_182259_116320/game.tw-pddl",
+    "task_type": "pick_heat_then_place_in_recep"
+  },
+  {
+    "id": "test:0095",
+    "gamefile": "json_2.1.1/valid_unseen/pick_heat_then_place_in_recep-Apple-None-Fridge-10/trial_T20190906_182353_418140/game.tw-pddl",
+    "task_type": "pick_heat_then_place_in_recep"
+  },
+  {
+    "id": "test:0096",
+    "gamefile": "json_2.1.1/valid_unseen/pick_heat_then_place_in_recep-Apple-None-Fridge-10/trial_T20190906_182435_622538/game.tw-pddl",
+    "task_type": "pick_heat_then_place_in_recep"
+  },
+  {
+    "id": "test:0097",
+    "gamefile": "json_2.1.1/valid_unseen/pick_heat_then_place_in_recep-Apple-None-GarbageCan-10/trial_T20190908_145050_918567/game.tw-pddl",
+    "task_type": "pick_heat_then_place_in_recep"
+  },
+  {
+    "id": "test:0098",
+    "gamefile": "json_2.1.1/valid_unseen/pick_heat_then_place_in_recep-Apple-None-GarbageCan-10/trial_T20190908_145143_820541/game.tw-pddl",
+    "task_type": "pick_heat_then_place_in_recep"
+  },
+  {
+    "id": "test:0099",
+    "gamefile": "json_2.1.1/valid_unseen/pick_heat_then_place_in_recep-Apple-None-GarbageCan-10/trial_T20190908_145356_918528/game.tw-pddl",
+    "task_type": "pick_heat_then_place_in_recep"
+  },
+  {
+    "id": "test:0100",
+    "gamefile": "json_2.1.1/valid_unseen/pick_heat_then_place_in_recep-Cup-None-Cabinet-10/trial_T20190907_083346_800823/game.tw-pddl",
+    "task_type": "pick_heat_then_place_in_recep"
+  },
+  {
+    "id": "test:0101",
+    "gamefile": "json_2.1.1/valid_unseen/pick_heat_then_place_in_recep-Cup-None-Cabinet-10/trial_T20190907_083429_887065/game.tw-pddl",
+    "task_type": "pick_heat_then_place_in_recep"
+  },
+  {
+    "id": "test:0102",
+    "gamefile": "json_2.1.1/valid_unseen/pick_heat_then_place_in_recep-Cup-None-Cabinet-10/trial_T20190907_083507_594820/game.tw-pddl",
+    "task_type": "pick_heat_then_place_in_recep"
+  },
+  {
+    "id": "test:0103",
+    "gamefile": "json_2.1.1/valid_unseen/pick_heat_then_place_in_recep-Egg-None-GarbageCan-10/trial_T20190908_113432_673307/game.tw-pddl",
+    "task_type": "pick_heat_then_place_in_recep"
+  },
+  {
+    "id": "test:0104",
+    "gamefile": "json_2.1.1/valid_unseen/pick_heat_then_place_in_recep-Egg-None-GarbageCan-10/trial_T20190908_113523_123938/game.tw-pddl",
+    "task_type": "pick_heat_then_place_in_recep"
+  },
+  {
+    "id": "test:0105",
+    "gamefile": "json_2.1.1/valid_unseen/pick_heat_then_place_in_recep-Egg-None-GarbageCan-10/trial_T20190908_113610_425142/game.tw-pddl",
+    "task_type": "pick_heat_then_place_in_recep"
+  },
+  {
+    "id": "test:0106",
+    "gamefile": "json_2.1.1/valid_unseen/pick_heat_then_place_in_recep-Mug-None-Cabinet-10/trial_T20190909_021100_341887/game.tw-pddl",
+    "task_type": "pick_heat_then_place_in_recep"
+  },
+  {
+    "id": "test:0107",
+    "gamefile": "json_2.1.1/valid_unseen/pick_heat_then_place_in_recep-Mug-None-Cabinet-10/trial_T20190909_021200_669381/game.tw-pddl",
+    "task_type": "pick_heat_then_place_in_recep"
+  },
+  {
+    "id": "test:0108",
+    "gamefile": "json_2.1.1/valid_unseen/pick_heat_then_place_in_recep-Mug-None-Cabinet-10/trial_T20190909_021247_306737/game.tw-pddl",
+    "task_type": "pick_heat_then_place_in_recep"
+  },
+  {
+    "id": "test:0109",
+    "gamefile": "json_2.1.1/valid_unseen/pick_heat_then_place_in_recep-Mug-None-CoffeeMachine-10/trial_T20190907_171806_406231/game.tw-pddl",
+    "task_type": "pick_heat_then_place_in_recep"
+  },
+  {
+    "id": "test:0110",
+    "gamefile": "json_2.1.1/valid_unseen/pick_heat_then_place_in_recep-Mug-None-CoffeeMachine-10/trial_T20190907_171850_960211/game.tw-pddl",
+    "task_type": "pick_heat_then_place_in_recep"
+  },
+  {
+    "id": "test:0111",
+    "gamefile": "json_2.1.1/valid_unseen/pick_heat_then_place_in_recep-Mug-None-CoffeeMachine-10/trial_T20190907_171933_349922/game.tw-pddl",
+    "task_type": "pick_heat_then_place_in_recep"
+  },
+  {
+    "id": "test:0112",
+    "gamefile": "json_2.1.1/valid_unseen/pick_heat_then_place_in_recep-Potato-None-GarbageCan-10/trial_T20190907_161745_664033/game.tw-pddl",
+    "task_type": "pick_heat_then_place_in_recep"
+  },
+  {
+    "id": "test:0113",
+    "gamefile": "json_2.1.1/valid_unseen/pick_heat_then_place_in_recep-Potato-None-GarbageCan-10/trial_T20190907_161853_945788/game.tw-pddl",
+    "task_type": "pick_heat_then_place_in_recep"
+  },
+  {
+    "id": "test:0114",
+    "gamefile": "json_2.1.1/valid_unseen/pick_heat_then_place_in_recep-Tomato-None-GarbageCan-10/trial_T20190908_225046_020282/game.tw-pddl",
+    "task_type": "pick_heat_then_place_in_recep"
+  },
+  {
+    "id": "test:0115",
+    "gamefile": "json_2.1.1/valid_unseen/pick_heat_then_place_in_recep-Tomato-None-GarbageCan-10/trial_T20190908_225359_617900/game.tw-pddl",
+    "task_type": "pick_heat_then_place_in_recep"
+  },
+  {
+    "id": "test:0116",
+    "gamefile": "json_2.1.1/valid_unseen/pick_heat_then_place_in_recep-Tomato-None-GarbageCan-10/trial_T20190908_225453_272533/game.tw-pddl",
+    "task_type": "pick_heat_then_place_in_recep"
+  },
+  {
+    "id": "test:0117",
+    "gamefile": "json_2.1.1/valid_unseen/pick_two_obj_and_place-CD-None-Safe-308/trial_T20190907_050942_897916/game.tw-pddl",
+    "task_type": "pick_two_obj_and_place"
+  },
+  {
+    "id": "test:0118",
+    "gamefile": "json_2.1.1/valid_unseen/pick_two_obj_and_place-CD-None-Safe-308/trial_T20190907_051013_060265/game.tw-pddl",
+    "task_type": "pick_two_obj_and_place"
+  },
+  {
+    "id": "test:0119",
+    "gamefile": "json_2.1.1/valid_unseen/pick_two_obj_and_place-CD-None-Safe-308/trial_T20190907_051056_585414/game.tw-pddl",
+    "task_type": "pick_two_obj_and_place"
+  },
+  {
+    "id": "test:0120",
+    "gamefile": "json_2.1.1/valid_unseen/pick_two_obj_and_place-KeyChain-None-Safe-219/trial_T20190909_011803_423115/game.tw-pddl",
+    "task_type": "pick_two_obj_and_place"
+  },
+  {
+    "id": "test:0121",
+    "gamefile": "json_2.1.1/valid_unseen/pick_two_obj_and_place-KeyChain-None-Safe-219/trial_T20190909_012027_782483/game.tw-pddl",
+    "task_type": "pick_two_obj_and_place"
+  },
+  {
+    "id": "test:0122",
+    "gamefile": "json_2.1.1/valid_unseen/pick_two_obj_and_place-PepperShaker-None-Drawer-10/trial_T20190908_010306_215435/game.tw-pddl",
+    "task_type": "pick_two_obj_and_place"
+  },
+  {
+    "id": "test:0123",
+    "gamefile": "json_2.1.1/valid_unseen/pick_two_obj_and_place-PepperShaker-None-Drawer-10/trial_T20190912_221016_460197/game.tw-pddl",
+    "task_type": "pick_two_obj_and_place"
+  },
+  {
+    "id": "test:0124",
+    "gamefile": "json_2.1.1/valid_unseen/pick_two_obj_and_place-PepperShaker-None-Drawer-10/trial_T20190912_221141_608117/game.tw-pddl",
+    "task_type": "pick_two_obj_and_place"
+  },
+  {
+    "id": "test:0125",
+    "gamefile": "json_2.1.1/valid_unseen/pick_two_obj_and_place-Pillow-None-Sofa-219/trial_T20190907_163240_345855/game.tw-pddl",
+    "task_type": "pick_two_obj_and_place"
+  },
+  {
+    "id": "test:0126",
+    "gamefile": "json_2.1.1/valid_unseen/pick_two_obj_and_place-Pillow-None-Sofa-219/trial_T20190907_163327_486300/game.tw-pddl",
+    "task_type": "pick_two_obj_and_place"
+  },
+  {
+    "id": "test:0127",
+    "gamefile": "json_2.1.1/valid_unseen/pick_two_obj_and_place-Pillow-None-Sofa-219/trial_T20190907_163408_914117/game.tw-pddl",
+    "task_type": "pick_two_obj_and_place"
+  },
+  {
+    "id": "test:0128",
+    "gamefile": "json_2.1.1/valid_unseen/pick_two_obj_and_place-SoapBar-None-Cabinet-424/trial_T20190909_081720_491733/game.tw-pddl",
+    "task_type": "pick_two_obj_and_place"
+  },
+  {
+    "id": "test:0129",
+    "gamefile": "json_2.1.1/valid_unseen/pick_two_obj_and_place-SoapBar-None-Cabinet-424/trial_T20190909_081746_857594/game.tw-pddl",
+    "task_type": "pick_two_obj_and_place"
+  },
+  {
+    "id": "test:0130",
+    "gamefile": "json_2.1.1/valid_unseen/pick_two_obj_and_place-SoapBar-None-GarbageCan-424/trial_T20190909_064053_839817/game.tw-pddl",
+    "task_type": "pick_two_obj_and_place"
+  },
+  {
+    "id": "test:0131",
+    "gamefile": "json_2.1.1/valid_unseen/pick_two_obj_and_place-SoapBar-None-GarbageCan-424/trial_T20190909_064221_368939/game.tw-pddl",
+    "task_type": "pick_two_obj_and_place"
+  },
+  {
+    "id": "test:0132",
+    "gamefile": "json_2.1.1/valid_unseen/pick_two_obj_and_place-SoapBar-None-GarbageCan-424/trial_T20190909_064309_357168/game.tw-pddl",
+    "task_type": "pick_two_obj_and_place"
+  },
+  {
+    "id": "test:0133",
+    "gamefile": "json_2.1.1/valid_unseen/pick_two_obj_and_place-ToiletPaper-None-Cabinet-424/trial_T20190906_202926_527010/game.tw-pddl",
+    "task_type": "pick_two_obj_and_place"
+  }
+]
@@ -0,0 +1,197 @@
+[
+  {
+    "id": "train:0000",
+    "gamefile": "json_2.1.1/train/look_at_obj_in_light-AlarmClock-None-DeskLamp-305/trial_T20190908_082736_108723/game.tw-pddl",
+    "task_type": "look_at_obj_in_light"
+  },
+  {
+    "id": "train:0001",
+    "gamefile": "json_2.1.1/train/look_at_obj_in_light-CD-None-DeskLamp-304/trial_T20190907_185649_782438/game.tw-pddl",
+    "task_type": "look_at_obj_in_light"
+  },
+  {
+    "id": "train:0002",
+    "gamefile": "json_2.1.1/train/look_at_obj_in_light-CD-None-DeskLamp-320/trial_T20190907_224439_174735/game.tw-pddl",
+    "task_type": "look_at_obj_in_light"
+  },
+  {
+    "id": "train:0003",
+    "gamefile": "json_2.1.1/train/look_at_obj_in_light-Pillow-None-DeskLamp-316/trial_T20190908_232421_645610/game.tw-pddl",
+    "task_type": "look_at_obj_in_light"
+  },
+  {
+    "id": "train:0004",
+    "gamefile": "json_2.1.1/train/look_at_obj_in_light-Statue-None-DeskLamp-319/trial_T20190907_035546_167548/game.tw-pddl",
+    "task_type": "look_at_obj_in_light"
+  },
+  {
+    "id": "train:0005",
+    "gamefile": "json_2.1.1/train/pick_and_place_simple-CellPhone-None-Shelf-313/trial_T20190908_123725_452958/game.tw-pddl",
+    "task_type": "pick_and_place_simple"
+  },
+  {
+    "id": "train:0006",
+    "gamefile": "json_2.1.1/train/pick_and_place_simple-Newspaper-None-Sofa-211/trial_T20190906_175004_203092/game.tw-pddl",
+    "task_type": "pick_and_place_simple"
+  },
+  {
+    "id": "train:0007",
+    "gamefile": "json_2.1.1/train/pick_and_place_simple-Pencil-None-Desk-302/trial_T20190908_032836_462632/game.tw-pddl",
+    "task_type": "pick_and_place_simple"
+  },
+  {
+    "id": "train:0008",
+    "gamefile": "json_2.1.1/train/pick_and_place_simple-SoapBar-None-GarbageCan-416/trial_T20190908_020839_714699/game.tw-pddl",
+    "task_type": "pick_and_place_simple"
+  },
+  {
+    "id": "train:0009",
+    "gamefile": "json_2.1.1/train/pick_and_place_simple-Statue-None-CoffeeTable-222/trial_T20190907_131249_788749/game.tw-pddl",
+    "task_type": "pick_and_place_simple"
+  },
+  {
+    "id": "train:0010",
+    "gamefile": "json_2.1.1/train/pick_and_place_simple-ToiletPaper-None-ToiletPaperHanger-406/trial_T20190908_122807_136741/game.tw-pddl",
+    "task_type": "pick_and_place_simple"
+  },
+  {
+    "id": "train:0011",
+    "gamefile": "json_2.1.1/train/pick_and_place_simple-ToiletPaper-None-ToiletPaperHanger-415/trial_T20190908_050443_333939/game.tw-pddl",
+    "task_type": "pick_and_place_simple"
+  },
+  {
+    "id": "train:0012",
+    "gamefile": "json_2.1.1/train/pick_clean_then_place_in_recep-Apple-None-DiningTable-4/trial_T20190908_104413_450768/game.tw-pddl",
+    "task_type": "pick_clean_then_place_in_recep"
+  },
+  {
+    "id": "train:0013",
+    "gamefile": "json_2.1.1/train/pick_clean_then_place_in_recep-DishSponge-None-Shelf-20/trial_T20190907_222429_992578/game.tw-pddl",
+    "task_type": "pick_clean_then_place_in_recep"
+  },
+  {
+    "id": "train:0014",
+    "gamefile": "json_2.1.1/train/pick_clean_then_place_in_recep-DishSponge-None-Shelf-401/trial_T20190908_072225_397518/game.tw-pddl",
+    "task_type": "pick_clean_then_place_in_recep"
+  },
+  {
+    "id": "train:0015",
+    "gamefile": "json_2.1.1/train/pick_clean_then_place_in_recep-Kettle-None-Cabinet-2/trial_T20190909_043103_418752/game.tw-pddl",
+    "task_type": "pick_clean_then_place_in_recep"
+  },
+  {
+    "id": "train:0016",
+    "gamefile": "json_2.1.1/train/pick_clean_then_place_in_recep-Knife-None-Drawer-22/trial_T20190907_224827_746945/game.tw-pddl",
+    "task_type": "pick_clean_then_place_in_recep"
+  },
+  {
+    "id": "train:0017",
+    "gamefile": "json_2.1.1/train/pick_clean_then_place_in_recep-Lettuce-None-DiningTable-20/trial_T20190906_191148_519826/game.tw-pddl",
+    "task_type": "pick_clean_then_place_in_recep"
+  },
+  {
+    "id": "train:0018",
+    "gamefile": "json_2.1.1/train/pick_clean_then_place_in_recep-Lettuce-None-Fridge-13/trial_T20190908_203022_601787/game.tw-pddl",
+    "task_type": "pick_clean_then_place_in_recep"
+  },
+  {
+    "id": "train:0019",
+    "gamefile": "json_2.1.1/train/pick_clean_then_place_in_recep-Plate-None-Fridge-5/trial_T20190909_112954_869911/game.tw-pddl",
+    "task_type": "pick_clean_then_place_in_recep"
+  },
+  {
+    "id": "train:0020",
+    "gamefile": "json_2.1.1/train/pick_clean_then_place_in_recep-Spoon-None-DiningTable-18/trial_T20190909_102159_277894/game.tw-pddl",
+    "task_type": "pick_clean_then_place_in_recep"
+  },
+  {
+    "id": "train:0021",
+    "gamefile": "json_2.1.1/train/pick_cool_then_place_in_recep-Bread-None-CounterTop-1/trial_T20190908_212439_711334/game.tw-pddl",
+    "task_type": "pick_cool_then_place_in_recep"
+  },
+  {
+    "id": "train:0022",
+    "gamefile": "json_2.1.1/train/pick_cool_then_place_in_recep-Bread-None-CounterTop-15/trial_T20190909_085448_256298/game.tw-pddl",
+    "task_type": "pick_cool_then_place_in_recep"
+  },
+  {
+    "id": "train:0023",
+    "gamefile": "json_2.1.1/train/pick_cool_then_place_in_recep-Bread-None-CounterTop-16/trial_T20190908_143948_082471/game.tw-pddl",
+    "task_type": "pick_cool_then_place_in_recep"
+  },
+  {
+    "id": "train:0024",
+    "gamefile": "json_2.1.1/train/pick_cool_then_place_in_recep-Pan-None-StoveBurner-27/trial_T20190906_212619_469871/game.tw-pddl",
+    "task_type": "pick_cool_then_place_in_recep"
+  },
+  {
+    "id": "train:0025",
+    "gamefile": "json_2.1.1/train/pick_cool_then_place_in_recep-Plate-None-DiningTable-17/trial_T20190909_122939_032098/game.tw-pddl",
+    "task_type": "pick_cool_then_place_in_recep"
+  },
+  {
+    "id": "train:0026",
+    "gamefile": "json_2.1.1/train/pick_cool_then_place_in_recep-Pot-None-CounterTop-1/trial_T20190909_124252_504581/game.tw-pddl",
+    "task_type": "pick_cool_then_place_in_recep"
+  },
+  {
+    "id": "train:0027",
+    "gamefile": "json_2.1.1/train/pick_heat_then_place_in_recep-Apple-None-Fridge-20/trial_T20190908_013911_274341/game.tw-pddl",
+    "task_type": "pick_heat_then_place_in_recep"
+  },
+  {
+    "id": "train:0028",
+    "gamefile": "json_2.1.1/train/pick_heat_then_place_in_recep-Egg-None-CounterTop-12/trial_T20190908_215527_416490/game.tw-pddl",
+    "task_type": "pick_heat_then_place_in_recep"
+  },
+  {
+    "id": "train:0029",
+    "gamefile": "json_2.1.1/train/pick_heat_then_place_in_recep-Mug-None-CoffeeMachine-1/trial_T20190907_222924_821086/game.tw-pddl",
+    "task_type": "pick_heat_then_place_in_recep"
+  },
+  {
+    "id": "train:0030",
+    "gamefile": "json_2.1.1/train/pick_heat_then_place_in_recep-Mug-None-CoffeeMachine-28/trial_T20190908_062730_537428/game.tw-pddl",
+    "task_type": "pick_heat_then_place_in_recep"
+  },
+  {
+    "id": "train:0031",
+    "gamefile": "json_2.1.1/train/pick_heat_then_place_in_recep-Plate-None-Cabinet-13/trial_T20190907_062749_759882/game.tw-pddl",
+    "task_type": "pick_heat_then_place_in_recep"
+  },
+  {
+    "id": "train:0032",
+    "gamefile": "json_2.1.1/train/pick_heat_then_place_in_recep-Potato-None-Fridge-2/trial_T20190909_030845_198194/game.tw-pddl",
+    "task_type": "pick_heat_then_place_in_recep"
+  },
+  {
+    "id": "train:0033",
+    "gamefile": "json_2.1.1/train/pick_heat_then_place_in_recep-Tomato-None-CounterTop-26/trial_T20190907_005525_499114/game.tw-pddl",
+    "task_type": "pick_heat_then_place_in_recep"
+  },
+  {
+    "id": "train:0034",
+    "gamefile": "json_2.1.1/train/pick_two_obj_and_place-CD-None-Drawer-319/trial_T20190907_145515_348252/game.tw-pddl",
+    "task_type": "pick_two_obj_and_place"
+  },
+  {
+    "id": "train:0035",
+    "gamefile": "json_2.1.1/train/pick_two_obj_and_place-Candle-None-Drawer-427/trial_T20190909_043917_251333/game.tw-pddl",
+    "task_type": "pick_two_obj_and_place"
+  },
+  {
+    "id": "train:0036",
+    "gamefile": "json_2.1.1/train/pick_two_obj_and_place-KeyChain-None-ArmChair-222/trial_T20190909_100312_677332/game.tw-pddl",
+    "task_type": "pick_two_obj_and_place"
+  },
+  {
+    "id": "train:0037",
+    "gamefile": "json_2.1.1/train/pick_two_obj_and_place-Newspaper-None-Sofa-212/trial_T20190908_112632_208041/game.tw-pddl",
+    "task_type": "pick_two_obj_and_place"
+  },
+  {
+    "id": "train:0038",
+    "gamefile": "json_2.1.1/train/pick_two_obj_and_place-SaltShaker-None-SideTable-21/trial_T20190909_041626_844806/game.tw-pddl",
+    "task_type": "pick_two_obj_and_place"
+  }
+]
@@ -0,0 +1,92 @@
+[
+  {
+    "id": "val:0000",
+    "gamefile": "json_2.1.1/valid_seen/look_at_obj_in_light-AlarmClock-None-DeskLamp-323/trial_T20190909_044715_250790/game.tw-pddl",
+    "task_type": "look_at_obj_in_light"
+  },
+  {
+    "id": "val:0001",
+    "gamefile": "json_2.1.1/valid_seen/look_at_obj_in_light-Bowl-None-DeskLamp-301/trial_T20190909_150719_492274/game.tw-pddl",
+    "task_type": "look_at_obj_in_light"
+  },
+  {
+    "id": "val:0002",
+    "gamefile": "json_2.1.1/valid_seen/look_at_obj_in_light-Pillow-None-DeskLamp-323/trial_T20190908_053153_077977/game.tw-pddl",
+    "task_type": "look_at_obj_in_light"
+  },
+  {
+    "id": "val:0003",
+    "gamefile": "json_2.1.1/valid_seen/pick_and_place_simple-Mug-None-SideTable-329/trial_T20190909_032318_169393/game.tw-pddl",
+    "task_type": "pick_and_place_simple"
+  },
+  {
+    "id": "val:0004",
+    "gamefile": "json_2.1.1/valid_seen/pick_and_place_simple-Mug-None-SideTable-329/trial_T20190909_032340_274147/game.tw-pddl",
+    "task_type": "pick_and_place_simple"
+  },
+  {
+    "id": "val:0005",
+    "gamefile": "json_2.1.1/valid_seen/pick_and_place_simple-Pencil-None-Desk-310/trial_T20190909_113054_894334/game.tw-pddl",
+    "task_type": "pick_and_place_simple"
+  },
+  {
+    "id": "val:0006",
+    "gamefile": "json_2.1.1/valid_seen/pick_clean_then_place_in_recep-ButterKnife-None-Drawer-30/trial_T20190908_052007_212776/game.tw-pddl",
+    "task_type": "pick_clean_then_place_in_recep"
+  },
+  {
+    "id": "val:0007",
+    "gamefile": "json_2.1.1/valid_seen/pick_clean_then_place_in_recep-ButterKnife-None-Drawer-8/trial_T20190909_124425_112757/game.tw-pddl",
+    "task_type": "pick_clean_then_place_in_recep"
+  },
+  {
+    "id": "val:0008",
+    "gamefile": "json_2.1.1/valid_seen/pick_clean_then_place_in_recep-SoapBar-None-Cabinet-402/trial_T20190908_055221_984342/game.tw-pddl",
+    "task_type": "pick_clean_then_place_in_recep"
+  },
+  {
+    "id": "val:0009",
+    "gamefile": "json_2.1.1/valid_seen/pick_clean_then_place_in_recep-SoapBar-None-Toilet-410/trial_T20190906_201106_979461/game.tw-pddl",
+    "task_type": "pick_clean_then_place_in_recep"
+  },
+  {
+    "id": "val:0010",
+    "gamefile": "json_2.1.1/valid_seen/pick_cool_then_place_in_recep-Apple-None-Microwave-19/trial_T20190906_210937_878489/game.tw-pddl",
+    "task_type": "pick_cool_then_place_in_recep"
+  },
+  {
+    "id": "val:0011",
+    "gamefile": "json_2.1.1/valid_seen/pick_cool_then_place_in_recep-Plate-None-CounterTop-1/trial_T20190906_205324_559361/game.tw-pddl",
+    "task_type": "pick_cool_then_place_in_recep"
+  },
+  {
+    "id": "val:0012",
+    "gamefile": "json_2.1.1/valid_seen/pick_cool_then_place_in_recep-Tomato-None-Microwave-18/trial_T20190909_012524_159092/game.tw-pddl",
+    "task_type": "pick_cool_then_place_in_recep"
+  },
+  {
+    "id": "val:0013",
+    "gamefile": "json_2.1.1/valid_seen/pick_heat_then_place_in_recep-Apple-None-DiningTable-26/trial_T20190907_060234_011675/game.tw-pddl",
+    "task_type": "pick_heat_then_place_in_recep"
+  },
+  {
+    "id": "val:0014",
+    "gamefile": "json_2.1.1/valid_seen/pick_heat_then_place_in_recep-Tomato-None-Fridge-15/trial_T20190909_020200_054379/game.tw-pddl",
+    "task_type": "pick_heat_then_place_in_recep"
+  },
+  {
+    "id": "val:0015",
+    "gamefile": "json_2.1.1/valid_seen/pick_heat_then_place_in_recep-Tomato-None-Fridge-23/trial_T20190909_082320_103350/game.tw-pddl",
+    "task_type": "pick_heat_then_place_in_recep"
+  },
+  {
+    "id": "val:0016",
+    "gamefile": "json_2.1.1/valid_seen/pick_two_obj_and_place-Book-None-Desk-313/trial_T20190908_125930_920681/game.tw-pddl",
+    "task_type": "pick_two_obj_and_place"
+  },
+  {
+    "id": "val:0017",
+    "gamefile": "json_2.1.1/valid_seen/pick_two_obj_and_place-CreditCard-None-Safe-323/trial_T20190907_001129_214240/game.tw-pddl",
+    "task_type": "pick_two_obj_and_place"
+  }
+]
@@ -0,0 +1,36 @@
+{
+  "benchmark": "DocVQA",
+  "manifest_type": "id_split",
+  "source_repo": "lmms-lab/DocVQA",
+  "source_repo_type": "dataset",
+  "source_url": "https://huggingface.co/datasets/lmms-lab/DocVQA",
+  "source_revision": "539088ef8a8ada01ac8e2e6d4e372586748a265e",
+  "source_config": "DocVQA",
+  "source_split": "validation",
+  "source_split_name": "docvqa_validation_10pct",
+  "split_method": "10% subset sampled from the DocVQA validation split",
+  "counts": {
+    "train": 107,
+    "val": 53,
+    "test": 374
+  },
+  "item_fields": [
+    "id",
+    "questionId",
+    "docId",
+    "image_path",
+    "ucsf_document_id",
+    "ucsf_document_page_no",
+    "topic",
+    "source_dataset",
+    "source_config",
+    "source_split",
+    "sample_seed"
+  ],
+  "notes": [
+    "This is a split manifest, not the full DocVQA payload.",
+    "Materialize full CSV rows and image files before evaluation.",
+    "This manifest corresponds to docvqa_validation_10pct.",
+    "All released train/val/test items originate from a 10% subset of the official DocVQA validation split."
+  ]
+}
@@ -0,0 +1,691 @@
+[
+  {
+    "id": "62409",
+    "questionId": "62409",
+    "docId": "8554",
+    "image_path": "data/docvqa_images/q62409_d8554.png",
+    "ucsf_document_id": "pgjw0227",
+    "ucsf_document_page_no": "5",
+    "topic": "table/list",
+    "source_dataset": "lmms-lab/DocVQA",
+    "source_config": "DocVQA",
+    "source_split": "validation",
+    "sample_seed": "full_validation_5349"
+  },
+  {
+    "id": "50961",
+    "questionId": "50961",
+    "docId": "549",
+    "image_path": "data/docvqa_images/q50961_d549.png",
+    "ucsf_document_id": "qtjf0226",
+    "ucsf_document_page_no": "2",
+    "topic": "free_text",
+    "source_dataset": "lmms-lab/DocVQA",
+    "source_config": "DocVQA",
+    "source_split": "validation",
+    "sample_seed": "full_validation_5349"
+  },
+  {
+    "id": "46461",
+    "questionId": "46461",
+    "docId": "13361",
+    "image_path": "data/docvqa_images/q46461_d13361.png",
+    "ucsf_document_id": "ysbw0217",
+    "ucsf_document_page_no": "5",
+    "topic": "layout",
+    "source_dataset": "lmms-lab/DocVQA",
+    "source_config": "DocVQA",
+    "source_split": "validation",
+    "sample_seed": "full_validation_5349"
+  },
+  {
+    "id": "3041",
+    "questionId": "3041",
+    "docId": "1204",
+    "image_path": "data/docvqa_images/q3041_d1204.png",
+    "ucsf_document_id": "xfjv0228",
+    "ucsf_document_page_no": "3",
+    "topic": "form",
+    "source_dataset": "lmms-lab/DocVQA",
+    "source_config": "DocVQA",
+    "source_split": "validation",
+    "sample_seed": "full_validation_5349"
+  },
+  {
+    "id": "41716",
+    "questionId": "41716",
+    "docId": "11835",
+    "image_path": "data/docvqa_images/q41716_d11835.png",
+    "ucsf_document_id": "qjgn0226",
+    "ucsf_document_page_no": "131",
+    "topic": "form",
+    "source_dataset": "lmms-lab/DocVQA",
+    "source_config": "DocVQA",
+    "source_split": "validation",
+    "sample_seed": "full_validation_5349"
+  },
+  {
+    "id": "61123",
+    "questionId": "61123",
+    "docId": "7374",
+    "image_path": "data/docvqa_images/q61123_d7374.png",
+    "ucsf_document_id": "mldg0227",
+    "ucsf_document_page_no": "5",
+    "topic": "layout",
+    "source_dataset": "lmms-lab/DocVQA",
+    "source_config": "DocVQA",
+    "source_split": "validation",
+    "sample_seed": "full_validation_5349"
+  },
+  {
+    "id": "43068",
+    "questionId": "43068",
+    "docId": "12393",
+    "image_path": "data/docvqa_images/q43068_d12393.png",
+    "ucsf_document_id": "rmwn0226",
+    "ucsf_document_page_no": "52",
+    "topic": "figure/diagram",
+    "source_dataset": "lmms-lab/DocVQA",
+    "source_config": "DocVQA",
+    "source_split": "validation",
+    "sample_seed": "full_validation_5349"
+  },
+  {
+    "id": "51221",
+    "questionId": "51221",
+    "docId": "764",
+    "image_path": "data/docvqa_images/q51221_d764.png",
+    "ucsf_document_id": "kzbn0226",
+    "ucsf_document_page_no": "14",
+    "topic": "layout",
+    "source_dataset": "lmms-lab/DocVQA",
+    "source_config": "DocVQA",
+    "source_split": "validation",
+    "sample_seed": "full_validation_5349"
+  },
+  {
+    "id": "6397",
+    "questionId": "6397",
+    "docId": "2242",
+    "image_path": "data/docvqa_images/q6397_d2242.png",
+    "ucsf_document_id": "jkcn0000",
+    "ucsf_document_page_no": "2",
+    "topic": "form",
+    "source_dataset": "lmms-lab/DocVQA",
+    "source_config": "DocVQA",
+    "source_split": "validation",
+    "sample_seed": "full_validation_5349"
+  },
+  {
+    "id": "57428",
+    "questionId": "57428",
+    "docId": "4779",
+    "image_path": "data/docvqa_images/q57428_d4779.png",
+    "ucsf_document_id": "rnbx0223",
+    "ucsf_document_page_no": "208",
+    "topic": "Image/Photo",
+    "source_dataset": "lmms-lab/DocVQA",
+    "source_config": "DocVQA",
+    "source_split": "validation",
+    "sample_seed": "full_validation_5349"
+  },
+  {
+    "id": "3135",
+    "questionId": "3135",
+    "docId": "1221",
+    "image_path": "data/docvqa_images/q3135_d1221.png",
+    "ucsf_document_id": "ngph0227",
+    "ucsf_document_page_no": "5",
+    "topic": "layout",
+    "source_dataset": "lmms-lab/DocVQA",
+    "source_config": "DocVQA",
+    "source_split": "validation",
+    "sample_seed": "full_validation_5349"
+  },
+  {
+    "id": "18819",
+    "questionId": "18819",
+    "docId": "5749",
+    "image_path": "data/docvqa_images/q18819_d5749.png",
+    "ucsf_document_id": "jhfd0079",
+    "ucsf_document_page_no": "9",
+    "topic": "form",
+    "source_dataset": "lmms-lab/DocVQA",
+    "source_config": "DocVQA",
+    "source_split": "validation",
+    "sample_seed": "full_validation_5349"
+  },
+  {
+    "id": "15382",
+    "questionId": "15382",
+    "docId": "4890",
+    "image_path": "data/docvqa_images/q15382_d4890.png",
+    "ucsf_document_id": "kjvw0217",
+    "ucsf_document_page_no": "3",
+    "topic": "table/list",
+    "source_dataset": "lmms-lab/DocVQA",
+    "source_config": "DocVQA",
+    "source_split": "validation",
+    "sample_seed": "full_validation_5349"
+  },
+  {
+    "id": "5772",
+    "questionId": "5772",
+    "docId": "1940",
+    "image_path": "data/docvqa_images/q5772_d1940.png",
+    "ucsf_document_id": "pzyw0224",
+    "ucsf_document_page_no": "10",
+    "topic": "form",
+    "source_dataset": "lmms-lab/DocVQA",
+    "source_config": "DocVQA",
+    "source_split": "validation",
+    "sample_seed": "full_validation_5349"
+  },
+  {
+    "id": "49077",
+    "questionId": "49077",
+    "docId": "14179",
+    "image_path": "data/docvqa_images/q49077_d14179.png",
+    "ucsf_document_id": "nrxb0228",
+    "ucsf_document_page_no": "3",
+    "topic": "layout",
+    "source_dataset": "lmms-lab/DocVQA",
+    "source_config": "DocVQA",
+    "source_split": "validation",
+    "sample_seed": "full_validation_5349"
+  },
+  {
+    "id": "58519",
+    "questionId": "58519",
+    "docId": "5347",
+    "image_path": "data/docvqa_images/q58519_d5347.png",
+    "ucsf_document_id": "sjbw0217",
+    "ucsf_document_page_no": "11",
+    "topic": "table/list",
+    "source_dataset": "lmms-lab/DocVQA",
+    "source_config": "DocVQA",
+    "source_split": "validation",
+    "sample_seed": "full_validation_5349"
+  },
+  {
+    "id": "50720",
+    "questionId": "50720",
+    "docId": "281",
+    "image_path": "data/docvqa_images/q50720_d281.png",
+    "ucsf_document_id": "nrcj0037",
+    "ucsf_document_page_no": "7",
+    "topic": "table/list",
+    "source_dataset": "lmms-lab/DocVQA",
+    "source_config": "DocVQA",
+    "source_split": "validation",
+    "sample_seed": "full_validation_5349"
+  },
+  {
+    "id": "56785",
+    "questionId": "56785",
+    "docId": "14289",
+    "image_path": "data/docvqa_images/q56785_d14289.png",
+    "ucsf_document_id": "xkbv0228",
+    "ucsf_document_page_no": "1",
+    "topic": "layout",
+    "source_dataset": "lmms-lab/DocVQA",
+    "source_config": "DocVQA",
+    "source_split": "validation",
+    "sample_seed": "full_validation_5349"
+  },
+  {
+    "id": "59653",
+    "questionId": "59653",
+    "docId": "6579",
+    "image_path": "data/docvqa_images/q59653_d6579.png",
+    "ucsf_document_id": "mzbx0227",
+    "ucsf_document_page_no": "2",
+    "topic": "layout",
+    "source_dataset": "lmms-lab/DocVQA",
+    "source_config": "DocVQA",
+    "source_split": "validation",
+    "sample_seed": "full_validation_5349"
+  },
+  {
+    "id": "61791",
+    "questionId": "61791",
+    "docId": "8072",
+    "image_path": "data/docvqa_images/q61791_d8072.png",
+    "ucsf_document_id": "hfmf0227",
+    "ucsf_document_page_no": "1",
+    "topic": "layout",
+    "source_dataset": "lmms-lab/DocVQA",
+    "source_config": "DocVQA",
+    "source_split": "validation",
+    "sample_seed": "full_validation_5349"
+  },
+  {
+    "id": "37229",
+    "questionId": "37229",
+    "docId": "10742",
+    "image_path": "data/docvqa_images/q37229_d10742.png",
+    "ucsf_document_id": "nkcd0227",
+    "ucsf_document_page_no": "2",
+    "topic": "free_text",
+    "source_dataset": "lmms-lab/DocVQA",
+    "source_config": "DocVQA",
+    "source_split": "validation",
+    "sample_seed": "full_validation_5349"
+  },
+  {
+    "id": "60407",
+    "questionId": "60407",
+    "docId": "7135",
+    "image_path": "data/docvqa_images/q60407_d7135.png",
+    "ucsf_document_id": "gkpk0226",
+    "ucsf_document_page_no": "1",
+    "topic": "free_text",
+    "source_dataset": "lmms-lab/DocVQA",
+    "source_config": "DocVQA",
+    "source_split": "validation",
+    "sample_seed": "full_validation_5349"
+  },
+  {
+    "id": "64420",
+    "questionId": "64420",
+    "docId": "10230",
+    "image_path": "data/docvqa_images/q64420_d10230.png",
+    "ucsf_document_id": "jnjm0223",
+    "ucsf_document_page_no": "107",
+    "topic": "table/list",
+    "source_dataset": "lmms-lab/DocVQA",
+    "source_config": "DocVQA",
+    "source_split": "validation",
+    "sample_seed": "full_validation_5349"
+  },
+  {
+    "id": "47365",
+    "questionId": "47365",
+    "docId": "13813",
+    "image_path": "data/docvqa_images/q47365_d13813.png",
+    "ucsf_document_id": "nxym0227",
+    "ucsf_document_page_no": "28",
+    "topic": "table/list",
+    "source_dataset": "lmms-lab/DocVQA",
+    "source_config": "DocVQA",
+    "source_split": "validation",
+    "sample_seed": "full_validation_5349"
+  },
+  {
+    "id": "47458",
+    "questionId": "47458",
+    "docId": "13639",
+    "image_path": "data/docvqa_images/q47458_d13639.png",
+    "ucsf_document_id": "skdv0228",
+    "ucsf_document_page_no": "5",
+    "topic": "layout",
+    "source_dataset": "lmms-lab/DocVQA",
+    "source_config": "DocVQA",
+    "source_split": "validation",
+    "sample_seed": "full_validation_5349"
+  },
+  {
+    "id": "7621",
+    "questionId": "7621",
+    "docId": "2668",
+    "image_path": "data/docvqa_images/q7621_d2668.png",
+    "ucsf_document_id": "flxn0020",
+    "ucsf_document_page_no": "1",
+    "topic": "form",
+    "source_dataset": "lmms-lab/DocVQA",
+    "source_config": "DocVQA",
+    "source_split": "validation",
+    "sample_seed": "full_validation_5349"
+  },
+  {
+    "id": "53575",
+    "questionId": "53575",
+    "docId": "2766",
+    "image_path": "data/docvqa_images/q53575_d2766.png",
+    "ucsf_document_id": "hsfn0020",
+    "ucsf_document_page_no": "2",
+    "topic": "free_text|table/list",
+    "source_dataset": "lmms-lab/DocVQA",
+    "source_config": "DocVQA",
+    "source_split": "validation",
+    "sample_seed": "full_validation_5349"
+  },
+  {
+    "id": "60913",
+    "questionId": "60913",
+    "docId": "7349",
+    "image_path": "data/docvqa_images/q60913_d7349.png",
+    "ucsf_document_id": "jzhd0227",
+    "ucsf_document_page_no": "61",
+    "topic": "free_text",
+    "source_dataset": "lmms-lab/DocVQA",
+    "source_config": "DocVQA",
+    "source_split": "validation",
+    "sample_seed": "full_validation_5349"
+  },
+  {
+    "id": "60454",
+    "questionId": "60454",
+    "docId": "7163",
+    "image_path": "data/docvqa_images/q60454_d7163.png",
+    "ucsf_document_id": "jgyk0226",
+    "ucsf_document_page_no": "1",
+    "topic": "free_text",
+    "source_dataset": "lmms-lab/DocVQA",
+    "source_config": "DocVQA",
+    "source_split": "validation",
+    "sample_seed": "full_validation_5349"
+  },
+  {
+    "id": "57978",
+    "questionId": "57978",
+    "docId": "4920",
+    "image_path": "data/docvqa_images/q57978_d4920.png",
+    "ucsf_document_id": "lkvw0217",
+    "ucsf_document_page_no": "2",
+    "topic": "table/list",
+    "source_dataset": "lmms-lab/DocVQA",
+    "source_config": "DocVQA",
+    "source_split": "validation",
+    "sample_seed": "full_validation_5349"
+  },
+  {
+    "id": "64547",
+    "questionId": "64547",
+    "docId": "10361",
+    "image_path": "data/docvqa_images/q64547_d10361.png",
+    "ucsf_document_id": "lpdl0226",
+    "ucsf_document_page_no": "32",
+    "topic": "table/list",
+    "source_dataset": "lmms-lab/DocVQA",
+    "source_config": "DocVQA",
+    "source_split": "validation",
+    "sample_seed": "full_validation_5349"
+  },
+  {
+    "id": "59481",
+    "questionId": "59481",
+    "docId": "6243",
+    "image_path": "data/docvqa_images/q59481_d6243.png",
+    "ucsf_document_id": "psgv0228",
+    "ucsf_document_page_no": "5",
+    "topic": "free_text",
+    "source_dataset": "lmms-lab/DocVQA",
+    "source_config": "DocVQA",
+    "source_split": "validation",
+    "sample_seed": "full_validation_5349"
+  },
+  {
+    "id": "61472",
+    "questionId": "61472",
+    "docId": "7757",
+    "image_path": "data/docvqa_images/q61472_d7757.png",
+    "ucsf_document_id": "ymkp0227",
+    "ucsf_document_page_no": "13",
+    "topic": "handwritten|table/list",
+    "source_dataset": "lmms-lab/DocVQA",
+    "source_config": "DocVQA",
+    "source_split": "validation",
+    "sample_seed": "full_validation_5349"
+  },
+  {
+    "id": "5673",
+    "questionId": "5673",
+    "docId": "1908",
+    "image_path": "data/docvqa_images/q5673_d1908.png",
+    "ucsf_document_id": "lldj0224",
+    "ucsf_document_page_no": "2",
+    "topic": "form",
+    "source_dataset": "lmms-lab/DocVQA",
+    "source_config": "DocVQA",
+    "source_split": "validation",
+    "sample_seed": "full_validation_5349"
+  },
+  {
+    "id": "49109",
+    "questionId": "49109",
+    "docId": "13644",
+    "image_path": "data/docvqa_images/q49109_d13644.png",
+    "ucsf_document_id": "mzdv0228",
+    "ucsf_document_page_no": "1",
+    "topic": "table/list",
+    "source_dataset": "lmms-lab/DocVQA",
+    "source_config": "DocVQA",
+    "source_split": "validation",
+    "sample_seed": "full_validation_5349"
+  },
+  {
+    "id": "46123",
+    "questionId": "46123",
+    "docId": "13503",
+    "image_path": "data/docvqa_images/q46123_d13503.png",
+    "ucsf_document_id": "xmww0217",
+    "ucsf_document_page_no": "17",
+    "topic": "table/list",
+    "source_dataset": "lmms-lab/DocVQA",
+    "source_config": "DocVQA",
+    "source_split": "validation",
+    "sample_seed": "full_validation_5349"
+  },
+  {
+    "id": "48158",
+    "questionId": "48158",
+    "docId": "13976",
+    "image_path": "data/docvqa_images/q48158_d13976.png",
+    "ucsf_document_id": "zqhm0227",
+    "ucsf_document_page_no": "1",
+    "topic": "free_text",
+    "source_dataset": "lmms-lab/DocVQA",
+    "source_config": "DocVQA",
+    "source_split": "validation",
+    "sample_seed": "full_validation_5349"
+  },
+  {
+    "id": "1955",
+    "questionId": "1955",
+    "docId": "892",
+    "image_path": "data/docvqa_images/q1955_d892.png",
+    "ucsf_document_id": "jsbn0226",
+    "ucsf_document_page_no": "2",
+    "topic": "table/list",
+    "source_dataset": "lmms-lab/DocVQA",
+    "source_config": "DocVQA",
+    "source_split": "validation",
+    "sample_seed": "full_validation_5349"
+  },
+  {
+    "id": "8127",
+    "questionId": "8127",
+    "docId": "2754",
+    "image_path": "data/docvqa_images/q8127_d2754.png",
+    "ucsf_document_id": "xtvn0020",
+    "ucsf_document_page_no": "2",
+    "topic": "form",
+    "source_dataset": "lmms-lab/DocVQA",
+    "source_config": "DocVQA",
+    "source_split": "validation",
+    "sample_seed": "full_validation_5349"
+  },
+  {
+    "id": "57431",
+    "questionId": "57431",
+    "docId": "4779",
+    "image_path": "data/docvqa_images/q57431_d4779.png",
+    "ucsf_document_id": "rnbx0223",
+    "ucsf_document_page_no": "208",
+    "topic": "Image/Photo",
+    "source_dataset": "lmms-lab/DocVQA",
+    "source_config": "DocVQA",
+    "source_split": "validation",
+    "sample_seed": "full_validation_5349"
+  },
+  {
+    "id": "64306",
+    "questionId": "64306",
+    "docId": "10149",
+    "image_path": "data/docvqa_images/q64306_d10149.png",
+    "ucsf_document_id": "lpjm0223",
+    "ucsf_document_page_no": "23",
+    "topic": "table/list",
+    "source_dataset": "lmms-lab/DocVQA",
+    "source_config": "DocVQA",
+    "source_split": "validation",
+    "sample_seed": "full_validation_5349"
+  },
+  {
+    "id": "64887",
+    "questionId": "64887",
+    "docId": "9754",
+    "image_path": "data/docvqa_images/q64887_d9754.png",
+    "ucsf_document_id": "szpg0227",
+    "ucsf_document_page_no": "9",
+    "topic": "table/list",
+    "source_dataset": "lmms-lab/DocVQA",
+    "source_config": "DocVQA",
+    "source_split": "validation",
+    "sample_seed": "full_validation_5349"
+  },
+  {
+    "id": "58680",
+    "questionId": "58680",
+    "docId": "5545",
+    "image_path": "data/docvqa_images/q58680_d5545.png",
+    "ucsf_document_id": "hhwh0078",
+    "ucsf_document_page_no": "1",
+    "topic": "table/list",
+    "source_dataset": "lmms-lab/DocVQA",
+    "source_config": "DocVQA",
+    "source_split": "validation",
+    "sample_seed": "full_validation_5349"
+  },
+  {
+    "id": "5287",
+    "questionId": "5287",
+    "docId": "1785",
+    "image_path": "data/docvqa_images/q5287_d1785.png",
+    "ucsf_document_id": "mtnh0227",
+    "ucsf_document_page_no": "10",
+    "topic": "form",
+    "source_dataset": "lmms-lab/DocVQA",
+    "source_config": "DocVQA",
+    "source_split": "validation",
+    "sample_seed": "full_validation_5349"
+  },
+  {
+    "id": "55471",
+    "questionId": "55471",
+    "docId": "4340",
+    "image_path": "data/docvqa_images/q55471_d4340.png",
+    "ucsf_document_id": "fsgj0223",
+    "ucsf_document_page_no": "96",
+    "topic": "free_text",
+    "source_dataset": "lmms-lab/DocVQA",
+    "source_config": "DocVQA",
+    "source_split": "validation",
+    "sample_seed": "full_validation_5349"
+  },
+  {
+    "id": "53095",
+    "questionId": "53095",
+    "docId": "296",
+    "image_path": "data/docvqa_images/q53095_d296.png",
+    "ucsf_document_id": "qhxj0037",
+    "ucsf_document_page_no": "3",
+    "topic": "layout",
+    "source_dataset": "lmms-lab/DocVQA",
+    "source_config": "DocVQA",
+    "source_split": "validation",
+    "sample_seed": "full_validation_5349"
+  },
+  {
+    "id": "53726",
+    "questionId": "53726",
+    "docId": "2008",
+    "image_path": "data/docvqa_images/q53726_d2008.png",
+    "ucsf_document_id": "hhnf0094",
+    "ucsf_document_page_no": "5",
+    "topic": "layout",
+    "source_dataset": "lmms-lab/DocVQA",
+    "source_config": "DocVQA",
+    "source_split": "validation",
+    "sample_seed": "full_validation_5349"
+  },
+  {
+    "id": "57321",
+    "questionId": "57321",
+    "docId": "4722",
+    "image_path": "data/docvqa_images/q57321_d4722.png",
+    "ucsf_document_id": "xybx0223",
+    "ucsf_document_page_no": "32",
+    "topic": "table/list",
+    "source_dataset": "lmms-lab/DocVQA",
+    "source_config": "DocVQA",
+    "source_split": "validation",
+    "sample_seed": "full_validation_5349"
+  },
+  {
+    "id": "26659",
+    "questionId": "26659",
+    "docId": "7470",
+    "image_path": "data/docvqa_images/q26659_d7470.png",
+    "ucsf_document_id": "lhmg0227",
+    "ucsf_document_page_no": "1",
+    "topic": "layout",
+    "source_dataset": "lmms-lab/DocVQA",
+    "source_config": "DocVQA",
+    "source_split": "validation",
+    "sample_seed": "full_validation_5349"
+  },
+  {
+    "id": "38920",
+    "questionId": "38920",
+    "docId": "11157",
+    "image_path": "data/docvqa_images/q38920_d11157.png",
+    "ucsf_document_id": "klnf0227",
+    "ucsf_document_page_no": "1",
+    "topic": "table/list|layout",
+    "source_dataset": "lmms-lab/DocVQA",
+    "source_config": "DocVQA",
+    "source_split": "validation",
+    "sample_seed": "full_validation_5349"
+  },
+  {
+    "id": "50837",
+    "questionId": "50837",
+    "docId": "14742",
+    "image_path": "data/docvqa_images/q50837_d14742.png",
+    "ucsf_document_id": "ysmc0228",
+    "ucsf_document_page_no": "4",
+    "topic": "table/list",
+    "source_dataset": "lmms-lab/DocVQA",
+    "source_config": "DocVQA",
+    "source_split": "validation",
+    "sample_seed": "full_validation_5349"
+  },
+  {
+    "id": "59615",
+    "questionId": "59615",
+    "docId": "6569",
+    "image_path": "data/docvqa_images/q59615_d6569.png",
+    "ucsf_document_id": "hnnp0227",
+    "ucsf_document_page_no": "45",
+    "topic": "handwritten|table/list|layout",
+    "source_dataset": "lmms-lab/DocVQA",
+    "source_config": "DocVQA",
+    "source_split": "validation",
+    "sample_seed": "full_validation_5349"
+  },
+  {
+    "id": "58687",
+    "questionId": "58687",
+    "docId": "5545",
+    "image_path": "data/docvqa_images/q58687_d5545.png",
+    "ucsf_document_id": "hhwh0078",
+    "ucsf_document_page_no": "1",
+    "topic": "table/list",
+    "source_dataset": "lmms-lab/DocVQA",
+    "source_config": "DocVQA",
+    "source_split": "validation",
+    "sample_seed": "full_validation_5349"
+  }
+]
@@ -0,0 +1,34 @@
+{
+  "benchmark": "LiveMathematicianBench",
+  "manifest_type": "id_split",
+  "source_repo": "LiveMathematicianBench/LiveMathematicianBench",
+  "source_repo_type": "dataset",
+  "source_url": "https://huggingface.co/datasets/LiveMathematicianBench/LiveMathematicianBench",
+  "source_revision": "b72450f6ce96c26158d64d945a5d31ef7727be41",
+  "source_files": [
+    "data/202511/qa_202511_final.json",
+    "data/202512/qa_202512_final.json",
+    "data/202601/qa_202601_final.json",
+    "data/202602/qa_202602_final.json"
+  ],
+  "split_mode": "ratio",
+  "split_ratio": "2:1:7",
+  "split_seed": 42,
+  "counts": {
+    "train": 35,
+    "val": 18,
+    "test": 124
+  },
+  "item_fields": [
+    "id",
+    "month",
+    "no",
+    "paper_link",
+    "source_file"
+  ],
+  "id_format": "<month>:<no>",
+  "notes": [
+    "This is an ID split manifest, not the full LiveMathematicianBench payload.",
+    "Materialize full split items from the official LiveMathematicianBench raw qa_*_final.json files before evaluation."
+  ]
+}
@@ -0,0 +1,870 @@
+[
+  {
+    "id": "202602:12",
+    "month": "202602",
+    "no": 12,
+    "paper_link": "http://arxiv.org/abs/2602.07171v1",
+    "source_file": "data/202602/qa_202602_final.json"
+  },
+  {
+    "id": "202601:3",
+    "month": "202601",
+    "no": 3,
+    "paper_link": "http://arxiv.org/abs/2601.01447v1",
+    "source_file": "data/202601/qa_202601_final.json"
+  },
+  {
+    "id": "202511:4",
+    "month": "202511",
+    "no": 4,
+    "paper_link": "http://arxiv.org/abs/2511.23123v1",
+    "source_file": "data/202511/qa_202511_final.json"
+  },
+  {
+    "id": "202601:20",
+    "month": "202601",
+    "no": 20,
+    "paper_link": "http://arxiv.org/abs/2601.13212v1",
+    "source_file": "data/202601/qa_202601_final.json"
+  },
+  {
+    "id": "202601:42",
+    "month": "202601",
+    "no": 42,
+    "paper_link": "http://arxiv.org/abs/2601.09348v1",
+    "source_file": "data/202601/qa_202601_final.json"
+  },
+  {
+    "id": "202512:38",
+    "month": "202512",
+    "no": 38,
+    "paper_link": "http://arxiv.org/abs/2512.19831v2",
+    "source_file": "data/202512/qa_202512_final.json"
+  },
+  {
+    "id": "202512:4",
+    "month": "202512",
+    "no": 4,
+    "paper_link": "http://arxiv.org/abs/2512.03141v1",
+    "source_file": "data/202512/qa_202512_final.json"
+  },
+  {
+    "id": "202602:4",
+    "month": "202602",
+    "no": 4,
+    "paper_link": "http://arxiv.org/abs/2602.14368v1",
+    "source_file": "data/202602/qa_202602_final.json"
+  },
+  {
+    "id": "202511:15",
+    "month": "202511",
+    "no": 15,
+    "paper_link": "http://arxiv.org/abs/2511.17325v1",
+    "source_file": "data/202511/qa_202511_final.json"
+  },
+  {
+    "id": "202602:32",
+    "month": "202602",
+    "no": 32,
+    "paper_link": "http://arxiv.org/abs/2602.14817v1",
+    "source_file": "data/202602/qa_202602_final.json"
+  },
+  {
+    "id": "202512:51",
+    "month": "202512",
+    "no": 51,
+    "paper_link": "http://arxiv.org/abs/2512.14581v1",
+    "source_file": "data/202512/qa_202512_final.json"
+  },
+  {
+    "id": "202512:26",
+    "month": "202512",
+    "no": 26,
+    "paper_link": "http://arxiv.org/abs/2512.19586v1",
+    "source_file": "data/202512/qa_202512_final.json"
+  },
+  {
+    "id": "202601:13",
+    "month": "202601",
+    "no": 13,
+    "paper_link": "http://arxiv.org/abs/2601.10017v1",
+    "source_file": "data/202601/qa_202601_final.json"
+  },
+  {
+    "id": "202602:1",
+    "month": "202602",
+    "no": 1,
+    "paper_link": "http://arxiv.org/abs/2602.23137v1",
+    "source_file": "data/202602/qa_202602_final.json"
+  },
+  {
+    "id": "202511:18",
+    "month": "202511",
+    "no": 18,
+    "paper_link": "http://arxiv.org/abs/2511.10795v1",
+    "source_file": "data/202511/qa_202511_final.json"
+  },
+  {
+    "id": "202512:5",
+    "month": "202512",
+    "no": 5,
+    "paper_link": "http://arxiv.org/abs/2512.00348v1",
+    "source_file": "data/202512/qa_202512_final.json"
+  },
+  {
+    "id": "202511:19",
+    "month": "202511",
+    "no": 19,
+    "paper_link": "http://arxiv.org/abs/2511.06951v1",
+    "source_file": "data/202511/qa_202511_final.json"
+  },
+  {
+    "id": "202602:40",
+    "month": "202602",
+    "no": 40,
+    "paper_link": "http://arxiv.org/abs/2602.20462v1",
+    "source_file": "data/202602/qa_202602_final.json"
+  },
+  {
+    "id": "202602:29",
+    "month": "202602",
+    "no": 29,
+    "paper_link": "http://arxiv.org/abs/2602.10676v1",
+    "source_file": "data/202602/qa_202602_final.json"
+  },
+  {
+    "id": "202512:35",
+    "month": "202512",
+    "no": 35,
+    "paper_link": "http://arxiv.org/abs/2512.08840v1",
+    "source_file": "data/202512/qa_202512_final.json"
+  },
+  {
+    "id": "202512:48",
+    "month": "202512",
+    "no": 48,
+    "paper_link": "http://arxiv.org/abs/2512.03482v1",
+    "source_file": "data/202512/qa_202512_final.json"
+  },
+  {
+    "id": "202512:52",
+    "month": "202512",
+    "no": 52,
+    "paper_link": "http://arxiv.org/abs/2512.11246v1",
+    "source_file": "data/202512/qa_202512_final.json"
+  },
+  {
+    "id": "202512:44",
+    "month": "202512",
+    "no": 44,
+    "paper_link": "http://arxiv.org/abs/2512.10385v1",
+    "source_file": "data/202512/qa_202512_final.json"
+  },
+  {
+    "id": "202511:28",
+    "month": "202511",
+    "no": 28,
+    "paper_link": "http://arxiv.org/abs/2511.03812v1",
+    "source_file": "data/202511/qa_202511_final.json"
+  },
+  {
+    "id": "202601:43",
+    "month": "202601",
+    "no": 43,
+    "paper_link": "http://arxiv.org/abs/2601.22555v1",
+    "source_file": "data/202601/qa_202601_final.json"
+  },
+  {
+    "id": "202602:9",
+    "month": "202602",
+    "no": 9,
+    "paper_link": "http://arxiv.org/abs/2602.19882v1",
+    "source_file": "data/202602/qa_202602_final.json"
+  },
+  {
+    "id": "202512:23",
+    "month": "202512",
+    "no": 23,
+    "paper_link": "http://arxiv.org/abs/2512.09180v1",
+    "source_file": "data/202512/qa_202512_final.json"
+  },
+  {
+    "id": "202602:21",
+    "month": "202602",
+    "no": 21,
+    "paper_link": "http://arxiv.org/abs/2602.10509v1",
+    "source_file": "data/202602/qa_202602_final.json"
+  },
+  {
+    "id": "202511:5",
+    "month": "202511",
+    "no": 5,
+    "paper_link": "http://arxiv.org/abs/2511.20164v1",
+    "source_file": "data/202511/qa_202511_final.json"
+  },
+  {
+    "id": "202601:35",
+    "month": "202601",
+    "no": 35,
+    "paper_link": "http://arxiv.org/abs/2601.15606v1",
+    "source_file": "data/202601/qa_202601_final.json"
+  },
+  {
+    "id": "202602:50",
+    "month": "202602",
+    "no": 50,
+    "paper_link": "http://arxiv.org/abs/2602.05652v1",
+    "source_file": "data/202602/qa_202602_final.json"
+  },
+  {
+    "id": "202512:13",
+    "month": "202512",
+    "no": 13,
+    "paper_link": "http://arxiv.org/abs/2512.22861v1",
+    "source_file": "data/202512/qa_202512_final.json"
+  },
+  {
+    "id": "202602:49",
+    "month": "202602",
+    "no": 49,
+    "paper_link": "http://arxiv.org/abs/2602.07167v1",
+    "source_file": "data/202602/qa_202602_final.json"
+  },
+  {
+    "id": "202602:18",
+    "month": "202602",
+    "no": 18,
+    "paper_link": "http://arxiv.org/abs/2602.20124v1",
+    "source_file": "data/202602/qa_202602_final.json"
+  },
+  {
+    "id": "202601:15",
+    "month": "202601",
+    "no": 15,
+    "paper_link": "http://arxiv.org/abs/2601.05327v1",
+    "source_file": "data/202601/qa_202601_final.json"
+  },
+  {
+    "id": "202601:21",
+    "month": "202601",
+    "no": 21,
+    "paper_link": "http://arxiv.org/abs/2601.04994v1",
+    "source_file": "data/202601/qa_202601_final.json"
+  },
+  {
+    "id": "202601:32",
+    "month": "202601",
+    "no": 32,
+    "paper_link": "http://arxiv.org/abs/2601.09183v1",
+    "source_file": "data/202601/qa_202601_final.json"
+  },
+  {
+    "id": "202602:34",
+    "month": "202602",
+    "no": 34,
+    "paper_link": "http://arxiv.org/abs/2602.21118v1",
+    "source_file": "data/202602/qa_202602_final.json"
+  },
+  {
+    "id": "202602:20",
+    "month": "202602",
+    "no": 20,
+    "paper_link": "http://arxiv.org/abs/2602.16506v1",
+    "source_file": "data/202602/qa_202602_final.json"
+  },
+  {
+    "id": "202602:5",
+    "month": "202602",
+    "no": 5,
+    "paper_link": "http://arxiv.org/abs/2602.09806v1",
+    "source_file": "data/202602/qa_202602_final.json"
+  },
+  {
+    "id": "202512:40",
+    "month": "202512",
+    "no": 40,
+    "paper_link": "http://arxiv.org/abs/2512.16535v1",
+    "source_file": "data/202512/qa_202512_final.json"
+  },
+  {
+    "id": "202511:22",
+    "month": "202511",
+    "no": 22,
+    "paper_link": "http://arxiv.org/abs/2511.07607v2",
+    "source_file": "data/202511/qa_202511_final.json"
+  },
+  {
+    "id": "202601:36",
+    "month": "202601",
+    "no": 36,
+    "paper_link": "http://arxiv.org/abs/2601.12457v1",
+    "source_file": "data/202601/qa_202601_final.json"
+  },
+  {
+    "id": "202512:49",
+    "month": "202512",
+    "no": 49,
+    "paper_link": "http://arxiv.org/abs/2512.21565v1",
+    "source_file": "data/202512/qa_202512_final.json"
+  },
+  {
+    "id": "202511:10",
+    "month": "202511",
+    "no": 10,
+    "paper_link": "http://arxiv.org/abs/2511.06484v1",
+    "source_file": "data/202511/qa_202511_final.json"
+  },
+  {
+    "id": "202601:2",
+    "month": "202601",
+    "no": 2,
+    "paper_link": "http://arxiv.org/abs/2601.07068v4",
+    "source_file": "data/202601/qa_202601_final.json"
+  },
+  {
+    "id": "202602:19",
+    "month": "202602",
+    "no": 19,
+    "paper_link": "http://arxiv.org/abs/2602.18179v1",
+    "source_file": "data/202602/qa_202602_final.json"
+  },
+  {
+    "id": "202601:9",
+    "month": "202601",
+    "no": 9,
+    "paper_link": "http://arxiv.org/abs/2601.17765v1",
+    "source_file": "data/202601/qa_202601_final.json"
+  },
+  {
+    "id": "202512:6",
+    "month": "202512",
+    "no": 6,
+    "paper_link": "http://arxiv.org/abs/2512.23079v1",
+    "source_file": "data/202512/qa_202512_final.json"
+  },
+  {
+    "id": "202601:5",
+    "month": "202601",
+    "no": 5,
+    "paper_link": "http://arxiv.org/abs/2601.20344v1",
+    "source_file": "data/202601/qa_202601_final.json"
+  },
+  {
+    "id": "202602:14",
+    "month": "202602",
+    "no": 14,
+    "paper_link": "http://arxiv.org/abs/2602.09177v2",
+    "source_file": "data/202602/qa_202602_final.json"
+  },
+  {
+    "id": "202512:17",
+    "month": "202512",
+    "no": 17,
+    "paper_link": "http://arxiv.org/abs/2512.11657v2",
+    "source_file": "data/202512/qa_202512_final.json"
+  },
+  {
+    "id": "202512:19",
+    "month": "202512",
+    "no": 19,
+    "paper_link": "http://arxiv.org/abs/2512.16655v2",
+    "source_file": "data/202512/qa_202512_final.json"
+  },
+  {
+    "id": "202602:33",
+    "month": "202602",
+    "no": 33,
+    "paper_link": "http://arxiv.org/abs/2602.13734v1",
+    "source_file": "data/202602/qa_202602_final.json"
+  },
+  {
+    "id": "202512:18",
+    "month": "202512",
+    "no": 18,
+    "paper_link": "http://arxiv.org/abs/2512.22960v1",
+    "source_file": "data/202512/qa_202512_final.json"
+  },
+  {
+    "id": "202601:26",
+    "month": "202601",
+    "no": 26,
+    "paper_link": "http://arxiv.org/abs/2601.06814v1",
+    "source_file": "data/202601/qa_202601_final.json"
+  },
+  {
+    "id": "202601:1",
+    "month": "202601",
+    "no": 1,
+    "paper_link": "http://arxiv.org/abs/2601.18276v1",
+    "source_file": "data/202601/qa_202601_final.json"
+  },
+  {
+    "id": "202512:30",
+    "month": "202512",
+    "no": 30,
+    "paper_link": "http://arxiv.org/abs/2512.07260v1",
+    "source_file": "data/202512/qa_202512_final.json"
+  },
+  {
+    "id": "202602:44",
+    "month": "202602",
+    "no": 44,
+    "paper_link": "http://arxiv.org/abs/2602.01138v1",
+    "source_file": "data/202602/qa_202602_final.json"
+  },
+  {
+    "id": "202512:20",
+    "month": "202512",
+    "no": 20,
+    "paper_link": "http://arxiv.org/abs/2512.14575v1",
+    "source_file": "data/202512/qa_202512_final.json"
+  },
+  {
+    "id": "202511:13",
+    "month": "202511",
+    "no": 13,
+    "paper_link": "http://arxiv.org/abs/2511.16910v1",
+    "source_file": "data/202511/qa_202511_final.json"
+  },
+  {
+    "id": "202601:30",
+    "month": "202601",
+    "no": 30,
+    "paper_link": "http://arxiv.org/abs/2601.12140v1",
+    "source_file": "data/202601/qa_202601_final.json"
+  },
+  {
+    "id": "202601:40",
+    "month": "202601",
+    "no": 40,
+    "paper_link": "http://arxiv.org/abs/2601.05146v1",
+    "source_file": "data/202601/qa_202601_final.json"
+  },
+  {
+    "id": "202601:29",
+    "month": "202601",
+    "no": 29,
+    "paper_link": "http://arxiv.org/abs/2601.12846v1",
+    "source_file": "data/202601/qa_202601_final.json"
+  },
+  {
+    "id": "202511:11",
+    "month": "202511",
+    "no": 11,
+    "paper_link": "http://arxiv.org/abs/2511.17548v1",
+    "source_file": "data/202511/qa_202511_final.json"
+  },
+  {
+    "id": "202512:9",
+    "month": "202512",
+    "no": 9,
+    "paper_link": "http://arxiv.org/abs/2512.08817v1",
+    "source_file": "data/202512/qa_202512_final.json"
+  },
+  {
+    "id": "202601:18",
+    "month": "202601",
+    "no": 18,
+    "paper_link": "http://arxiv.org/abs/2601.01797v1",
+    "source_file": "data/202601/qa_202601_final.json"
+  },
+  {
+    "id": "202512:1",
+    "month": "202512",
+    "no": 1,
+    "paper_link": "http://arxiv.org/abs/2512.20055v1",
+    "source_file": "data/202512/qa_202512_final.json"
+  },
+  {
+    "id": "202601:4",
+    "month": "202601",
+    "no": 4,
+    "paper_link": "http://arxiv.org/abs/2601.21223v1",
+    "source_file": "data/202601/qa_202601_final.json"
+  },
+  {
+    "id": "202511:6",
+    "month": "202511",
+    "no": 6,
+    "paper_link": "http://arxiv.org/abs/2511.14959v1",
+    "source_file": "data/202511/qa_202511_final.json"
+  },
+  {
+    "id": "202602:38",
+    "month": "202602",
+    "no": 38,
+    "paper_link": "http://arxiv.org/abs/2602.08398v2",
+    "source_file": "data/202602/qa_202602_final.json"
+  },
+  {
+    "id": "202601:10",
+    "month": "202601",
+    "no": 10,
+    "paper_link": "http://arxiv.org/abs/2601.15524v1",
+    "source_file": "data/202601/qa_202601_final.json"
+  },
+  {
+    "id": "202602:11",
+    "month": "202602",
+    "no": 11,
+    "paper_link": "http://arxiv.org/abs/2602.11045v1",
+    "source_file": "data/202602/qa_202602_final.json"
+  },
+  {
+    "id": "202512:45",
+    "month": "202512",
+    "no": 45,
+    "paper_link": "http://arxiv.org/abs/2512.08395v1",
+    "source_file": "data/202512/qa_202512_final.json"
+  },
+  {
+    "id": "202601:12",
+    "month": "202601",
+    "no": 12,
+    "paper_link": "http://arxiv.org/abs/2601.11877v1",
+    "source_file": "data/202601/qa_202601_final.json"
+  },
+  {
+    "id": "202512:47",
+    "month": "202512",
+    "no": 47,
+    "paper_link": "http://arxiv.org/abs/2512.09683v1",
+    "source_file": "data/202512/qa_202512_final.json"
+  },
+  {
+    "id": "202511:21",
+    "month": "202511",
+    "no": 21,
+    "paper_link": "http://arxiv.org/abs/2511.21288v1",
+    "source_file": "data/202511/qa_202511_final.json"
+  },
+  {
+    "id": "202601:16",
+    "month": "202601",
+    "no": 16,
+    "paper_link": "http://arxiv.org/abs/2601.05008v2",
+    "source_file": "data/202601/qa_202601_final.json"
+  },
+  {
+    "id": "202512:3",
+    "month": "202512",
+    "no": 3,
+    "paper_link": "http://arxiv.org/abs/2512.13450v1",
+    "source_file": "data/202512/qa_202512_final.json"
+  },
+  {
+    "id": "202601:37",
+    "month": "202601",
+    "no": 37,
+    "paper_link": "http://arxiv.org/abs/2601.09443v1",
+    "source_file": "data/202601/qa_202601_final.json"
+  },
+  {
+    "id": "202511:12",
+    "month": "202511",
+    "no": 12,
+    "paper_link": "http://arxiv.org/abs/2511.04978v1",
+    "source_file": "data/202511/qa_202511_final.json"
+  },
+  {
+    "id": "202512:39",
+    "month": "202512",
+    "no": 39,
+    "paper_link": "http://arxiv.org/abs/2512.19003v1",
+    "source_file": "data/202512/qa_202512_final.json"
+  },
+  {
+    "id": "202601:8",
+    "month": "202601",
+    "no": 8,
+    "paper_link": "http://arxiv.org/abs/2601.19754v1",
+    "source_file": "data/202601/qa_202601_final.json"
+  },
+  {
+    "id": "202601:11",
+    "month": "202601",
+    "no": 11,
+    "paper_link": "http://arxiv.org/abs/2601.13552v1",
+    "source_file": "data/202601/qa_202601_final.json"
+  },
+  {
+    "id": "202511:25",
+    "month": "202511",
+    "no": 25,
+    "paper_link": "http://arxiv.org/abs/2511.10548v1",
+    "source_file": "data/202511/qa_202511_final.json"
+  },
+  {
+    "id": "202601:17",
+    "month": "202601",
+    "no": 17,
+    "paper_link": "http://arxiv.org/abs/2601.02655v2",
+    "source_file": "data/202601/qa_202601_final.json"
+  },
+  {
+    "id": "202602:36",
+    "month": "202602",
+    "no": 36,
+    "paper_link": "http://arxiv.org/abs/2602.13001v1",
+    "source_file": "data/202602/qa_202602_final.json"
+  },
+  {
+    "id": "202602:43",
+    "month": "202602",
+    "no": 43,
+    "paper_link": "http://arxiv.org/abs/2602.06897v1",
+    "source_file": "data/202602/qa_202602_final.json"
+  },
+  {
+    "id": "202601:6",
+    "month": "202601",
+    "no": 6,
+    "paper_link": "http://arxiv.org/abs/2601.04747v1",
+    "source_file": "data/202601/qa_202601_final.json"
+  },
+  {
+    "id": "202602:35",
+    "month": "202602",
+    "no": 35,
+    "paper_link": "http://arxiv.org/abs/2602.20938v1",
+    "source_file": "data/202602/qa_202602_final.json"
+  },
+  {
+    "id": "202512:11",
+    "month": "202512",
+    "no": 11,
+    "paper_link": "http://arxiv.org/abs/2512.03294v1",
+    "source_file": "data/202512/qa_202512_final.json"
+  },
+  {
+    "id": "202602:23",
+    "month": "202602",
+    "no": 23,
+    "paper_link": "http://arxiv.org/abs/2602.09201v1",
+    "source_file": "data/202602/qa_202602_final.json"
+  },
+  {
+    "id": "202601:7",
+    "month": "202601",
+    "no": 7,
+    "paper_link": "http://arxiv.org/abs/2601.02859v1",
+    "source_file": "data/202601/qa_202601_final.json"
+  },
+  {
+    "id": "202602:39",
+    "month": "202602",
+    "no": 39,
+    "paper_link": "http://arxiv.org/abs/2602.21659v1",
+    "source_file": "data/202602/qa_202602_final.json"
+  },
+  {
+    "id": "202512:12",
+    "month": "202512",
+    "no": 12,
+    "paper_link": "http://arxiv.org/abs/2512.00690v3",
+    "source_file": "data/202512/qa_202512_final.json"
+  },
+  {
+    "id": "202511:2",
+    "month": "202511",
+    "no": 2,
+    "paper_link": "http://arxiv.org/abs/2511.19681v1",
+    "source_file": "data/202511/qa_202511_final.json"
+  },
+  {
+    "id": "202512:43",
+    "month": "202512",
+    "no": 43,
+    "paper_link": "http://arxiv.org/abs/2512.10820v1",
+    "source_file": "data/202512/qa_202512_final.json"
+  },
+  {
+    "id": "202602:24",
+    "month": "202602",
+    "no": 24,
+    "paper_link": "http://arxiv.org/abs/2602.08680v1",
+    "source_file": "data/202602/qa_202602_final.json"
+  },
+  {
+    "id": "202601:34",
+    "month": "202601",
+    "no": 34,
+    "paper_link": "http://arxiv.org/abs/2601.07318v1",
+    "source_file": "data/202601/qa_202601_final.json"
+  },
+  {
+    "id": "202512:28",
+    "month": "202512",
+    "no": 28,
+    "paper_link": "http://arxiv.org/abs/2512.11294v1",
+    "source_file": "data/202512/qa_202512_final.json"
+  },
+  {
+    "id": "202601:27",
+    "month": "202601",
+    "no": 27,
+    "paper_link": "http://arxiv.org/abs/2601.05692v1",
+    "source_file": "data/202601/qa_202601_final.json"
+  },
+  {
+    "id": "202602:42",
+    "month": "202602",
+    "no": 42,
+    "paper_link": "http://arxiv.org/abs/2602.09749v1",
+    "source_file": "data/202602/qa_202602_final.json"
+  },
+  {
+    "id": "202512:22",
+    "month": "202512",
+    "no": 22,
+    "paper_link": "http://arxiv.org/abs/2512.11658v1",
+    "source_file": "data/202512/qa_202512_final.json"
+  },
+  {
+    "id": "202602:17",
+    "month": "202602",
+    "no": 17,
+    "paper_link": "http://arxiv.org/abs/2602.22504v1",
+    "source_file": "data/202602/qa_202602_final.json"
+  },
+  {
+    "id": "202602:48",
+    "month": "202602",
+    "no": 48,
+    "paper_link": "http://arxiv.org/abs/2602.08760v1",
+    "source_file": "data/202602/qa_202602_final.json"
+  },
+  {
+    "id": "202602:28",
+    "month": "202602",
+    "no": 28,
+    "paper_link": "http://arxiv.org/abs/2602.11595v1",
+    "source_file": "data/202602/qa_202602_final.json"
+  },
+  {
+    "id": "202602:3",
+    "month": "202602",
+    "no": 3,
+    "paper_link": "http://arxiv.org/abs/2602.17369v1",
+    "source_file": "data/202602/qa_202602_final.json"
+  },
+  {
+    "id": "202512:31",
+    "month": "202512",
+    "no": 31,
+    "paper_link": "http://arxiv.org/abs/2512.23668v1",
+    "source_file": "data/202512/qa_202512_final.json"
+  },
+  {
+    "id": "202512:27",
+    "month": "202512",
+    "no": 27,
+    "paper_link": "http://arxiv.org/abs/2512.16505v1",
+    "source_file": "data/202512/qa_202512_final.json"
+  },
+  {
+    "id": "202511:24",
+    "month": "202511",
+    "no": 24,
+    "paper_link": "http://arxiv.org/abs/2511.12549v2",
+    "source_file": "data/202511/qa_202511_final.json"
+  },
+  {
+    "id": "202511:8",
+    "month": "202511",
+    "no": 8,
+    "paper_link": "http://arxiv.org/abs/2511.12657v1",
+    "source_file": "data/202511/qa_202511_final.json"
+  },
+  {
+    "id": "202511:9",
+    "month": "202511",
+    "no": 9,
+    "paper_link": "http://arxiv.org/abs/2511.09015v1",
+    "source_file": "data/202511/qa_202511_final.json"
+  },
+  {
+    "id": "202601:28",
+    "month": "202601",
+    "no": 28,
+    "paper_link": "http://arxiv.org/abs/2601.14825v1",
+    "source_file": "data/202601/qa_202601_final.json"
+  },
+  {
+    "id": "202602:25",
+    "month": "202602",
+    "no": 25,
+    "paper_link": "http://arxiv.org/abs/2602.16048v3",
+    "source_file": "data/202602/qa_202602_final.json"
+  },
+  {
+    "id": "202511:23",
+    "month": "202511",
+    "no": 23,
+    "paper_link": "http://arxiv.org/abs/2511.06595v1",
+    "source_file": "data/202511/qa_202511_final.json"
+  },
+  {
+    "id": "202602:13",
+    "month": "202602",
+    "no": 13,
+    "paper_link": "http://arxiv.org/abs/2602.12261v1",
+    "source_file": "data/202602/qa_202602_final.json"
+  },
+  {
+    "id": "202511:27",
+    "month": "202511",
+    "no": 27,
+    "paper_link": "http://arxiv.org/abs/2511.04407v1",
+    "source_file": "data/202511/qa_202511_final.json"
+  },
+  {
+    "id": "202512:7",
+    "month": "202512",
+    "no": 7,
+    "paper_link": "http://arxiv.org/abs/2512.09490v1",
+    "source_file": "data/202512/qa_202512_final.json"
+  },
+  {
+    "id": "202512:29",
+    "month": "202512",
+    "no": 29,
+    "paper_link": "http://arxiv.org/abs/2512.08562v1",
+    "source_file": "data/202512/qa_202512_final.json"
+  },
+  {
+    "id": "202512:34",
+    "month": "202512",
+    "no": 34,
+    "paper_link": "http://arxiv.org/abs/2512.09598v1",
+    "source_file": "data/202512/qa_202512_final.json"
+  },
+  {
+    "id": "202512:42",
+    "month": "202512",
+    "no": 42,
+    "paper_link": "http://arxiv.org/abs/2512.10845v1",
+    "source_file": "data/202512/qa_202512_final.json"
+  },
+  {
+    "id": "202511:7",
+    "month": "202511",
+    "no": 7,
+    "paper_link": "http://arxiv.org/abs/2511.13976v1",
+    "source_file": "data/202511/qa_202511_final.json"
+  },
+  {
+    "id": "202511:29",
+    "month": "202511",
+    "no": 29,
+    "paper_link": "http://arxiv.org/abs/2511.03722v1",
+    "source_file": "data/202511/qa_202511_final.json"
+  },
+  {
+    "id": "202602:37",
+    "month": "202602",
+    "no": 37,
+    "paper_link": "http://arxiv.org/abs/2602.08644v1",
+    "source_file": "data/202602/qa_202602_final.json"
+  }
+]
@@ -0,0 +1,247 @@
+[
+  {
+    "id": "202602:22",
+    "month": "202602",
+    "no": 22,
+    "paper_link": "http://arxiv.org/abs/2602.10700v1",
+    "source_file": "data/202602/qa_202602_final.json"
+  },
+  {
+    "id": "202512:8",
+    "month": "202512",
+    "no": 8,
+    "paper_link": "http://arxiv.org/abs/2512.08863v1",
+    "source_file": "data/202512/qa_202512_final.json"
+  },
+  {
+    "id": "202511:16",
+    "month": "202511",
+    "no": 16,
+    "paper_link": "http://arxiv.org/abs/2511.15668v1",
+    "source_file": "data/202511/qa_202511_final.json"
+  },
+  {
+    "id": "202601:44",
+    "month": "202601",
+    "no": 44,
+    "paper_link": "http://arxiv.org/abs/2601.21267v1",
+    "source_file": "data/202601/qa_202601_final.json"
+  },
+  {
+    "id": "202511:14",
+    "month": "202511",
+    "no": 14,
+    "paper_link": "http://arxiv.org/abs/2511.13447v1",
+    "source_file": "data/202511/qa_202511_final.json"
+  },
+  {
+    "id": "202602:30",
+    "month": "202602",
+    "no": 30,
+    "paper_link": "http://arxiv.org/abs/2602.16692v1",
+    "source_file": "data/202602/qa_202602_final.json"
+  },
+  {
+    "id": "202602:2",
+    "month": "202602",
+    "no": 2,
+    "paper_link": "http://arxiv.org/abs/2602.22933v1",
+    "source_file": "data/202602/qa_202602_final.json"
+  },
+  {
+    "id": "202601:41",
+    "month": "202601",
+    "no": 41,
+    "paper_link": "http://arxiv.org/abs/2601.01164v1",
+    "source_file": "data/202601/qa_202601_final.json"
+  },
+  {
+    "id": "202601:23",
+    "month": "202601",
+    "no": 23,
+    "paper_link": "http://arxiv.org/abs/2601.02528v1",
+    "source_file": "data/202601/qa_202601_final.json"
+  },
+  {
+    "id": "202511:20",
+    "month": "202511",
+    "no": 20,
+    "paper_link": "http://arxiv.org/abs/2511.02963v1",
+    "source_file": "data/202511/qa_202511_final.json"
+  },
+  {
+    "id": "202601:22",
+    "month": "202601",
+    "no": 22,
+    "paper_link": "http://arxiv.org/abs/2601.03984v1",
+    "source_file": "data/202601/qa_202601_final.json"
+  },
+  {
+    "id": "202512:14",
+    "month": "202512",
+    "no": 14,
+    "paper_link": "http://arxiv.org/abs/2512.22459v1",
+    "source_file": "data/202512/qa_202512_final.json"
+  },
+  {
+    "id": "202511:26",
+    "month": "202511",
+    "no": 26,
+    "paper_link": "http://arxiv.org/abs/2511.07817v1",
+    "source_file": "data/202511/qa_202511_final.json"
+  },
+  {
+    "id": "202511:3",
+    "month": "202511",
+    "no": 3,
+    "paper_link": "http://arxiv.org/abs/2511.11409v1",
+    "source_file": "data/202511/qa_202511_final.json"
+  },
+  {
+    "id": "202601:33",
+    "month": "202601",
+    "no": 33,
+    "paper_link": "http://arxiv.org/abs/2601.07747v1",
+    "source_file": "data/202601/qa_202601_final.json"
+  },
+  {
+    "id": "202602:7",
+    "month": "202602",
+    "no": 7,
+    "paper_link": "http://arxiv.org/abs/2602.22912v1",
+    "source_file": "data/202602/qa_202602_final.json"
+  },
+  {
+    "id": "202602:27",
+    "month": "202602",
+    "no": 27,
+    "paper_link": "http://arxiv.org/abs/2602.13968v1",
+    "source_file": "data/202602/qa_202602_final.json"
+  },
+  {
+    "id": "202602:31",
+    "month": "202602",
+    "no": 31,
+    "paper_link": "http://arxiv.org/abs/2602.15528v1",
+    "source_file": "data/202602/qa_202602_final.json"
+  },
+  {
+    "id": "202602:41",
+    "month": "202602",
+    "no": 41,
+    "paper_link": "http://arxiv.org/abs/2602.10707v1",
+    "source_file": "data/202602/qa_202602_final.json"
+  },
+  {
+    "id": "202512:25",
+    "month": "202512",
+    "no": 25,
+    "paper_link": "http://arxiv.org/abs/2512.04531v1",
+    "source_file": "data/202512/qa_202512_final.json"
+  },
+  {
+    "id": "202602:10",
+    "month": "202602",
+    "no": 10,
+    "paper_link": "http://arxiv.org/abs/2602.17863v1",
+    "source_file": "data/202602/qa_202602_final.json"
+  },
+  {
+    "id": "202602:16",
+    "month": "202602",
+    "no": 16,
+    "paper_link": "http://arxiv.org/abs/2602.02723v1",
+    "source_file": "data/202602/qa_202602_final.json"
+  },
+  {
+    "id": "202512:16",
+    "month": "202512",
+    "no": 16,
+    "paper_link": "http://arxiv.org/abs/2512.11601v1",
+    "source_file": "data/202512/qa_202512_final.json"
+  },
+  {
+    "id": "202512:2",
+    "month": "202512",
+    "no": 2,
+    "paper_link": "http://arxiv.org/abs/2512.16120v1",
+    "source_file": "data/202512/qa_202512_final.json"
+  },
+  {
+    "id": "202512:24",
+    "month": "202512",
+    "no": 24,
+    "paper_link": "http://arxiv.org/abs/2512.08391v1",
+    "source_file": "data/202512/qa_202512_final.json"
+  },
+  {
+    "id": "202512:32",
+    "month": "202512",
+    "no": 32,
+    "paper_link": "http://arxiv.org/abs/2512.23224v1",
+    "source_file": "data/202512/qa_202512_final.json"
+  },
+  {
+    "id": "202602:47",
+    "month": "202602",
+    "no": 47,
+    "paper_link": "http://arxiv.org/abs/2602.10391v1",
+    "source_file": "data/202602/qa_202602_final.json"
+  },
+  {
+    "id": "202602:46",
+    "month": "202602",
+    "no": 46,
+    "paper_link": "http://arxiv.org/abs/2602.13727v2",
+    "source_file": "data/202602/qa_202602_final.json"
+  },
+  {
+    "id": "202512:21",
+    "month": "202512",
+    "no": 21,
+    "paper_link": "http://arxiv.org/abs/2512.12835v1",
+    "source_file": "data/202512/qa_202512_final.json"
+  },
+  {
+    "id": "202512:33",
+    "month": "202512",
+    "no": 33,
+    "paper_link": "http://arxiv.org/abs/2512.19500v1",
+    "source_file": "data/202512/qa_202512_final.json"
+  },
+  {
+    "id": "202602:45",
+    "month": "202602",
+    "no": 45,
+    "paper_link": "http://arxiv.org/abs/2602.23912v1",
+    "source_file": "data/202602/qa_202602_final.json"
+  },
+  {
+    "id": "202602:26",
+    "month": "202602",
+    "no": 26,
+    "paper_link": "http://arxiv.org/abs/2602.14658v1",
+    "source_file": "data/202602/qa_202602_final.json"
+  },
+  {
+    "id": "202512:41",
+    "month": "202512",
+    "no": 41,
+    "paper_link": "http://arxiv.org/abs/2512.15177v1",
+    "source_file": "data/202512/qa_202512_final.json"
+  },
+  {
+    "id": "202601:38",
+    "month": "202601",
+    "no": 38,
+    "paper_link": "http://arxiv.org/abs/2601.07817v1",
+    "source_file": "data/202601/qa_202601_final.json"
+  },
+  {
+    "id": "202601:14",
+    "month": "202601",
+    "no": 14,
+    "paper_link": "http://arxiv.org/abs/2601.08704v1",
+    "source_file": "data/202601/qa_202601_final.json"
+  }
+]
@@ -0,0 +1,128 @@
+[
+  {
+    "id": "202602:8",
+    "month": "202602",
+    "no": 8,
+    "paper_link": "http://arxiv.org/abs/2602.19529v1",
+    "source_file": "data/202602/qa_202602_final.json"
+  },
+  {
+    "id": "202512:50",
+    "month": "202512",
+    "no": 50,
+    "paper_link": "http://arxiv.org/abs/2512.15277v1",
+    "source_file": "data/202512/qa_202512_final.json"
+  },
+  {
+    "id": "202512:36",
+    "month": "202512",
+    "no": 36,
+    "paper_link": "http://arxiv.org/abs/2512.06696v1",
+    "source_file": "data/202512/qa_202512_final.json"
+  },
+  {
+    "id": "202511:1",
+    "month": "202511",
+    "no": 1,
+    "paper_link": "http://arxiv.org/abs/2511.04651v1",
+    "source_file": "data/202511/qa_202511_final.json"
+  },
+  {
+    "id": "202601:31",
+    "month": "202601",
+    "no": 31,
+    "paper_link": "http://arxiv.org/abs/2601.10298v1",
+    "source_file": "data/202601/qa_202601_final.json"
+  },
+  {
+    "id": "202511:17",
+    "month": "202511",
+    "no": 17,
+    "paper_link": "http://arxiv.org/abs/2511.13215v1",
+    "source_file": "data/202511/qa_202511_final.json"
+  },
+  {
+    "id": "202512:37",
+    "month": "202512",
+    "no": 37,
+    "paper_link": "http://arxiv.org/abs/2512.20498v1",
+    "source_file": "data/202512/qa_202512_final.json"
+  },
+  {
+    "id": "202601:39",
+    "month": "202601",
+    "no": 39,
+    "paper_link": "http://arxiv.org/abs/2601.06601v2",
+    "source_file": "data/202601/qa_202601_final.json"
+  },
+  {
+    "id": "202601:25",
+    "month": "202601",
+    "no": 25,
+    "paper_link": "http://arxiv.org/abs/2601.10996v3",
+    "source_file": "data/202601/qa_202601_final.json"
+  },
+  {
+    "id": "202601:24",
+    "month": "202601",
+    "no": 24,
+    "paper_link": "http://arxiv.org/abs/2601.12250v1",
+    "source_file": "data/202601/qa_202601_final.json"
+  },
+  {
+    "id": "202601:45",
+    "month": "202601",
+    "no": 45,
+    "paper_link": "http://arxiv.org/abs/2601.12113v1",
+    "source_file": "data/202601/qa_202601_final.json"
+  },
+  {
+    "id": "202601:19",
+    "month": "202601",
+    "no": 19,
+    "paper_link": "http://arxiv.org/abs/2601.00779v1",
+    "source_file": "data/202601/qa_202601_final.json"
+  },
+  {
+    "id": "202512:10",
+    "month": "202512",
+    "no": 10,
+    "paper_link": "http://arxiv.org/abs/2512.07073v2",
+    "source_file": "data/202512/qa_202512_final.json"
+  },
+  {
+    "id": "202601:46",
+    "month": "202601",
+    "no": 46,
+    "paper_link": "http://arxiv.org/abs/2601.07793v2",
+    "source_file": "data/202601/qa_202601_final.json"
+  },
+  {
+    "id": "202512:15",
+    "month": "202512",
+    "no": 15,
+    "paper_link": "http://arxiv.org/abs/2512.16165v1",
+    "source_file": "data/202512/qa_202512_final.json"
+  },
+  {
+    "id": "202602:15",
+    "month": "202602",
+    "no": 15,
+    "paper_link": "http://arxiv.org/abs/2602.05303v1",
+    "source_file": "data/202602/qa_202602_final.json"
+  },
+  {
+    "id": "202602:6",
+    "month": "202602",
+    "no": 6,
+    "paper_link": "http://arxiv.org/abs/2602.01571v1",
+    "source_file": "data/202602/qa_202602_final.json"
+  },
+  {
+    "id": "202512:46",
+    "month": "202512",
+    "no": 46,
+    "paper_link": "http://arxiv.org/abs/2512.05945v1",
+    "source_file": "data/202512/qa_202512_final.json"
+  }
+]
@@ -0,0 +1,27 @@
+{
+  "benchmark": "OfficeQA",
+  "manifest_type": "id_split",
+  "source_repo": "databricks/officeqa",
+  "source_repo_type": "dataset",
+  "source_url": "https://huggingface.co/datasets/databricks/officeqa",
+  "source_revision": "8ecbf18d3833daf4750a903d14963e4c4c1d4cd8",
+  "source_file": "officeqa_full.csv",
+  "source_split_name": "officeqa_split",
+  "counts": {
+    "train": 50,
+    "val": 24,
+    "test": 172
+  },
+  "item_fields": [
+    "id",
+    "uid",
+    "category",
+    "source_files",
+    "source_docs",
+    "source_split"
+  ],
+  "notes": [
+    "This is a split manifest, not the full OfficeQA payload.",
+    "The official OfficeQA CSV is gated on Hugging Face; materialization requires authorized access."
+  ]
+}
@@ -0,0 +1,402 @@
+[
+  {
+    "id": "UID0002",
+    "uid": "UID0002",
+    "category": "easy",
+    "source_files": "treasury_bulletin_1944_01.txt",
+    "source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/january-1944-6565?page=18",
+    "source_split": "train"
+  },
+  {
+    "id": "UID0007",
+    "uid": "UID0007",
+    "category": "hard",
+    "source_files": "treasury_bulletin_1950_02.txt",
+    "source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/february-1950-6637?page=15",
+    "source_split": "train"
+  },
+  {
+    "id": "UID0014",
+    "uid": "UID0014",
+    "category": "easy",
+    "source_files": "treasury_bulletin_1942_07.txt\r\ntreasury_bulletin_2001_12.txt",
+    "source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/july-1942-6547?page=76\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/december-2001-7108?page=17&deep=true",
+    "source_split": "train"
+  },
+  {
+    "id": "UID0017",
+    "uid": "UID0017",
+    "category": "hard",
+    "source_files": "treasury_bulletin_1982_08.txt",
+    "source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/august-1982-7028?page=13",
+    "source_split": "train"
+  },
+  {
+    "id": "UID0018",
+    "uid": "UID0018",
+    "category": "hard",
+    "source_files": "treasury_bulletin_1985_03.txt\r\ntreasury_bulletin_1986_03.txt\r\ntreasury_bulletin_1987_03.txt\r\ntreasury_bulletin_1988_03.txt",
+    "source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/march-1985-7040?page=22\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/march-1986-7045?page=26\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/march-1987-7049?page=24\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/march-1988-7052?page=36",
+    "source_split": "train"
+  },
+  {
+    "id": "UID0019",
+    "uid": "UID0019",
+    "category": "hard",
+    "source_files": "treasury_bulletin_2016_09.txt",
+    "source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/september-2016-533966?page=54\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/september-2016-533966?page=58",
+    "source_split": "train"
+  },
+  {
+    "id": "UID0026",
+    "uid": "UID0026",
+    "category": "easy",
+    "source_files": "treasury_bulletin_1963_01.txt\r\ntreasury_bulletin_1962_01.txt",
+    "source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/january-1963-6793?page=88\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/january-1962-6781?page=82&deep=true",
+    "source_split": "train"
+  },
+  {
+    "id": "UID0028",
+    "uid": "UID0028",
+    "category": "hard",
+    "source_files": "treasury_bulletin_1970_06.txt\r\ntreasury_bulletin_1964_12.txt",
+    "source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/june-1970-6882?page=89&deep=true\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/december-1964-6816?page=25&deep=true",
+    "source_split": "train"
+  },
+  {
+    "id": "UID0030",
+    "uid": "UID0030",
+    "category": "hard",
+    "source_files": "treasury_bulletin_1990_09.txt",
+    "source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/september-1990-7062?page=19&deep=true",
+    "source_split": "train"
+  },
+  {
+    "id": "UID0031",
+    "uid": "UID0031",
+    "category": "hard",
+    "source_files": "treasury_bulletin_1992_03.txt",
+    "source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/march-1992-7068?page=158&deep=true",
+    "source_split": "train"
+  },
+  {
+    "id": "UID0033",
+    "uid": "UID0033",
+    "category": "easy",
+    "source_files": "treasury_bulletin_1977_04.txt",
+    "source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/april-1977-6964?page=9",
+    "source_split": "train"
+  },
+  {
+    "id": "UID0034",
+    "uid": "UID0034",
+    "category": "easy",
+    "source_files": "treasury_bulletin_1992_06.txt",
+    "source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/june-1992-7069?page=32",
+    "source_split": "train"
+  },
+  {
+    "id": "UID0044",
+    "uid": "UID0044",
+    "category": "hard",
+    "source_files": "treasury_bulletin_1939_04.txt",
+    "source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/april-1939-6506?page=61",
+    "source_split": "train"
+  },
+  {
+    "id": "UID0046",
+    "uid": "UID0046",
+    "category": "easy",
+    "source_files": "treasury_bulletin_1988_09.txt",
+    "source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/september-1988-7054?page=37",
+    "source_split": "train"
+  },
+  {
+    "id": "UID0049",
+    "uid": "UID0049",
+    "category": "hard",
+    "source_files": "treasury_bulletin_1942_02.txt",
+    "source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/february-1942-6542?page=19&deep=true",
+    "source_split": "train"
+  },
+  {
+    "id": "UID0056",
+    "uid": "UID0056",
+    "category": "hard",
+    "source_files": "treasury_bulletin_1991_09.txt",
+    "source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/september-1991-7066?page=30&deep=true",
+    "source_split": "train"
+  },
+  {
+    "id": "UID0063",
+    "uid": "UID0063",
+    "category": "easy",
+    "source_files": "treasury_bulletin_1990_06.txt",
+    "source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/june-1990-7061?page=127",
+    "source_split": "train"
+  },
+  {
+    "id": "UID0065",
+    "uid": "UID0065",
+    "category": "hard",
+    "source_files": "treasury_bulletin_1998_06.txt\r\ntreasury_bulletin_1995_12.txt",
+    "source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/june-1998-7094?page=7\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/december-1995-7084?page=16",
+    "source_split": "train"
+  },
+  {
+    "id": "UID0073",
+    "uid": "UID0073",
+    "category": "hard",
+    "source_files": "treasury_bulletin_1982_03.txt",
+    "source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/march-1982-7023?page=24",
+    "source_split": "train"
+  },
+  {
+    "id": "UID0079",
+    "uid": "UID0079",
+    "category": "easy",
+    "source_files": "treasury_bulletin_2011_12.txt\r\ntreasury_bulletin_2013_12.txt\r\ntreasury_bulletin_2015_12.txt\r\ntreasury_bulletin_2017_12.txt\r\ntreasury_bulletin_2019_12.txt",
+    "source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/december-2011-7149?page=25\r\n \r\n https://fraser.stlouisfed.org/title/treasury-bulletin-407/december-2013-7155?page=24\r\n \r\n https://fraser.stlouisfed.org/title/treasury-bulletin-407/december-2015-519209?page=23\r\n \r\n https://fraser.stlouisfed.org/title/treasury-bulletin-407/december-2017-575188?page=23\r\n \r\n https://fraser.stlouisfed.org/title/treasury-bulletin-407/december-2019-584842?page=22",
+    "source_split": "train"
+  },
+  {
+    "id": "UID0083",
+    "uid": "UID0083",
+    "category": "hard",
+    "source_files": "treasury_bulletin_1981_12.txt",
+    "source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/december-1981-7020?page=24",
+    "source_split": "train"
+  },
+  {
+    "id": "UID0085",
+    "uid": "UID0085",
+    "category": "hard",
+    "source_files": "treasury_bulletin_2019_12.txt\r\ntreasury_bulletin_2018_12.txt",
+    "source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/december-2019-584842?page=23\r\n\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/december-2018-581283?page=22",
+    "source_split": "train"
+  },
+  {
+    "id": "UID0087",
+    "uid": "UID0087",
+    "category": "easy",
+    "source_files": "treasury_bulletin_2013_12.txt",
+    "source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/december-2013-7155?page=17",
+    "source_split": "train"
+  },
+  {
+    "id": "UID0092",
+    "uid": "UID0092",
+    "category": "easy",
+    "source_files": "treasury_bulletin_1987_12.txt\r\ntreasury_bulletin_1992_12.txt",
+    "source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/december-1987-7051?page=69\r\n\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/december-1992-7071?page=84",
+    "source_split": "train"
+  },
+  {
+    "id": "UID0098",
+    "uid": "UID0098",
+    "category": "easy",
+    "source_files": "treasury_bulletin_2020_12.txt\r\ntreasury_bulletin_2024_12.txt",
+    "source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/december-2020-598551?page=21\r\n \r\n https://fraser.stlouisfed.org/title/treasury-bulletin-407/december-2024-679984?page=22",
+    "source_split": "train"
+  },
+  {
+    "id": "UID0101",
+    "uid": "UID0101",
+    "category": "hard",
+    "source_files": "treasury_bulletin_2011_12.txt\r\ntreasury_bulletin_2019_12.txt",
+    "source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/december-2011-7149?page=25\r\n \r\n https://fraser.stlouisfed.org/title/treasury-bulletin-407/december-2019-584842?page=22",
+    "source_split": "train"
+  },
+  {
+    "id": "UID0110",
+    "uid": "UID0110",
+    "category": "hard",
+    "source_files": "treasury_bulletin_2020_03.txt\r\ntreasury_bulletin_2016_03.txt",
+    "source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/march-2020-587316?page=10\r\n \r\n https://fraser.stlouisfed.org/title/treasury-bulletin-407/march-2016-527290?page=9",
+    "source_split": "train"
+  },
+  {
+    "id": "UID0115",
+    "uid": "UID0115",
+    "category": "easy",
+    "source_files": "treasury_bulletin_1980_02.txt\r\ntreasury_bulletin_1981_02.txt\r\ntreasury_bulletin_1982_02.txt",
+    "source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/february-1980-6998?page=27\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/february-1981-7010?page=38\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/february-1982-7022?page=31",
+    "source_split": "train"
+  },
+  {
+    "id": "UID0122",
+    "uid": "UID0122",
+    "category": "hard",
+    "source_files": "treasury_bulletin_2001_03.txt\r\ntreasury_bulletin_2002_03.txt\r\ntreasury_bulletin_2003_03.txt",
+    "source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/march-2001-7105?page=112\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/march-2002-7110?page=115\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/march-2003-7109?page=113",
+    "source_split": "train"
+  },
+  {
+    "id": "UID0123",
+    "uid": "UID0123",
+    "category": "hard",
+    "source_files": "treasury_bulletin_1941_12.txt",
+    "source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/december-1941-6540?page=91",
+    "source_split": "train"
+  },
+  {
+    "id": "UID0133",
+    "uid": "UID0133",
+    "category": "hard",
+    "source_files": "treasury_bulletin_2004_09.txt\r\ntreasury_bulletin_2013_12.txt",
+    "source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/september-2004-7119?page=63\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/december-2013-7155?page=68",
+    "source_split": "train"
+  },
+  {
+    "id": "UID0141",
+    "uid": "UID0141",
+    "category": "easy",
+    "source_files": "treasury_bulletin_1962_04.txt\r\ntreasury_bulletin_1963_04.txt\r\ntreasury_bulletin_1964_04.txt",
+    "source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/april-1962-6784?page=75\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/april-1963-6796?page=79\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/april-1964-6808?page=82",
+    "source_split": "train"
+  },
+  {
+    "id": "UID0144",
+    "uid": "UID0144",
+    "category": "easy",
+    "source_files": "treasury_bulletin_1980_11.txt\r\ntreasury_bulletin_1981_11.txt",
+    "source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/november-1980-7007?page=76\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/november-1981-7019?page=67",
+    "source_split": "train"
+  },
+  {
+    "id": "UID0145",
+    "uid": "UID0145",
+    "category": "easy",
+    "source_files": "treasury_bulletin_1943_01.txt",
+    "source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/january-1943-6553?page=41",
+    "source_split": "train"
+  },
+  {
+    "id": "UID0150",
+    "uid": "UID0150",
+    "category": "hard",
+    "source_files": "treasury_bulletin_1972_04.txt\r\ntreasury_bulletin_1973_04.txt\r\ntreasury_bulletin_1974_04.txt\r\ntreasury_bulletin_1975_04.txt\r\ntreasury_bulletin_1976_04.txt",
+    "source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/april-1972-6905?page=89\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/april-1973-6916?page=91\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/april-1974-6927?page=88\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/april-1975-6940?page=88\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/april-1976-6952?page=104",
+    "source_split": "train"
+  },
+  {
+    "id": "UID0151",
+    "uid": "UID0151",
+    "category": "easy",
+    "source_files": "treasury_bulletin_1953_03.txt",
+    "source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/march-1953-6674?page=54",
+    "source_split": "train"
+  },
+  {
+    "id": "UID0162",
+    "uid": "UID0162",
+    "category": "easy",
+    "source_files": "treasury_bulletin_2011_06.txt\r\ntreasury_bulletin_2012_06.txt\r\ntreasury_bulletin_2013_06.txt",
+    "source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/june-2011-7129?page=105\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/june-2012-7151?page=105\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/june-2013-7153?page=104",
+    "source_split": "train"
+  },
+  {
+    "id": "UID0163",
+    "uid": "UID0163",
+    "category": "easy",
+    "source_files": "treasury_bulletin_1981_12.txt",
+    "source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/december-1981-7020?page=28",
+    "source_split": "train"
+  },
+  {
+    "id": "UID0165",
+    "uid": "UID0165",
+    "category": "hard",
+    "source_files": "treasury_bulletin_2010_03.txt",
+    "source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/march-2010-7143?page=49",
+    "source_split": "train"
+  },
+  {
+    "id": "UID0166",
+    "uid": "UID0166",
+    "category": "easy",
+    "source_files": "treasury_bulletin_1943_03.txt\r\ntreasury_bulletin_1944_03.txt\r\ntreasury_bulletin_1945_03.txt",
+    "source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/march-1943-6555?page=68\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/march-1944-6567?page=83\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/march-1945-6577?page=71",
+    "source_split": "train"
+  },
+  {
+    "id": "UID0169",
+    "uid": "UID0169",
+    "category": "hard",
+    "source_files": "treasury_bulletin_1982_03.txt",
+    "source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/march-1982-7023?page=73",
+    "source_split": "train"
+  },
+  {
+    "id": "UID0189",
+    "uid": "UID0189",
+    "category": "easy",
+    "source_files": "treasury_bulletin_1970_08.txt\r\ntreasury_bulletin_1970_09.txt",
+    "source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/august-1970-6884?page=70\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/september-1970-6885?page=70",
+    "source_split": "train"
+  },
+  {
+    "id": "UID0195",
+    "uid": "UID0195",
+    "category": "hard",
+    "source_files": "treasury_bulletin_1956_08.txt",
+    "source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/august-1956-6715?page=59",
+    "source_split": "train"
+  },
+  {
+    "id": "UID0202",
+    "uid": "UID0202",
+    "category": "easy",
+    "source_files": "treasury_bulletin_1939_07.txt\r\ntreasury_bulletin_1939_08.txt\r\ntreasury_bulletin_1939_09.txt\r\ntreasury_bulletin_1939_10.txt",
+    "source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/july-1939-6509?page=99\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/august-1939-6510?page=107\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/september-1939-6511?page=60\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/october-1939-6520?page=62",
+    "source_split": "train"
+  },
+  {
+    "id": "UID0212",
+    "uid": "UID0212",
+    "category": "hard",
+    "source_files": "treasury_bulletin_1964_01.txt",
+    "source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/january-1964-6805?page=99",
+    "source_split": "train"
+  },
+  {
+    "id": "UID0222",
+    "uid": "UID0222",
+    "category": "hard",
+    "source_files": "treasury_bulletin_2001_06.txt\r\ntreasury_bulletin_2006_06.txt",
+    "source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/june-2001-7106?page=50\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/june-2006-7126?page=50",
+    "source_split": "train"
+  },
+  {
+    "id": "UID0228",
+    "uid": "UID0228",
+    "category": "hard",
+    "source_files": "treasury_bulletin_1956_03.txt\r\ntreasury_bulletin_1956_04.txt",
+    "source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/march-1956-6710?page=22\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/april-1956-6711?page=22",
+    "source_split": "train"
+  },
+  {
+    "id": "UID0229",
+    "uid": "UID0229",
+    "category": "easy",
+    "source_files": "treasury_bulletin_2005_03.txt\r\ntreasury_bulletin_2006_03.txt\r\ntreasury_bulletin_2007_03.txt\r\ntreasury_bulletin_2008_03.txt",
+    "source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/march-2005-7121?page=109\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/march-2006-7125?page=106\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/march-2007-7130?page=109\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/march-2008-7134?page=107",
+    "source_split": "train"
+  },
+  {
+    "id": "UID0238",
+    "uid": "UID0238",
+    "category": "hard",
+    "source_files": "treasury_bulletin_1982_03.txt",
+    "source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/march-1982-7023?page=44",
+    "source_split": "train"
+  },
+  {
+    "id": "UID0241",
+    "uid": "UID0241",
+    "category": "easy",
+    "source_files": "treasury_bulletin_1963_06.txt",
+    "source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/june-1963-6798?page=13",
+    "source_split": "train"
+  }
+]
@@ -0,0 +1,194 @@
+[
+  {
+    "id": "UID0001",
+    "uid": "UID0001",
+    "category": "hard",
+    "source_files": "treasury_bulletin_1941_01.txt",
+    "source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/january-1941-6529?page=15",
+    "source_split": "val"
+  },
+  {
+    "id": "UID0027",
+    "uid": "UID0027",
+    "category": "hard",
+    "source_files": "treasury_bulletin_1970_06.txt",
+    "source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/june-1970-6882?page=89&deep=true",
+    "source_split": "val"
+  },
+  {
+    "id": "UID0039",
+    "uid": "UID0039",
+    "category": "hard",
+    "source_files": "treasury_bulletin_2004_03.txt",
+    "source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/march-2004-7117?page=20\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/march-2004-7117?page=21&deep=true",
+    "source_split": "val"
+  },
+  {
+    "id": "UID0041",
+    "uid": "UID0041",
+    "category": "easy",
+    "source_files": "treasury_bulletin_1970_10.txt",
+    "source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/october-1970-6886?page=35",
+    "source_split": "val"
+  },
+  {
+    "id": "UID0052",
+    "uid": "UID0052",
+    "category": "easy",
+    "source_files": "treasury_bulletin_2000_06.txt",
+    "source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/june-2000-7102?page=56",
+    "source_split": "val"
+  },
+  {
+    "id": "UID0070",
+    "uid": "UID0070",
+    "category": "easy",
+    "source_files": "treasury_bulletin_1939_01.txt\r\ntreasury_bulletin_1939_02.txt\r\ntreasury_bulletin_1939_03.txt\r\ntreasury_bulletin_1939_04.txt\r\ntreasury_bulletin_1939_05.txt\r\ntreasury_bulletin_1939_06.txt\r\ntreasury_bulletin_1939_07.txt\r\ntreasury_bulletin_1939_08.txt\r\ntreasury_bulletin_1939_09.txt\r\ntreasury_bulletin_1939_10.txt\r\ntreasury_bulletin_1939_11.txt\r\ntreasury_bulletin_1939_12.txt",
+    "source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/january-1939-6518?page=81\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/february-1939-6505?page=111\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/march-1939-6519?page=117\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/april-1939-6506?page=95\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/may-1939-6507?page=109\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/june-1939-6508?page=117\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/july-1939-6509?page=109\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/august-1939-6510?page=117\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/september-1939-6511?page=66&deep=true \r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/october-1939-6520?page=68\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/november-1939-6512?page=70\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/december-1939-6513?page=72",
+    "source_split": "val"
+  },
+  {
+    "id": "UID0072",
+    "uid": "UID0072",
+    "category": "easy",
+    "source_files": "treasury_bulletin_2011_12.txt\r\ntreasury_bulletin_2016_12.txt",
+    "source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/december-2011-7149?page=58\r\n\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/december-2016-535293?page=57",
+    "source_split": "val"
+  },
+  {
+    "id": "UID0086",
+    "uid": "UID0086",
+    "category": "hard",
+    "source_files": "treasury_bulletin_2022_12.txt",
+    "source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/december-2022-627778?page=86",
+    "source_split": "val"
+  },
+  {
+    "id": "UID0091",
+    "uid": "UID0091",
+    "category": "easy",
+    "source_files": "treasury_bulletin_1940_12.txt\r\ntreasury_bulletin_1941_12.txt",
+    "source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/december-1940-6528?page=21\r\n\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/december-1941-6540?page=64",
+    "source_split": "val"
+  },
+  {
+    "id": "UID0109",
+    "uid": "UID0109",
+    "category": "hard",
+    "source_files": "treasury_bulletin_2015_12.txt\r\ntreasury_bulletin_2020_12.txt",
+    "source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/december-2015-519209?page=21\r\n \r\n https://fraser.stlouisfed.org/title/treasury-bulletin-407/december-2020-598551?page=24",
+    "source_split": "val"
+  },
+  {
+    "id": "UID0142",
+    "uid": "UID0142",
+    "category": "easy",
+    "source_files": "treasury_bulletin_1944_03.txt",
+    "source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/march-1944-6567?page=93",
+    "source_split": "val"
+  },
+  {
+    "id": "UID0154",
+    "uid": "UID0154",
+    "category": "hard",
+    "source_files": "treasury_bulletin_1977_03.txt\r\ntreasury_bulletin_1978_03.txt",
+    "source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/march-1977-6963?page=83\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/march-1978-6975?page=84",
+    "source_split": "val"
+  },
+  {
+    "id": "UID0159",
+    "uid": "UID0159",
+    "category": "easy",
+    "source_files": "treasury_bulletin_2000_09.txt",
+    "source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/september-2000-7103?page=109",
+    "source_split": "val"
+  },
+  {
+    "id": "UID0161",
+    "uid": "UID0161",
+    "category": "hard",
+    "source_files": "treasury_bulletin_1980_03.txt\r\ntreasury_bulletin_1985_03.txt",
+    "source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/march-1980-6999?page=88\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/march-1985-7040?page=48",
+    "source_split": "val"
+  },
+  {
+    "id": "UID0170",
+    "uid": "UID0170",
+    "category": "hard",
+    "source_files": "treasury_bulletin_1960_03.txt",
+    "source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/march-1960-6759?page=64",
+    "source_split": "val"
+  },
+  {
+    "id": "UID0190",
+    "uid": "UID0190",
+    "category": "hard",
+    "source_files": "treasury_bulletin_1939_10.txt\r\ntreasury_bulletin_1939_11.txt",
+    "source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/october-1939-6520?page=14\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/november-1939-6512?page=14",
+    "source_split": "val"
+  },
+  {
+    "id": "UID0213",
+    "uid": "UID0213",
+    "category": "hard",
+    "source_files": "treasury_bulletin_1947_04.txt\r\ntreasury_bulletin_1948_04.txt",
+    "source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/april-1947-6603?page=28\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/april-1948-6615?page=18",
+    "source_split": "val"
+  },
+  {
+    "id": "UID0217",
+    "uid": "UID0217",
+    "category": "easy",
+    "source_files": "treasury_bulletin_1963_10.txt",
+    "source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/october-1963-6802?page=15",
+    "source_split": "val"
+  },
+  {
+    "id": "UID0220",
+    "uid": "UID0220",
+    "category": "hard",
+    "source_files": "treasury_bulletin_1939_02.txt",
+    "source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/february-1939-6505?page=25",
+    "source_split": "val"
+  },
+  {
+    "id": "UID0233",
+    "uid": "UID0233",
+    "category": "hard",
+    "source_files": "treasury_bulletin_1948_04.txt\r\ntreasury_bulletin_1958_04.txt",
+    "source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/april-1948-6615?page=42\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/april-1958-6735?page=54",
+    "source_split": "val"
+  },
+  {
+    "id": "UID0234",
+    "uid": "UID0234",
+    "category": "easy",
+    "source_files": "treasury_bulletin_1958_01.txt\r\ntreasury_bulletin_1958_02.txt",
+    "source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/january-1958-6732?page=28\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/february-1958-6733?page=32",
+    "source_split": "val"
+  },
+  {
+    "id": "UID0235",
+    "uid": "UID0235",
+    "category": "easy",
+    "source_files": "treasury_bulletin_1948_04.txt\r\ntreasury_bulletin_1948_05.txt",
+    "source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/april-1948-6615?page=27\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/may-1948-6616?page=27",
+    "source_split": "val"
+  },
+  {
+    "id": "UID0239",
+    "uid": "UID0239",
+    "category": "easy",
+    "source_files": "treasury_bulletin_1953_01.txt\r\ntreasury_bulletin_1954_01.txt",
+    "source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/january-1953-6672?page=62\r\nhttp://fraser.stlouisfed.org/title/treasury-bulletin-407/january-1954-6684?page=51",
+    "source_split": "val"
+  },
+  {
+    "id": "UID0240",
+    "uid": "UID0240",
+    "category": "hard",
+    "source_files": "treasury_bulletin_1957_12.txt",
+    "source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/december-1957-6731?page=26",
+    "source_split": "val"
+  }
+]
@@ -0,0 +1,21 @@
+{
+  "benchmark": "SearchQA",
+  "manifest_type": "id_split",
+  "source_repo": "lucadiliello/searchqa",
+  "source_repo_type": "dataset",
+  "source_url": "https://huggingface.co/datasets/lucadiliello/searchqa",
+  "source_id_field": "key",
+  "counts": {
+    "train": 400,
+    "val": 200,
+    "test": 1400
+  },
+  "item_fields": [
+    "id"
+  ],
+  "notes": [
+    "This is a split manifest, not the full SearchQA payload.",
+    "Materialize full split items from lucadiliello/searchqa before evaluation.",
+    "The IDs in items.json exactly match the key field in lucadiliello/searchqa."
+  ]
+}
@@ -0,0 +1,24 @@
+{
+  "benchmark": "SpreadsheetBench",
+  "manifest_type": "id_split",
+  "source_repo": "KAKA22/SpreadsheetBench",
+  "source_repo_type": "dataset",
+  "source_url": "https://huggingface.co/datasets/KAKA22/SpreadsheetBench",
+  "source_revision": "ab0b742b0fc95b946f212d80ac7771b5531272e4",
+  "source_file": "spreadsheetbench_verified_400.tar.gz",
+  "source_split_name": "spreadsheetbench_split",
+  "counts": {
+    "train": 80,
+    "val": 40,
+    "test": 280
+  },
+  "item_fields": [
+    "id",
+    "spreadsheet_path",
+    "instruction_type"
+  ],
+  "notes": [
+    "This is a split manifest, not the full SpreadsheetBench payload.",
+    "Materialize full task JSON rows plus spreadsheet files from SpreadsheetBench Verified 400 before evaluation."
+  ]
+}
@@ -0,0 +1,402 @@
+[
+  {
+    "id": "32438",
+    "spreadsheet_path": "spreadsheet/32438",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "398-14",
+    "spreadsheet_path": "spreadsheet/398-14",
+    "instruction_type": "Sheet-Level Manipulation"
+  },
+  {
+    "id": "47766",
+    "spreadsheet_path": "spreadsheet/47766",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "48365",
+    "spreadsheet_path": "spreadsheet/48365",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "32255",
+    "spreadsheet_path": "spreadsheet/32255",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "10747",
+    "spreadsheet_path": "spreadsheet/10747",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "50916",
+    "spreadsheet_path": "spreadsheet/50916",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "577-40",
+    "spreadsheet_path": "spreadsheet/577-40",
+    "instruction_type": "Sheet-Level Manipulation"
+  },
+  {
+    "id": "35742",
+    "spreadsheet_path": "spreadsheet/35742",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "46121",
+    "spreadsheet_path": "spreadsheet/46121",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "51090",
+    "spreadsheet_path": "spreadsheet/51090",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "51249",
+    "spreadsheet_path": "spreadsheet/51249",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "82-30",
+    "spreadsheet_path": "spreadsheet/82-30",
+    "instruction_type": "Sheet-Level Manipulation"
+  },
+  {
+    "id": "56274",
+    "spreadsheet_path": "spreadsheet/56274",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "57445",
+    "spreadsheet_path": "spreadsheet/57445",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "46646",
+    "spreadsheet_path": "spreadsheet/46646",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "105-24",
+    "spreadsheet_path": "spreadsheet/105-24",
+    "instruction_type": "Sheet-Level Manipulation"
+  },
+  {
+    "id": "6239",
+    "spreadsheet_path": "spreadsheet/6239",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "414-20",
+    "spreadsheet_path": "spreadsheet/414-20",
+    "instruction_type": "Sheet-Level Manipulation"
+  },
+  {
+    "id": "165-23",
+    "spreadsheet_path": "spreadsheet/165-23",
+    "instruction_type": "Sheet-Level Manipulation"
+  },
+  {
+    "id": "40892",
+    "spreadsheet_path": "spreadsheet/40892",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "48745",
+    "spreadsheet_path": "spreadsheet/48745",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "32612",
+    "spreadsheet_path": "spreadsheet/32612",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "325-44",
+    "spreadsheet_path": "spreadsheet/325-44",
+    "instruction_type": "Sheet-Level Manipulation"
+  },
+  {
+    "id": "262-17",
+    "spreadsheet_path": "spreadsheet/262-17",
+    "instruction_type": "Sheet-Level Manipulation"
+  },
+  {
+    "id": "141-20",
+    "spreadsheet_path": "spreadsheet/141-20",
+    "instruction_type": "Sheet-Level Manipulation"
+  },
+  {
+    "id": "52216",
+    "spreadsheet_path": "spreadsheet/52216",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "22-47",
+    "spreadsheet_path": "spreadsheet/22-47",
+    "instruction_type": "Sheet-Level Manipulation"
+  },
+  {
+    "id": "55421",
+    "spreadsheet_path": "spreadsheet/55421",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "56427",
+    "spreadsheet_path": "spreadsheet/56427",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "36097",
+    "spreadsheet_path": "spreadsheet/36097",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "32902",
+    "spreadsheet_path": "spreadsheet/32902",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "32023",
+    "spreadsheet_path": "spreadsheet/32023",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "1818",
+    "spreadsheet_path": "spreadsheet/1818",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "170-13",
+    "spreadsheet_path": "spreadsheet/170-13",
+    "instruction_type": "Sheet-Level Manipulation"
+  },
+  {
+    "id": "66-24",
+    "spreadsheet_path": "spreadsheet/66-24",
+    "instruction_type": "Sheet-Level Manipulation"
+  },
+  {
+    "id": "58949",
+    "spreadsheet_path": "spreadsheet/58949",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "42354",
+    "spreadsheet_path": "spreadsheet/42354",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "194-19",
+    "spreadsheet_path": "spreadsheet/194-19",
+    "instruction_type": "Sheet-Level Manipulation"
+  },
+  {
+    "id": "31915",
+    "spreadsheet_path": "spreadsheet/31915",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "58499",
+    "spreadsheet_path": "spreadsheet/58499",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "45372",
+    "spreadsheet_path": "spreadsheet/45372",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "11842",
+    "spreadsheet_path": "spreadsheet/11842",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "57558",
+    "spreadsheet_path": "spreadsheet/57558",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "472-15",
+    "spreadsheet_path": "spreadsheet/472-15",
+    "instruction_type": "Sheet-Level Manipulation"
+  },
+  {
+    "id": "55060",
+    "spreadsheet_path": "spreadsheet/55060",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "31011",
+    "spreadsheet_path": "spreadsheet/31011",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "408-39",
+    "spreadsheet_path": "spreadsheet/408-39",
+    "instruction_type": "Sheet-Level Manipulation"
+  },
+  {
+    "id": "54085",
+    "spreadsheet_path": "spreadsheet/54085",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "39903",
+    "spreadsheet_path": "spreadsheet/39903",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "48983",
+    "spreadsheet_path": "spreadsheet/48983",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "108-24",
+    "spreadsheet_path": "spreadsheet/108-24",
+    "instruction_type": "Sheet-Level Manipulation"
+  },
+  {
+    "id": "58484",
+    "spreadsheet_path": "spreadsheet/58484",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "118-50",
+    "spreadsheet_path": "spreadsheet/118-50",
+    "instruction_type": "Sheet-Level Manipulation"
+  },
+  {
+    "id": "10452",
+    "spreadsheet_path": "spreadsheet/10452",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "39931",
+    "spreadsheet_path": "spreadsheet/39931",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "3413",
+    "spreadsheet_path": "spreadsheet/3413",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "247-24",
+    "spreadsheet_path": "spreadsheet/247-24",
+    "instruction_type": "Sheet-Level Manipulation"
+  },
+  {
+    "id": "56786",
+    "spreadsheet_path": "spreadsheet/56786",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "55965",
+    "spreadsheet_path": "spreadsheet/55965",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "379-36",
+    "spreadsheet_path": "spreadsheet/379-36",
+    "instruction_type": "Sheet-Level Manipulation"
+  },
+  {
+    "id": "58109",
+    "spreadsheet_path": "spreadsheet/58109",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "433-47",
+    "spreadsheet_path": "spreadsheet/433-47",
+    "instruction_type": "Sheet-Level Manipulation"
+  },
+  {
+    "id": "192-22",
+    "spreadsheet_path": "spreadsheet/192-22",
+    "instruction_type": "Sheet-Level Manipulation"
+  },
+  {
+    "id": "49333",
+    "spreadsheet_path": "spreadsheet/49333",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "493-18",
+    "spreadsheet_path": "spreadsheet/493-18",
+    "instruction_type": "Sheet-Level Manipulation"
+  },
+  {
+    "id": "54638",
+    "spreadsheet_path": "spreadsheet/54638",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "34033",
+    "spreadsheet_path": "spreadsheet/34033",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "30930",
+    "spreadsheet_path": "spreadsheet/30930",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "585-41",
+    "spreadsheet_path": "spreadsheet/585-41",
+    "instruction_type": "Sheet-Level Manipulation"
+  },
+  {
+    "id": "32337",
+    "spreadsheet_path": "spreadsheet/32337",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "55427",
+    "spreadsheet_path": "spreadsheet/55427",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "263-1",
+    "spreadsheet_path": "spreadsheet/263-1",
+    "instruction_type": "Sheet-Level Manipulation"
+  },
+  {
+    "id": "254-34",
+    "spreadsheet_path": "spreadsheet/254-34",
+    "instruction_type": "Sheet-Level Manipulation"
+  },
+  {
+    "id": "57113",
+    "spreadsheet_path": "spreadsheet/57113",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "57743",
+    "spreadsheet_path": "spreadsheet/57743",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "43589",
+    "spreadsheet_path": "spreadsheet/43589",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "250-20",
+    "spreadsheet_path": "spreadsheet/250-20",
+    "instruction_type": "Sheet-Level Manipulation"
+  },
+  {
+    "id": "48080",
+    "spreadsheet_path": "spreadsheet/48080",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "370-43",
+    "spreadsheet_path": "spreadsheet/370-43",
+    "instruction_type": "Sheet-Level Manipulation"
+  }
+]
@@ -0,0 +1,202 @@
+[
+  {
+    "id": "45635",
+    "spreadsheet_path": "spreadsheet/45635",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "560-12",
+    "spreadsheet_path": "spreadsheet/560-12",
+    "instruction_type": "Sheet-Level Manipulation"
+  },
+  {
+    "id": "55049",
+    "spreadsheet_path": "spreadsheet/55049",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "9569",
+    "spreadsheet_path": "spreadsheet/9569",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "7902",
+    "spreadsheet_path": "spreadsheet/7902",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "227-40",
+    "spreadsheet_path": "spreadsheet/227-40",
+    "instruction_type": "Sheet-Level Manipulation"
+  },
+  {
+    "id": "463-17",
+    "spreadsheet_path": "spreadsheet/463-17",
+    "instruction_type": "Sheet-Level Manipulation"
+  },
+  {
+    "id": "54144",
+    "spreadsheet_path": "spreadsheet/54144",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "80-42",
+    "spreadsheet_path": "spreadsheet/80-42",
+    "instruction_type": "Sheet-Level Manipulation"
+  },
+  {
+    "id": "2768",
+    "spreadsheet_path": "spreadsheet/2768",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "37456",
+    "spreadsheet_path": "spreadsheet/37456",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "12864",
+    "spreadsheet_path": "spreadsheet/12864",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "55979",
+    "spreadsheet_path": "spreadsheet/55979",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "48620",
+    "spreadsheet_path": "spreadsheet/48620",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "48588",
+    "spreadsheet_path": "spreadsheet/48588",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "395-36",
+    "spreadsheet_path": "spreadsheet/395-36",
+    "instruction_type": "Sheet-Level Manipulation"
+  },
+  {
+    "id": "382-10",
+    "spreadsheet_path": "spreadsheet/382-10",
+    "instruction_type": "Sheet-Level Manipulation"
+  },
+  {
+    "id": "59595",
+    "spreadsheet_path": "spreadsheet/59595",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "53383",
+    "spreadsheet_path": "spreadsheet/53383",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "48921",
+    "spreadsheet_path": "spreadsheet/48921",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "416-15",
+    "spreadsheet_path": "spreadsheet/416-15",
+    "instruction_type": "Sheet-Level Manipulation"
+  },
+  {
+    "id": "47798",
+    "spreadsheet_path": "spreadsheet/47798",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "56563",
+    "spreadsheet_path": "spreadsheet/56563",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "46897",
+    "spreadsheet_path": "spreadsheet/46897",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "9726",
+    "spreadsheet_path": "spreadsheet/9726",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "50768",
+    "spreadsheet_path": "spreadsheet/50768",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "51-12",
+    "spreadsheet_path": "spreadsheet/51-12",
+    "instruction_type": "Sheet-Level Manipulation"
+  },
+  {
+    "id": "31628",
+    "spreadsheet_path": "spreadsheet/31628",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "39046",
+    "spreadsheet_path": "spreadsheet/39046",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "8942",
+    "spreadsheet_path": "spreadsheet/8942",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "48527",
+    "spreadsheet_path": "spreadsheet/48527",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "59196",
+    "spreadsheet_path": "spreadsheet/59196",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "6698",
+    "spreadsheet_path": "spreadsheet/6698",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "43436",
+    "spreadsheet_path": "spreadsheet/43436",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "38462",
+    "spreadsheet_path": "spreadsheet/38462",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "402-43",
+    "spreadsheet_path": "spreadsheet/402-43",
+    "instruction_type": "Sheet-Level Manipulation"
+  },
+  {
+    "id": "267-18",
+    "spreadsheet_path": "spreadsheet/267-18",
+    "instruction_type": "Sheet-Level Manipulation"
+  },
+  {
+    "id": "37378",
+    "spreadsheet_path": "spreadsheet/37378",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "53647",
+    "spreadsheet_path": "spreadsheet/53647",
+    "instruction_type": "Cell-Level Manipulation"
+  },
+  {
+    "id": "142-12",
+    "spreadsheet_path": "spreadsheet/142-12",
+    "instruction_type": "Sheet-Level Manipulation"
+  }
+]
@@ -1,181 +1,393 @@
 # Add a New Benchmark

-Extend SkillOpt with your own benchmark in ~100 lines of code.
+Extend SkillOpt with your own benchmark in ~200 lines of code. We will use
+a tiny worked example, `docfaithful`, that scores a target model on
+how faithfully it answers questions grounded in a small reference doc.

-## Overview
+> **Working reference.** The easiest way to copy-cargo-cult a new env is
+> to read [`skillopt/envs/officeqa/`](https://github.com/microsoft/SkillOpt/tree/main/skillopt/envs/officeqa).
+> Everything below is the same shape, simplified.

-To add a benchmark, you need:
+## What you need to build

-1. **Data Loader** — Loads and splits your dataset
-2. **Environment Adapter** — Executes tasks and returns scores
-3. **Config** — YAML configuration file
+To add a benchmark you implement four things:

-## Step 1: Create the Benchmark Package
+1. **A `SplitDataLoader` subclass** — knows how to load train / val / test
+   item dicts from disk.
+2. **A rollout helper** — runs the target model on a batch of items
+   under the current skill and scores each prediction.
+3. **An `EnvAdapter` subclass** — wires the loader + rollout helper into
+   SkillOpt's lifecycle (`build_*_env`, `rollout`, `reflect`,
+   `get_task_types`).
+4. **A YAML config** — references your env name plus the standard
+   train / optimizer / gradient knobs.
+
+Then one line in `scripts/train.py`'s `_register_builtins()` makes it
+discoverable.
+
+---
+
+## Step 1 — Create the package

 ```bash
-mkdir -p skillopt/envs/my_benchmark
-touch skillopt/envs/my_benchmark/__init__.py
+mkdir -p skillopt/envs/docfaithful
+touch skillopt/envs/docfaithful/__init__.py
 ```

-## Step 2: Implement the Data Loader
+## Step 2 — Implement the data loader

-Create `skillopt/envs/my_benchmark/loader.py`:
+`skillopt/envs/docfaithful/loader.py`:

 ```python
-from skillopt.data.base import DataLoader, DataItem
+from __future__ import annotations

-class MyBenchmarkDataLoader(DataLoader):
-    """Load and split your benchmark data."""
-    
-    def __init__(self, data_dir: str, **kwargs):
-        super().__init__(**kwargs)
-        self.data_dir = data_dir
-    
-    def setup(self, cfg: dict):
-        """Initialize splits based on config."""
-        self.split_mode = cfg.get('split_mode', 'ratio')
-        # Load your data here
-        self.items = self._load_items()
-        self._create_splits(cfg)
-    
-    def _load_items(self) -> list[DataItem]:
-        """Load raw data into DataItem objects."""
-        items = []
-        # TODO: Load your data
-        for entry in your_data:
-            items.append(DataItem(
-                id=entry['id'],
-                input=entry['question'],
-                ground_truth=entry['answer'],
-                metadata=entry.get('metadata', {})
-            ))
-        return items
-    
-    def get_split_items(self, split: str) -> list[DataItem]:
-        """Return items for a given split (train/valid/test)."""
-        return self.splits[split]
+import json
+from pathlib import Path
+
+from skillopt.datasets.base import SplitDataLoader
+
+
+def _normalize(raw: dict) -> dict:
+    """Make sure every item has an ``id``. Other keys are env-specific."""
+    return {
+        "id": str(raw["uid"]),
+        "question": raw["question"],
+        "ground_truth": raw["answer"],
+        "reference_text": raw.get("reference", ""),
+        "task_type": raw.get("category", "docfaithful"),
+    }
+
+
+class DocFaithfulDataLoader(SplitDataLoader):
+    """Load DocFaithful items from JSON files inside each split dir."""
+
+    def load_split_items(self, split_path: str) -> list[dict]:
+        # split_path is e.g. data/docfaithful_split/train/
+        json_files = sorted(Path(split_path).glob("*.json"))
+        if not json_files:
+            raise FileNotFoundError(f"No .json file found in {split_path}")
+        with json_files[0].open(encoding="utf-8") as f:
+            raw = json.load(f)
+        return [_normalize(item) for item in raw]
 ```

-## Step 3: Implement the Environment Adapter
+Only `load_split_items()` is mandatory. If you also want to support
+`split_mode="ratio"` (auto-split a single raw file into train/val/test),
+override `load_raw_items(data_path)` as well — see
+`skillopt/datasets/base.py` docstrings.

-Create `skillopt/envs/my_benchmark/env.py`:
+## Step 3 — Write the rollout helper
+
+`skillopt/envs/docfaithful/rollout.py`:

 ```python
-from skillopt.envs.base import EnvAdapter, TaskResult
+from __future__ import annotations

-class MyBenchmarkEnv(EnvAdapter):
-    """Execute tasks and evaluate results."""
-    
-    def __init__(self, cfg: dict):
-        super().__init__(cfg)
-    
-    async def execute(self, item: DataItem, skill: str, model) -> TaskResult:
-        """
-        Execute a single task.
-        
-        Args:
-            item: The data item to process
-            skill: Current skill document content
-            model: The target model instance
-            
-        Returns:
-            TaskResult with prediction, score, and trajectory
-        """
-        # Build prompt with skill document
-        prompt = self.build_prompt(item, skill)
-        
-        # Get model response
-        response = await model.generate(prompt)
-        
-        # Extract prediction
-        prediction = self.parse_response(response)
-        
-        # Score against ground truth
-        score = self.evaluate(prediction, item.ground_truth)
-        
-        return TaskResult(
-            item_id=item.id,
-            prediction=prediction,
-            score=score,
-            trajectory=[
-                {"role": "system", "content": skill},
-                {"role": "user", "content": item.input},
-                {"role": "assistant", "content": response}
-            ]
+import json
+import os
+from pathlib import Path
+
+from skillopt.model import chat_target
+
+
+def _score(prediction: str, ground_truth: str) -> tuple[int, float]:
+    """Trivial exact-match scorer. Replace with F1 / ROUGE / LLM-judge."""
+    p = (prediction or "").strip().lower()
+    g = (ground_truth or "").strip().lower()
+    hard = int(p == g and bool(g))
+    soft = 1.0 if hard else 0.0
+    return hard, soft
+
+
+def _rollout_one(item: dict, skill_content: str,
+                 *, max_completion_tokens: int) -> dict:
+    system = skill_content
+    user = (
+        f"Question: {item['question']}\n\n"
+        f"Reference:\n{item.get('reference_text', '')}\n\n"
+        "Answer:"
+    )
+    prediction, _usage = chat_target(
+        system=system,
+        user=user,
+        max_completion_tokens=max_completion_tokens,
+    )
+    hard, soft = _score(prediction, item.get("ground_truth", ""))
+    return {
+        "id": str(item["id"]),
+        "hard": hard,
+        "soft": soft,
+        "predicted_answer": prediction,
+        "question": item.get("question", ""),
+        "reference_text": item.get("reference_text", ""),
+        "task_type": item.get("task_type", "docfaithful"),
+    }
+
+
+def run_batch(*, items: list[dict], skill_content: str, out_root: str,
+              workers: int = 4, max_completion_tokens: int = 4096) -> list[dict]:
+    """Run a batch of episodes sequentially or with a thread pool."""
+    os.makedirs(out_root, exist_ok=True)
+    # For brevity we go sequentially — swap in concurrent.futures.ThreadPoolExecutor
+    # when network / model latency dominates.
+    results = [
+        _rollout_one(item, skill_content,
+                     max_completion_tokens=max_completion_tokens)
+        for item in items
+    ]
+    Path(out_root, "rollouts.json").write_text(
+        json.dumps(results, ensure_ascii=False, indent=2)
+    )
+    return results
+```
+
+Two design points worth flagging:
+
+- **Scoring lives here, not in `EnvAdapter`.** There is no `evaluate()`
+  method on the ABC. Whatever signal you put in `hard` (0/1, or a float
+  in [0, 1] for smoothed reward) and `soft` (float in [0, 1]) is what
+  the optimizer reads.
+- **Use `skillopt.model.chat_target`**, not raw OpenAI/Claude calls.
+  That routes through whichever **chat** target backend the user
+  configured (`openai_chat` / `claude_chat` / `qwen_chat` /
+  `minimax_chat`) without your adapter caring. Exec-style backends
+  (`codex_exec`, `claude_code_exec`) need env-specific rollout code —
+  see `skillopt/envs/swebench/` for an example.
+
+## Step 4 — Implement the environment adapter
+
+`skillopt/envs/docfaithful/adapter.py`:
+
+```python
+from __future__ import annotations
+
+import os
+
+from skillopt.datasets.base import BatchSpec
+from skillopt.envs.base import EnvAdapter
+from skillopt.envs.docfaithful.loader import DocFaithfulDataLoader
+from skillopt.envs.docfaithful.rollout import run_batch
+from skillopt.gradient.reflect import run_minibatch_reflect
+
+
+class DocFaithfulAdapter(EnvAdapter):
+    """SkillOpt adapter for the DocFaithful benchmark."""
+
+    def __init__(
+        self,
+        split_dir: str = "",
+        data_path: str = "",
+        split_mode: str = "split_dir",
+        split_ratio: str = "2:1:7",
+        split_seed: int = 42,
+        split_output_dir: str = "",
+        workers: int = 4,
+        analyst_workers: int = 4,
+        failure_only: bool = False,
+        minibatch_size: int = 8,
+        edit_budget: int = 4,
+        seed: int = 42,
+        limit: int = 0,
+        max_completion_tokens: int = 4096,
+    ) -> None:
+        self.workers = workers
+        self.analyst_workers = analyst_workers
+        self.failure_only = failure_only
+        self.minibatch_size = minibatch_size
+        self.edit_budget = edit_budget
+        self.max_completion_tokens = int(max_completion_tokens)
+        self.dataloader = DocFaithfulDataLoader(
+            split_dir=split_dir,
+            data_path=data_path,
+            split_mode=split_mode,
+            split_ratio=split_ratio,
+            split_seed=split_seed,
+            split_output_dir=split_output_dir,
+            seed=seed,
+            limit=limit,
        )
-    
-    def evaluate(self, prediction: str, ground_truth: str) -> float:
-        """
-        Score a prediction against ground truth.
-        
-        Returns:
-            Float between 0.0 and 1.0
-        """
-        # TODO: Implement your scoring logic
-        # Examples: exact match, F1, ANLS, etc.
-        return float(prediction.strip() == ground_truth.strip())
-    
-    def build_prompt(self, item, skill: str) -> str:
-        """Combine skill document with task input."""
-        return f"{skill}\n\n---\n\nQuestion: {item.input}"
-    
-    def parse_response(self, response: str) -> str:
-        """Extract the answer from model response."""
-        return response.strip()
+
+    # ── Lifecycle ───────────────────────────────────────────────────────
+
+    def setup(self, cfg: dict) -> None:
+        super().setup(cfg)
+        self.dataloader.setup(cfg)
+
+    def get_dataloader(self):
+        return self.dataloader
+
+    # ── Env construction ────────────────────────────────────────────────
+
+    def build_env_from_batch(self, batch: BatchSpec, **kwargs):
+        # For dataset-backed envs the "manager" is just the items list.
+        return list(batch.payload or [])
+
+    def build_train_env(self, batch_size: int, seed: int, **kwargs):
+        batch = self.dataloader.build_train_batch(
+            batch_size=batch_size, seed=seed, **kwargs
+        )
+        return self.build_env_from_batch(batch, **kwargs)
+
+    def build_eval_env(self, env_num: int, split: str, seed: int, **kwargs):
+        batch = self.dataloader.build_eval_batch(
+            env_num=env_num, split=split, seed=seed, **kwargs
+        )
+        return self.build_env_from_batch(batch, **kwargs)
+
+    # ── The two real action methods ─────────────────────────────────────
+
+    def rollout(self, env_manager, skill_content: str,
+                out_dir: str, **kwargs) -> list[dict]:
+        items: list[dict] = env_manager
+        return run_batch(
+            items=items,
+            skill_content=skill_content,
+            out_root=out_dir,
+            workers=self.workers,
+            max_completion_tokens=self.max_completion_tokens,
+        )
+
+    def reflect(self, results: list[dict], skill_content: str,
+                out_dir: str, **kwargs) -> list[dict | None]:
+        return run_minibatch_reflect(
+            results=results,
+            skill_content=skill_content,
+            prediction_dir=kwargs.get(
+                "prediction_dir", os.path.join(out_dir, "predictions")
+            ),
+            patches_dir=kwargs.get(
+                "patches_dir", os.path.join(out_dir, "patches")
+            ),
+            workers=self.analyst_workers,
+            failure_only=self.failure_only,
+            minibatch_size=self.minibatch_size,
+            edit_budget=self.edit_budget,
+            random_seed=kwargs.get("random_seed"),
+            error_system=self.get_error_minibatch_prompt(),
+            success_system=self.get_success_minibatch_prompt(),
+            step_buffer_context=kwargs.get("step_buffer_context", ""),
+            update_mode=getattr(self, "_cfg", {}).get("skill_update_mode", "patch"),
+        )
+
+    def get_task_types(self) -> list[str]:
+        seen: list[str] = []
+        for item in (
+            self.dataloader.train_items
+            + self.dataloader.val_items
+            + self.dataloader.test_items
+        ):
+            tt = str(item.get("task_type") or "docfaithful")
+            if tt not in seen:
+                seen.append(tt)
+        return seen or ["docfaithful"]
 ```

-## Step 4: Register the Benchmark
+### What the rollout actually does

-Add to `skillopt/envs/__init__.py`:
+Look back at `run_batch` from Step 3 — it sends each `item["question"]`
+to the target model with `skill_content` as the system prompt, scores
+the answer against `item["ground_truth"]`, and returns a list of dicts:

 ```python
-from .my_benchmark.env import MyBenchmarkEnv
-from .my_benchmark.loader import MyBenchmarkDataLoader
-
-BENCHMARK_REGISTRY = {
-    # ... existing benchmarks ...
-    'my_benchmark': {
-        'env': MyBenchmarkEnv,
-        'loader': MyBenchmarkDataLoader,
-    },
-}
+[
+    {"id": "ex_001", "hard": 1, "soft": 0.92,
+     "predicted_answer": "...", "question": "...",
+     "reference_text": item["reference_text"]},
+    {"id": "ex_002", "hard": 0, "soft": 0.13, "fail_reason": "...", ...},
+    ...
+]
 ```

-## Step 5: Create Config
+The trainer only requires `id`, `hard`, `soft`. The rest is preserved on
+`RolloutResult.extras` (see `skillopt/types.py`) and is what your
+`reflect()` consumes via `run_minibatch_reflect`.

-Create `configs/my_benchmark/default.yaml`:
+## Step 5 — Register the adapter
+
+Edit [`scripts/train.py`](https://github.com/microsoft/SkillOpt/blob/main/scripts/train.py)
+and add to `_register_builtins()`:
+
+```python
+    try:
+        from skillopt.envs.docfaithful.adapter import DocFaithfulAdapter
+        _ENV_REGISTRY["docfaithful"] = DocFaithfulAdapter
+    except ImportError:
+        pass  # docfaithful deps not installed — skip
+```
+
+There is **no `BENCHMARK_REGISTRY` dict in `skillopt/envs/__init__.py`** —
+the registry lives in `scripts/train.py` and is populated lazily so that
+optional deps don't break `--help`.
+
+## Step 6 — Create the YAML config
+
+`configs/docfaithful/default.yaml`:

 ```yaml
-_base_: ['../_base_/default.yaml']
+_base_: ../_base_/default.yaml      # NOTE: string, not list

-env:
-  name: my_benchmark
-  data_path: data/my_benchmark
-  split_mode: ratio
-  split_ratio: "2:1:7"
+model:
+  reasoning_effort: medium

 train:
+  batch_size: 16
+  accumulation: 1
  num_epochs: 4
-  batch_size: 40
+
+gradient:
+  minibatch_size: 8
+  merge_batch_size: 8

 optimizer:
  learning_rate: 4
-  lr_scheduler: cosine
-  use_slow_update: true
-  use_meta_skill: true

-gradient:
-  analyst_workers: 16
+env:
+  name: docfaithful
+  # Optional: a seed skill document. Create this file (or any markdown
+  # file) yourself before the first run, or omit the key to let SkillOpt
+  # start from an empty skill.
+  skill_init: skillopt/envs/docfaithful/skills/initial.md
+  split_mode: split_dir
+  split_dir: data/docfaithful_split
+  workers: 4
+  max_completion_tokens: 4096
+  limit: 0
 ```

-## Step 6: Run
+> ⚠️ `_base_` is currently parsed as a **string path**, not a list. Write
+> `_base_: ../_base_/default.yaml`, not `_base_: ['../_base_/default.yaml']`.
+> See [`skillopt/config.py`](https://github.com/microsoft/SkillOpt/blob/main/skillopt/config.py)
+> if you want to add list-form inheritance.
+
+## Step 7 — Run

 ```bash
-python scripts/train.py --config configs/my_benchmark/default.yaml
+# If you set skill_init above, create the seed skill first:
+#   mkdir -p skillopt/envs/docfaithful/skills
+#   echo "# DocFaithful initial skill" > skillopt/envs/docfaithful/skills/initial.md
+
+python scripts/train.py --config configs/docfaithful/default.yaml
 ```

+If you get `ValueError: Unknown environment 'docfaithful'. Available: [...]`,
+you forgot Step 5.
+
+If you get `TypeError: Can't instantiate abstract class DocFaithfulAdapter`,
+you forgot to implement one of the five abstract methods on `EnvAdapter`:
+`build_train_env`, `build_eval_env`, `rollout`, `reflect`,
+`get_task_types`.
+
 ## Tips

-!!! tip
-    - Use a small `batch_size` (10-20) for initial testing
-    - The `evaluate()` method is critical — a noisy metric will confuse the optimizer
+- Start with `train.batch_size: 4` and `limit: 10` while debugging.
+- The `evaluate` half lives **inside your `rollout`**, not as a separate
+  method — there is no `evaluate()` in the `EnvAdapter` ABC. Score the
+  prediction in `run_batch` and put the score on each result dict's
+  `hard` / `soft`.
+- Noisy scoring kills the optimizer. Spend time on `run_batch`'s scoring
+  before you spend time on prompts.
+- If your benchmark needs heavy optional deps (selenium, vllm, ...),
+  wrap the registration block with `try / except ImportError` (Step 5)
+  so people without those deps can still `--help`.
+- Copy `skillopt/envs/_template/` as a starting skeleton — it now
+  implements the real abstract methods.
@@ -1,911 +0,0 @@
-<!DOCTYPE html>
-<html lang="en">
-<head>
-<meta charset="UTF-8">
-<meta name="viewport" content="width=device-width, initial-scale=1.0">
-<title>SkillOpt — Documentation &amp; Reproduction Guide</title>
-<meta name="description" content="Complete documentation and reproduction guide for SkillOpt: installation, data preparation, training, configuration reference, framework internals, and API reference.">
-<link rel="icon" type="image/svg+xml" href="data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 23 23'%3E%3Crect width='10' height='10' fill='%23F25022'/%3E%3Crect x='13' width='10' height='10' fill='%237FBA00'/%3E%3Crect y='13' width='10' height='10' fill='%2300A4EF'/%3E%3Crect x='13' y='13' width='10' height='10' fill='%23FFB900'/%3E%3C/svg%3E">
-<style>
-  :root {
-    --bg: #ffffff;
-    --bg-soft: #f7f8fb;
-    --sidebar-bg: #fbfcfe;
-    --ink: #1f2733;
-    --muted: #5b6675;
-    --quiet: #8a94a3;
-    --line: #e6e9ef;
-    --line-strong: #d3d9e3;
-    --brand: #4f46e5;
-    --brand-soft: #eef0fe;
-    --accent: #0ea5e9;
-    --green: #16a34a;
-    --amber: #d97706;
-    --red: #dc2626;
-    --code-bg: #0f172a;
-    --code-ink: #e2e8f0;
-    --inline-code-bg: #eef1f6;
-    --inline-code-ink: #b3146b;
-    --sidebar-w: 300px;
-    --toc-w: 220px;
-    --mono: "SFMono-Regular", "JetBrains Mono", Consolas, "Liberation Mono", monospace;
-    --sans: "Inter", -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif;
-  }
-  * { box-sizing: border-box; }
-  html { scroll-behavior: smooth; }
-  body {
-    margin: 0;
-    font-family: var(--sans);
-    color: var(--ink);
-    background: var(--bg);
-    font-size: 15px;
-    line-height: 1.65;
-    -webkit-font-smoothing: antialiased;
-  }
-
-  /* ── Top bar ─────────────────────────────────────────── */
-  header.topbar {
-    position: sticky; top: 0; z-index: 40;
-    height: 56px;
-    display: flex; align-items: center; gap: 14px;
-    padding: 0 20px;
-    background: rgba(255,255,255,0.92);
-    backdrop-filter: blur(8px);
-    border-bottom: 1px solid var(--line);
-  }
-  .topbar .logo { width: 22px; height: 22px; flex: none; }
-  .topbar .brand { font-weight: 700; font-size: 16px; letter-spacing: -0.01em; }
-  .topbar .brand span { color: var(--brand); }
-  .topbar .tag { color: var(--quiet); font-size: 13px; border-left: 1px solid var(--line-strong); padding-left: 14px; }
-  .topbar .spacer { flex: 1; }
-  .topbar a.gh {
-    display: inline-flex; align-items: center; gap: 6px;
-    font-size: 13px; font-weight: 600; color: var(--muted);
-    text-decoration: none; padding: 6px 12px; border: 1px solid var(--line-strong);
-    border-radius: 8px;
-  }
-  .topbar a.gh:hover { color: var(--brand); border-color: var(--brand); }
-  #menuBtn {
-    display: none; background: none; border: 1px solid var(--line-strong);
-    border-radius: 8px; width: 38px; height: 34px; cursor: pointer; font-size: 18px; color: var(--muted);
-  }
-
-  /* ── Layout ──────────────────────────────────────────── */
-  .layout { display: flex; align-items: flex-start; }
-
-  /* ── Sidebar (left nav) ──────────────────────────────── */
-  nav.sidebar {
-    position: sticky; top: 56px;
-    width: var(--sidebar-w); flex: none;
-    height: calc(100vh - 56px);
-    overflow-y: auto;
-    background: var(--sidebar-bg);
-    border-right: 1px solid var(--line);
-    padding: 22px 14px 60px 20px;
-  }
-  nav.sidebar .group { margin-bottom: 22px; }
-  nav.sidebar .group > .glabel {
-    display: flex; align-items: center; gap: 8px;
-    font-size: 11.5px; font-weight: 700; text-transform: uppercase;
-    letter-spacing: 0.07em; color: var(--quiet);
-    margin: 0 0 8px 2px;
-  }
-  nav.sidebar .group > .glabel .num {
-    display: inline-flex; align-items: center; justify-content: center;
-    width: 18px; height: 18px; border-radius: 5px;
-    background: var(--brand-soft); color: var(--brand);
-    font-size: 11px; font-weight: 700;
-  }
-  nav.sidebar a {
-    display: block; text-decoration: none;
-    color: var(--muted); font-size: 13.5px;
-    padding: 5px 10px; border-radius: 7px; margin: 1px 0;
-    border-left: 2px solid transparent;
-  }
-  nav.sidebar a:hover { background: #eef1f6; color: var(--ink); }
-  nav.sidebar a.active {
-    color: var(--brand); background: var(--brand-soft);
-    border-left-color: var(--brand); font-weight: 600;
-  }
-
-  /* ── Content ─────────────────────────────────────────── */
-  main.content {
-    flex: 1; min-width: 0;
-    padding: 38px 46px 120px;
-    max-width: 900px;
-  }
-  main.content section { scroll-margin-top: 72px; }
-  main h1 { font-size: 30px; line-height: 1.2; letter-spacing: -0.02em; margin: 0 0 8px; }
-  main h2 {
-    font-size: 23px; letter-spacing: -0.015em; margin: 52px 0 14px;
-    padding-bottom: 10px; border-bottom: 1px solid var(--line);
-  }
-  main section:first-of-type h2 { margin-top: 8px; }
-  main h3 { font-size: 17.5px; margin: 30px 0 10px; letter-spacing: -0.01em; }
-  main h4 { font-size: 15px; margin: 22px 0 8px; color: var(--ink); }
-  main p { margin: 12px 0; color: #2c3645; }
-  main ul, main ol { margin: 12px 0; padding-left: 22px; }
-  main li { margin: 5px 0; }
-  main a { color: var(--brand); text-decoration: none; }
-  main a:hover { text-decoration: underline; }
-  .lead { font-size: 16.5px; color: var(--muted); margin: 6px 0 4px; }
-  .eyebrow { color: var(--brand); font-weight: 700; font-size: 12.5px; letter-spacing: 0.08em; text-transform: uppercase; }
-
-  /* code */
-  code {
-    font-family: var(--mono); font-size: 0.86em;
-    background: var(--inline-code-bg); color: var(--inline-code-ink);
-    padding: 2px 6px; border-radius: 5px;
-  }
-  pre {
-    background: var(--code-bg); color: var(--code-ink);
-    border-radius: 12px; padding: 16px 18px; overflow-x: auto;
-    font-family: var(--mono); font-size: 13px; line-height: 1.6;
-    margin: 14px 0; border: 1px solid #1e293b;
-  }
-  pre code { background: none; color: inherit; padding: 0; font-size: inherit; }
-  .tok-c { color: #7c8aa5; }   /* comment */
-  .tok-k { color: #c4b5fd; }   /* keyword */
-  .tok-s { color: #86efac; }   /* string */
-  .tok-f { color: #93c5fd; }   /* flag/path */
-  .tok-n { color: #fca5a5; }   /* number/value */
-
-  /* tables */
-  .table-wrap { overflow-x: auto; margin: 16px 0; border: 1px solid var(--line); border-radius: 12px; }
-  table { border-collapse: collapse; width: 100%; font-size: 13.5px; }
-  th, td { text-align: left; padding: 9px 13px; border-bottom: 1px solid var(--line); vertical-align: top; }
-  thead th { background: var(--bg-soft); font-weight: 700; color: var(--ink); white-space: nowrap; }
-  tbody tr:last-child td { border-bottom: none; }
-  td code { white-space: nowrap; }
-  td.def { color: var(--muted); font-family: var(--mono); font-size: 12px; }
-
-  /* callouts */
-  .note { border-radius: 10px; padding: 12px 16px; margin: 16px 0; border: 1px solid; font-size: 14px; }
-  .note p { margin: 4px 0; }
-  .note .nh { font-weight: 700; display: block; margin-bottom: 2px; }
-  .note.info  { background: #eff6ff; border-color: #bfdbfe; }
-  .note.info .nh { color: #1d4ed8; }
-  .note.tip   { background: #ecfdf5; border-color: #a7f3d0; }
-  .note.tip .nh { color: #047857; }
-  .note.warn  { background: #fffbeb; border-color: #fde68a; }
-  .note.warn .nh { color: #b45309; }
-
-  .pill { display:inline-block; font-size: 11px; font-weight:700; padding: 1px 8px; border-radius: 999px; vertical-align: middle; }
-  .pill.def { background:#eef2ff; color:#4338ca; }
-  .pill.opt { background:#f1f5f9; color:#475569; }
-
-  /* card grid */
-  .cards { display: grid; grid-template-columns: repeat(auto-fit, minmax(230px,1fr)); gap: 14px; margin: 18px 0; }
-  .card { border: 1px solid var(--line); border-radius: 12px; padding: 16px; background: var(--bg-soft); }
-  .card h4 { margin: 0 0 6px; font-size: 14.5px; }
-  .card p { margin: 0; font-size: 13px; color: var(--muted); }
-
-  /* anchor link on hover */
-  .anchor { color: var(--quiet); text-decoration: none; font-weight: 400; opacity: 0; margin-left: 8px; font-size: 0.8em; }
-  h2:hover .anchor, h3:hover .anchor { opacity: 1; }
-
-  /* ── Right TOC ───────────────────────────────────────── */
-  aside.toc {
-    position: sticky; top: 56px;
-    width: var(--toc-w); flex: none;
-    height: calc(100vh - 56px); overflow-y: auto;
-    padding: 38px 18px; border-left: 1px solid var(--line);
-  }
-  aside.toc .tl { font-size: 11.5px; font-weight: 700; text-transform: uppercase; letter-spacing: 0.07em; color: var(--quiet); margin-bottom: 10px; }
-  aside.toc a { display: block; color: var(--muted); text-decoration: none; font-size: 12.5px; padding: 4px 8px; border-left: 2px solid var(--line); line-height: 1.45; }
-  aside.toc a:hover { color: var(--ink); }
-  aside.toc a.active { color: var(--brand); border-left-color: var(--brand); font-weight: 600; }
-
-  .footer-note { margin-top: 60px; padding-top: 20px; border-top: 1px solid var(--line); color: var(--quiet); font-size: 13px; }
-
-  /* responsive */
-  @media (max-width: 1180px) { aside.toc { display: none; } }
-  @media (max-width: 860px) {
-    #menuBtn { display: inline-block; }
-    nav.sidebar {
-      position: fixed; left: 0; top: 56px; z-index: 35;
-      transform: translateX(-100%); transition: transform 0.22s ease;
-      box-shadow: 0 16px 40px rgba(15,23,42,0.18);
-    }
-    nav.sidebar.open { transform: translateX(0); }
-    main.content { padding: 28px 20px 100px; }
-    .topbar .tag { display: none; }
-  }
-</style>
-</head>
-<body>
-
-<header class="topbar">
-  <button id="menuBtn" aria-label="Toggle navigation">&#9776;</button>
-  <svg class="logo" viewBox="0 0 23 23"><rect width="10" height="10" fill="#F25022"/><rect x="13" width="10" height="10" fill="#7FBA00"/><rect y="13" width="10" height="10" fill="#00A4EF"/><rect x="13" y="13" width="10" height="10" fill="#FFB900"/></svg>
-  <span class="brand">Skill<span>Opt</span></span>
-  <span class="tag">Documentation &amp; Reproduction Guide</span>
-  <span class="spacer"></span>
-  <a class="gh" href="https://github.com/microsoft/SkillOpt" target="_blank" rel="noopener">GitHub ↗</a>
-  <a class="gh" href="https://arxiv.org/abs/2605.23904" target="_blank" rel="noopener">Paper ↗</a>
-</header>
-
-<div class="layout">
-
-  <!-- ───────────── LEFT NAV ───────────── -->
-  <nav class="sidebar" id="sidebar">
-    <div class="group">
-      <div class="glabel"><span class="num">1</span> Overview</div>
-      <a href="#what-is">What is SkillOpt</a>
-      <a href="#analogy">DL ↔ SkillOpt analogy</a>
-      <a href="#features">Key features</a>
-      <a href="#layout">Repository layout</a>
-    </div>
-    <div class="group">
-      <div class="glabel"><span class="num">2</span> Installation</div>
-      <a href="#requirements">Requirements</a>
-      <a href="#install">Install the package</a>
-      <a href="#credentials">Configure credentials</a>
-      <a href="#verify">Verify installation</a>
-    </div>
-    <div class="group">
-      <div class="glabel"><span class="num">3</span> Data Preparation</div>
-      <a href="#split-dir">Split directory format</a>
-      <a href="#item-schema">Item JSON schema</a>
-      <a href="#split-modes">Split modes</a>
-    </div>
-    <div class="group">
-      <div class="glabel"><span class="num">4</span> Quick Start</div>
-      <a href="#train">Train a skill</a>
-      <a href="#eval">Evaluate a skill</a>
-      <a href="#outputs">Output structure</a>
-      <a href="#resume">Auto-resume</a>
-    </div>
-    <div class="group">
-      <div class="glabel"><span class="num">5</span> How It Works</div>
-      <a href="#loop">The training loop</a>
-      <a href="#stages">The six per-step stages</a>
-      <a href="#gate">Validation gate</a>
-      <a href="#slow-update">Slow update (momentum)</a>
-      <a href="#meta-skill">Meta skill (memory)</a>
-      <a href="#skill-doc">Skill document anatomy</a>
-    </div>
-    <div class="group">
-      <div class="glabel"><span class="num">6</span> Configuration</div>
-      <a href="#config-system">Config system</a>
-      <a href="#cfg-model">model.*</a>
-      <a href="#cfg-train">train.*</a>
-      <a href="#cfg-gradient">gradient.*</a>
-      <a href="#cfg-optimizer">optimizer.*</a>
-      <a href="#cfg-evaluation">evaluation.*</a>
-      <a href="#cfg-env">env.*</a>
-    </div>
-    <div class="group">
-      <div class="glabel"><span class="num">7</span> Benchmarks</div>
-      <a href="#bench-list">Supported benchmarks</a>
-      <a href="#bench-new">Add a new benchmark</a>
-    </div>
-    <div class="group">
-      <div class="glabel"><span class="num">8</span> API Reference</div>
-      <a href="#module-map">Module map</a>
-      <a href="#functions">Core functions</a>
-      <a href="#cli">CLI scripts</a>
-      <a href="#webui">WebUI</a>
-    </div>
-  </nav>
-
-  <!-- ───────────── MAIN CONTENT ───────────── -->
-  <main class="content">
-
-    <span class="eyebrow">Microsoft Research</span>
-    <h1>SkillOpt Documentation &amp; Reproduction Guide</h1>
-    <p class="lead">Train agent skills like you train neural networks — with epochs, (mini-)batch size, learning rates, and validation gates — but without touching any model weights.</p>
-    <p>This guide walks you from a clean checkout to a reproduced result and a full reference for every configuration knob and core function. It is generated from, and kept consistent with, the current state of the codebase.</p>
-
-    <!-- ===================== 1. OVERVIEW ===================== -->
-    <section id="what-is">
-      <h2>1.1 What is SkillOpt <a class="anchor" href="#what-is">#</a></h2>
-      <p><strong>SkillOpt</strong> is a text-space optimizer that improves a <em>frozen</em> language agent by iteratively editing a natural-language <strong>skill document</strong> — never the model weights. The skill document is a Markdown file that conditions a target model as it executes tasks. SkillOpt treats this document as the "weights" and runs a training loop that mirrors deep-learning training: rollout (forward pass), reflect (backward pass / gradients), select &amp; apply edits (optimizer step), and a validation gate (accept/reject).</p>
-      <p>Two roles split every model call:</p>
-      <ul>
-        <li><strong>Target</strong> — executes tasks using the current skill document (the agent being improved).</li>
-        <li><strong>Optimizer</strong> — analyzes the target's trajectories and proposes edits to the skill document.</li>
-      </ul>
-      <p>The same loop drives six benchmarks out of the box (QA, document QA, embodied agents, math, spreadsheet code generation, and tool-augmented QA).</p>
-    </section>
-
-    <section id="analogy">
-      <h2>1.2 Deep-Learning ↔ SkillOpt Analogy <a class="anchor" href="#analogy">#</a></h2>
-      <p>Every concept below maps to a concrete code construct, so deep-learning intuitions transfer directly to hyperparameter tuning.</p>
-      <div class="table-wrap">
-      <table>
-        <thead><tr><th>Deep learning</th><th>SkillOpt</th><th>Where it lives</th></tr></thead>
-        <tbody>
-          <tr><td>Model weights</td><td>Skill document (Markdown)</td><td><code>skillopt/optimizer/skill.py</code></td></tr>
-          <tr><td>Forward pass</td><td>Rollout — target runs tasks</td><td><code>envs/&lt;bench&gt;/rollout.py</code></td></tr>
-          <tr><td>Loss / score</td><td>Task evaluator</td><td><code>envs/&lt;bench&gt;/evaluator.py</code></td></tr>
-          <tr><td>Backprop / gradients</td><td>Reflect → edit patches</td><td><code>gradient/reflect.py</code></td></tr>
-          <tr><td>Gradient aggregation</td><td>Hierarchical patch merge</td><td><code>gradient/aggregate.py</code></td></tr>
-          <tr><td>Gradient clipping</td><td>Rank &amp; select top-k edits</td><td><code>optimizer/clip.py</code></td></tr>
-          <tr><td>Learning rate</td><td><code>optimizer.learning_rate</code> (edits/step)</td><td><code>optimizer/scheduler.py</code></td></tr>
-          <tr><td>LR scheduler</td><td><code>lr_scheduler</code> (cosine/linear/…)</td><td><code>optimizer/scheduler.py</code></td></tr>
-          <tr><td>Optimizer step</td><td>Apply patches to the document</td><td><code>optimizer/skill.py</code></td></tr>
-          <tr><td>Validation set</td><td>Selection split (<code>valid_seen</code>)</td><td><code>evaluation/gate.py</code></td></tr>
-          <tr><td>Early stopping / accept</td><td>Validation gate</td><td><code>evaluation/gate.py</code></td></tr>
-          <tr><td>Momentum</td><td>Slow update (epoch boundary)</td><td><code>optimizer/slow_update.py</code></td></tr>
-          <tr><td>Meta-learning</td><td>Meta skill (cross-epoch memory)</td><td><code>optimizer/meta_skill.py</code></td></tr>
-          <tr><td>Batch / minibatch</td><td><code>batch_size</code> / <code>minibatch_size</code></td><td><code>engine/trainer.py</code></td></tr>
-          <tr><td>Epoch</td><td>Epoch (+ slow update &amp; meta skill)</td><td><code>engine/trainer.py</code></td></tr>
-        </tbody>
-      </table>
-      </div>
-      <div class="note tip"><span class="nh">What transfers from DL</span>
-        <p>Cosine schedule tends to beat constant; moderate learning rates (≈4–16 edits/step) beat very high/low; slow update curbs cross-epoch forgetting; meta-skill memory improves reflection quality. Conversely, bigger rollout batches and many epochs show diminishing returns — skills converge in ~2–4 epochs.</p>
-      </div>
-    </section>
-
-    <section id="features">
-      <h2>1.3 Key Features <a class="anchor" href="#features">#</a></h2>
-      <div class="cards">
-        <div class="card"><h4>Validation gating</h4><p>Every candidate skill is scored on a held-out selection split and only accepted if it beats the current/best skill.</p></div>
-        <div class="card"><h4>Slow update</h4><p>Epoch-boundary longitudinal comparison writes guidance into a protected region — momentum against forgetting. Force-injected or selection-gated.</p></div>
-        <div class="card"><h4>Meta skill</h4><p>Optimizer-side memory that reflects on what worked across epochs and feeds back into reflection.</p></div>
-        <div class="card"><h4>Pluggable backends</h4><p>OpenAI / Azure OpenAI, Anthropic Claude, local Qwen (vLLM), plus Codex/Claude-Code exec backends for the target.</p></div>
-        <div class="card"><h4>Six benchmarks</h4><p>SearchQA, DocVQA, ALFWorld, LiveMathematicianBench, SpreadsheetBench, OfficeQA — each a self-contained env module.</p></div>
-        <div class="card"><h4>Auto-resume</h4><p>Every run is checkpointed step-by-step; re-running the same command continues from the last completed step.</p></div>
-      </div>
-    </section>
-
-    <section id="layout">
-      <h2>1.4 Repository Layout <a class="anchor" href="#layout">#</a></h2>
-<pre><code><span class="tok-c"># top level</span>
-configs/            <span class="tok-c"># YAML configs (_base_ + per-benchmark)</span>
-scripts/            <span class="tok-c"># train.py, eval_only.py CLIs</span>
-ckpt/               <span class="tok-c"># packaged reference skills (e.g. gpt5.5_skill.md)</span>
-docs/               <span class="tok-c"># this guide + mkdocs sources</span>
-skillopt/           <span class="tok-c"># the package</span>
- ├─ config.py        <span class="tok-c"># YAML loading, _base_ inheritance, flatten</span>
- ├─ engine/trainer.py<span class="tok-c"># the training loop (ReflACTTrainer)</span>
- ├─ gradient/        <span class="tok-c"># reflect.py (analyst), aggregate.py (merge)</span>
- ├─ optimizer/       <span class="tok-c"># skill edits, scheduler, clip, slow_update, meta_skill</span>
- ├─ evaluation/      <span class="tok-c"># gate.py (accept/reject logic)</span>
- ├─ model/           <span class="tok-c"># backend clients + routing</span>
- └─ envs/&lt;benchmark&gt;/ <span class="tok-c"># adapter, dataloader, rollout, evaluator, reflect</span></code></pre>
-    </section>
-
-    <!-- ===================== 2. INSTALLATION ===================== -->
-    <section id="requirements">
-      <h2>2.1 Requirements <a class="anchor" href="#requirements">#</a></h2>
-      <ul>
-        <li>Python ≥ 3.10</li>
-        <li>Credentials for at least one model backend (Azure OpenAI, OpenAI-compatible, Anthropic, or a local Qwen server)</li>
-        <li>Benchmark datasets are <strong>not</strong> bundled — prepare your own splits (see §3)</li>
-      </ul>
-    </section>
-
-    <section id="install">
-      <h2>2.2 Install the Package <a class="anchor" href="#install">#</a></h2>
-<pre><code><span class="tok-k">git</span> clone https://github.com/microsoft/SkillOpt.git
-<span class="tok-k">cd</span> SkillOpt
-<span class="tok-k">pip</span> install -e .
-
-<span class="tok-c"># Optional extras (install only what you need):</span>
-<span class="tok-k">pip</span> install -e <span class="tok-s">".[alfworld]"</span>   <span class="tok-c"># ALFWorld benchmark</span>
-<span class="tok-k">pip</span> install -e <span class="tok-s">".[claude]"</span>     <span class="tok-c"># Anthropic Claude backend</span>
-<span class="tok-k">pip</span> install -e <span class="tok-s">".[qwen]"</span>       <span class="tok-c"># local Qwen backend</span>
-<span class="tok-k">pip</span> install -e <span class="tok-s">".[webui]"</span>      <span class="tok-c"># monitoring dashboard</span>
-
-<span class="tok-c"># ALFWorld also needs its data assets:</span>
-<span class="tok-k">alfworld-download</span></code></pre>
-    </section>
-
-    <section id="credentials">
-      <h2>2.3 Configure Credentials <a class="anchor" href="#credentials">#</a></h2>
-      <p>Copy the template and fill in whichever backend you will use:</p>
-<pre><code><span class="tok-k">cp</span> .env.example .env
-<span class="tok-c"># edit .env, then:</span>
-<span class="tok-k">set</span> -a; <span class="tok-k">source</span> .env; <span class="tok-k">set</span> +a</code></pre>
-      <div class="note info"><span class="nh">One env-var family for all OpenAI modes</span>
-        <p>SkillOpt reuses the <code>AZURE_OPENAI_*</code> variable names even for plain OpenAI — there is no separate <code>OPENAI_API_KEY</code> knob. <code>AZURE_OPENAI_ENDPOINT</code> is required for every OpenAI auth mode.</p>
-      </div>
-      <h4>Azure OpenAI (default)</h4>
-<pre><code><span class="tok-k">export</span> AZURE_OPENAI_ENDPOINT=<span class="tok-s">"https://your-resource.openai.azure.com/"</span>
-<span class="tok-k">export</span> AZURE_OPENAI_API_VERSION=<span class="tok-s">"2024-12-01-preview"</span>
-<span class="tok-c"># Auth option 1 — API key:</span>
-<span class="tok-k">export</span> AZURE_OPENAI_API_KEY=<span class="tok-s">"your-key"</span>
-<span class="tok-c"># Auth option 2 — Azure CLI (no key; recommended on Azure VMs):</span>
-<span class="tok-k">export</span> AZURE_OPENAI_AUTH_MODE=azure_cli
-<span class="tok-c"># Auth option 3 — Managed Identity:</span>
-<span class="tok-k">export</span> AZURE_OPENAI_AUTH_MODE=managed_identity
-<span class="tok-k">export</span> AZURE_OPENAI_MANAGED_IDENTITY_CLIENT_ID=<span class="tok-s">"your-client-id"</span></code></pre>
-      <h4>OpenAI-compatible endpoint</h4>
-<pre><code><span class="tok-k">export</span> AZURE_OPENAI_ENDPOINT=<span class="tok-s">"https://api.openai.com/v1"</span>
-<span class="tok-k">export</span> AZURE_OPENAI_API_KEY=<span class="tok-s">"sk-..."</span>
-<span class="tok-k">export</span> AZURE_OPENAI_AUTH_MODE=openai_compatible</code></pre>
-      <h4>Anthropic Claude / local Qwen</h4>
-<pre><code><span class="tok-k">export</span> ANTHROPIC_API_KEY=<span class="tok-s">"sk-ant-..."</span>          <span class="tok-c"># claude_chat backend</span>
-
-<span class="tok-k">export</span> QWEN_CHAT_BASE_URL=<span class="tok-s">"http://localhost:8000/v1"</span> <span class="tok-c"># local vLLM</span>
-<span class="tok-k">export</span> QWEN_CHAT_MODEL=<span class="tok-s">"Qwen/Qwen3.5-4B"</span></code></pre>
-    </section>
-
-    <section id="verify">
-      <h2>2.4 Verify Installation <a class="anchor" href="#verify">#</a></h2>
-<pre><code><span class="tok-k">python</span> -c <span class="tok-s">"import skillopt; print('SkillOpt ready!')"</span></code></pre>
-    </section>
-
-    <!-- ===================== 3. DATA ===================== -->
-    <section id="split-dir">
-      <h2>3.1 Split Directory Format <a class="anchor" href="#split-dir">#</a></h2>
-      <p>With <code>env.split_mode: split_dir</code> (the recommended, deterministic mode), SkillOpt reads a directory containing <code>train/</code>, <code>val/</code>, and <code>test/</code> subfolders, each holding a JSON array of task items:</p>
-<pre><code>data/my_split/
- ├─ train/items.json   <span class="tok-c"># used for rollout (the "train split")</span>
- ├─ val/items.json     <span class="tok-c"># selection split → validation gate (valid_seen)</span>
- └─ test/items.json    <span class="tok-c"># held-out final eval (valid_unseen)</span></code></pre>
-      <div class="note info"><span class="nh">Split naming</span>
-        <p>Internally the splits are referred to as <code>train</code>, <code>valid_seen</code> (validation/selection), and <code>valid_unseen</code> (test). The <code>--split</code> flag of <code>eval_only.py</code> uses these names.</p>
-      </div>
-    </section>
-
-    <section id="item-schema">
-      <h2>3.2 Item JSON Schema <a class="anchor" href="#item-schema">#</a></h2>
-      <p>Required fields depend on the benchmark; consult <code>skillopt/envs/&lt;benchmark&gt;/dataloader.py</code> for the exact contract. A SearchQA item, for example:</p>
-<pre><code>[
-  {
-    <span class="tok-f">"id"</span>:       <span class="tok-s">"unique_item_id"</span>,
-    <span class="tok-f">"question"</span>: <span class="tok-s">"Who wrote the novel ..."</span>,
-    <span class="tok-f">"context"</span>:  <span class="tok-s">"[DOC] relevant passage text ..."</span>,
-    <span class="tok-f">"answers"</span>:  [<span class="tok-s">"expected answer"</span>]
-  }
-]</code></pre>
-      <div class="note warn"><span class="nh">Datasets not included</span>
-        <p>This repository ships no benchmark data. Prepare your own splits in the format above before training.</p>
-      </div>
-    </section>
-
-    <section id="split-modes">
-      <h2>3.3 Split Modes <a class="anchor" href="#split-modes">#</a></h2>
-      <div class="table-wrap"><table>
-        <thead><tr><th><code>env.split_mode</code></th><th>Behavior</th></tr></thead>
-        <tbody>
-          <tr><td><code>split_dir</code></td><td>Use a pre-built directory with explicit <code>train/val/test</code> folders (set <code>env.split_dir</code>). Deterministic and reproducible.</td></tr>
-          <tr><td><code>ratio</code></td><td>Build a deterministic split on the fly from a single <code>env.data_path</code>, using <code>split_seed</code> (and a train:val:test ratio). Convenient for quick experiments.</td></tr>
-        </tbody>
-      </table></div>
-    </section>
-
-    <!-- ===================== 4. QUICK START ===================== -->
-    <section id="train">
-      <h2>4.1 Train a Skill <a class="anchor" href="#train">#</a></h2>
-<pre><code><span class="tok-c"># Minimal SearchQA run</span>
-<span class="tok-k">python</span> scripts/train.py \
-    <span class="tok-f">--config</span> configs/searchqa/default.yaml \
-    <span class="tok-f">--split_dir</span> /path/to/your/searchqa_split \
-    <span class="tok-f">--azure_openai_endpoint</span> https://your-resource.openai.azure.com/ \
-    <span class="tok-f">--optimizer_model</span> gpt-5.5 \
-    <span class="tok-f">--target_model</span> gpt-5.5</code></pre>
-      <p>Swap the config for another benchmark (e.g. <code>configs/livemathematicianbench/default.yaml</code>, <code>configs/alfworld/default.yaml</code>). Common CLI arguments:</p>
-      <div class="table-wrap"><table>
-        <thead><tr><th>Argument</th><th>Description</th></tr></thead>
-        <tbody>
-          <tr><td><code>--config</code></td><td>Benchmark config YAML (required)</td></tr>
-          <tr><td><code>--split_dir</code></td><td>Path to the data split directory</td></tr>
-          <tr><td><code>--azure_openai_endpoint</code></td><td>Azure OpenAI endpoint URL</td></tr>
-          <tr><td><code>--optimizer_model</code> / <code>--target_model</code></td><td>Deployment names for optimizer / target</td></tr>
-          <tr><td><code>--num_epochs</code> / <code>--batch_size</code></td><td>Epochs and rollout batch size</td></tr>
-          <tr><td><code>--out_root</code></td><td>Output directory</td></tr>
-          <tr><td><code>--cfg-options k=v ...</code></td><td>Override any config key (see §6.1)</td></tr>
-        </tbody>
-      </table></div>
-    </section>
-
-    <section id="eval">
-      <h2>4.2 Evaluate a Skill <a class="anchor" href="#eval">#</a></h2>
-      <p>Evaluate any skill document (a packaged reference skill, or a trained run's <code>best_skill.md</code>) without training:</p>
-<pre><code><span class="tok-c"># Evaluate the packaged GPT-5.5 SearchQA skill on the test split</span>
-<span class="tok-k">python</span> scripts/eval_only.py \
-  <span class="tok-f">--config</span> configs/searchqa/default.yaml \
-  <span class="tok-f">--skill</span> ckpt/searchqa/gpt5.5_skill.md \
-  <span class="tok-f">--split</span> valid_unseen \
-  <span class="tok-f">--split_dir</span> /path/to/searchqa_split \
-  <span class="tok-f">--azure_openai_endpoint</span> https://your-resource.openai.azure.com/</code></pre>
-      <div class="table-wrap"><table>
-        <thead><tr><th><code>--split</code></th><th>Meaning</th></tr></thead>
-        <tbody>
-          <tr><td><code>valid_unseen</code></td><td>Test set (held-out)</td></tr>
-          <tr><td><code>valid_seen</code></td><td>Validation / selection set</td></tr>
-          <tr><td><code>train</code></td><td>Training set</td></tr>
-          <tr><td><code>all</code></td><td>All splits combined (default)</td></tr>
-        </tbody>
-      </table></div>
-    </section>
-
-    <section id="outputs">
-      <h2>4.3 Output Structure <a class="anchor" href="#outputs">#</a></h2>
-<pre><code>outputs/&lt;run_name&gt;/
- ├─ config.json          <span class="tok-c"># flattened runtime config</span>
- ├─ history.json         <span class="tok-c"># per-step training history</span>
- ├─ runtime_state.json   <span class="tok-c"># resume checkpoint</span>
- ├─ best_skill.md        <span class="tok-c"># best validated skill document</span>
- ├─ skills/skill_vXXXX.md<span class="tok-c"># skill snapshot per step</span>
- ├─ steps/step_XXXX/     <span class="tok-c"># per-step artifacts (patches, evals)</span>
- ├─ slow_update/epoch_XX/<span class="tok-c"># slow-update logs &amp; rollouts</span>
- └─ meta_skill/epoch_XX/ <span class="tok-c"># meta-skill logs</span></code></pre>
-    </section>
-
-    <section id="resume">
-      <h2>4.4 Auto-Resume <a class="anchor" href="#resume">#</a></h2>
-      <p>Each completed step persists its state to <code>runtime_state.json</code> and a <code>steps/step_XXXX/</code> directory. Re-running the <em>same command</em> against the same <code>out_root</code> detects finished work and continues from the last completed step — including epoch-boundary slow-update and meta-skill stages.</p>
-    </section>
-
-    <!-- ===================== 5. HOW IT WORKS ===================== -->
-    <section id="loop">
-      <h2>5.1 The Training Loop <a class="anchor" href="#loop">#</a></h2>
-      <p>The loop lives in <code>ReflACTTrainer</code> (<code>skillopt/engine/trainer.py</code>). Each epoch runs a series of optimization steps over rollout batches, then performs two epoch-boundary stages.</p>
-<pre><code><span class="tok-k">for</span> epoch <span class="tok-k">in</span> epochs:
-    <span class="tok-k">for</span> step <span class="tok-k">in</span> steps:
-        1. Rollout    <span class="tok-c"># target executes a batch of tasks</span>
-        2. Reflect    <span class="tok-c"># optimizer analyzes trajectories → edit patches</span>
-        3. Aggregate  <span class="tok-c"># hierarchically merge similar patches</span>
-        4. Select     <span class="tok-c"># rank &amp; clip edits to the learning rate</span>
-        5. Update     <span class="tok-c"># apply patches → candidate skill</span>
-        6. Gate       <span class="tok-c"># score on selection split → accept / reject</span>
-
-    <span class="tok-c"># epoch boundary (from epoch 2 onward)</span>
-    Slow update   <span class="tok-c"># longitudinal comparison → protected guidance</span>
-    Meta skill    <span class="tok-c"># cross-epoch optimizer memory</span></code></pre>
-    </section>
-
-    <section id="stages">
-      <h2>5.2 The Six Per-Step Stages <a class="anchor" href="#stages">#</a></h2>
-      <div class="table-wrap"><table>
-        <thead><tr><th>Stage</th><th>What happens</th><th>Source</th></tr></thead>
-        <tbody>
-          <tr><td><strong>1. Rollout</strong></td><td>The target model runs each task in the batch with the current skill as context, producing trajectories and scores.</td><td><code>envs/&lt;b&gt;/rollout.py</code></td></tr>
-          <tr><td><strong>2. Reflect</strong></td><td>The optimizer runs an error analyst (and optional success analyst) over minibatches of trajectories, emitting structured edit patches. Runs in parallel across <code>analyst_workers</code>.</td><td><code>gradient/reflect.py</code></td></tr>
-          <tr><td><strong>3. Aggregate</strong></td><td>Semantically similar patches are merged hierarchically to remove redundancy.</td><td><code>gradient/aggregate.py</code> → <code>merge_patches</code></td></tr>
-          <tr><td><strong>4. Select</strong></td><td>Patches are ranked and clipped to the current learning rate (max edits this step), set by the scheduler.</td><td><code>optimizer/clip.py</code> → <code>rank_and_select</code></td></tr>
-          <tr><td><strong>5. Update</strong></td><td>Selected edits are applied to the skill document, producing a candidate skill (patch / rewrite modes).</td><td><code>optimizer/skill.py</code>, <code>update_modes.py</code></td></tr>
-          <tr><td><strong>6. Gate</strong></td><td>The candidate is scored on the selection split and accepted only if it improves (see §5.3).</td><td><code>evaluation/gate.py</code> → <code>evaluate_gate</code></td></tr>
-        </tbody>
-      </table></div>
-    </section>
-
-    <section id="gate">
-      <h2>5.3 Validation Gate <a class="anchor" href="#gate">#</a></h2>
-      <p><code>evaluate_gate</code> is a pure decision function. It compares the candidate's selection-set score against the <em>current</em> and <em>best</em> skills:</p>
-      <ul>
-        <li><strong>accept_new_best</strong> — candidate &gt; current <em>and</em> candidate &gt; best → becomes both current and best.</li>
-        <li><strong>accept</strong> — candidate &gt; current but ≤ best → becomes current only.</li>
-        <li><strong>reject</strong> — candidate ≤ current → discarded; current/best unchanged.</li>
-      </ul>
-      <p>The comparison metric is configurable via <code>evaluation.gate_metric</code>:</p>
-      <div class="table-wrap"><table>
-        <thead><tr><th>Metric</th><th>Score used</th></tr></thead>
-        <tbody>
-          <tr><td><code>hard</code> <span class="pill def">default</span></td><td>Exact-match / discrete score</td></tr>
-          <tr><td><code>soft</code></td><td>Partial-credit / continuous score</td></tr>
-          <tr><td><code>mixed</code></td><td>Weighted blend, controlled by <code>gate_mixed_weight</code></td></tr>
-        </tbody>
-      </table></div>
-      <div class="note info"><span class="nh">When to use soft/mixed</span>
-        <p>The <code>soft</code>/<code>mixed</code> metrics (contributed config <code>configs/examples/soft_gate.yaml</code>) help when the selection split is small and rewards are continuous, where a discrete hard gate may reject every candidate and stall training. Paper numbers use the default <code>hard</code> gate.</p>
-      </div>
-    </section>
-
-    <section id="slow-update">
-      <h2>5.4 Slow Update (Momentum) <a class="anchor" href="#slow-update">#</a></h2>
-      <p>At each epoch boundary (from epoch 2), the slow update rolls out both the <em>previous</em> epoch's skill and the <em>current</em> skill on the same sampled tasks, categorizes items (improved / regressed / persistent-fail / stable-success), and asks the optimizer to write a free-form <strong>guidance</strong> block. This guidance lands in a <strong>protected region</strong> of the skill that step-level edits cannot touch — only the slow update overwrites it. It is SkillOpt's analogue of momentum, countering cross-epoch forgetting.</p>
-      <p>Acceptance has two modes, selected by <code>optimizer.slow_update_gate_with_selection</code>:</p>
-      <div class="table-wrap"><table>
-        <thead><tr><th>Mode</th><th>Behavior</th></tr></thead>
-        <tbody>
-          <tr><td><code>false</code> <span class="pill def">default</span> — force-injected</td><td>Guidance is injected into both current and best skills unconditionally. The longitudinal guidance always persists; it is not gated by step-level selection scores.</td></tr>
-          <tr><td><code>true</code> — gated</td><td>The slow-update candidate is scored on the selection split and accepted/rejected through the same validation gate as step-level updates.</td></tr>
-        </tbody>
-      </table></div>
-    </section>
-
-    <section id="meta-skill">
-      <h2>5.5 Meta Skill (Optimizer Memory) <a class="anchor" href="#meta-skill">#</a></h2>
-      <p>The meta skill is <strong>optimizer-side memory</strong> — it never modifies the target skill document. At the end of each epoch (skipped for epoch 1), the optimizer compares the previous and current epoch's last-step skills on the same sampled tasks and writes a compact, evidence-based reflection on what kind of edits helped or hurt. That memory is then injected as extra context into the next epoch's reflect / merge / learning-rate / ranking stages, so the optimizer accumulates strategy across the run.</p>
-    </section>
-
-    <section id="skill-doc">
-      <h2>5.6 Skill Document Anatomy <a class="anchor" href="#skill-doc">#</a></h2>
-      <p>A skill document is plain Markdown. Initial skills can be empty (learn from scratch) or seeded with domain knowledge via <code>env.skill_init</code>. During training the document accrues rules, patterns, and edge-case handling through accepted edit patches. A dedicated protected region holds the slow-update guidance, delimited by HTML-comment markers:</p>
-<pre><code><span class="tok-c"># Question Answering Skill</span>
-
-<span class="tok-c">## Learned rules ...</span>
- When the context contains multiple candidates, prefer ...
-
-<span class="tok-c">&lt;!-- SLOW_UPDATE_START --&gt;</span>
-<span class="tok-c"># (epoch-level longitudinal guidance — only the slow update writes here)</span>
-<span class="tok-c">&lt;!-- SLOW_UPDATE_END --&gt;</span></code></pre>
-      <p>Helpers in <code>optimizer/slow_update.py</code> manage this region: <code>inject_empty_slow_update_field</code> (placeholder at epoch 1), <code>extract_slow_update_field</code> (read), and <code>replace_slow_update_field</code> (overwrite). Step-level edits are blocked from modifying anything inside the markers.</p>
-    </section>
-
-    <!-- ===================== 6. CONFIGURATION ===================== -->
-    <section id="config-system">
-      <h2>6.1 Configuration System <a class="anchor" href="#config-system">#</a></h2>
-      <p>Configs are <strong>structured YAML</strong> with section blocks (<code>model</code>, <code>train</code>, <code>gradient</code>, <code>optimizer</code>, <code>evaluation</code>, <code>env</code>) and <code>_base_</code> inheritance. A benchmark config inherits the shared defaults and overrides only what differs:</p>
-<pre><code><span class="tok-c"># configs/searchqa/default.yaml</span>
-<span class="tok-f">_base_</span>: ../_base_/default.yaml
-<span class="tok-f">train</span>:
-  <span class="tok-f">train_size</span>: <span class="tok-n">400</span>
-  <span class="tok-f">batch_size</span>: <span class="tok-n">40</span>
-<span class="tok-f">optimizer</span>:
-  <span class="tok-f">learning_rate</span>: <span class="tok-n">4</span>
-<span class="tok-f">env</span>:
-  <span class="tok-f">name</span>: searchqa
-  <span class="tok-f">split_dir</span>: data/searchqa_split</code></pre>
-      <p>Override any key at the command line without editing files:</p>
-<pre><code><span class="tok-k">python</span> scripts/train.py --config configs/searchqa/default.yaml \
-  <span class="tok-f">--cfg-options</span> optimizer.learning_rate=<span class="tok-n">16</span> optimizer.lr_scheduler=linear</code></pre>
-      <div class="note info"><span class="nh">Reading the tables below</span>
-        <p>Each section lists the key (relative to its YAML block), type, default (from <code>configs/_base_/default.yaml</code>), allowed values, and meaning. Defaults shown are the shipped base defaults.</p>
-      </div>
-    </section>
-
-    <section id="cfg-model">
-      <h2>6.2 <code>model.*</code> <a class="anchor" href="#cfg-model">#</a></h2>
-      <div class="table-wrap"><table>
-        <thead><tr><th>Key</th><th>Type</th><th>Default</th><th>Description / options</th></tr></thead>
-        <tbody>
-          <tr><td><code>backend</code></td><td>str</td><td class="def">azure_openai</td><td>High-level backend label for the run.</td></tr>
-          <tr><td><code>optimizer</code></td><td>str</td><td class="def">gpt-5.5</td><td>Optimizer model deployment (writes skill edits).</td></tr>
-          <tr><td><code>target</code></td><td>str</td><td class="def">gpt-5.5</td><td>Target model deployment (executes tasks).</td></tr>
-          <tr><td><code>optimizer_backend</code></td><td>str</td><td class="def">openai_chat</td><td>Client path for the optimizer: <code>openai_chat</code> or <code>claude_chat</code>.</td></tr>
-          <tr><td><code>target_backend</code></td><td>str</td><td class="def">openai_chat</td><td>Client path for the target: <code>openai_chat</code> / <code>claude_chat</code> / <code>qwen_chat</code> / <code>codex_exec</code> / <code>claude_code_exec</code>.</td></tr>
-          <tr><td><code>reasoning_effort</code></td><td>str</td><td class="def">medium</td><td><code>low</code> / <code>medium</code> / <code>high</code> / <code>xhigh</code> / <code>max</code> (or empty).</td></tr>
-          <tr><td><code>rewrite_reasoning_effort</code></td><td>str</td><td class="def">""</td><td>Override effort for full-rewrite calls (empty = inherit).</td></tr>
-          <tr><td><code>rewrite_max_completion_tokens</code></td><td>int</td><td class="def">64000</td><td>Token cap for full-rewrite optimizer calls.</td></tr>
-          <tr><td><code>azure_openai_endpoint</code></td><td>str</td><td class="def">""</td><td>Azure resource URL (or via <code>AZURE_OPENAI_ENDPOINT</code>).</td></tr>
-          <tr><td><code>azure_openai_api_version</code></td><td>str</td><td class="def">2024-12-01-preview</td><td>Azure API version header.</td></tr>
-          <tr><td><code>azure_openai_auth_mode</code></td><td>str</td><td class="def">""</td><td><code>api_key</code> / <code>azure_cli</code> / <code>managed_identity</code> / <code>openai_compatible</code> (empty → env default).</td></tr>
-        </tbody>
-      </table></div>
-      <div class="note info"><span class="nh">Separate optimizer / target endpoints</span>
-        <p>Every <code>azure_openai_*</code> key also has <code>optimizer_azure_openai_*</code> and <code>target_azure_openai_*</code> variants, letting you point the optimizer and target at different Azure resources. Exec backends (<code>codex_exec</code>, <code>claude_code_exec</code>) add their own <code>codex_exec_*</code> / <code>claude_code_exec_*</code> knobs (sandbox, reasoning effort, SDK mode, etc.).</p>
-      </div>
-    </section>
-
-    <section id="cfg-train">
-      <h2>6.3 <code>train.*</code> <a class="anchor" href="#cfg-train">#</a></h2>
-      <div class="table-wrap"><table>
-        <thead><tr><th>Key</th><th>Type</th><th>Default</th><th>DL analogy</th><th>Description</th></tr></thead>
-        <tbody>
-          <tr><td><code>num_epochs</code></td><td>int</td><td class="def">4</td><td>Epochs</td><td>Number of training epochs.</td></tr>
-          <tr><td><code>train_size</code></td><td>int</td><td class="def">0</td><td>Train-set size</td><td>0 = derive from the dataset split. (Fixed by split size when using <code>split_dir</code>.)</td></tr>
-          <tr><td><code>batch_size</code></td><td>int</td><td class="def">40</td><td>Batch size</td><td>Tasks rolled out per optimization step.</td></tr>
-          <tr><td><code>accumulation</code></td><td>int</td><td class="def">1</td><td>Grad accumulation</td><td>Accumulation rounds per step.</td></tr>
-          <tr><td><code>seed</code></td><td>int</td><td class="def">42</td><td>Random seed</td><td>Reproducibility seed.</td></tr>
-        </tbody>
-      </table></div>
-    </section>
-
-    <section id="cfg-gradient">
-      <h2>6.4 <code>gradient.*</code> <a class="anchor" href="#cfg-gradient">#</a></h2>
-      <div class="table-wrap"><table>
-        <thead><tr><th>Key</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>
-        <tbody>
-          <tr><td><code>minibatch_size</code></td><td>int</td><td class="def">8</td><td>Trajectories per reflect minibatch.</td></tr>
-          <tr><td><code>merge_batch_size</code></td><td>int</td><td class="def">8</td><td>Patches per merge batch during aggregation.</td></tr>
-          <tr><td><code>analyst_workers</code></td><td>int</td><td class="def">16</td><td>Parallel reflection workers (data parallelism).</td></tr>
-          <tr><td><code>max_analyst_rounds</code></td><td>int</td><td class="def">3</td><td>Max rounds of analyst reflection per step.</td></tr>
-          <tr><td><code>failure_only</code></td><td>bool</td><td class="def">false</td><td>Reflect only on failed trajectories when true.</td></tr>
-        </tbody>
-      </table></div>
-    </section>
-
-    <section id="cfg-optimizer">
-      <h2>6.5 <code>optimizer.*</code> <a class="anchor" href="#cfg-optimizer">#</a></h2>
-      <div class="table-wrap"><table>
-        <thead><tr><th>Key</th><th>Type</th><th>Default</th><th>DL analogy</th><th>Description / options</th></tr></thead>
-        <tbody>
-          <tr><td><code>learning_rate</code></td><td>int</td><td class="def">4</td><td>Learning rate</td><td>Max edit patches applied per step (the "edit budget").</td></tr>
-          <tr><td><code>min_learning_rate</code></td><td>int</td><td class="def">2</td><td>Min LR</td><td>Floor edit budget for decaying schedulers.</td></tr>
-          <tr><td><code>lr_scheduler</code></td><td>str</td><td class="def">cosine</td><td>LR schedule</td><td><code>constant</code> / <code>linear</code> / <code>cosine</code> / <code>autonomous</code>.</td></tr>
-          <tr><td><code>lr_control_mode</code></td><td>str</td><td class="def">fixed</td><td>—</td><td><code>fixed</code> / <code>autonomous</code> / <code>none</code>.</td></tr>
-          <tr><td><code>skill_update_mode</code></td><td>str</td><td class="def">patch</td><td>—</td><td><code>patch</code> / <code>rewrite_from_suggestions</code> / <code>full_rewrite_minibatch</code>.</td></tr>
-          <tr><td><code>use_slow_update</code></td><td>bool</td><td class="def">true</td><td>Momentum</td><td>Enable epoch-boundary slow update.</td></tr>
-          <tr><td><code>slow_update_samples</code></td><td>int</td><td class="def">20</td><td>—</td><td>Tasks sampled for the longitudinal comparison.</td></tr>
-          <tr><td><code>slow_update_gate_with_selection</code></td><td>bool</td><td class="def">false</td><td>—</td><td><code>false</code> = force-inject guidance; <code>true</code> = gate it on the selection split (see §5.4).</td></tr>
-          <tr><td><code>longitudinal_pair_policy</code></td><td>str</td><td class="def">mixed</td><td>—</td><td><code>mixed</code> / <code>changed</code> / <code>unchanged</code> — which comparison pairs to keep.</td></tr>
-          <tr><td><code>use_meta_skill</code></td><td>bool</td><td class="def">true</td><td>Meta-learning</td><td>Enable cross-epoch optimizer memory.</td></tr>
-        </tbody>
-      </table></div>
-    </section>
-
-    <section id="cfg-evaluation">
-      <h2>6.6 <code>evaluation.*</code> <a class="anchor" href="#cfg-evaluation">#</a></h2>
-      <div class="table-wrap"><table>
-        <thead><tr><th>Key</th><th>Type</th><th>Default</th><th>Description / options</th></tr></thead>
-        <tbody>
-          <tr><td><code>use_gate</code></td><td>bool</td><td class="def">true</td><td>Validation gating is mandatory in this branch (must remain <code>true</code>).</td></tr>
-          <tr><td><code>gate_metric</code></td><td>str</td><td class="def">hard</td><td><code>hard</code> / <code>soft</code> / <code>mixed</code> — score used by the gate (see §5.3).</td></tr>
-          <tr><td><code>gate_mixed_weight</code></td><td>float</td><td class="def">0.5</td><td>Weight on the soft score when <code>gate_metric = mixed</code>.</td></tr>
-          <tr><td><code>sel_env_num</code></td><td>int</td><td class="def">0</td><td>Selection-split eval size (0 = use full split).</td></tr>
-          <tr><td><code>test_env_num</code></td><td>int</td><td class="def">0</td><td>Test-split eval size (0 = use full split).</td></tr>
-          <tr><td><code>eval_test</code></td><td>bool</td><td class="def">true</td><td>Run a final test evaluation after training.</td></tr>
-        </tbody>
-      </table></div>
-      <div class="note warn"><span class="nh">Gate is required</span>
-        <p>Setting <code>evaluation.use_gate: false</code> raises an error — validation gating cannot be disabled in this branch.</p>
-      </div>
-    </section>
-
-    <section id="cfg-env">
-      <h2>6.7 <code>env.*</code> <a class="anchor" href="#cfg-env">#</a></h2>
-      <div class="table-wrap"><table>
-        <thead><tr><th>Key</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>
-        <tbody>
-          <tr><td><code>name</code></td><td>str</td><td class="def">""</td><td>Benchmark name (<code>searchqa</code>, <code>docvqa</code>, <code>alfworld</code>, …). Selects the env module.</td></tr>
-          <tr><td><code>skill_init</code></td><td>str</td><td class="def">""</td><td>Path to a seed skill (empty = start from scratch).</td></tr>
-          <tr><td><code>split_mode</code></td><td>str</td><td class="def">ratio</td><td><code>ratio</code> or <code>split_dir</code> (see §3.3).</td></tr>
-          <tr><td><code>split_dir</code></td><td>str</td><td class="def">""</td><td>Pre-split directory (when <code>split_mode = split_dir</code>).</td></tr>
-          <tr><td><code>data_path</code></td><td>str</td><td class="def">""</td><td>Single dataset path (when <code>split_mode = ratio</code>).</td></tr>
-          <tr><td><code>split_seed</code></td><td>int</td><td class="def">42</td><td>Seed for deterministic ratio splitting.</td></tr>
-          <tr><td><code>exec_timeout</code></td><td>int</td><td class="def">120</td><td>Per-task target/code-agent timeout (seconds).</td></tr>
-          <tr><td><code>out_root</code></td><td>str</td><td class="def">""</td><td>Output directory for the run.</td></tr>
-        </tbody>
-      </table></div>
-      <div class="note info"><span class="nh">Benchmark-specific env keys</span>
-        <p>Env blocks may carry extra benchmark-specific keys (e.g. <code>max_turns</code>, <code>workers</code>, <code>max_completion_tokens</code>, <code>limit</code>). Unmapped env keys are passed straight through to the benchmark adapter — check the relevant <code>configs/&lt;benchmark&gt;/default.yaml</code>.</p>
-      </div>
-    </section>
-
-    <!-- ===================== 7. BENCHMARKS ===================== -->
-    <section id="bench-list">
-      <h2>7.1 Supported Benchmarks <a class="anchor" href="#bench-list">#</a></h2>
-      <div class="table-wrap"><table>
-        <thead><tr><th>Benchmark</th><th>Type</th><th>Config</th></tr></thead>
-        <tbody>
-          <tr><td>SearchQA</td><td>Question answering</td><td><code>configs/searchqa/default.yaml</code></td></tr>
-          <tr><td>DocVQA</td><td>Document QA</td><td><code>configs/docvqa/default.yaml</code></td></tr>
-          <tr><td>ALFWorld</td><td>Embodied agent</td><td><code>configs/alfworld/default.yaml</code></td></tr>
-          <tr><td>LiveMathematicianBench</td><td>Math reasoning</td><td><code>configs/livemathematicianbench/default.yaml</code></td></tr>
-          <tr><td>SpreadsheetBench</td><td>Spreadsheet code generation</td><td><code>configs/spreadsheetbench/default.yaml</code></td></tr>
-          <tr><td>OfficeQA</td><td>Tool-augmented QA</td><td><code>configs/officeqa/default.yaml</code></td></tr>
-        </tbody>
-      </table></div>
-      <p>Each benchmark is a self-contained module under <code>skillopt/envs/&lt;benchmark&gt;/</code> with an <code>adapter.py</code>, <code>dataloader.py</code>, <code>rollout.py</code>, and <code>evaluator.py</code> (some add a custom <code>reflect.py</code>). Packaged reference skills live in <code>ckpt/&lt;benchmark&gt;/</code>.</p>
-    </section>
-
-    <section id="bench-new">
-      <h2>7.2 Add a New Benchmark <a class="anchor" href="#bench-new">#</a></h2>
-      <p>Use <code>skillopt/envs/_template/</code> as a starting point. At minimum, implement:</p>
-      <ol>
-        <li><strong>Dataloader</strong> — read your item JSON into the framework's item dicts (<code>dataloader.py</code>).</li>
-        <li><strong>Rollout</strong> — run the target on one item with the current skill and return a trajectory + score (<code>rollout.py</code>).</li>
-        <li><strong>Evaluator</strong> — score predictions against ground truth (<code>evaluator.py</code>).</li>
-        <li><strong>Adapter</strong> — wire the above into the trainer's expected interface and register the env name (<code>adapter.py</code>).</li>
-      </ol>
-      <p>Then add a <code>configs/&lt;name&gt;/default.yaml</code> inheriting <code>_base_/default.yaml</code> and set <code>env.name</code> to your new benchmark.</p>
-    </section>
-
-    <!-- ===================== 8. API REFERENCE ===================== -->
-    <section id="module-map">
-      <h2>8.1 Module Map <a class="anchor" href="#module-map">#</a></h2>
-      <div class="table-wrap"><table>
-        <thead><tr><th>Module</th><th>Responsibility</th></tr></thead>
-        <tbody>
-          <tr><td><code>skillopt/config.py</code></td><td>Load structured YAML, resolve <code>_base_</code> inheritance, flatten to the trainer's flat dict, apply CLI overrides.</td></tr>
-          <tr><td><code>skillopt/engine/trainer.py</code></td><td><code>ReflACTTrainer</code> — orchestrates the whole loop, gating, slow update, meta skill, resume, and artifact writing.</td></tr>
-          <tr><td><code>skillopt/gradient/</code></td><td>Reflection ("backward pass"): <code>reflect.py</code> analysts, <code>aggregate.py</code> patch merging.</td></tr>
-          <tr><td><code>skillopt/optimizer/</code></td><td>The "optimizer": edit application, learning-rate scheduling, edit selection, slow update, meta skill, rewrite modes.</td></tr>
-          <tr><td><code>skillopt/evaluation/gate.py</code></td><td>Pure accept/reject decision and metric selection.</td></tr>
-          <tr><td><code>skillopt/model/</code></td><td>Backend clients (OpenAI/Azure, Claude, Qwen, Codex/Claude-Code exec) and routing.</td></tr>
-          <tr><td><code>skillopt/envs/&lt;b&gt;/</code></td><td>Per-benchmark dataloader, rollout, evaluator, adapter.</td></tr>
-        </tbody>
-      </table></div>
-    </section>
-
-    <section id="functions">
-      <h2>8.2 Core Functions <a class="anchor" href="#functions">#</a></h2>
-      <div class="table-wrap"><table>
-        <thead><tr><th>Function</th><th>File</th><th>Purpose</th></tr></thead>
-        <tbody>
-          <tr><td><code>load_config</code> / <code>flatten_config</code> / <code>apply_overrides</code></td><td><code>config.py</code></td><td>Load YAML with inheritance; flatten sections; apply <code>key=value</code> overrides.</td></tr>
-          <tr><td><code>run_minibatch_reflect</code></td><td><code>gradient/reflect.py</code></td><td>Run error/success analysts over trajectory minibatches → edit patches.</td></tr>
-          <tr><td><code>merge_patches</code></td><td><code>gradient/aggregate.py</code></td><td>Hierarchically merge semantically similar patches.</td></tr>
-          <tr><td><code>rank_and_select</code></td><td><code>optimizer/clip.py</code></td><td>Rank edits and clip to the learning-rate budget.</td></tr>
-          <tr><td><code>build_scheduler</code></td><td><code>optimizer/scheduler.py</code></td><td>Construct the LR (edit-budget) scheduler: constant/linear/cosine/autonomous.</td></tr>
-          <tr><td><code>decide_autonomous_learning_rate</code></td><td><code>optimizer/lr_autonomous.py</code></td><td>Let the optimizer pick the next learning rate (autonomous mode).</td></tr>
-          <tr><td><code>apply_patch</code> / <code>apply_edit</code></td><td><code>optimizer/skill.py</code></td><td>Apply edits to the skill document (respecting the protected region).</td></tr>
-          <tr><td><code>rewrite_skill_from_suggestions</code></td><td><code>optimizer/rewrite.py</code></td><td>Full-rewrite update mode from accumulated suggestions.</td></tr>
-          <tr><td><code>evaluate_gate</code> / <code>select_gate_score</code></td><td><code>evaluation/gate.py</code></td><td>Accept/reject decision; compute hard/soft/mixed score.</td></tr>
-          <tr><td><code>run_slow_update</code></td><td><code>optimizer/slow_update.py</code></td><td>Produce epoch-boundary longitudinal guidance.</td></tr>
-          <tr><td><code>replace_slow_update_field</code> / <code>extract_slow_update_field</code></td><td><code>optimizer/slow_update.py</code></td><td>Read/overwrite the protected guidance region.</td></tr>
-          <tr><td><code>run_meta_skill</code> / <code>format_meta_skill_context</code></td><td><code>optimizer/meta_skill.py</code></td><td>Generate cross-epoch optimizer memory and render it into reflection context.</td></tr>
-        </tbody>
-      </table></div>
-    </section>
-
-    <section id="cli">
-      <h2>8.3 CLI Scripts <a class="anchor" href="#cli">#</a></h2>
-      <h4>scripts/train.py</h4>
-      <p>Runs a full training loop. Required: <code>--config</code>. Override config via <code>--cfg-options section.key=value …</code> or legacy flat flags (<code>--num_epochs</code>, <code>--batch_size</code>, <code>--optimizer_model</code>, <code>--target_model</code>, <code>--lr_scheduler</code>, <code>--edit_budget</code>, <code>--split_dir</code>, …).</p>
-      <h4>scripts/eval_only.py</h4>
-      <p>Evaluates a skill document without training. Required: <code>--config</code> and <code>--skill</code>. Use <code>--split</code> to choose <code>train</code> / <code>valid_seen</code> / <code>valid_unseen</code> / <code>all</code>.</p>
-<pre><code><span class="tok-k">python</span> scripts/eval_only.py \
-  --config configs/searchqa/default.yaml \
-  --skill outputs/my_run/best_skill.md \
-  --split valid_unseen</code></pre>
-    </section>
-
-    <section id="webui">
-      <h2>8.4 WebUI <a class="anchor" href="#webui">#</a></h2>
-      <p>An optional Gradio dashboard to configure parameters and monitor runs:</p>
-<pre><code><span class="tok-k">pip</span> install -e <span class="tok-s">".[webui]"</span>
-<span class="tok-k">python</span> -m skillopt_webui.app          <span class="tok-c"># http://localhost:7860</span>
-<span class="tok-k">python</span> -m skillopt_webui.app --share  <span class="tok-c"># public share link</span></code></pre>
-      <div class="table-wrap"><table>
-        <thead><tr><th>Flag</th><th>Default</th><th>Description</th></tr></thead>
-        <tbody>
-          <tr><td><code>--port</code></td><td class="def">7860</td><td>Server port.</td></tr>
-          <tr><td><code>--host</code></td><td class="def">0.0.0.0</td><td>Bind address.</td></tr>
-          <tr><td><code>--share</code></td><td class="def">off</td><td>Create a public Gradio share link.</td></tr>
-        </tbody>
-      </table></div>
-
-      <div class="footer-note">
-        SkillOpt — Executive Strategy for Self-Evolving Agent Skills ·
-        <a href="https://github.com/microsoft/SkillOpt">github.com/microsoft/SkillOpt</a> ·
-        <a href="https://arxiv.org/abs/2605.23904">arXiv:2605.23904</a><br>
-        This guide reflects the current configuration defaults in <code>configs/_base_/default.yaml</code>. When in doubt, the code is the source of truth.
-      </div>
-    </section>
-
-  </main>
-
-  <!-- ───────────── RIGHT TOC ───────────── -->
-  <aside class="toc" id="toc">
-    <div class="tl">On this page</div>
-    <div id="tocLinks"></div>
-  </aside>
-
-</div>
-
-<script>
-(function () {
-  // Build right-hand "On this page" from <h2> elements
-  var sections = Array.prototype.slice.call(document.querySelectorAll('main.content section[id]'));
-  var tocLinks = document.getElementById('tocLinks');
-  var h2s = Array.prototype.slice.call(document.querySelectorAll('main.content h2'));
-  h2s.forEach(function (h) {
-    var sec = h.closest('section');
-    if (!sec || !sec.id) return;
-    var a = document.createElement('a');
-    a.href = '#' + sec.id;
-    a.textContent = h.textContent.replace(/#$/, '').trim();
-    a.dataset.target = sec.id;
-    tocLinks.appendChild(a);
-  });
-
-  var sideLinks = Array.prototype.slice.call(document.querySelectorAll('nav.sidebar a'));
-  var tocAnchors = Array.prototype.slice.call(tocLinks.querySelectorAll('a'));
-
-  function setActive(id) {
-    sideLinks.forEach(function (a) {
-      a.classList.toggle('active', a.getAttribute('href') === '#' + id);
-    });
-    tocAnchors.forEach(function (a) {
-      a.classList.toggle('active', a.dataset.target === id);
-    });
-  }
-
-  // Scroll spy
-  var observer = new IntersectionObserver(function (entries) {
-    entries.forEach(function (e) {
-      if (e.isIntersecting) setActive(e.target.id);
-    });
-  }, { rootMargin: '-64px 0px -75% 0px', threshold: 0 });
-  sections.forEach(function (s) { observer.observe(s); });
-
-  // Mobile sidebar toggle
-  var btn = document.getElementById('menuBtn');
-  var sidebar = document.getElementById('sidebar');
-  btn.addEventListener('click', function () { sidebar.classList.toggle('open'); });
-  sideLinks.forEach(function (a) {
-    a.addEventListener('click', function () { sidebar.classList.remove('open'); });
-  });
-})();
-</script>
-</body>
-</html>
@@ -1,81 +1,195 @@
 # API Reference

+This page documents the public Python API SkillOpt exposes for **extending the
+framework** with new environments / benchmarks. For ready-made adapters,
+browse [`skillopt/envs/`](https://github.com/microsoft/SkillOpt/tree/main/skillopt/envs).
+
+> **Source of truth.** The classes below are real Python ABCs defined in
+> `skillopt/envs/base.py`, `skillopt/datasets/base.py`, `skillopt/types.py`,
+> and `skillopt/evaluation/gate.py`. If this page ever drifts, the code
+> wins — please open an issue.
+
+---
+
 ## Core Classes

 ### `EnvAdapter`

-Abstract base class for benchmark environments.
+`skillopt/envs/base.py` — abstract adapter that connects the SkillOpt
+trainer to an environment (benchmark, simulator, REST API, ...).
+Subclasses **must** implement the five abstract methods below.

 ```python
+from abc import ABC, abstractmethod
+from skillopt.datasets.base import BaseDataLoader, BatchSpec
+
 class EnvAdapter(ABC):
-    async def execute(self, item, skill, model) -> TaskResult
-    def evaluate(self, prediction, ground_truth) -> float
-    def build_prompt(self, item, skill) -> str
+
+    # ── Lifecycle hooks (have defaults; override only if needed) ────────
+
+    def setup(self, cfg: dict) -> None: ...
+    def get_dataloader(self) -> BaseDataLoader | None: ...
+    def requires_ray(self) -> bool: ...                 # default False
+
+    # ── Abstract methods (subclasses MUST implement) ────────────────────
+
+    @abstractmethod
+    def build_train_env(self, batch_size: int, seed: int, **kwargs):
+        """Return an environment-manager object to be passed to rollout()."""
+
+    @abstractmethod
+    def build_eval_env(self, env_num: int, split: str, seed: int, **kwargs):
+        """Like build_train_env() but for a fixed eval split."""
+
+    @abstractmethod
+    def rollout(self, env_manager, skill_content: str,
+                out_dir: str, **kwargs) -> list[dict]:
+        """Run a batch of episodes with the current skill.
+
+        Each returned dict MUST contain:
+          - "id":   str        episode/task identifier
+          - "hard": int (0|1)  pass/fail (may be float 0.0-1.0 if smoothed)
+          - "soft": float      partial-credit score in [0.0, 1.0]
+        It MAY contain env-specific extra keys (parsed into RolloutResult.extras).
+        """
+
+    @abstractmethod
+    def reflect(self, results: list[dict], skill_content: str,
+                out_dir: str, **kwargs) -> list[dict | None]:
+        """Turn rollout results into a list of raw patch dicts.
+
+        Each dict (or None to drop the slot) MUST contain:
+          - "patch":       {"edits": [...]}     a Patch.to_dict() payload
+          - "source_type": "failure" | "success"
+        """
+
+    @abstractmethod
+    def get_task_types(self) -> list[str]:
+        """Distinct task-type strings used for stratified sampling."""
 ```

-### `DataLoader`
+The trainer also calls a few default-implemented helpers on every adapter:
+`build_reference_text`, `get_reference_metadata`, `attach_reference_context`,
+`select_representative_items`, and `build_env_from_batch`. Read the docstrings
+in `skillopt/envs/base.py` if you need to override any of these — most
+benchmarks don't.

-Abstract base class for data loading and splitting.
+### `BaseDataLoader` / `SplitDataLoader`
+
+`skillopt/datasets/base.py` — episode-planning loaders.

 ```python
-class DataLoader(ABC):
-    def setup(self, cfg: dict) -> None
-    def get_split_items(self, split: str) -> list[DataItem]
+class BaseDataLoader(ABC):
+    def setup(self, cfg: dict) -> None: ...
+    @abstractmethod
+    def build_train_batch(self, batch_size: int, seed: int, **kwargs) -> BatchSpec: ...
+    @abstractmethod
+    def build_eval_batch(self, env_num: int, split: str, seed: int, **kwargs) -> BatchSpec: ...
+
+class SplitDataLoader(BaseDataLoader):
+    """Concrete base for dataset-backed envs with on-disk train/val/test splits.
+
+    Subclasses only need to implement load_split_items() (and optionally
+    load_raw_items() if you also want ``split_mode='ratio'``).
+    """
+    def load_split_items(self, split_path: str) -> list[dict]: ...
+    def load_raw_items(self, data_path: str) -> list[dict]: ...   # optional
 ```

-### `ModelBackend`
+`SplitDataLoader` handles two layout modes:

-Abstract base class for LLM backends.
+| `split_mode` | What it expects |
+|---|---|
+| `"split_dir"` | A directory with `train/`, `val/`, `test/` subdirs already split. |
+| `"ratio"` | A raw dataset path + `split_ratio: "2:1:7"` style string. |
+
+In either case the items returned by `load_split_items()` are plain
+`dict` objects with at minimum an `"id"` key.
+
+### `BatchSpec`
+
+`skillopt/datasets/base.py` — a slotted dataclass describing one batch
+request the trainer hands to the adapter.

 ```python
-class ModelBackend(ABC):
-    async def generate(self, messages, **kwargs) -> ModelResponse
-    async def generate_with_tools(self, messages, tools, **kwargs) -> ModelResponse
-```
-
-### `Trainer`
-
-Main training loop orchestrator.
-
-```python
-class Trainer:
-    def __init__(self, cfg: dict)
-    async def train(self) -> TrainResult
-    async def evaluate(self, skill: str, split: str) -> EvalResult
-```
-
-## Data Classes
-
-### `DataItem`
-
-```python
-@dataclass
-class DataItem:
-    id: str
-    input: str
-    ground_truth: str
+@dataclass(slots=True)
+class BatchSpec:
+    phase: str                 # "train" | "eval"
+    split: str                 # "train" | "val" | "test" | "valid_seen" | ...
+    seed: int
+    batch_size: int
+    payload: object | None = None     # what the loader produced (e.g. list[dict])
    metadata: dict = field(default_factory=dict)
 ```

-### `TaskResult`
+### `Edit` / `Patch`
+
+`skillopt/types.py` — the I/O types Reflect / Aggregate / Update produce
+and consume.

 ```python
+EditOp = Literal["append", "insert_after", "replace", "delete"]
+
@dataclass
-class TaskResult:
-    item_id: str
-    prediction: str
-    score: float
-    trajectory: list[dict]
+class Edit:
+    op: EditOp
+    content: str = ""
+    target: str = ""
+    support_count: int | None = None
+    source_type: Literal["failure", "success"] | None = None
+    merge_level: int | None = None
+    update_origin: str = ""
+    update_target: str = ""
+
+@dataclass
+class Patch:
+    edits: list[Edit] = field(default_factory=list)
+    reasoning: str = ""
+    ranking_details: dict[str, Any] | None = None
 ```

-### `ModelResponse`
+Both types support `to_dict()` / `from_dict()` for serialization.

-```python
-@dataclass
-class ModelResponse:
-    content: str
-    usage: dict
-    model: str
-```
+### `RolloutResult`

-For detailed source code, see the [`skillopt/`](https://github.com/microsoft/SkillOpt/tree/main/skillopt) directory.
+`skillopt/types.py` — the normalised rollout return type. The trainer
+calls `RolloutResult.from_dict(...)` on each dict returned from
+`EnvAdapter.rollout()`, so the only **hard** requirement on those dicts is
+the three keys above (`id`, `hard`, `soft`). Extra fields are preserved
+into `RolloutResult.extras`.
+
+### `GateResult` / `GateAction`
+
+`skillopt/evaluation/gate.py` — the validation-gate decision types
+returned each epoch.
+
+---
+
+## Registering an environment
+
+Environments are not registered via decorators or a `BENCHMARK_REGISTRY`
+dict. The trainer keeps a lazy registry inside `scripts/train.py` —
+`_ENV_REGISTRY` — populated by `_register_builtins()`. To add a new env
+you append a `try / except ImportError` block there. See
+[Add a New Benchmark](../guide/new-benchmark.md) for the full step-by-step.
+
+---
+
+## Backends (model layer)
+
+The model layer lives under `skillopt.model.*`. Backends are selected
+via `model.optimizer_backend` and `model.target_backend` in the config —
+not via a base class subclass. Supported values (as of this writing):
+
+| Backend | Optimizer? | Target? |
+|---|---|---|
+| `openai_chat` | ✓ | ✓ |
+| `claude_chat` | ✓ | ✓ |
+| `qwen_chat` | ✓ | ✓ |
+| `minimax_chat` | ✓ | ✓ |
+| `codex_exec` | — | ✓ |
+| `claude_code_exec` | — | ✓ |
+
+See `skillopt/model/backend_config.py` for the live whitelist and
+[`docs/reference/config.md`](./config.md) for the per-backend
+configuration keys.
@@ -10,6 +10,12 @@ Complete reference for all SkillOpt configuration parameters.
 | `model.optimizer` | str | `gpt-5.5` | Optimizer model (for reflection & slow update) |
 | `model.target` | str | `gpt-5.5` | Target model (for rollout execution) |
 | `model.reasoning_effort` | str | `medium` | Reasoning effort level |
+| `model.optimizer_backend` | str | `openai_chat` | Optimizer backend: `openai_chat` / `claude_chat` / `qwen_chat` / `minimax_chat` |
+| `model.target_backend` | str | `openai_chat` | Target backend: chat backends plus execution harnesses |
+| `model.qwen_chat_base_url` | str | `http://localhost:8000/v1` | Shared Qwen/vLLM OpenAI-compatible endpoint |
+| `model.qwen_chat_enable_thinking` | bool | `false` | Shared Qwen thinking flag |
+| `model.optimizer_qwen_chat_base_url` | str | — | Optimizer-specific Qwen/vLLM endpoint; overrides shared `qwen_chat_base_url` |
+| `model.target_qwen_chat_base_url` | str | — | Target-specific Qwen/vLLM endpoint; overrides shared `qwen_chat_base_url` |

 ## Training (`train`)

@@ -70,3 +76,10 @@ Complete reference for all SkillOpt configuration parameters.
 | `AZURE_OPENAI_API_KEY` / `model.azure_openai_api_key` | Azure API key |
 | `OPENAI_API_KEY` | OpenAI API key (for `openai_chat` backend) |
 | `ANTHROPIC_API_KEY` | Anthropic API key (for `claude_code_exec` backend) |
+| `QWEN_CHAT_BASE_URL` | Shared local vLLM endpoint for `qwen_chat` |
+| `QWEN_CHAT_MODEL` | Shared served model name for `qwen_chat` |
+| `QWEN_CHAT_API_KEY` | Optional API key for the shared Qwen endpoint |
+| `OPTIMIZER_QWEN_CHAT_BASE_URL` | Optimizer-specific local vLLM endpoint |
+| `OPTIMIZER_QWEN_CHAT_MODEL` | Optimizer-specific served model name |
+| `TARGET_QWEN_CHAT_BASE_URL` | Target-specific local vLLM endpoint |
+| `TARGET_QWEN_CHAT_MODEL` | Target-specific served model name |
@@ -137,7 +137,7 @@ def parse_args() -> argparse.Namespace:
    # Legacy flat CLI overrides (still work, prefer --cfg-options for new usage)
    p.add_argument("--env", type=str)
    p.add_argument("--backend", type=str,
-                   choices=["azure_openai", "codex", "codex_exec", "claude", "claude_chat", "claude_code_exec", "qwen", "qwen_chat"])
+                   choices=["azure_openai", "codex", "codex_exec", "claude", "claude_chat", "claude_code_exec", "qwen", "qwen_chat", "minimax", "minimax_chat"])
    p.add_argument("--optimizer_model", type=str)
    p.add_argument("--target_model", type=str)
    p.add_argument("--optimizer_backend", type=str)
@@ -173,6 +173,24 @@ def parse_args() -> argparse.Namespace:
    p.add_argument("--qwen_chat_timeout_seconds", type=float)
    p.add_argument("--qwen_chat_max_tokens", type=int)
    p.add_argument("--qwen_chat_enable_thinking", type=_BOOL)
+    p.add_argument("--optimizer_qwen_chat_base_url", type=str)
+    p.add_argument("--optimizer_qwen_chat_api_key", type=str)
+    p.add_argument("--optimizer_qwen_chat_temperature", type=float)
+    p.add_argument("--optimizer_qwen_chat_timeout_seconds", type=float)
+    p.add_argument("--optimizer_qwen_chat_max_tokens", type=int)
+    p.add_argument("--optimizer_qwen_chat_enable_thinking", type=_BOOL)
+    p.add_argument("--target_qwen_chat_base_url", type=str)
+    p.add_argument("--target_qwen_chat_api_key", type=str)
+    p.add_argument("--target_qwen_chat_temperature", type=float)
+    p.add_argument("--target_qwen_chat_timeout_seconds", type=float)
+    p.add_argument("--target_qwen_chat_max_tokens", type=int)
+    p.add_argument("--target_qwen_chat_enable_thinking", type=_BOOL)
+    p.add_argument("--minimax_base_url", type=str)
+    p.add_argument("--minimax_api_key", type=str)
+    p.add_argument("--minimax_model", type=str)
+    p.add_argument("--minimax_temperature", type=float)
+    p.add_argument("--minimax_max_tokens", type=int)
+    p.add_argument("--minimax_enable_thinking", type=_BOOL)
    p.add_argument("--codex_exec_path", type=str)
    p.add_argument("--codex_exec_sandbox", type=str)
    p.add_argument("--codex_exec_profile", type=str)
@@ -289,6 +307,24 @@ _LEGACY_TO_STRUCTURED: dict[str, str] = {
    "qwen_chat_timeout_seconds": "model.qwen_chat_timeout_seconds",
    "qwen_chat_max_tokens": "model.qwen_chat_max_tokens",
    "qwen_chat_enable_thinking": "model.qwen_chat_enable_thinking",
+    "optimizer_qwen_chat_base_url": "model.optimizer_qwen_chat_base_url",
+    "optimizer_qwen_chat_api_key": "model.optimizer_qwen_chat_api_key",
+    "optimizer_qwen_chat_temperature": "model.optimizer_qwen_chat_temperature",
+    "optimizer_qwen_chat_timeout_seconds": "model.optimizer_qwen_chat_timeout_seconds",
+    "optimizer_qwen_chat_max_tokens": "model.optimizer_qwen_chat_max_tokens",
+    "optimizer_qwen_chat_enable_thinking": "model.optimizer_qwen_chat_enable_thinking",
+    "target_qwen_chat_base_url": "model.target_qwen_chat_base_url",
+    "target_qwen_chat_api_key": "model.target_qwen_chat_api_key",
+    "target_qwen_chat_temperature": "model.target_qwen_chat_temperature",
+    "target_qwen_chat_timeout_seconds": "model.target_qwen_chat_timeout_seconds",
+    "target_qwen_chat_max_tokens": "model.target_qwen_chat_max_tokens",
+    "target_qwen_chat_enable_thinking": "model.target_qwen_chat_enable_thinking",
+    "minimax_base_url": "model.minimax_base_url",
+    "minimax_api_key": "model.minimax_api_key",
+    "minimax_model": "model.minimax_model",
+    "minimax_temperature": "model.minimax_temperature",
+    "minimax_max_tokens": "model.minimax_max_tokens",
+    "minimax_enable_thinking": "model.minimax_enable_thinking",
    "codex_exec_path": "model.codex_exec_path",
    "codex_exec_sandbox": "model.codex_exec_sandbox",
    "codex_exec_profile": "model.codex_exec_profile",
@@ -403,6 +439,9 @@ def load_config(args: argparse.Namespace) -> dict:
        elif backend in {"qwen", "qwen_chat"}:
            flat.setdefault("optimizer_backend", "openai_chat")
            flat.setdefault("target_backend", "qwen_chat")
+        elif backend in {"minimax", "minimax_chat"}:
+            flat.setdefault("optimizer_backend", "openai_chat")
+            flat.setdefault("target_backend", "minimax_chat")
        else:
            flat.setdefault("optimizer_backend", "openai_chat")
            flat.setdefault("target_backend", "openai_chat")
@@ -416,6 +455,12 @@ def load_config(args: argparse.Namespace) -> dict:
            and not _has_model_override("model.optimizer", "optimizer_model")
        ):
            flat["optimizer_model"] = default_model_for_backend("claude_chat")
+    if flat.get("optimizer_backend") == "qwen_chat":
+        if (
+            str(flat.get("optimizer_model", "") or "").strip() in _OPENAI_DEFAULT_MODEL_SENTINELS
+            and not _has_model_override("model.optimizer", "optimizer_model")
+        ):
+            flat["optimizer_model"] = default_model_for_backend("qwen_chat")
    if flat.get("target_backend") == "claude_chat":
        if (
            str(flat.get("target_model", "") or "").strip() in _OPENAI_DEFAULT_MODEL_SENTINELS
@@ -434,6 +479,15 @@ def load_config(args: argparse.Namespace) -> dict:
            and not _has_model_override("model.target", "target_model")
        ):
            flat["target_model"] = default_model_for_backend("qwen_chat")
+    if flat.get("target_backend") == "minimax_chat":
+        if (
+            str(flat.get("target_model", "") or "").strip() in _OPENAI_DEFAULT_MODEL_SENTINELS
+            and not _has_model_override("model.target", "target_model")
+        ):
+            flat["target_model"] = (
+                flat.get("minimax_model")
+                or default_model_for_backend("minimax_chat")
+            )

    # Auto-generate output root
    if not flat.get("out_root"):
@@ -79,6 +79,24 @@ _FLATTEN_MAP: dict[str, str] = {
    "model.qwen_chat_timeout_seconds": "qwen_chat_timeout_seconds",
    "model.qwen_chat_max_tokens": "qwen_chat_max_tokens",
    "model.qwen_chat_enable_thinking": "qwen_chat_enable_thinking",
+    "model.optimizer_qwen_chat_base_url": "optimizer_qwen_chat_base_url",
+    "model.optimizer_qwen_chat_api_key": "optimizer_qwen_chat_api_key",
+    "model.optimizer_qwen_chat_temperature": "optimizer_qwen_chat_temperature",
+    "model.optimizer_qwen_chat_timeout_seconds": "optimizer_qwen_chat_timeout_seconds",
+    "model.optimizer_qwen_chat_max_tokens": "optimizer_qwen_chat_max_tokens",
+    "model.optimizer_qwen_chat_enable_thinking": "optimizer_qwen_chat_enable_thinking",
+    "model.target_qwen_chat_base_url": "target_qwen_chat_base_url",
+    "model.target_qwen_chat_api_key": "target_qwen_chat_api_key",
+    "model.target_qwen_chat_temperature": "target_qwen_chat_temperature",
+    "model.target_qwen_chat_timeout_seconds": "target_qwen_chat_timeout_seconds",
+    "model.target_qwen_chat_max_tokens": "target_qwen_chat_max_tokens",
+    "model.target_qwen_chat_enable_thinking": "target_qwen_chat_enable_thinking",
+    "model.minimax_base_url": "minimax_base_url",
+    "model.minimax_api_key": "minimax_api_key",
+    "model.minimax_model": "minimax_model",
+    "model.minimax_temperature": "minimax_temperature",
+    "model.minimax_max_tokens": "minimax_max_tokens",
+    "model.minimax_enable_thinking": "minimax_enable_thinking",
    "train.num_epochs": "num_epochs",
    "train.train_size": "train_size",
    "train.steps_per_epoch": "steps_per_epoch",
@@ -51,6 +51,7 @@ from skillopt.model import (
    configure_azure_openai,
    configure_claude_code_exec,
    configure_codex_exec,
+    configure_minimax_chat,
    configure_qwen_chat,
    get_token_summary,
    reset_token_tracker,
@@ -628,14 +629,36 @@ class ReflACTTrainer:
            effort=cfg.get("claude_code_exec_effort", cfg.get("reasoning_effort", "medium")),
            max_thinking_tokens=cfg.get("claude_code_exec_max_thinking_tokens", 16384),
        )
-        configure_qwen_chat(
-            base_url=cfg.get("qwen_chat_base_url") or None,
-            api_key=cfg.get("qwen_chat_api_key") or None,
-            temperature=cfg.get("qwen_chat_temperature"),
-            timeout_seconds=cfg.get("qwen_chat_timeout_seconds"),
-            max_tokens=cfg.get("qwen_chat_max_tokens"),
-            enable_thinking=cfg.get("qwen_chat_enable_thinking"),
+        configure_qwen_chat(
+            base_url=cfg.get("qwen_chat_base_url") or None,
+            api_key=cfg.get("qwen_chat_api_key") or None,
+            temperature=cfg.get("qwen_chat_temperature"),
+            timeout_seconds=cfg.get("qwen_chat_timeout_seconds"),
+            max_tokens=cfg.get("qwen_chat_max_tokens"),
+            enable_thinking=cfg.get("qwen_chat_enable_thinking"),
+            optimizer_base_url=cfg.get("optimizer_qwen_chat_base_url") or None,
+            optimizer_api_key=cfg.get("optimizer_qwen_chat_api_key") or None,
+            optimizer_temperature=cfg.get("optimizer_qwen_chat_temperature"),
+            optimizer_timeout_seconds=cfg.get("optimizer_qwen_chat_timeout_seconds"),
+            optimizer_max_tokens=cfg.get("optimizer_qwen_chat_max_tokens"),
+            optimizer_enable_thinking=cfg.get("optimizer_qwen_chat_enable_thinking"),
+            target_base_url=cfg.get("target_qwen_chat_base_url") or None,
+            target_api_key=cfg.get("target_qwen_chat_api_key") or None,
+            target_temperature=cfg.get("target_qwen_chat_temperature"),
+            target_timeout_seconds=cfg.get("target_qwen_chat_timeout_seconds"),
+            target_max_tokens=cfg.get("target_qwen_chat_max_tokens"),
+            target_enable_thinking=cfg.get("target_qwen_chat_enable_thinking"),
+        )
+        configure_minimax_chat(
+            base_url=cfg.get("minimax_base_url") or None,
+            api_key=cfg.get("minimax_api_key") or None,
+            temperature=cfg.get("minimax_temperature"),
+            max_tokens=cfg.get("minimax_max_tokens"),
+            enable_thinking=cfg.get("minimax_enable_thinking"),
        )
+        minimax_model_cfg = cfg.get("minimax_model")
+        if minimax_model_cfg and cfg.get("target_backend") == "minimax_chat":
+            set_target_deployment(str(minimax_model_cfg))
        os.environ["REFLACT_CODEX_TRACE_TO_OPTIMIZER"] = (
            "1"
            if target_backend == "codex_exec" and cfg.get("codex_trace_to_optimizer", False)
@@ -4,16 +4,40 @@ This directory provides scaffold files for adding a new benchmark to SkillOpt.

 ## Files

- `env_template.py` — Environment adapter template
- `loader_template.py` — Data loader template
- `config_template.yaml` — Config file template
+- `env_template.py` — Environment adapter template (subclasses
+  `EnvAdapter`; implements the 5 abstract methods so the file is
+  instantiable out of the box).
+- `loader_template.py` — Data loader template (subclasses
+  `SplitDataLoader`; implements `load_split_items` for `.json`/`.jsonl`).
+- `config_template.yaml` — Config file template.

 ## Usage

-1. Copy this directory: `cp -r skillopt/envs/_template skillopt/envs/your_benchmark`
-2. Rename files: remove `_template` suffix
-3. Implement the `TODO` sections
-4. Register in `skillopt/envs/__init__.py`
-5. Create config at `configs/your_benchmark/default.yaml`
+1. **Copy the directory:**
+   ```bash
+   cp -r skillopt/envs/_template skillopt/envs/your_benchmark
+   ```
+2. **Rename the files** (drop the `_template` suffix):
+   ```bash
+   cd skillopt/envs/your_benchmark
+   mv env_template.py    adapter.py
+   mv loader_template.py loader.py
+   ```
+   …and inside each file rename the classes
+   (`TemplateBenchmarkEnv → YourBenchmarkAdapter`,
+   `TemplateBenchmarkLoader → YourBenchmarkLoader`)
+   and fix the cross-import in `adapter.py`.
+3. **Implement the TODO blocks** inside `adapter.py:rollout` and the
+   `_normalize_item` helper in `loader.py`. If you want real reflection,
+   uncomment the `run_minibatch_reflect` block in `adapter.py:reflect`.
+4. **Register** the adapter — add a `try / except ImportError` block in
+   `scripts/train.py`'s `_register_builtins()` mapping the registry key
+   to your `YourBenchmarkAdapter` class. There is no
+   `BENCHMARK_REGISTRY` dict in `skillopt/envs/__init__.py`; the live
+   registry is `_ENV_REGISTRY` in `scripts/train.py`.
+5. **Create the config** at `configs/your_benchmark/default.yaml`
+   (start from `config_template.yaml`). `_base_` is a **string path**,
+   not a list.

-See the [documentation](../../docs/guide/new-benchmark.md) for the full guide.
+See the [Add a New Benchmark guide](../../../docs/guide/new-benchmark.md)
+for the full step-by-step with a worked `docfaithful` example.
@@ -4,27 +4,36 @@
 # Copy this file to configs/<your_benchmark>/default.yaml
 # and customize the values below.

-# Inherit global defaults
-_base_: ['../_base_/default.yaml']
+# Inherit global defaults.
+# NOTE: `_base_` is a string path, not a list.
+_base_: ../_base_/default.yaml

 # ── Environment ──────────────────────────────────
 env:
-  name: your_benchmark            # Must match registry key
-  data_path: data/your_benchmark  # Path to your data
+  name: your_benchmark            # Must match the key registered in scripts/train.py
+  # Optional: a seed skill document. Create this file yourself before the
+  # first run, or omit the key to start from an empty skill.
+  # skill_init: skillopt/envs/your_benchmark/skills/initial.md
+  data_path: data/your_benchmark  # Path to your data (for split_mode: ratio)
+  split_dir: ""                   # Set this and use split_mode: split_dir for pre-split data
  split_mode: ratio               # "ratio" or "split_dir"
-  split_ratio: "2:1:7"            # train:val:test
-  exec_timeout: 120               # Per-task timeout (seconds)
+  split_ratio: "2:1:7"            # train:val:test (used when split_mode: ratio)
+  workers: 4                      # Parallel rollout workers
+  max_completion_tokens: 4096     # Cap per target-model call
+  limit: 0                        # 0 = no limit; small int = debug sample

 # ── Training ─────────────────────────────────────
 train:
-  num_epochs: 4                   # Number of epochs
-  batch_size: 40                  # Tasks per step (batch size)
+  num_epochs: 4
+  batch_size: 40
+  accumulation: 1
  seed: 42

 # ── Gradient (Reflection) ───────────────────────
 gradient:
  analyst_workers: 16             # Parallel reflection workers
  minibatch_size: 8
+  merge_batch_size: 8

 # ── Optimizer ────────────────────────────────────
 optimizer:
@@ -39,7 +48,8 @@ evaluation:
  eval_test: true                 # Run test eval after training

 # ── Model ────────────────────────────────────────
+# Override only what differs from the inherited defaults.
 model:
-  backend: azure_openai           # azure_openai | openai_chat | claude_code_exec | qwen
-  optimizer: gpt-4o
-  target: gpt-4o
+  optimizer_backend: openai_chat  # openai_chat | claude_chat | qwen_chat | minimax_chat
+  target_backend: openai_chat     # … plus codex_exec / claude_code_exec for target only
+  reasoning_effort: medium
@@ -4,89 +4,193 @@ Benchmark Environment Template
 Copy this file and implement the TODO sections to add a new benchmark.

 The EnvAdapter is responsible for:
-1. Executing tasks using the target model + current skill document
-2. Evaluating predictions against ground truth
-3. Returning structured results for the training loop
+  1. Building per-batch environment managers (train and eval splits).
+  2. Running rollouts under the current skill document.
+  3. Reflecting on those rollouts into raw patch dicts.
+  4. Reporting the distinct task types in your data (for stratified
+     sampling).
+
+For a fully worked example see ``skillopt/envs/officeqa/``.
 """
+from __future__ import annotations
+
+import os
+
+from skillopt.datasets.base import BatchSpec
 from skillopt.envs.base import EnvAdapter
+from skillopt.envs._template.loader_template import TemplateBenchmarkLoader
+# When you wire in real reflection, also import:
+# from skillopt.gradient.reflect import run_minibatch_reflect


 class TemplateBenchmarkEnv(EnvAdapter):
    """
    Environment adapter for <Your Benchmark Name>.
-    
-    Rename this class and implement the abstract methods below.
+
+    Rename this class. Each abstract method below is required by
+    :class:`skillopt.envs.base.EnvAdapter`. The template implementations
+    are minimal so this file is importable and instantiable; replace the
+    TODOs with real logic.
    """

-    def __init__(self, cfg: dict):
-        super().__init__(cfg)
-        # TODO: Initialize benchmark-specific state
-        # Example: self.tools = load_tools(cfg)
+    def __init__(
+        self,
+        split_dir: str = "",
+        data_path: str = "",
+        split_mode: str = "split_dir",
+        split_ratio: str = "2:1:7",
+        split_seed: int = 42,
+        split_output_dir: str = "",
+        workers: int = 4,
+        analyst_workers: int = 4,
+        failure_only: bool = False,
+        minibatch_size: int = 8,
+        edit_budget: int = 4,
+        seed: int = 42,
+        limit: int = 0,
+        max_completion_tokens: int = 4096,
+    ) -> None:
+        self.workers = workers
+        self.analyst_workers = analyst_workers
+        self.failure_only = failure_only
+        self.minibatch_size = minibatch_size
+        self.edit_budget = edit_budget
+        self.max_completion_tokens = int(max_completion_tokens)
+        self.dataloader = TemplateBenchmarkLoader(
+            split_dir=split_dir,
+            data_path=data_path,
+            split_mode=split_mode,
+            split_ratio=split_ratio,
+            split_seed=split_seed,
+            split_output_dir=split_output_dir,
+            seed=seed,
+            limit=limit,
+        )

-    async def execute(self, item, skill: str, model):
+    # ── Lifecycle hooks ────────────────────────────────────────────────
+
+    def setup(self, cfg: dict) -> None:
+        super().setup(cfg)
+        self.dataloader.setup(cfg)
+
+    def get_dataloader(self):
+        return self.dataloader
+
+    # ── Batch → env manager ────────────────────────────────────────────
+
+    def build_env_from_batch(self, batch: BatchSpec, **kwargs):
+        # Dataset-backed envs typically just pass items straight through.
+        return list(batch.payload or [])
+
+    def build_train_env(self, batch_size: int, seed: int, **kwargs):
+        batch = self.dataloader.build_train_batch(
+            batch_size=batch_size, seed=seed, **kwargs
+        )
+        return self.build_env_from_batch(batch, **kwargs)
+
+    def build_eval_env(self, env_num: int, split: str, seed: int, **kwargs):
+        batch = self.dataloader.build_eval_batch(
+            env_num=env_num, split=split, seed=seed, **kwargs
+        )
+        return self.build_env_from_batch(batch, **kwargs)
+
+    # ── Rollout: run episodes under current skill ──────────────────────
+
+    def rollout(
+        self,
+        env_manager,
+        skill_content: str,
+        out_dir: str,
+        **kwargs,
+    ) -> list[dict]:
        """
-        Execute a single task with the target model.
+        Run a batch of episodes under the current skill.

-        Args:
-            item: DataItem with .id, .input, .ground_truth, .metadata
-            skill: Current skill document content (Markdown string)
-            model: Target model backend instance
-
-        Returns:
-            TaskResult with prediction, score, and trajectory
+        TODO: replace this loop with your real rollout. For each item:
+          1. Build the prompt using `skill_content` as the system message.
+          2. Call your target model.
+          3. Score the prediction.
+          4. Return a dict with at minimum: ``id`` (str), ``hard`` (0|1),
+             ``soft`` (float in [0, 1]). Add any env-specific extras you
+             need for reflect() — they will be preserved on
+             ``RolloutResult.extras``.
        """
-        # Step 1: Build the prompt combining skill + task input
-        prompt = self.build_prompt(item, skill)
+        items: list[dict] = env_manager
+        results: list[dict] = []
+        for item in items:
+            # ── REPLACE THIS BLOCK WITH YOUR REAL ROLLOUT ──
+            results.append(
+                {
+                    "id": str(item.get("id", "")),
+                    "hard": 0,
+                    "soft": 0.0,
+                    "predicted_answer": "",
+                    "question": item.get("question", ""),
+                    "fail_reason": "template rollout — not implemented",
+                }
+            )
+        return results

-        # Step 2: Call the target model
-        # TODO: Customize the message format for your benchmark
-        messages = [
-            {"role": "system", "content": skill},
-            {"role": "user", "content": item.input},
-        ]
-        response = await model.generate(messages)
+    # ── Reflect: turn rollout results into patch dicts ─────────────────

-        # Step 3: Parse the model response into a prediction
-        prediction = self.parse_response(response.content)
-
-        # Step 4: Score the prediction
-        score = self.evaluate(prediction, item.ground_truth)
-
-        # Step 5: Return structured result
-        return {
-            "item_id": item.id,
-            "prediction": prediction,
-            "score": score,
-            "trajectory": messages + [{"role": "assistant", "content": response.content}],
-        }
-
-    def evaluate(self, prediction: str, ground_truth: str) -> float:
+    def reflect(
+        self,
+        results: list[dict],
+        skill_content: str,
+        out_dir: str,
+        **kwargs,
+    ) -> list[dict | None]:
        """
-        Score a prediction against the ground truth.
+        Turn rollouts into a list of raw patch dicts (or None to drop).

-        Returns:
-            Float between 0.0 (wrong) and 1.0 (correct)
-        
-        TODO: Implement your scoring metric. Common options:
-        - Exact match: float(pred.strip().lower() == gt.strip().lower())
-        - F1 score: compute token overlap
-        - ANLS: for document QA tasks
-        - Custom: any float in [0, 1]
-        """
-        # Placeholder — exact match
-        return float(prediction.strip().lower() == ground_truth.strip().lower())
+        Each non-None dict MUST have:
+          - "patch":       {"edits": [...]}     a Patch.to_dict() payload
+          - "source_type": "failure" | "success"

-    def build_prompt(self, item, skill: str) -> str:
-        """Combine skill document with task input."""
-        return f"{skill}\n\n---\n\nQuestion: {item.input}"
+        Most benchmarks delegate to
+        :func:`skillopt.gradient.reflect.run_minibatch_reflect` which
+        will call the optimizer model with the
+        ``analyst_error_*`` / ``analyst_success_*`` prompts. To enable it,
+        uncomment the import above and call:

-    def parse_response(self, response: str) -> str:
+            from skillopt.gradient.reflect import run_minibatch_reflect
+            return run_minibatch_reflect(
+                results=results,
+                skill_content=skill_content,
+                prediction_dir=kwargs.get(
+                    "prediction_dir", os.path.join(out_dir, "predictions")
+                ),
+                patches_dir=kwargs.get(
+                    "patches_dir", os.path.join(out_dir, "patches")
+                ),
+                workers=self.analyst_workers,
+                failure_only=self.failure_only,
+                minibatch_size=self.minibatch_size,
+                edit_budget=self.edit_budget,
+                random_seed=kwargs.get("random_seed"),
+                error_system=self.get_error_minibatch_prompt(),
+                success_system=self.get_success_minibatch_prompt(),
+                step_buffer_context=kwargs.get("step_buffer_context", ""),
+                update_mode=getattr(self, "_cfg", {}).get(
+                    "skill_update_mode", "patch"
+                ),
+            )
        """
-        Extract the answer from the model's raw response.
-        
-        TODO: Implement extraction logic. For example:
-        - Extract text after "Answer:" 
-        - Parse JSON output
-        - Extract from code blocks
-        """
-        return response.strip()
+        # Template default: produce no patches (no-op trainer step).
+        return [None for _ in results]
+
+    # ── Stratification hint ────────────────────────────────────────────
+
+    def get_task_types(self) -> list[str]:
+        """Distinct task-type strings used for stratified sampling."""
+        seen: list[str] = []
+        all_items = (
+            self.dataloader.train_items
+            + self.dataloader.val_items
+            + self.dataloader.test_items
+        )
+        for item in all_items:
+            tt = str(item.get("task_type") or "template")
+            if tt not in seen:
+                seen.append(tt)
+        return seen or ["template"]
@@ -1,103 +1,87 @@
 """
 Benchmark Data Loader Template
 ================================
-Copy this file and implement the TODO sections to load your benchmark data.
+Copy this file and implement ``load_split_items`` to load your benchmark
+data. The loader is a :class:`skillopt.datasets.base.SplitDataLoader`
+subclass — the base class handles both ``split_mode="split_dir"`` (read
+an existing train/val/test layout) and ``split_mode="ratio"`` (build the
+splits from a single raw file deterministically).

-The DataLoader is responsible for:
-1. Loading raw data from disk
-2. Splitting into train / validation / test sets
-3. Providing DataItem objects to the training loop
+For a fully worked example see
+``skillopt/envs/officeqa/dataloader.py``.
 """
+from __future__ import annotations
+
+import json
 from pathlib import Path

+from skillopt.datasets.base import SplitDataLoader

-class TemplateBenchmarkLoader:
+
+def _normalize_item(raw: dict) -> dict:
+    """
+    Normalise one raw entry into the dict shape SkillOpt expects.
+
+    The only **hard** requirement is ``"id"`` (str). Add whatever extra
+    fields your :class:`TemplateBenchmarkEnv.rollout` needs.
+    """
+    return {
+        "id": str(raw.get("uid") or raw.get("id") or ""),
+        "question": str(raw.get("question") or raw.get("prompt") or ""),
+        "ground_truth": str(raw.get("ground_truth") or raw.get("answer") or ""),
+        "task_type": str(raw.get("category") or raw.get("task_type") or "template"),
+        # ── add benchmark-specific keys here ──
+    }
+
+
+class TemplateBenchmarkLoader(SplitDataLoader):
    """
    Data loader for <Your Benchmark Name>.
-    
-    Rename this class and implement the methods below.
+
+    Subclass note: you usually only need to implement
+    :meth:`load_split_items`. The base class drives ``setup(cfg)``,
+    materialises ratio-mode splits, exposes ``train_items``,
+    ``val_items``, ``test_items``, and builds ``BatchSpec`` objects on
+    demand.
+
+    If you want to support ``split_mode="ratio"`` (auto-split a single
+    file into train/val/test), also implement
+    :meth:`load_raw_items(data_path)` returning the full list of items.
    """

-    def __init__(self, data_dir: str = "data/your_benchmark", **kwargs):
-        self.data_dir = Path(data_dir)
-        self.items = []
-        self.splits = {}
+    def load_split_items(self, split_path: str) -> list[dict]:
+        """Load all items for one split directory.

-    def setup(self, cfg: dict):
+        ``split_path`` is e.g. ``data/your_benchmark/train/``. Return a
+        list of dicts, each shaped like :func:`_normalize_item`'s output.
        """
-        Initialize the loader with config.
-        
-        Called once before training starts.
-        
-        Args:
-            cfg: Dict with keys like 'split_mode', 'train_ratio', 'val_ratio', etc.
-        """
-        # Step 1: Load raw data
-        self.items = self._load_items()
+        path = Path(split_path)

-        # Step 2: Create splits
-        split_mode = cfg.get("split_mode", "ratio")
-        if split_mode == "ratio":
-            self._split_by_ratio(
-                train_ratio=cfg.get("train_ratio", 0.7),
-                val_ratio=cfg.get("val_ratio", 0.15),
-            )
-        elif split_mode == "split_dir":
-            self._load_predefined_splits(cfg.get("split_dir", self.data_dir))
+        json_files = sorted(path.glob("*.json"))
+        if json_files:
+            with json_files[0].open(encoding="utf-8") as f:
+                payload = json.load(f)
+            if not isinstance(payload, list):
+                raise ValueError(
+                    f"Expected JSON array at top level of {json_files[0]}"
+                )
+            return [_normalize_item(row) for row in payload]

-    def _load_items(self) -> list:
-        """
-        Load raw data into structured items.
-        
-        TODO: Implement data loading. Each item should have at minimum:
-        - id: unique identifier
-        - input: the task input (question, instruction, etc.)
-        - ground_truth: the expected answer
-        - metadata: optional dict with extra info
-        
-        Example:
-            items = []
-            for path in self.data_dir.glob("*.json"):
-                data = json.loads(path.read_text())
-                for entry in data:
-                    items.append({
-                        "id": entry["id"],
-                        "input": entry["question"],
-                        "ground_truth": entry["answer"],
-                        "metadata": {"source": path.name},
-                    })
+        jsonl_files = sorted(path.glob("*.jsonl"))
+        if jsonl_files:
+            items: list[dict] = []
+            with jsonl_files[0].open(encoding="utf-8") as f:
+                for line in f:
+                    line = line.strip()
+                    if not line:
+                        continue
+                    items.append(_normalize_item(json.loads(line)))
            return items
-        """
-        raise NotImplementedError("Implement _load_items() for your benchmark")

-    def _split_by_ratio(self, train_ratio: float, val_ratio: float):
-        """Split items by ratio."""
-        import random
-        random.shuffle(self.items)
-        n = len(self.items)
-        n_train = int(n * train_ratio)
-        n_val = int(n * val_ratio)
-        self.splits = {
-            "train": self.items[:n_train],
-            "valid": self.items[n_train:n_train + n_val],
-            "test": self.items[n_train + n_val:],
-        }
+        raise FileNotFoundError(
+            f"No .json or .jsonl file found in {split_path}"
+        )

-    def _load_predefined_splits(self, split_dir):
-        """Load from pre-split directories."""
-        # TODO: Implement if your benchmark has pre-defined splits
-        raise NotImplementedError
-
-    def get_split_items(self, split: str) -> list:
-        """
-        Return items for a given split.
-        
-        Args:
-            split: One of "train", "valid", "test"
-            
-        Returns:
-            List of data items for the requested split
-        """
-        if split not in self.splits:
-            raise ValueError(f"Unknown split '{split}'. Available: {list(self.splits.keys())}")
-        return self.splits[split]
+    # Optional — only needed if you intend to use ``split_mode='ratio'``.
+    # def load_raw_items(self, data_path: str) -> list[dict]:
+    #     ...
@@ -6,6 +6,7 @@ from typing import Any

 from skillopt.model import azure_openai as _openai
 from skillopt.model import claude_backend as _claude
+from skillopt.model import minimax_backend as _minimax
 from skillopt.model import qwen_backend as _qwen
 from skillopt.model.backend_config import (  # noqa: F401
    configure_claude_code_exec,
@@ -50,6 +51,10 @@ def set_backend(name: str | None) -> str:
        set_optimizer_backend("openai_chat")
        set_target_backend("qwen_chat")
        return "qwen_chat"
+    if normalized in {"minimax", "minimax_chat"}:
+        set_optimizer_backend("openai_chat")
+        set_target_backend("minimax_chat")
+        return "minimax_chat"
    raise ValueError(f"Unsupported legacy backend: {name!r}")


@@ -59,12 +64,16 @@ def get_backend_name() -> str:
    target = get_target_backend()
    if optimizer == "claude_chat" and target == "claude_chat":
        return "claude_chat"
+    if optimizer == "qwen_chat" and target == "qwen_chat":
+        return "qwen_chat"
    if optimizer == "openai_chat" and target == "openai_chat":
        return "azure_openai"
    if optimizer == "openai_chat" and target == "codex_exec":
        return "codex"
    if optimizer == "openai_chat" and target == "qwen_chat":
        return "qwen_chat"
+    if optimizer == "openai_chat" and target == "minimax_chat":
+        return "minimax_chat"
    return f"{optimizer}+{target}"


@@ -86,6 +95,16 @@ def chat_optimizer(
            stage=stage,
            timeout=timeout,
        )
+    if get_optimizer_backend() == "qwen_chat":
+        return _qwen.chat_optimizer(
+            system=system,
+            user=user,
+            max_completion_tokens=max_completion_tokens,
+            retries=retries,
+            stage=stage,
+            reasoning_effort=reasoning_effort,
+            timeout=timeout,
+        )
    return _openai.chat_optimizer(
        system=system,
        user=user,
@@ -124,9 +143,18 @@ def chat_target(
            stage=stage,
            reasoning_effort=reasoning_effort,
        )
+    if get_target_backend() == "minimax_chat":
+        return _minimax.chat_target(
+            system=system,
+            user=user,
+            max_completion_tokens=max_completion_tokens,
+            retries=retries,
+            stage=stage,
+            reasoning_effort=reasoning_effort,
+        )
    if not is_target_chat_backend():
        raise NotImplementedError(
-            "chat_target is only supported with target_backend=openai_chat, claude_chat, or qwen_chat. "
+            "chat_target is only supported with target_backend=openai_chat, claude_chat, qwen_chat, or minimax_chat. "
            "Exec backends are handled in environment-specific rollout code."
        )
    return _openai.chat_target(
@@ -163,6 +191,18 @@ def chat_optimizer_messages(
            return_message=return_message,
            timeout=timeout,
        )
+    if get_optimizer_backend() == "qwen_chat":
+        return _qwen.chat_optimizer_messages(
+            messages=messages,
+            max_completion_tokens=max_completion_tokens,
+            retries=retries,
+            stage=stage,
+            reasoning_effort=reasoning_effort,
+            tools=tools,
+            tool_choice=tool_choice,
+            return_message=return_message,
+            timeout=timeout,
+        )
    return _openai.chat_optimizer_messages(
        messages=messages,
        max_completion_tokens=max_completion_tokens,
@@ -210,9 +250,20 @@ def chat_target_messages(
            tool_choice=tool_choice,
            return_message=return_message,
        )
+    if get_target_backend() == "minimax_chat":
+        return _minimax.chat_target_messages(
+            messages=messages,
+            max_completion_tokens=max_completion_tokens,
+            retries=retries,
+            stage=stage,
+            reasoning_effort=reasoning_effort,
+            tools=tools,
+            tool_choice=tool_choice,
+            return_message=return_message,
+        )
    if not is_target_chat_backend():
        raise NotImplementedError(
-            "chat_target_messages is only supported with target_backend=openai_chat, claude_chat, or qwen_chat. "
+            "chat_target_messages is only supported with target_backend=openai_chat, claude_chat, qwen_chat, or minimax_chat. "
            "Exec backends are handled in environment-specific rollout code."
        )
    return _openai.chat_target_messages(
@@ -301,6 +352,17 @@ def get_token_summary() -> dict:
        summary[stage]["prompt_tokens"] += values["prompt_tokens"]
        summary[stage]["completion_tokens"] += values["completion_tokens"]
        summary[stage]["total_tokens"] += values["total_tokens"]
+    minimax_summary = _minimax.get_token_summary()
+    for stage, values in minimax_summary.items():
+        if stage == "_total":
+            continue
+        if stage not in summary:
+            summary[stage] = values
+            continue
+        summary[stage]["calls"] += values["calls"]
+        summary[stage]["prompt_tokens"] += values["prompt_tokens"]
+        summary[stage]["completion_tokens"] += values["completion_tokens"]
+        summary[stage]["total_tokens"] += values["total_tokens"]
    total = {
        "calls": 0,
        "prompt_tokens": 0,
@@ -322,6 +384,7 @@ def reset_token_tracker() -> None:
    _openai.reset_token_tracker()
    _claude.reset_token_tracker()
    _qwen.reset_token_tracker()
+    _minimax.reset_token_tracker()


 def configure_azure_openai(
@@ -375,6 +438,18 @@ def configure_qwen_chat(
    timeout_seconds: float | str | None = None,
    max_tokens: int | str | None = None,
    enable_thinking: bool | str | None = None,
+    optimizer_base_url: str | None = None,
+    optimizer_api_key: str | None = None,
+    optimizer_temperature: float | str | None = None,
+    optimizer_timeout_seconds: float | str | None = None,
+    optimizer_max_tokens: int | str | None = None,
+    optimizer_enable_thinking: bool | str | None = None,
+    target_base_url: str | None = None,
+    target_api_key: str | None = None,
+    target_temperature: float | str | None = None,
+    target_timeout_seconds: float | str | None = None,
+    target_max_tokens: int | str | None = None,
+    target_enable_thinking: bool | str | None = None,
 ) -> None:
    _qwen.configure_qwen_chat(
        base_url=base_url,
@@ -383,6 +458,37 @@ def configure_qwen_chat(
        timeout_seconds=timeout_seconds,
        max_tokens=max_tokens,
        enable_thinking=enable_thinking,
+        optimizer_base_url=optimizer_base_url,
+        optimizer_api_key=optimizer_api_key,
+        optimizer_temperature=optimizer_temperature,
+        optimizer_timeout_seconds=optimizer_timeout_seconds,
+        optimizer_max_tokens=optimizer_max_tokens,
+        optimizer_enable_thinking=optimizer_enable_thinking,
+        target_base_url=target_base_url,
+        target_api_key=target_api_key,
+        target_temperature=target_temperature,
+        target_timeout_seconds=target_timeout_seconds,
+        target_max_tokens=target_max_tokens,
+        target_enable_thinking=target_enable_thinking,
+    )
+
+
+def configure_minimax_chat(
+    *,
+    base_url: str | None = None,
+    api_key: str | None = None,
+    temperature: float | str | None = None,
+    timeout_seconds: float | str | None = None,
+    max_tokens: int | str | None = None,
+    enable_thinking: bool | str | None = None,
+) -> None:
+    _minimax.configure_minimax_chat(
+        base_url=base_url,
+        api_key=api_key,
+        temperature=temperature,
+        timeout_seconds=timeout_seconds,
+        max_tokens=max_tokens,
+        enable_thinking=enable_thinking,
    )


@@ -390,14 +496,17 @@ def set_reasoning_effort(effort: str | None) -> None:
    _openai.set_reasoning_effort(effort)
    _claude.set_reasoning_effort(effort)
    _qwen.set_reasoning_effort(effort)
+    _minimax.set_reasoning_effort(effort)


 def set_target_deployment(deployment: str) -> None:
    _openai.set_target_deployment(deployment)
    _claude.set_target_deployment(deployment)
    _qwen.set_target_deployment(deployment)
+    _minimax.set_target_deployment(deployment)


 def set_optimizer_deployment(deployment: str) -> None:
    _openai.set_optimizer_deployment(deployment)
    _claude.set_optimizer_deployment(deployment)
+    _qwen.set_optimizer_deployment(deployment)
@@ -336,9 +336,10 @@ def get_target_client() -> AzureOpenAI | OpenAI:
            from skillopt.model.backend_config import get_target_backend
            if get_target_backend() == "qwen_chat":
                from skillopt.model import qwen_backend as _qwen
+                target_config = _qwen.TARGET_CONFIG
                _target_client = OpenAI(
-                    base_url=_qwen.BASE_URL,
-                    api_key=_qwen.API_KEY or "dummy",
+                    base_url=target_config.base_url,
+                    api_key=target_config.api_key or "dummy",
                )
            else:
                _target_client = _make_client("target")
@@ -49,10 +49,10 @@ CLAUDE_CODE_EXEC_MAX_THINKING_TOKENS = max(
 def set_optimizer_backend(backend: str) -> None:
    global OPTIMIZER_BACKEND
    OPTIMIZER_BACKEND = normalize_backend_name(backend or "openai_chat")
-    if OPTIMIZER_BACKEND not in {"openai_chat", "claude_chat"}:
+    if OPTIMIZER_BACKEND not in {"openai_chat", "claude_chat", "qwen_chat", "minimax_chat"}:
        raise ValueError(
            f"Unsupported optimizer backend: {OPTIMIZER_BACKEND!r}. "
-            "Supported values are 'openai_chat' and 'claude_chat'."
+            "Supported values are 'openai_chat', 'claude_chat', 'qwen_chat', and 'minimax_chat'."
        )
    os.environ["OPTIMIZER_BACKEND"] = OPTIMIZER_BACKEND

@@ -64,10 +64,10 @@ def get_optimizer_backend() -> str:
 def set_target_backend(backend: str) -> None:
    global TARGET_BACKEND
    TARGET_BACKEND = normalize_backend_name(backend or "openai_chat")
-    if TARGET_BACKEND not in {"openai_chat", "claude_chat", "qwen_chat", "codex_exec", "claude_code_exec"}:
+    if TARGET_BACKEND not in {"openai_chat", "claude_chat", "qwen_chat", "minimax_chat", "codex_exec", "claude_code_exec"}:
        raise ValueError(
            f"Unsupported target backend: {TARGET_BACKEND!r}. "
-            "Supported values are 'openai_chat', 'claude_chat', 'qwen_chat', 'codex_exec', and 'claude_code_exec'."
+            "Supported values are 'openai_chat', 'claude_chat', 'qwen_chat', 'minimax_chat', 'codex_exec', and 'claude_code_exec'."
        )
    os.environ["TARGET_BACKEND"] = TARGET_BACKEND

@@ -81,11 +81,11 @@ def is_target_exec_backend() -> bool:


 def is_optimizer_chat_backend() -> bool:
-    return OPTIMIZER_BACKEND in {"openai_chat", "claude_chat"}
+    return OPTIMIZER_BACKEND in {"openai_chat", "claude_chat", "qwen_chat", "minimax_chat"}


 def is_target_chat_backend() -> bool:
-    return TARGET_BACKEND in {"openai_chat", "claude_chat", "qwen_chat"}
+    return TARGET_BACKEND in {"openai_chat", "claude_chat", "qwen_chat", "minimax_chat"}


 def configure_codex_exec(
@@ -25,6 +25,7 @@ _BACKEND_DEFAULT_MODELS = {
    "claude_chat": "claude-sonnet-4-6",
    "claude_code_exec": "claude-sonnet-4-6",
    "qwen_chat": "Qwen/Qwen3.5-4B",
+    "minimax_chat": "MiniMax-M2.7",
 }

 _BACKEND_ALIASES = {
@@ -41,6 +42,8 @@ _BACKEND_ALIASES = {
    "anthropic": "claude_chat",
    "qwen": "qwen_chat",
    "qwen_chat": "qwen_chat",
+    "minimax": "minimax_chat",
+    "minimax_chat": "minimax_chat",
 }


@@ -0,0 +1,277 @@
+"""OpenAI-compatible MiniMax chat backend for the target path."""
+from __future__ import annotations
+
+import json
+import os
+import threading
+import time
+import urllib.error
+import urllib.request
+from typing import Any
+
+from skillopt.model.common import (
+    CompatAssistantMessage,
+    CompatToolCall,
+    CompatToolFunction,
+    TokenTracker,
+    default_model_for_backend,
+)
+
+BASE_URL = os.environ.get("MINIMAX_BASE_URL", "https://api.minimax.io/v1")
+API_KEY = os.environ.get("MINIMAX_API_KEY", "")
+TIMEOUT_SECONDS = float(os.environ.get("MINIMAX_TIMEOUT_SECONDS", "300") or 300)
+MAX_TOKENS = int(os.environ.get("MINIMAX_MAX_TOKENS", "8000") or 8000)
+TEMPERATURE: float | None = None
+_raw_temperature = os.environ.get("MINIMAX_TEMPERATURE", "0.7").strip()
+if _raw_temperature:
+    TEMPERATURE = float(_raw_temperature)
+ENABLE_THINKING = os.environ.get("MINIMAX_ENABLE_THINKING", "false").strip().lower() in {
+    "1",
+    "true",
+    "yes",
+    "on",
+}
+
+TARGET_DEPLOYMENT = os.environ.get(
+    "TARGET_DEPLOYMENT",
+    default_model_for_backend("minimax_chat"),
+)
+
+_config_lock = threading.Lock()
+tracker = TokenTracker()
+
+
+def _chat_url() -> str:
+    base = BASE_URL.rstrip("/")
+    if base.endswith("/chat/completions"):
+        return base
+    return f"{base}/chat/completions"
+
+
+def _json_safe(value: Any) -> Any:
+    if value is None or isinstance(value, (str, int, float, bool)):
+        return value
+    if isinstance(value, list):
+        return [_json_safe(item) for item in value]
+    if isinstance(value, dict):
+        return {str(key): _json_safe(val) for key, val in value.items()}
+    model_dump = getattr(value, "model_dump", None)
+    if callable(model_dump):
+        try:
+            return model_dump(mode="json")
+        except TypeError:
+            return model_dump()
+    return str(value)
+
+
+def _usage_from_payload(payload: dict[str, Any]) -> dict[str, int]:
+    usage = payload.get("usage") or {}
+    prompt_tokens = int(usage.get("prompt_tokens") or usage.get("input_tokens") or 0)
+    completion_tokens = int(usage.get("completion_tokens") or usage.get("output_tokens") or 0)
+    total_tokens = int(usage.get("total_tokens") or (prompt_tokens + completion_tokens))
+    return {
+        "prompt_tokens": prompt_tokens,
+        "completion_tokens": completion_tokens,
+        "total_tokens": total_tokens,
+    }
+
+
+def _compat_message_from_payload(message: dict[str, Any], choice: dict[str, Any]) -> CompatAssistantMessage:
+    content = message.get("content") or ""
+    if not isinstance(content, str):
+        content = json.dumps(content, ensure_ascii=False)
+    tool_calls: list[CompatToolCall] = []
+    for index, tool_call in enumerate(message.get("tool_calls") or [], start=1):
+        function = tool_call.get("function") or {}
+        tool_calls.append(
+            CompatToolCall(
+                id=str(tool_call.get("id") or f"minimax_tool_{index}"),
+                type=str(tool_call.get("type") or "function"),
+                function=CompatToolFunction(
+                    name=str(function.get("name") or ""),
+                    arguments=str(function.get("arguments") or "{}"),
+                ),
+            )
+        )
+    return CompatAssistantMessage(
+        content=content,
+        tool_calls=tool_calls,
+        metadata={
+            "finish_reason": choice.get("finish_reason"),
+            "choice0": _json_safe(choice),
+        },
+    )
+
+
+def _post_chat_completion(payload: dict[str, Any], timeout: float | None) -> dict[str, Any]:
+    headers = {"Content-Type": "application/json"}
+    if API_KEY:
+        headers["Authorization"] = f"Bearer {API_KEY}"
+    req = urllib.request.Request(
+        _chat_url(),
+        data=json.dumps(payload, ensure_ascii=False).encode("utf-8"),
+        headers=headers,
+        method="POST",
+    )
+    try:
+        with urllib.request.urlopen(req, timeout=timeout or TIMEOUT_SECONDS) as resp:
+            raw = resp.read().decode("utf-8")
+    except urllib.error.HTTPError as e:
+        body = e.read().decode("utf-8", errors="replace")
+        raise RuntimeError(f"MiniMax chat API returned HTTP {e.code}: {body}") from e
+    except urllib.error.URLError as e:
+        raise RuntimeError(f"MiniMax chat API request failed: {e}") from e
+    try:
+        return json.loads(raw)
+    except json.JSONDecodeError as e:
+        raise RuntimeError(f"MiniMax chat API returned non-JSON response: {raw[:1000]}") from e
+
+
+def _chat_messages_impl(
+    messages: list[dict[str, Any]],
+    max_completion_tokens: int,
+    retries: int,
+    stage: str,
+    *,
+    tools: list[dict[str, Any]] | None = None,
+    tool_choice: str | dict[str, Any] | None = None,
+    return_message: bool = False,
+    deployment: str | None = None,
+    timeout: float | None = None,
+) -> tuple[Any, dict[str, int]]:
+    payload: dict[str, Any] = {
+        "model": deployment or TARGET_DEPLOYMENT,
+        "messages": _json_safe(messages),
+        "max_tokens": min(max_completion_tokens, MAX_TOKENS),
+    }
+    payload["chat_template_kwargs"] = {"enable_thinking": ENABLE_THINKING}
+    if TEMPERATURE is not None:
+        payload["temperature"] = TEMPERATURE
+    if tools:
+        payload["tools"] = _json_safe(tools)
+        if tool_choice is not None:
+            payload["tool_choice"] = _json_safe(tool_choice)
+
+    last_err: Exception | None = None
+    for attempt in range(retries):
+        try:
+            data = _post_chat_completion(payload, timeout)
+            choices = data.get("choices") or []
+            if not choices:
+                raise RuntimeError(f"MiniMax chat API returned no choices: {data}")
+            choice0 = choices[0]
+            message = choice0.get("message") or {}
+            text = message.get("content") or ""
+            if not isinstance(text, str):
+                text = json.dumps(text, ensure_ascii=False)
+            usage_info = _usage_from_payload(data)
+            tracker.record(stage, usage_info["prompt_tokens"], usage_info["completion_tokens"])
+            if return_message:
+                return _compat_message_from_payload(message, choice0), usage_info
+            return text, usage_info
+        except Exception as e:  # noqa: BLE001
+            last_err = e
+            time.sleep(min(2 ** attempt, 30))
+    raise RuntimeError(f"MiniMax chat call failed after {retries} retries: {last_err}")
+
+
+def configure_minimax_chat(
+    *,
+    base_url: str | None = None,
+    api_key: str | None = None,
+    temperature: float | str | None = None,
+    timeout_seconds: float | str | None = None,
+    max_tokens: int | str | None = None,
+    enable_thinking: bool | str | None = None,
+) -> None:
+    global BASE_URL, API_KEY, TEMPERATURE, TIMEOUT_SECONDS, MAX_TOKENS, ENABLE_THINKING
+    with _config_lock:
+        if base_url is not None:
+            BASE_URL = str(base_url).strip() or BASE_URL
+            os.environ["MINIMAX_BASE_URL"] = BASE_URL
+        if api_key is not None:
+            API_KEY = str(api_key).strip()
+            os.environ["MINIMAX_API_KEY"] = API_KEY
+        if temperature is not None:
+            raw = str(temperature).strip()
+            TEMPERATURE = float(raw) if raw else None
+            os.environ["MINIMAX_TEMPERATURE"] = raw
+        if timeout_seconds is not None:
+            TIMEOUT_SECONDS = float(timeout_seconds)
+            os.environ["MINIMAX_TIMEOUT_SECONDS"] = str(timeout_seconds)
+        if max_tokens is not None:
+            MAX_TOKENS = int(max_tokens)
+            os.environ["MINIMAX_MAX_TOKENS"] = str(max_tokens)
+        if enable_thinking is not None:
+            if isinstance(enable_thinking, str):
+                ENABLE_THINKING = enable_thinking.strip().lower() in {"1", "true", "yes", "on"}
+            else:
+                ENABLE_THINKING = bool(enable_thinking)
+            os.environ["MINIMAX_ENABLE_THINKING"] = "true" if ENABLE_THINKING else "false"
+
+
+def get_max_tokens() -> int:
+    return MAX_TOKENS
+
+
+def chat_target(
+    system: str,
+    user: str,
+    max_completion_tokens: int = 16384,
+    retries: int = 5,
+    stage: str = "target",
+    reasoning_effort: str | None = None,
+    timeout: float | None = None,
+) -> tuple[str, dict[str, int]]:
+    del reasoning_effort
+    messages = [{"role": "system", "content": system}, {"role": "user", "content": user}]
+    return _chat_messages_impl(
+        messages,
+        max_completion_tokens,
+        retries,
+        stage,
+        timeout=timeout,
+    )
+
+
+def chat_target_messages(
+    messages: list[dict[str, Any]],
+    max_completion_tokens: int = 16384,
+    retries: int = 5,
+    stage: str = "target",
+    reasoning_effort: str | None = None,
+    *,
+    tools: list[dict[str, Any]] | None = None,
+    tool_choice: str | dict[str, Any] | None = None,
+    return_message: bool = False,
+    timeout: float | None = None,
+) -> tuple[Any, dict[str, int]]:
+    del reasoning_effort
+    return _chat_messages_impl(
+        messages,
+        max_completion_tokens,
+        retries,
+        stage,
+        tools=tools,
+        tool_choice=tool_choice,
+        return_message=return_message,
+        timeout=timeout,
+    )
+
+
+def get_token_summary() -> dict[str, dict[str, int]]:
+    return tracker.summary()
+
+
+def reset_token_tracker() -> None:
+    tracker.reset()
+
+
+def set_reasoning_effort(effort: str | None) -> None:
+    del effort
+
+
+def set_target_deployment(deployment: str) -> None:
+    global TARGET_DEPLOYMENT
+    TARGET_DEPLOYMENT = deployment or default_model_for_backend("minimax_chat")
+    os.environ["TARGET_DEPLOYMENT"] = TARGET_DEPLOYMENT
@@ -1,6 +1,7 @@
-"""OpenAI-compatible Qwen chat backend for the target path."""
+"""OpenAI-compatible Qwen chat backend for optimizer and target paths."""
 from __future__ import annotations

+from dataclasses import dataclass
 import json
 import os
 import threading
@@ -17,32 +18,72 @@ from skillopt.model.common import (
    default_model_for_backend,
 )

-BASE_URL = os.environ.get("QWEN_CHAT_BASE_URL", "http://localhost:8000/v1")
-API_KEY = os.environ.get("QWEN_CHAT_API_KEY", "")
-TIMEOUT_SECONDS = float(os.environ.get("QWEN_CHAT_TIMEOUT_SECONDS", "300") or 300)
-MAX_TOKENS = int(os.environ.get("QWEN_CHAT_MAX_TOKENS", "8000") or 8000)
-TEMPERATURE: float | None = None
-_raw_temperature = os.environ.get("QWEN_CHAT_TEMPERATURE", "0.7").strip()
-if _raw_temperature:
-    TEMPERATURE = float(_raw_temperature)
-ENABLE_THINKING = os.environ.get("QWEN_CHAT_ENABLE_THINKING", "false").strip().lower() in {
-    "1",
-    "true",
-    "yes",
-    "on",
-}

-TARGET_DEPLOYMENT = os.environ.get(
-    "TARGET_DEPLOYMENT",
-    default_model_for_backend("qwen_chat"),
-)
+@dataclass
+class QwenChatConfig:
+    base_url: str
+    api_key: str
+    timeout_seconds: float
+    max_tokens: int
+    temperature: float | None
+    enable_thinking: bool
+    deployment: str
+
+
+def _parse_bool(value: Any, default: bool = False) -> bool:
+    if value is None:
+        return default
+    return str(value).strip().lower() in {"1", "true", "yes", "on"}
+
+
+def _parse_optional_float(value: Any) -> float | None:
+    if value is None:
+        return None
+    raw = str(value).strip()
+    return float(raw) if raw else None
+
+
+def _parse_int(value: Any, default: int) -> int:
+    if value is None:
+        return default
+    raw = str(value).strip()
+    return int(raw) if raw else default
+
+
+def _role_env(role: str, key: str, default: str) -> str:
+    role_key = f"{role.upper()}_QWEN_CHAT_{key}"
+    generic_key = f"QWEN_CHAT_{key}"
+    return os.environ.get(role_key) or os.environ.get(generic_key) or default
+
+
+def _initial_config(role: str) -> QwenChatConfig:
+    role_upper = role.upper()
+    deployment_env = "OPTIMIZER_DEPLOYMENT" if role == "optimizer" else "TARGET_DEPLOYMENT"
+    return QwenChatConfig(
+        base_url=_role_env(role, "BASE_URL", "http://localhost:8000/v1"),
+        api_key=_role_env(role, "API_KEY", ""),
+        timeout_seconds=float(_role_env(role, "TIMEOUT_SECONDS", "300") or 300),
+        max_tokens=_parse_int(_role_env(role, "MAX_TOKENS", "8000"), 8000),
+        temperature=_parse_optional_float(_role_env(role, "TEMPERATURE", "0.7")),
+        enable_thinking=_parse_bool(_role_env(role, "ENABLE_THINKING", "false")),
+        deployment=(
+            os.environ.get(f"{role_upper}_QWEN_CHAT_MODEL")
+            or os.environ.get("QWEN_CHAT_MODEL")
+            or os.environ.get(deployment_env)
+            or default_model_for_backend("qwen_chat")
+        ),
+    )
+
+
+OPTIMIZER_CONFIG = _initial_config("optimizer")
+TARGET_CONFIG = _initial_config("target")

 _config_lock = threading.Lock()
 tracker = TokenTracker()


-def _chat_url() -> str:
-    base = BASE_URL.rstrip("/")
+def _chat_url(config: QwenChatConfig) -> str:
+    base = config.base_url.rstrip("/")
    if base.endswith("/chat/completions"):
        return base
    return f"{base}/chat/completions"
@@ -103,18 +144,22 @@ def _compat_message_from_payload(message: dict[str, Any], choice: dict[str, Any]
    )


-def _post_chat_completion(payload: dict[str, Any], timeout: float | None) -> dict[str, Any]:
+def _post_chat_completion(
+    payload: dict[str, Any],
+    timeout: float | None,
+    config: QwenChatConfig,
+) -> dict[str, Any]:
    headers = {"Content-Type": "application/json"}
-    if API_KEY:
-        headers["Authorization"] = f"Bearer {API_KEY}"
+    if config.api_key:
+        headers["Authorization"] = f"Bearer {config.api_key}"
    req = urllib.request.Request(
-        _chat_url(),
+        _chat_url(config),
        data=json.dumps(payload, ensure_ascii=False).encode("utf-8"),
        headers=headers,
        method="POST",
    )
    try:
-        with urllib.request.urlopen(req, timeout=timeout or TIMEOUT_SECONDS) as resp:
+        with urllib.request.urlopen(req, timeout=timeout or config.timeout_seconds) as resp:
            raw = resp.read().decode("utf-8")
    except urllib.error.HTTPError as e:
        body = e.read().decode("utf-8", errors="replace")
@@ -133,20 +178,22 @@ def _chat_messages_impl(
    retries: int,
    stage: str,
    *,
+    role: str,
    tools: list[dict[str, Any]] | None = None,
    tool_choice: str | dict[str, Any] | None = None,
    return_message: bool = False,
    deployment: str | None = None,
    timeout: float | None = None,
 ) -> tuple[Any, dict[str, int]]:
+    config = OPTIMIZER_CONFIG if role == "optimizer" else TARGET_CONFIG
    payload: dict[str, Any] = {
-        "model": deployment or TARGET_DEPLOYMENT,
+        "model": deployment or config.deployment,
        "messages": _json_safe(messages),
-        "max_tokens": min(max_completion_tokens, MAX_TOKENS),
+        "max_tokens": min(max_completion_tokens, config.max_tokens),
    }
-    payload["chat_template_kwargs"] = {"enable_thinking": ENABLE_THINKING}
-    if TEMPERATURE is not None:
-        payload["temperature"] = TEMPERATURE
+    payload["chat_template_kwargs"] = {"enable_thinking": config.enable_thinking}
+    if config.temperature is not None:
+        payload["temperature"] = config.temperature
    if tools:
        payload["tools"] = _json_safe(tools)
        if tool_choice is not None:
@@ -155,7 +202,7 @@ def _chat_messages_impl(
    last_err: Exception | None = None
    for attempt in range(retries):
        try:
-            data = _post_chat_completion(payload, timeout)
+            data = _post_chat_completion(payload, timeout, config)
            choices = data.get("choices") or []
            if not choices:
                raise RuntimeError(f"Qwen chat API returned no choices: {data}")
@@ -183,35 +230,134 @@ def configure_qwen_chat(
    timeout_seconds: float | str | None = None,
    max_tokens: int | str | None = None,
    enable_thinking: bool | str | None = None,
+    optimizer_base_url: str | None = None,
+    optimizer_api_key: str | None = None,
+    optimizer_temperature: float | str | None = None,
+    optimizer_timeout_seconds: float | str | None = None,
+    optimizer_max_tokens: int | str | None = None,
+    optimizer_enable_thinking: bool | str | None = None,
+    target_base_url: str | None = None,
+    target_api_key: str | None = None,
+    target_temperature: float | str | None = None,
+    target_timeout_seconds: float | str | None = None,
+    target_max_tokens: int | str | None = None,
+    target_enable_thinking: bool | str | None = None,
 ) -> None:
-    global BASE_URL, API_KEY, TEMPERATURE, TIMEOUT_SECONDS, MAX_TOKENS, ENABLE_THINKING
    with _config_lock:
        if base_url is not None:
-            BASE_URL = str(base_url).strip() or BASE_URL
-            os.environ["QWEN_CHAT_BASE_URL"] = BASE_URL
+            os.environ["QWEN_CHAT_BASE_URL"] = str(base_url).strip()
        if api_key is not None:
-            API_KEY = str(api_key).strip()
-            os.environ["QWEN_CHAT_API_KEY"] = API_KEY
+            os.environ["QWEN_CHAT_API_KEY"] = str(api_key).strip()
        if temperature is not None:
-            raw = str(temperature).strip()
-            TEMPERATURE = float(raw) if raw else None
-            os.environ["QWEN_CHAT_TEMPERATURE"] = raw
+            os.environ["QWEN_CHAT_TEMPERATURE"] = str(temperature).strip()
        if timeout_seconds is not None:
-            TIMEOUT_SECONDS = float(timeout_seconds)
            os.environ["QWEN_CHAT_TIMEOUT_SECONDS"] = str(timeout_seconds)
        if max_tokens is not None:
-            MAX_TOKENS = int(max_tokens)
            os.environ["QWEN_CHAT_MAX_TOKENS"] = str(max_tokens)
        if enable_thinking is not None:
-            if isinstance(enable_thinking, str):
-                ENABLE_THINKING = enable_thinking.strip().lower() in {"1", "true", "yes", "on"}
-            else:
-                ENABLE_THINKING = bool(enable_thinking)
-            os.environ["QWEN_CHAT_ENABLE_THINKING"] = "true" if ENABLE_THINKING else "false"
+            os.environ["QWEN_CHAT_ENABLE_THINKING"] = (
+                "true" if _parse_bool(enable_thinking) else "false"
+            )
+        _update_config(
+            OPTIMIZER_CONFIG,
+            "optimizer",
+            base_url=optimizer_base_url if optimizer_base_url is not None else base_url,
+            api_key=optimizer_api_key if optimizer_api_key is not None else api_key,
+            temperature=(
+                optimizer_temperature
+                if optimizer_temperature is not None
+                else temperature
+            ),
+            timeout_seconds=(
+                optimizer_timeout_seconds
+                if optimizer_timeout_seconds is not None
+                else timeout_seconds
+            ),
+            max_tokens=optimizer_max_tokens if optimizer_max_tokens is not None else max_tokens,
+            enable_thinking=(
+                optimizer_enable_thinking
+                if optimizer_enable_thinking is not None
+                else enable_thinking
+            ),
+        )
+        _update_config(
+            TARGET_CONFIG,
+            "target",
+            base_url=target_base_url if target_base_url is not None else base_url,
+            api_key=target_api_key if target_api_key is not None else api_key,
+            temperature=target_temperature if target_temperature is not None else temperature,
+            timeout_seconds=(
+                target_timeout_seconds
+                if target_timeout_seconds is not None
+                else timeout_seconds
+            ),
+            max_tokens=target_max_tokens if target_max_tokens is not None else max_tokens,
+            enable_thinking=(
+                target_enable_thinking
+                if target_enable_thinking is not None
+                else enable_thinking
+            ),
+        )
+
+
+def _update_config(
+    config: QwenChatConfig,
+    role: str,
+    *,
+    base_url: str | None = None,
+    api_key: str | None = None,
+    temperature: float | str | None = None,
+    timeout_seconds: float | str | None = None,
+    max_tokens: int | str | None = None,
+    enable_thinking: bool | str | None = None,
+) -> None:
+    env_prefix = role.upper()
+    if base_url is not None:
+        config.base_url = str(base_url).strip() or config.base_url
+        os.environ[f"{env_prefix}_QWEN_CHAT_BASE_URL"] = config.base_url
+    if api_key is not None:
+        config.api_key = str(api_key).strip()
+        os.environ[f"{env_prefix}_QWEN_CHAT_API_KEY"] = config.api_key
+    if temperature is not None:
+        raw = str(temperature).strip()
+        config.temperature = float(raw) if raw else None
+        os.environ[f"{env_prefix}_QWEN_CHAT_TEMPERATURE"] = raw
+    if timeout_seconds is not None:
+        config.timeout_seconds = float(timeout_seconds)
+        os.environ[f"{env_prefix}_QWEN_CHAT_TIMEOUT_SECONDS"] = str(timeout_seconds)
+    if max_tokens is not None:
+        config.max_tokens = int(max_tokens)
+        os.environ[f"{env_prefix}_QWEN_CHAT_MAX_TOKENS"] = str(max_tokens)
+    if enable_thinking is not None:
+        config.enable_thinking = _parse_bool(enable_thinking)
+        os.environ[f"{env_prefix}_QWEN_CHAT_ENABLE_THINKING"] = (
+            "true" if config.enable_thinking else "false"
+        )


 def get_max_tokens() -> int:
-    return MAX_TOKENS
+    return TARGET_CONFIG.max_tokens
+
+
+def chat_optimizer(
+    system: str,
+    user: str,
+    max_completion_tokens: int = 16384,
+    retries: int = 5,
+    stage: str = "optimizer",
+    reasoning_effort: str | None = None,
+    timeout: float | None = None,
+) -> tuple[str, dict[str, int]]:
+    del reasoning_effort
+    messages = [{"role": "system", "content": system}, {"role": "user", "content": user}]
+    return _chat_messages_impl(
+        messages,
+        max_completion_tokens,
+        retries,
+        stage,
+        role="optimizer",
+        timeout=timeout,
+    )


 def chat_target(
@@ -230,6 +376,33 @@ def chat_target(
        max_completion_tokens,
        retries,
        stage,
+        role="target",
+        timeout=timeout,
+    )
+
+
+def chat_optimizer_messages(
+    messages: list[dict[str, Any]],
+    max_completion_tokens: int = 16384,
+    retries: int = 5,
+    stage: str = "optimizer",
+    reasoning_effort: str | None = None,
+    *,
+    tools: list[dict[str, Any]] | None = None,
+    tool_choice: str | dict[str, Any] | None = None,
+    return_message: bool = False,
+    timeout: float | None = None,
+) -> tuple[Any, dict[str, int]]:
+    del reasoning_effort
+    return _chat_messages_impl(
+        messages,
+        max_completion_tokens,
+        retries,
+        stage,
+        role="optimizer",
+        tools=tools,
+        tool_choice=tool_choice,
+        return_message=return_message,
        timeout=timeout,
    )

@@ -252,6 +425,7 @@ def chat_target_messages(
        max_completion_tokens,
        retries,
        stage,
+        role="target",
        tools=tools,
        tool_choice=tool_choice,
        return_message=return_message,
@@ -272,6 +446,10 @@ def set_reasoning_effort(effort: str | None) -> None:


 def set_target_deployment(deployment: str) -> None:
-    global TARGET_DEPLOYMENT
-    TARGET_DEPLOYMENT = deployment or default_model_for_backend("qwen_chat")
-    os.environ["TARGET_DEPLOYMENT"] = TARGET_DEPLOYMENT
+    TARGET_CONFIG.deployment = deployment or default_model_for_backend("qwen_chat")
+    os.environ["TARGET_DEPLOYMENT"] = TARGET_CONFIG.deployment
+
+
+def set_optimizer_deployment(deployment: str) -> None:
+    OPTIMIZER_CONFIG.deployment = deployment or default_model_for_backend("qwen_chat")
+    os.environ["OPTIMIZER_DEPLOYMENT"] = OPTIMIZER_CONFIG.deployment
@@ -0,0 +1,112 @@
+"""Tests for skillopt.utils.json_utils."""
+from __future__ import annotations
+
+import pytest
+
+from skillopt.utils.json_utils import extract_json, extract_json_array
+
+
+class TestExtractJson:
+    """extract_json — extract a JSON object from LLM response text."""
+
+    def test_code_fence_json(self) -> None:
+        text = 'Some text\n```json\n{"key": "value", "num": 42}\n```\nmore text'
+        assert extract_json(text) == {"key": "value", "num": 42}
+
+    def test_bare_json_object(self) -> None:
+        text = 'The result is {"answer": "yes", "score": 0.95}.'
+        assert extract_json(text) == {"answer": "yes", "score": 0.95}
+
+    def test_code_fence_takes_precedence(self) -> None:
+        """If fence content parses successfully it should be preferred over bare."""
+        text = (
+            '```json\n{"source": "fence"}\n```\n'
+            'Then also {"source": "bare"}'
+        )
+        assert extract_json(text) == {"source": "fence"}
+
+    def test_broken_fence_falls_back_to_bare(self) -> None:
+        """When fence content is invalid JSON, fall back to bare {...} match."""
+        # Use invalid fence content that has no braces so the greedy bare
+        # regex doesn't swallow the valid object.
+        text = (
+            '```json\nnot json at all\n```\n'
+            'Answer: {"fallback": "yes"}'
+        )
+        assert extract_json(text) == {"fallback": "yes"}
+
+    def test_nested_json(self) -> None:
+        text = '```json\n{"outer": {"inner": [1, 2, 3]}}\n```'
+        assert extract_json(text) == {"outer": {"inner": [1, 2, 3]}}
+
+    def test_no_json_returns_none(self) -> None:
+        assert extract_json("Just plain text without JSON.") is None
+
+    def test_empty_string_returns_none(self) -> None:
+        assert extract_json("") is None
+
+    def test_malformed_json_returns_none(self) -> None:
+        assert extract_json("{broken") is None
+
+    def test_empty_json_object(self) -> None:
+        assert extract_json('{"empty": {}}') == {"empty": {}}
+
+    def test_json_with_escaped_chars(self) -> None:
+        text = '{"message": "hello\\nworld"}'
+        assert extract_json(text) == {"message": "hello\nworld"}
+
+    def test_only_fence_with_no_json_syntax(self) -> None:
+        """Code fences without valid JSON content should not match."""
+        text = "```\nplain code block\n```"
+        assert extract_json(text) is None
+
+
+class TestExtractJsonArray:
+    """extract_json_array — extract a JSON array from LLM response text."""
+
+    def test_code_fence_array(self) -> None:
+        text = '```json\n["a", "b", "c"]\n```'
+        assert extract_json_array(text) == ["a", "b", "c"]
+
+    def test_bare_array(self) -> None:
+        text = "The items are [1, 2, 3]."
+        assert extract_json_array(text) == [1, 2, 3]
+
+    def test_code_fence_takes_precedence(self) -> None:
+        text = (
+            '```json\n["from_fence"]\n```\n'
+            'also ["from_bare"]'
+        )
+        assert extract_json_array(text) == ["from_fence"]
+
+    def test_broken_fence_falls_back_to_bare(self) -> None:
+        text = (
+            '```json\nnot json at all\n```\n'
+            'values: [42]'
+        )
+        assert extract_json_array(text) == [42]
+
+    def test_nested_array(self) -> None:
+        text = '```json\n[[1, 2], [3, 4]]\n```'
+        assert extract_json_array(text) == [[1, 2], [3, 4]]
+
+    def test_no_array_returns_none(self) -> None:
+        assert extract_json_array("no brackets here") is None
+
+    def test_empty_string_returns_none(self) -> None:
+        assert extract_json_array("") is None
+
+    def test_malformed_array_returns_none(self) -> None:
+        assert extract_json_array("[1, 2, ") is None
+
+    def test_empty_json_array(self) -> None:
+        assert extract_json_array("[]") == []
+
+    def test_array_of_objects(self) -> None:
+        text = '[{"x": 1}, {"x": 2}]'
+        assert extract_json_array(text) == [{"x": 1}, {"x": 2}]
+
+    def test_object_not_confused_with_array(self) -> None:
+        """extract_json_array should not match a bare JSON object."""
+        text = '{"this is an object": true}'
+        assert extract_json_array(text) is None
@@ -0,0 +1,106 @@
+"""Tests for skillopt.utils.scoring."""
+from __future__ import annotations
+
+import pytest
+
+from skillopt.utils.scoring import compute_score, skill_hash
+
+
+class _ResultObject:
+    """Minimal object with hard/soft attrs (duck-typing path)."""
+
+    def __init__(self, hard: float, soft: float) -> None:
+        self.hard = hard
+        self.soft = soft
+
+
+class TestComputeScore:
+    """compute_score — hard/soft accuracy from a list of episode results."""
+
+    def test_empty_list_returns_zeros(self) -> None:
+        assert compute_score([]) == (0.0, 0.0)
+
+    def test_dict_results_happy_path(self) -> None:
+        results = [
+            {"hard": 1, "soft": 0.8},
+            {"hard": 0, "soft": 0.5},
+            {"hard": 1, "soft": 0.9},
+        ]
+        hard, soft = compute_score(results)
+        assert hard == pytest.approx(2 / 3)
+        assert soft == pytest.approx((0.8 + 0.5 + 0.9) / 3)
+
+    def test_object_results(self) -> None:
+        results = [
+            _ResultObject(1.0, 0.75),
+            _ResultObject(0.0, 0.25),
+        ]
+        hard, soft = compute_score(results)
+        assert hard == 0.5
+        assert soft == 0.5
+
+    def test_mixed_dict_and_object_results(self) -> None:
+        results = [
+            {"hard": 1, "soft": 1.0},
+            _ResultObject(0, 0.0),
+        ]
+        hard, soft = compute_score(results)
+        assert hard == 0.5
+        assert soft == 0.5
+
+    def test_missing_keys_default_to_zero(self) -> None:
+        results = [
+            {"hard": 1},
+            {},
+        ]
+        hard, soft = compute_score(results)
+        assert hard == 0.5
+        assert soft == 0.0
+
+    def test_single_result(self) -> None:
+        results = [{"hard": 1, "soft": 0.95}]
+        assert compute_score(results) == (1.0, 0.95)
+
+    def test_continuous_hard_values(self) -> None:
+        """Hard may be continuous 0.0-1.0 when using smoothed reward."""
+        results = [
+            {"hard": 0.75, "soft": 0.6},
+            {"hard": 0.25, "soft": 0.4},
+        ]
+        hard, soft = compute_score(results)
+        assert hard == 0.5
+        assert soft == 0.5
+
+
+class TestSkillHash:
+    """skill_hash — a short, deterministic hash of skill content."""
+
+    def test_deterministic(self) -> None:
+        assert skill_hash("hello") == skill_hash("hello")
+
+    def test_different_input_produces_different_hash(self) -> None:
+        assert skill_hash("hello") != skill_hash("world")
+
+    def test_empty_string(self) -> None:
+        h = skill_hash("")
+        assert isinstance(h, str)
+        assert len(h) == 16
+
+    def test_output_length(self) -> None:
+        h = skill_hash("some skill content here")
+        assert len(h) == 16
+
+    def test_hex_characters(self) -> None:
+        h = skill_hash("any content")
+        assert all(c in "0123456789abcdef" for c in h)
+
+    def test_unicode_content(self) -> None:
+        h1 = skill_hash("cafe")
+        h2 = skill_hash("cafe")
+        assert h1 == h2
+
+    def test_multiline_content(self) -> None:
+        content = "line1\nline2\nline3"
+        h = skill_hash(content)
+        assert len(h) == 16
+        assert isinstance(h, str)
@@ -0,0 +1,249 @@
+"""Tests for skillopt.types — Edit and Patch dataclass serialization."""
+from __future__ import annotations
+
+import pytest
+
+from skillopt.types import Edit, Patch
+
+
+# ── Edit ────────────────────────────────────────────────────────────────────
+
+
+class TestEditCreation:
+    """Edit dataclass construction."""
+
+    def test_minimal_edit(self) -> None:
+        e = Edit(op="append")
+        assert e.op == "append"
+        assert e.content == ""
+        assert e.target == ""
+        assert e.support_count is None
+        assert e.source_type is None
+        assert e.merge_level is None
+        assert e.update_origin == ""
+        assert e.update_target == ""
+
+    def test_full_edit(self) -> None:
+        e = Edit(
+            op="replace",
+            content="new content",
+            target="old content",
+            support_count=5,
+            source_type="failure",
+            merge_level=2,
+            update_origin="reflect",
+            update_target="skill",
+        )
+        assert e.op == "replace"
+        assert e.content == "new content"
+        assert e.target == "old content"
+        assert e.support_count == 5
+        assert e.source_type == "failure"
+        assert e.merge_level == 2
+        assert e.update_origin == "reflect"
+        assert e.update_target == "skill"
+
+    def test_insert_after_op(self) -> None:
+        e = Edit(op="insert_after", content="insertion", target="anchor")
+        assert e.op == "insert_after"
+        assert e.content == "insertion"
+        assert e.target == "anchor"
+
+    def test_delete_op(self) -> None:
+        e = Edit(op="delete", target="thing_to_remove")
+        assert e.op == "delete"
+        assert e.target == "thing_to_remove"
+
+
+class TestEditRoundTrip:
+    """Edit.to_dict() / Edit.from_dict() round-trip."""
+
+    def test_round_trip_minimal(self) -> None:
+        e = Edit(op="append")
+        d = e.to_dict()
+        restored = Edit.from_dict(d)
+        assert restored == e
+
+    def test_round_trip_full(self) -> None:
+        e = Edit(
+            op="replace",
+            content="new content",
+            target="old content",
+            support_count=3,
+            source_type="success",
+            merge_level=1,
+            update_origin="meta_reflect",
+            update_target="system_prompt",
+        )
+        d = e.to_dict()
+        restored = Edit.from_dict(d)
+        assert restored == e
+
+    def test_round_trip_delete_without_content(self) -> None:
+        e = Edit(op="delete", target="obsolete_line")
+        d = e.to_dict()
+        restored = Edit.from_dict(d)
+        assert restored == e
+
+    def test_optional_fields_omitted_when_default(self) -> None:
+        e = Edit(op="append")
+        d = e.to_dict()
+        assert d == {"op": "append", "content": ""}
+        # support_count, source_type, etc. should be absent
+        assert "support_count" not in d
+        assert "source_type" not in d
+        assert "merge_level" not in d
+        assert "target" not in d
+        assert "update_origin" not in d
+        assert "update_target" not in d
+
+    def test_from_dict_with_defaults(self) -> None:
+        d = {"op": "replace", "content": "abc"}
+        e = Edit.from_dict(d)
+        assert e.op == "replace"
+        assert e.content == "abc"
+        assert e.target == ""
+        assert e.support_count is None
+        assert e.source_type is None
+
+    def test_from_dict_with_extra_keys(self) -> None:
+        """Extra keys in dict should be ignored."""
+        d = {"op": "append", "content": "", "unknown_field": 42}
+        e = Edit.from_dict(d)
+        assert e.op == "append"
+        assert not hasattr(e, "unknown_field")
+
+
+class TestEditEdgeCases:
+    """Edge cases around Edit."""
+
+    def test_support_count_zero(self) -> None:
+        """0 is a valid support_count and should be serialized."""
+        e = Edit(op="append", support_count=0)
+        d = e.to_dict()
+        assert d["support_count"] == 0
+        restored = Edit.from_dict(d)
+        assert restored.support_count == 0
+
+    def test_merge_level_zero(self) -> None:
+        e = Edit(op="replace", merge_level=0)
+        d = e.to_dict()
+        assert d["merge_level"] == 0
+        restored = Edit.from_dict(d)
+        assert restored.merge_level == 0
+
+    def test_empty_target_stays_empty(self) -> None:
+        e = Edit(op="append", target="")
+        d = e.to_dict()
+        assert "target" not in d
+
+
+# ── Patch ───────────────────────────────────────────────────────────────────
+
+
+class TestPatchCreation:
+    """Patch dataclass construction."""
+
+    def test_empty_patch(self) -> None:
+        p = Patch()
+        assert p.edits == []
+        assert p.reasoning == ""
+        assert p.ranking_details is None
+
+    def test_patch_with_edits(self) -> None:
+        edits = [
+            Edit(op="append", content="step 1"),
+            Edit(op="append", content="step 2"),
+        ]
+        p = Patch(edits=edits, reasoning="Added two steps")
+        assert len(p.edits) == 2
+        assert p.reasoning == "Added two steps"
+
+    def test_patch_with_ranking_details(self) -> None:
+        p = Patch(ranking_details={"score": 0.95, "rank": 1})
+        assert p.ranking_details == {"score": 0.95, "rank": 1}
+
+
+class TestPatchRoundTrip:
+    """Patch.to_dict() / Patch.from_dict() round-trip."""
+
+    def test_round_trip_empty(self) -> None:
+        p = Patch()
+        d = p.to_dict()
+        restored = Patch.from_dict(d)
+        assert restored.edits == []
+        assert restored.reasoning == ""
+        assert restored.ranking_details is None
+
+    def test_round_trip_with_edits(self) -> None:
+        edits = [
+            Edit(op="insert_after", content="new step", target="existing step"),
+            Edit(op="replace", content="updated", target="old"),
+        ]
+        p = Patch(edits=edits, reasoning="Batch update")
+        d = p.to_dict()
+        restored = Patch.from_dict(d)
+        assert len(restored.edits) == 2
+        for original, restored_edit in zip(p.edits, restored.edits):
+            assert isinstance(restored_edit, Edit)
+            assert original == restored_edit
+        assert restored.reasoning == "Batch update"
+        assert restored.ranking_details is None
+
+    def test_round_trip_with_ranking_details(self) -> None:
+        details = {"strategy": "rouge", "scores": [0.9, 0.8, 0.7]}
+        p = Patch(
+            edits=[Edit(op="append", content="a")],
+            reasoning="selected best",
+            ranking_details=details,
+        )
+        d = p.to_dict()
+        restored = Patch.from_dict(d)
+        assert restored.ranking_details == details
+
+    def test_to_dict_contains_reasoning_and_edits(self) -> None:
+        p = Patch(edits=[Edit(op="append", content="test")], reasoning="reason")
+        d = p.to_dict()
+        assert "reasoning" in d
+        assert "edits" in d
+        assert isinstance(d["edits"], list)
+
+    def test_from_dict_preserves_edit_order(self) -> None:
+        edits = [
+            Edit(op="append", content="first"),
+            Edit(op="insert_after", content="second", target="first"),
+            Edit(op="append", content="third"),
+        ]
+        p = Patch(edits=edits, reasoning="ordered")
+        d = p.to_dict()
+        restored = Patch.from_dict(d)
+        assert restored.edits[0].content == "first"
+        assert restored.edits[1].content == "second"
+        assert restored.edits[2].content == "third"
+
+
+class TestPatchEdgeCases:
+    """Edge cases around Patch."""
+
+    def test_reasoning_empty_string(self) -> None:
+        p = Patch(reasoning="")
+        d = p.to_dict()
+        assert d["reasoning"] == ""
+
+    def test_zero_edits(self) -> None:
+        """Patch with explicitly empty edit list."""
+        p = Patch(edits=[])
+        d = p.to_dict()
+        assert d["edits"] == []
+
+    def test_nested_edit_from_dict_handles_dicts(self) -> None:
+        """from_dict should accept dicts in the 'edits' list."""
+        d = {
+            "reasoning": "test",
+            "edits": [{"op": "append", "content": "hello"}],
+        }
+        p = Patch.from_dict(d)
+        assert len(p.edits) == 1
+        assert isinstance(p.edits[0], Edit)
+        assert p.edits[0].op == "append"
+        assert p.edits[0].content == "hello"
Author	SHA1	Message	Date
Ziyang Gong	25da7cb2dd	Merge pull request #32 from Yif-Yang/fix/issue-30-docs-and-template Fix/issue 30 docs and template	2026-06-02 10:12:48 +08:00
Yifan Yang	4eb4c64b2a	envs/_template: make template instantiable against real EnvAdapter ABC The shipped env_template.py and loader_template.py described the same fictional async execute / evaluate / build_prompt API documented in docs/reference/api.md. As a result TemplateBenchmarkEnv(cfg) raised 'TypeError: Can't instantiate abstract class' for every copy-and-paste user who followed the in-tree scaffold. Rewrite the template so it's a working starting point: - env_template.py: TemplateBenchmarkEnv(EnvAdapter) now implements all five real abstract methods (build_train_env, build_eval_env, rollout, reflect, get_task_types) with no-op defaults documented as TODO. Instantiable today; pytest 60/60 still passes. - loader_template.py: TemplateBenchmarkLoader(SplitDataLoader) implements load_split_items for .json / .jsonl input and explains the optional load_raw_items override for split_mode="ratio". - README.md: usage steps now point at scripts/train.py's _ENV_REGISTRY (the real registry) instead of a non-existent BENCHMARK_REGISTRY in skillopt/envs/__init__.py, and link to the rewritten new-benchmark guide. - config_template.yaml: _base_ is a string path (not a list, which the loader rejects); skill_init is commented out with a note so the template config doesn't reference a file the user hasn't created. Verified locally: 'from skillopt.envs._template.env_template import TemplateBenchmarkEnv; TemplateBenchmarkEnv()' succeeds. Refs microsoft/SkillOpt#30. Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>	2026-06-01 20:15:12 +00:00
Yifan Yang	2ca2910649	docs: align API reference and Add-a-Benchmark guide with real EnvAdapter ABC docs/reference/api.md previously documented a fictional EnvAdapter API (execute / evaluate / build_prompt + DataItem / TaskResult) and a BENCHMARK_REGISTRY that never existed in code. Anyone following the documented contract would hit ImportError or TypeError on the first instantiation. Replace both pages with the real shape from skillopt/envs/base.py and skillopt/datasets/base.py: - EnvAdapter: build_train_env, build_eval_env, rollout, reflect, get_task_types (the 5 actual abstract methods). - Rollout dicts: id / hard / soft required; everything else preserved into RolloutResult.extras. - Reflect dicts: {patch, source_type} schema as consumed by run_minibatch_reflect. - BatchSpec: slotted-but-mutable dataclass matching the actual definition (payload defaults to None, metadata to dict()). - SplitDataLoader.load_split_items as the one mandatory loader method. - Registry: _ENV_REGISTRY in scripts/train.py (lazy try/except ImportError block), not a non-existent BENCHMARK_REGISTRY in skillopt/envs/__init__.py. - _base_: documented as a string path, since the current YAML loader only accepts strings. The new-benchmark.md guide now walks through a docfaithful worked example with a real rollout helper (chat_target + scorer) instead of hand-waving over the rollout step. Refs microsoft/SkillOpt#30. Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>	2026-06-01 20:14:54 +00:00
Yifan Yang	fb1a76371d	Merge pull request #29 from LifeIsSoSolong/codex/qwen-chat-optimizer-backend Support qwen_chat as optimizer backend	2026-06-02 03:27:50 +08:00
Yifan Yang	47063e1ceb	Merge pull request #27 from Oxygen56/test/add-core-utility-tests test: add unit test suite for core utility modules	2026-06-02 03:27:26 +08:00
hwq	181d71b737	Release data split manifests	2026-06-01 16:02:14 +00:00
kaikai-macbook	41012e2d5e	Support Qwen chat as optimizer backend	2026-06-01 16:44:49 +08:00
Claude Code Agent	dd8cd993b5	test: add unit test suite for core utility modules Add initial test infrastructure covering: - skillopt/utils/scoring.py (compute_score, skill_hash) - skillopt/utils/json_utils.py (extract_json, extract_json_array) - skillopt/types.py (Edit, Patch dataclass serialization) All tested functions are pure/deterministic with no LLM dependencies. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-01 02:04:22 +08:00
Yif Yang	8ebede0efd	Refine README for clarity on optimization results Removed redundant wording about math benchmarks.	2026-05-31 18:20:00 +08:00
Yif Yang	266fca72ab	docs: clarify optional features and ckpt artifacts	2026-05-31 09:36:25 +00:00
Yif Yang	9265545c45	docs: clarify README and paper-aligned skill artifacts	2026-05-31 09:23:07 +00:00
Yif Yang	b4850ce418	fix(minimax): wire YAML / CLI config through to backend PR #26 added a MiniMax chat backend but left three loose ends that silently dropped any YAML / CLI configuration of minimax_* keys: only the environment-variable path worked. - skillopt/config.py: add 6 model.minimax_* entries to _FLATTEN_MAP so the keys declared in configs/_base_/default.yaml actually survive flatten_config() (mirroring the existing model.qwen_chat_* block). - skillopt/engine/trainer.py: import configure_minimax_chat and call it alongside configure_qwen_chat, so cfg-supplied credentials, temperature, max_tokens, and enable_thinking reach the backend. Also apply cfg["minimax_model"] via set_target_deployment when the active target backend is minimax_chat. - scripts/train.py: add 6 --minimax_* CLI flags + the corresponding _CLI_TO_YAML entries, add 'minimax' / 'minimax_chat' to the --backend choices, auto-route to target_backend=minimax_chat, and pick the right default target_model for the new backend. Default behavior on existing backends (openai, claude, qwen, codex, claude_code_exec) is unchanged; all 8 shipped configs continue to load with gate_metric falling back to 'hard' for paper reproduction.	2026-05-31 08:22:20 +00:00
Yif Yang	643346c9f3	Merge pull request #26 from KovaForge/minimax-backend feat: add MiniMax as first-class chat backend Adds skillopt/model/minimax_backend.py (clean port of qwen_backend.py targeting MiniMax-M2.7 via https://api.minimax.io/v1) and registers it in the router, backend_config, and common defaults. Existing backends (openai_chat, claude_chat, qwen_chat, codex_exec, claude_code_exec) remain bit-for-bit unchanged. Verified via 10 import / routing / parity subtests; backward-compat sweep across the 8 shipped configs passes with no regression. A follow-up commit completes the YAML / CLI plumbing that this PR left half-wired (FLATTEN_MAP entries, trainer-level configure_minimax_chat call, and --minimax_* CLI args).	2026-05-31 08:20:39 +00:00
Declan Murphy	c6da31df44	fix: use correct MiniMax endpoint, model name, and add .venv to gitignore	2026-05-31 05:27:50 +08:00
Declan Murphy	e4201074aa	docs: add MiniMax config to default.yaml and .env.example default.yaml: - Add minimax_base_url, minimax_api_key, minimax_model, minimax_temperature, minimax_max_tokens, minimax_enable_thinking settings - Add optimizer_minimax_base_url, target_minimax_base_url per-role overrides - Add optimizer_minimax_api_key, target_minimax_api_key per-role overrides .env.example: - Add MINIMAX_BASE_URL, MINIMAX_API_KEY, MINIMAX_MODEL env var docs	2026-05-31 05:22:35 +08:00
Declan Murphy	309ea64ff4	feat: integrate MiniMax into model router, backend config, and common common.py: - Add minimax_chat → MiniMax/MiniMax-Text-01 to _BACKEND_DEFAULT_MODELS - Add minimax/minimax_chat aliases to _BACKEND_ALIASES backend_config.py: - Add minimax_chat to set_optimizer_backend() valid set - Add minimax_chat to set_target_backend() valid set - Add minimax_chat to is_optimizer_chat_backend() - Add minimax_chat to is_target_chat_backend() __init__.py: - Import minimax_backend as _minimax - Add minimax_chat to set_backend() legacy handler - Add minimax_chat to get_backend_name() reporting - Route chat_target() and chat_target_messages() to _minimax - Update NotImplementedError messages to list minimax_chat - Aggregate _minimax into get_token_summary() - Add _minimax.reset_token_tracker() - Add configure_minimax_chat() delegator - Add _minimax to set_reasoning_effort() and set_target_deployment()	2026-05-31 05:22:33 +08:00
Declan Murphy	d224d425f9	feat: add MiniMax chat backend module Port qwen_backend.py pattern to minimax_backend.py as a new OpenAI-compatible urllib-based backend. Includes: - BASE_URL defaulting to https://api.minimax.chat/v1 - API_KEY, TIMEOUT_SECONDS, MAX_TOKENS, TEMPERATURE env vars - ENABLE_THINKING support (MiniMax thinking mode) - configure_minimax_chat() runtime configurator - chat_target() and chat_target_messages() functions - TokenTracker integration and get_token_summary() - set_target_deployment() support - Default model: MiniMax/MiniMax-Text-01	2026-05-31 05:22:29 +08:00