Compare commits
1 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Cuzyoung
|
8acc2dd03e |
@@ -27,8 +27,3 @@ export AZURE_OPENAI_API_KEY=
|
||||
# ── Qwen Local Model (for qwen_chat backend) ────────────────────────
|
||||
# export QWEN_CHAT_BASE_URL=http://localhost:8000/v1
|
||||
# export QWEN_CHAT_MODEL=Qwen/Qwen3.5-4B
|
||||
|
||||
# ── MiniMax (for minimax_chat backend) ──────────────────────────────
|
||||
# export MINIMAX_BASE_URL=https://api.minimax.io/v1
|
||||
# export MINIMAX_API_KEY=...
|
||||
# export MINIMAX_MODEL=MiniMax-M2.7
|
||||
|
||||
+1
-15
@@ -5,20 +5,7 @@ build/
|
||||
dist/
|
||||
site/
|
||||
|
||||
data/*
|
||||
!data/README.md
|
||||
!data/searchqa_id_split/
|
||||
!data/searchqa_id_split/**
|
||||
!data/livemathematicianbench_id_split/
|
||||
!data/livemathematicianbench_id_split/**
|
||||
!data/docvqa_id_split/
|
||||
!data/docvqa_id_split/**
|
||||
!data/officeqa_id_split/
|
||||
!data/officeqa_id_split/**
|
||||
!data/spreadsheetbench_id_split/
|
||||
!data/spreadsheetbench_id_split/**
|
||||
!data/alfworld_path_split/
|
||||
!data/alfworld_path_split/**
|
||||
data/
|
||||
outputs/
|
||||
logs/
|
||||
external/
|
||||
@@ -53,4 +40,3 @@ docs/reflact_overview.html
|
||||
docs/render_ablation_paper_tables.py
|
||||
docs/让*
|
||||
.gradio/
|
||||
.venv
|
||||
|
||||
@@ -4,37 +4,7 @@
|
||||
|
||||
[](https://microsoft.github.io/SkillOpt/) [](https://arxiv.org/abs/2605.23904) [](https://youtu.be/JUBMDTCiM0M) [](https://www.python.org/) [](LICENSE)
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
Modern agent skills are usually hand-crafted, generated one-shot by a strong
|
||||
LLM, or evolved through loosely controlled self-revision — none of which
|
||||
behaves like a deep-learning optimizer for the skill itself, and none of
|
||||
which reliably improves over its starting point under feedback.
|
||||
|
||||
**SkillOpt treats the skill document as the trainable state of a frozen
|
||||
agent**, and trains it with the discipline that makes weight-space
|
||||
optimization reproducible. A separate optimizer model turns scored rollouts
|
||||
into bounded add / delete / replace edits on a single skill document; a
|
||||
candidate edit is accepted only when it strictly improves a held-out
|
||||
validation score. A textual learning-rate budget, a rejected-edit buffer,
|
||||
and an epoch-wise slow / meta update make skill training stable while
|
||||
adding **zero inference-time model calls** at deployment.
|
||||
|
||||
The deployed artifact is a compact `best_skill.md` (typically 300–2,000
|
||||
tokens) that runs against the unchanged target model. Across **six
|
||||
benchmarks, seven target models, and three execution harnesses** (direct
|
||||
chat, Codex CLI, Claude Code CLI), SkillOpt is best or tied-best on **all
|
||||
52 evaluated (model, benchmark, harness) cells** and on GPT-5.5 lifts the
|
||||
average no-skill accuracy by **+23.5 points in direct chat, +24.8 inside
|
||||
the Codex agentic loop, and +19.1 inside Claude Code**. Optimized skill
|
||||
artifacts transfer across model scales, between Codex and Claude Code
|
||||
harnesses, and to nearby benchmarks without further optimization.
|
||||
|
||||
For the full method, ablations, and per-cell results see the [paper](https://arxiv.org/abs/2605.23904); for a visual walkthrough of the loop see the [project page](https://microsoft.github.io/SkillOpt/); for deeper API / backend / benchmark docs see [`docs/`](docs/).
|
||||
|
||||
## 🎬 Demo Video
|
||||
## 🎬 SkillOpt Demo Video
|
||||
|
||||
https://github.com/user-attachments/assets/eb12d3bc-371c-467f-904d-91b61f339ed7
|
||||
|
||||
@@ -44,18 +14,39 @@ https://github.com/user-attachments/assets/eb12d3bc-371c-467f-904d-91b61f339ed7
|
||||
|
||||
---
|
||||
|
||||
## Documentation
|
||||
|
||||
A complete, self-contained **Documentation & Reproduction Guide** lives at
|
||||
[`docs/guideline.html`](docs/guideline.html). It covers installation, data
|
||||
preparation, training/eval commands, the full configuration reference, the
|
||||
framework internals (training loop, validation gate, slow update, meta skill),
|
||||
and an API/function reference — all in a single page with a left navigation
|
||||
sidebar.
|
||||
|
||||
Because GitHub shows raw source for `.html` files instead of rendering them,
|
||||
open the guide one of these ways:
|
||||
|
||||
- **Locally** — clone the repo and open `docs/guideline.html` in any browser
|
||||
(no build step required).
|
||||
- **Rendered online (no setup)** — via the htmlpreview proxy:
|
||||
[`htmlpreview.github.io/?…/docs/guideline.html`](https://htmlpreview.github.io/?https://github.com/microsoft/SkillOpt/blob/main/docs/guideline.html)
|
||||
- **GitHub Pages** — the repository's GitHub Pages site already serves the
|
||||
project homepage from the repo root, so the guide is reachable alongside it at
|
||||
`https://microsoft.github.io/SkillOpt/docs/guideline.html` (the homepage at
|
||||
`https://microsoft.github.io/SkillOpt/` is unaffected).
|
||||
|
||||
---
|
||||
|
||||
## Install
|
||||
|
||||
### Requirements
|
||||
|
||||
- Python 3.10+
|
||||
**Requirements:** Python 3.10+
|
||||
|
||||
```bash
|
||||
git clone https://github.com/microsoft/SkillOpt.git
|
||||
cd SkillOpt
|
||||
pip install -e .
|
||||
|
||||
# For the ALFWorld benchmark (optional):
|
||||
# For ALFWorld benchmark (optional):
|
||||
pip install -e ".[alfworld]"
|
||||
alfworld-download
|
||||
```
|
||||
@@ -68,8 +59,7 @@ cp .env.example .env
|
||||
source .env
|
||||
```
|
||||
|
||||
#### Azure OpenAI *(recommended)*
|
||||
|
||||
**Azure OpenAI** (recommended):
|
||||
```bash
|
||||
export AZURE_OPENAI_ENDPOINT="https://your-resource.openai.azure.com/"
|
||||
# Option 1: API key auth
|
||||
@@ -78,56 +68,74 @@ export AZURE_OPENAI_API_KEY="your-key"
|
||||
export AZURE_OPENAI_AUTH_MODE="azure_cli"
|
||||
```
|
||||
|
||||
> **Note:** `AZURE_OPENAI_ENDPOINT` is required for all three modes (`api_key`, `azure_cli`, `openai_compatible`). Without it, all LLM calls will fail.
|
||||
|
||||
#### OpenAI-compatible endpoints
|
||||
> **Note:** `AZURE_OPENAI_ENDPOINT` is required for all three modes (`api_key`, `azure_cli`,
|
||||
> `openai_compatible`). Without it, all LLM calls will fail.
|
||||
|
||||
**OpenAI-compatible endpoints**:
|
||||
```bash
|
||||
export AZURE_OPENAI_ENDPOINT="https://api.openai.com/v1"
|
||||
export AZURE_OPENAI_API_KEY="sk-..."
|
||||
export AZURE_OPENAI_AUTH_MODE="openai_compatible"
|
||||
```
|
||||
|
||||
This routes all calls through the plain OpenAI Python client (no Azure auth, no `api-version` header).
|
||||
This routes all calls through the plain OpenAI Python client (no Azure auth, no `api-version`
|
||||
header).
|
||||
|
||||
> **Note:** SkillOpt reuses the `AZURE_OPENAI_*` env var names even in this mode — there is no separate `OPENAI_API_KEY` knob.
|
||||
|
||||
#### Anthropic Claude
|
||||
> **Note:** SkillOpt reuses the `AZURE_OPENAI_*` env var names even in this mode — there is no
|
||||
> separate `OPENAI_API_KEY` knob.
|
||||
|
||||
**Anthropic Claude**:
|
||||
```bash
|
||||
export ANTHROPIC_API_KEY="sk-ant-..."
|
||||
```
|
||||
|
||||
#### Qwen *(local vLLM)*
|
||||
|
||||
**Qwen (local vLLM)**:
|
||||
```bash
|
||||
export QWEN_CHAT_BASE_URL="http://localhost:8000/v1"
|
||||
export QWEN_CHAT_MODEL="Qwen/Qwen3.5-4B"
|
||||
```
|
||||
|
||||
`qwen_chat` can also be used as the optimizer backend. When optimizer and
|
||||
target should point to different local vLLM services, use the role-specific
|
||||
settings:
|
||||
---
|
||||
|
||||
```bash
|
||||
python scripts/train.py \
|
||||
--config configs/searchqa/default.yaml \
|
||||
--optimizer_backend qwen_chat \
|
||||
--target_backend qwen_chat \
|
||||
--optimizer_model Qwen/Qwen3.5-4B \
|
||||
--target_model Qwen/Qwen3.5-4B \
|
||||
--optimizer_qwen_chat_base_url http://localhost:8001/v1 \
|
||||
--target_qwen_chat_base_url http://localhost:8000/v1
|
||||
## Data Preparation
|
||||
|
||||
SkillOpt expects data in a **split directory** with `train/`, `val/`, `test/` subdirectories, each containing a JSON file (e.g., `items.json`).
|
||||
|
||||
```
|
||||
data/my_split/
|
||||
├── train/items.json
|
||||
├── val/items.json
|
||||
└── test/items.json
|
||||
```
|
||||
|
||||
#### MiniMax
|
||||
Each JSON file is an array of task items. The required fields depend on the benchmark. For example, SearchQA items look like:
|
||||
|
||||
```bash
|
||||
export MINIMAX_BASE_URL="https://api.minimax.io/v1"
|
||||
export MINIMAX_API_KEY="..."
|
||||
export MINIMAX_MODEL="MiniMax-M2.7"
|
||||
```json
|
||||
[
|
||||
{
|
||||
"id": "unique_item_id",
|
||||
"question": "Who wrote the novel ...",
|
||||
"context": "[DOC] relevant passage text ...",
|
||||
"answers": ["expected answer"]
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
See `skillopt/envs/<benchmark>/dataloader.py` for the exact format each benchmark expects.
|
||||
|
||||
> **Note:** Benchmark datasets are not included in this repository. Prepare your own data following the format above.
|
||||
|
||||
### Supported Benchmarks
|
||||
|
||||
| Benchmark | Type | Config |
|
||||
|---|---|---|
|
||||
| SearchQA | QA | `configs/searchqa/default.yaml` |
|
||||
| ALFWorld | Embodied agent | `configs/alfworld/default.yaml` |
|
||||
| DocVQA | Document QA | `configs/docvqa/default.yaml` |
|
||||
| LiveMathematicianBench | Math | `configs/livemathematicianbench/default.yaml` |
|
||||
| SpreadsheetBench | Code generation | `configs/spreadsheetbench/default.yaml` |
|
||||
| OfficeQA | Tool-augmented QA | `configs/officeqa/default.yaml` |
|
||||
|
||||
---
|
||||
|
||||
## Quick Start
|
||||
@@ -154,7 +162,7 @@ python scripts/train.py \
|
||||
# Train on ALFWorld:
|
||||
python scripts/train.py \
|
||||
--config configs/alfworld/default.yaml \
|
||||
--split_dir data/alfworld_path_split \
|
||||
--split_dir /path/to/your/alfworld_split \
|
||||
--azure_openai_endpoint https://your-resource.openai.azure.com/ \
|
||||
--optimizer_model gpt-5.5 \
|
||||
--target_model gpt-5.5
|
||||
@@ -196,7 +204,8 @@ python scripts/eval_only.py \
|
||||
--azure_openai_endpoint https://your-resource.openai.azure.com/
|
||||
```
|
||||
|
||||
To evaluate a skill produced by your own training run, replace `--skill` with that run's best-skill path, for example `outputs/my_run/best_skill.md`.
|
||||
To evaluate a skill produced by a training run, replace `--skill` with that
|
||||
run's best-skill path, for example `outputs/my_run/best_skill.md`.
|
||||
|
||||
| Split | Description |
|
||||
|---|---|
|
||||
@@ -207,7 +216,7 @@ To evaluate a skill produced by your own training run, replace `--skill` with th
|
||||
|
||||
### Output Structure
|
||||
|
||||
Each training run writes to a structured output directory:
|
||||
Each run writes to a structured output directory:
|
||||
|
||||
```
|
||||
outputs/<run_name>/
|
||||
@@ -223,147 +232,26 @@ outputs/<run_name>/
|
||||
|
||||
Re-running the same command auto-resumes from the last completed step.
|
||||
|
||||
### Pretrained Skill Artifacts
|
||||
|
||||
We provide a subset of the paper's main Table 1 GPT-5.5 optimized skills in
|
||||
[`ckpt/`](ckpt/) as reference artifacts. Use them with `scripts/eval_only.py`
|
||||
to evaluate the provided skills on a matching data split without re-running
|
||||
training. See [`ckpt/README.md`](ckpt/README.md) for the full per-benchmark
|
||||
command. This is the first artifact batch; we plan to continue uploading
|
||||
the remaining optimized skills and benchmark split manifests as they are
|
||||
cleaned and verified.
|
||||
|
||||
---
|
||||
|
||||
## Data Preparation
|
||||
## Community-contributed configs
|
||||
|
||||
### Directory layout
|
||||
|
||||
SkillOpt expects data in a **split directory** with `train/`, `val/`, `test/` subdirectories, each containing a JSON file (e.g., `items.json`):
|
||||
|
||||
```
|
||||
data/my_split/
|
||||
├── train/items.json
|
||||
├── val/items.json
|
||||
└── test/items.json
|
||||
```
|
||||
|
||||
Each JSON file is an array of task items. The required fields depend on the benchmark. For example, SearchQA items look like:
|
||||
|
||||
```json
|
||||
[
|
||||
{
|
||||
"id": "unique_item_id",
|
||||
"question": "Who wrote the novel ...",
|
||||
"context": "[DOC] relevant passage text ...",
|
||||
"answers": ["expected answer"]
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
See `skillopt/envs/<benchmark>/dataloader.py` for the exact format each benchmark expects.
|
||||
|
||||
> **Note:** Most benchmark datasets are not included in this repository. Prepare your own data following the format above. The exact SearchQA split used in the paper is provided at [`data/searchqa_id_split/`](data/searchqa_id_split) (400 train / 200 val / 1400 test). We are preparing the remaining benchmark split manifests for upload.
|
||||
|
||||
### Supported Benchmarks
|
||||
|
||||
| Benchmark | Type | Config |
|
||||
|---|---|---|
|
||||
| SearchQA | QA | `configs/searchqa/default.yaml` |
|
||||
| ALFWorld | Embodied agent | `configs/alfworld/default.yaml` |
|
||||
| DocVQA | Document QA | `configs/docvqa/default.yaml` |
|
||||
| LiveMathematicianBench | Math | `configs/livemathematicianbench/default.yaml` |
|
||||
| SpreadsheetBench | Code generation | `configs/spreadsheetbench/default.yaml` |
|
||||
| OfficeQA | Tool-augmented QA | `configs/officeqa/default.yaml` |
|
||||
|
||||
---
|
||||
|
||||
## Configuration
|
||||
|
||||
### Default settings and paper-reproduction knobs
|
||||
|
||||
`configs/_base_/default.yaml` is the single source of truth for SkillOpt's
|
||||
runtime knobs. Out of the box, every included benchmark config inherits
|
||||
from it and keeps the paper protocol visible: 4 epochs, rollout batch 40,
|
||||
reflection minibatch 8, textual learning rate 4 with cosine decay, strict
|
||||
hard validation gating, and slow-update + meta-skill enabled. One detail to
|
||||
watch is slow-update acceptance: the current `main` default is the newer
|
||||
post-submission force-accept mode, while the paper protocol and the
|
||||
paper-aligned skills under `ckpt/` use the gated semantics described in
|
||||
paper Section 3.6.
|
||||
|
||||
### Slow-update acceptance mode
|
||||
|
||||
The epoch-boundary slow / meta update can be applied two ways, controlled
|
||||
by `optimizer.slow_update_gate_with_selection`:
|
||||
|
||||
```yaml
|
||||
optimizer:
|
||||
slow_update_gate_with_selection: false # current main default
|
||||
```
|
||||
|
||||
- **`false`** *(current `main` default)*: force-accept. The
|
||||
slow-update guidance is injected into both `current_skill` and
|
||||
`best_skill` unconditionally at the epoch boundary. This is the newer
|
||||
post-submission behavior on `main`.
|
||||
- **`true`** *(paper / ckpt-skill reproduction)*: gated, matching paper
|
||||
Section 3.6 verbatim. The slow-update candidate is evaluated on the
|
||||
selection split and accepted only if it passes the same validation gate
|
||||
as a step-level edit. Use this setting when re-running optimization to
|
||||
match the paper protocol and the provenance of the provided `ckpt/` skills.
|
||||
|
||||
The trainer prints which mode is active at startup
|
||||
(`[slow update] acceptance=...`). See issue #22 for the discussion that
|
||||
led to the flag.
|
||||
|
||||
### Gate metric (`hard` / `soft` / `mixed`)
|
||||
|
||||
The validation gate compares candidate vs. current skills on the selection
|
||||
split using `gate_metric`:
|
||||
|
||||
- **`hard`** *(default, paper)*: exact-match accuracy, strictly greater
|
||||
than the current score is required.
|
||||
- **`soft`**: per-item soft / partial-credit score. Useful when the
|
||||
selection split is small (e.g. ≤10 items) and the reward is continuous,
|
||||
where the discrete hard gate often rejects every candidate.
|
||||
- **`mixed`**: weighted average, `(1 - w) * hard + w * soft`, with `w`
|
||||
set by `gate_mixed_weight` (default `0.5`).
|
||||
|
||||
Default is `hard`. Use the optional feature config below to switch.
|
||||
|
||||
### Optional feature configs
|
||||
|
||||
These are **not** default SkillOpt settings — they are optional feature configs
|
||||
These are **not** default SkillOpt settings — they are reference configs
|
||||
contributed by users for specific scenarios. The paper-reported numbers
|
||||
were obtained with the default settings, not these.
|
||||
|
||||
- **[`configs/features/soft_gate.yaml`](configs/features/soft_gate.yaml)**
|
||||
*(PR #25, contributed by [@lvbaocheng](https://github.com/lvbaocheng))* —
|
||||
switches `gate_metric` to `soft` (or `mixed`). See the comment at the
|
||||
top of the file for when to use and when not to.
|
||||
- **`configs/examples/soft_gate.yaml`** *(PR #25, contributed by
|
||||
[@lvbaocheng](https://github.com/lvbaocheng))* — switches the
|
||||
validation gate from exact-match (`hard`) to soft / partial-credit
|
||||
(`soft` or `mixed`). Useful when the held-out **selection split is
|
||||
small** (e.g. ≤ ~10 items) and the **reward is continuous**, where the
|
||||
discrete hard gate often rejects every candidate and training stalls.
|
||||
See the comment at the top of the file for details and when not to use
|
||||
it.
|
||||
|
||||
---
|
||||
|
||||
## Extensibility & WebUI
|
||||
|
||||
### Adding a new backend
|
||||
|
||||
A backend = a chat / exec target (e.g. `openai_chat`, `claude_chat`,
|
||||
`qwen_chat`, `minimax_chat`, `codex_exec`, `claude_code_exec`). See
|
||||
[`docs/guide/new-backend.md`](docs/guide/new-backend.md) for the full
|
||||
contract; in short you add a `skillopt/model/<name>_backend.py` module,
|
||||
register it in `skillopt/model/common.py` + `backend_config.py`, and wire
|
||||
it through the router in `skillopt/model/__init__.py`. `qwen_backend.py`
|
||||
and `minimax_backend.py` are good templates.
|
||||
|
||||
### Adding a new benchmark
|
||||
|
||||
A benchmark = a `skillopt/envs/<name>/` package with a `dataloader.py`, a
|
||||
`rollout.py`, and an `initial.md` seed skill. See
|
||||
[`docs/guide/new-benchmark.md`](docs/guide/new-benchmark.md) for the full
|
||||
contract; the simplest reference is `skillopt/envs/searchqa/`.
|
||||
|
||||
### WebUI
|
||||
## WebUI
|
||||
|
||||
Launch the monitoring dashboard (optional):
|
||||
|
||||
@@ -378,6 +266,11 @@ python -m skillopt_webui.app
|
||||
| `--host` | `0.0.0.0` | Bind address |
|
||||
| `--share` | off | Create a public Gradio share link |
|
||||
|
||||
```bash
|
||||
# With public share link (useful for remote servers)
|
||||
python -m skillopt_webui.app --share
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Citation
|
||||
|
||||
@@ -1,79 +0,0 @@
|
||||
# Paper-aligned SkillOpt reference skills (GPT-5.5)
|
||||
|
||||
This folder provides a subset of the paper's main Table 1 GPT-5.5 optimized
|
||||
skills as reference artifacts — one `gpt5.5_skill.md` per currently included
|
||||
benchmark. You can plug them into `scripts/eval_only.py` to evaluate the
|
||||
provided skills on a given split without re-running the training loop.
|
||||
|
||||
> These are checkpoints associated with the paper, not a general-purpose
|
||||
> tool. They're here so you can verify the reported numbers and use the
|
||||
> skills as portable artifacts. If you want to *train* your own skill,
|
||||
> use `scripts/train.py` per the top-level README.
|
||||
>
|
||||
> This is the first artifact batch. We plan to continue uploading the
|
||||
> remaining optimized skills and benchmark split manifests as they are
|
||||
> cleaned and verified.
|
||||
|
||||
## What's here
|
||||
|
||||
| Benchmark | Skill artifact | Matching config |
|
||||
|---|---|---|
|
||||
| SearchQA | `ckpt/searchqa/gpt5.5_skill.md` | `configs/searchqa/default.yaml` |
|
||||
| ALFWorld | `ckpt/alfworld/gpt5.5_skill.md` | `configs/alfworld/default.yaml` |
|
||||
| DocVQA | `ckpt/docvqa/gpt5.5_skill.md` | `configs/docvqa/default.yaml` |
|
||||
| LiveMathematicianBench | `ckpt/livemath/gpt5.5_skill.md` | `configs/livemathematicianbench/default.yaml` |
|
||||
| OfficeQA | `ckpt/officeqa/gpt5.5_skill.md` | `configs/officeqa/default.yaml` |
|
||||
| SpreadsheetBench | `ckpt/spreadsheetbench/gpt5.5_skill.md` | `configs/spreadsheetbench/default.yaml` |
|
||||
|
||||
Each file is a plain Markdown skill document (~2k–13k chars). It contains a
|
||||
protected `SLOW_UPDATE` section at the end that holds epoch-wise
|
||||
longitudinal guidance — that's expected, not a formatting issue.
|
||||
|
||||
## How to evaluate a provided skill
|
||||
|
||||
`scripts/eval_only.py` runs a single skill against a data split without
|
||||
invoking the optimizer. Example for SearchQA against the test split:
|
||||
|
||||
```bash
|
||||
python scripts/eval_only.py \
|
||||
--config configs/searchqa/default.yaml \
|
||||
--skill ckpt/searchqa/gpt5.5_skill.md \
|
||||
--split valid_unseen \
|
||||
--split_dir data/searchqa_id_split \
|
||||
--azure_openai_endpoint https://your-resource.openai.azure.com/ \
|
||||
--target_model gpt-5.5
|
||||
```
|
||||
|
||||
Substitute the benchmark, config, skill path, and `--split_dir` to evaluate
|
||||
any of the other five. `--split valid_unseen` is the test split, `valid_seen`
|
||||
is the selection / validation split, `train` is the training split, and
|
||||
`all` runs all three.
|
||||
|
||||
## On comparing to the paper numbers
|
||||
|
||||
To compare against the paper-reported cells, use the same dataset split and
|
||||
scorer. SearchQA's split is checked in at `data/searchqa_id_split/` (400
|
||||
train / 200 selection / 1400 test). For the other benchmarks, point
|
||||
`--split_dir` at your own materialized split; the loader is deterministic
|
||||
from `split_seed` (default `42`) + `split_ratio` (default `2:1:7`) when
|
||||
`split_mode: ratio` is used, so a given `data_path` + seed reproduces
|
||||
across machines. Explicit per-benchmark split manifests are being prepared
|
||||
for upload — see issues #14 and #21.
|
||||
|
||||
## Why force-accept vs. gated slow-update matters
|
||||
|
||||
These `ckpt/` skills were produced with the gated slow-update semantics
|
||||
described in paper Section 3.6:
|
||||
|
||||
```yaml
|
||||
optimizer:
|
||||
slow_update_gate_with_selection: true
|
||||
```
|
||||
|
||||
Current `main` defaults to `false` (force-accept mode), a newer
|
||||
post-submission behavior where the slow-update guidance is written into
|
||||
`current_skill` and `best_skill` unconditionally at the epoch boundary. If
|
||||
you re-train with the current default, you may produce a *different*
|
||||
`best_skill.md` than the one checked in here. Both modes are supported;
|
||||
see the top-level README's "Configuration -> Slow-update acceptance mode"
|
||||
section.
|
||||
@@ -44,18 +44,6 @@ model:
|
||||
target_azure_openai_ad_scope: "https://cognitiveservices.azure.com/.default"
|
||||
target_azure_openai_managed_identity_client_id: ""
|
||||
|
||||
# MiniMax backend settings (minimax_chat target)
|
||||
minimax_base_url: "" # https://api.minimax.io/v1 if blank
|
||||
minimax_api_key: ""
|
||||
minimax_model: "MiniMax-M2.7"
|
||||
minimax_temperature: "0.7"
|
||||
minimax_max_tokens: "8000"
|
||||
minimax_enable_thinking: "false"
|
||||
optimizer_minimax_base_url: "" # per-role override
|
||||
target_minimax_base_url: "" # per-role override
|
||||
optimizer_minimax_api_key: ""
|
||||
target_minimax_api_key: ""
|
||||
|
||||
train:
|
||||
num_epochs: 4
|
||||
train_size: 0 # 0 = derive from dataset split when available
|
||||
|
||||
@@ -19,7 +19,7 @@ env:
|
||||
name: alfworld
|
||||
skill_init: skillopt/envs/alfworld/skills/initial.md
|
||||
split_mode: split_dir
|
||||
split_dir: data/alfworld_path_split
|
||||
split_dir: data/ablation_splits/alfworld/2-1-7_seed42
|
||||
data_path: ""
|
||||
split_output_dir: ""
|
||||
max_steps: 50
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Feature: soft / mixed validation-gate metric (community-contributed, PR #25)
|
||||
# Example: soft / mixed validation-gate metric (community-contributed, PR #25)
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
#
|
||||
# This is NOT a default SkillOpt setting and was NOT used to produce the
|
||||
@@ -28,7 +28,7 @@
|
||||
# and matches the design described in the paper.
|
||||
#
|
||||
# To use: inherit your env config from this file, e.g.
|
||||
# _base_: ../features/soft_gate.yaml
|
||||
# _base_: ../examples/soft_gate.yaml
|
||||
# or copy the `evaluation:` block below into your config.
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
@@ -9,7 +9,7 @@ env:
|
||||
name: livemathematicianbench
|
||||
skill_init: skillopt/envs/livemathematicianbench/skills/initial.md
|
||||
split_mode: split_dir
|
||||
split_dir: data/livemathematicianbench_split
|
||||
split_dir: data/ablation_splits/livemathematicianbench/2-1-7_seed42
|
||||
data_path: ""
|
||||
split_output_dir: ""
|
||||
max_turns: 1
|
||||
|
||||
-223
@@ -1,223 +0,0 @@
|
||||
# Data Manifests
|
||||
|
||||
This directory releases lightweight split manifests for the SkillOpt paper
|
||||
splits. These manifests are not full runnable benchmark payloads. To evaluate a
|
||||
benchmark, first materialize the full examples from the raw data source when
|
||||
needed, then point `--split_dir` at the split directory listed below.
|
||||
|
||||
In this README, "coverage" describes which part of the upstream benchmark the
|
||||
manifest references. It does not mean the released manifest directory contains
|
||||
the full runnable examples.
|
||||
|
||||
## Layout
|
||||
|
||||
Every released manifest directory uses the same file layout:
|
||||
|
||||
```text
|
||||
data/<benchmark>_<manifest_type>/
|
||||
|-- split_manifest.json
|
||||
|-- train/items.json
|
||||
|-- val/items.json
|
||||
`-- test/items.json
|
||||
```
|
||||
|
||||
`split_manifest.json` records source metadata, split counts, and item fields.
|
||||
Each `items.json` contains only stable IDs or source-path hints.
|
||||
|
||||
## Released Splits
|
||||
|
||||
| Manifest directory | Benchmark | Counts | Coverage | Raw data source | `split_dir` |
|
||||
|---|---|---:|---|---|---|
|
||||
| `searchqa_id_split/` | SearchQA | 400 / 200 / 1400 | Official HF dataset IDs | [lucadiliello/searchqa](https://huggingface.co/datasets/lucadiliello/searchqa) | `data/searchqa_split` |
|
||||
| `livemathematicianbench_id_split/` | LiveMathematicianBench | 35 / 18 / 124 | Four official monthly files | [LiveMathematicianBench/LiveMathematicianBench](https://huggingface.co/datasets/LiveMathematicianBench/LiveMathematicianBench) | `data/livemathematicianbench_split` |
|
||||
| `docvqa_id_split/` | DocVQA | 107 / 53 / 374 | 10% subset of validation | [lmms-lab/DocVQA](https://huggingface.co/datasets/lmms-lab/DocVQA) | `data/docvqa/splits` |
|
||||
| `officeqa_id_split/` | OfficeQA | 50 / 24 / 172 | OfficeQA Full | [databricks/officeqa](https://huggingface.co/datasets/databricks/officeqa) | `data/officeqa_split` |
|
||||
| `spreadsheetbench_id_split/` | SpreadsheetBench | 80 / 40 / 280 | SpreadsheetBench Verified 400 | [KAKA22/SpreadsheetBench](https://huggingface.co/datasets/KAKA22/SpreadsheetBench) | `data/spreadsheetbench_split` |
|
||||
| `alfworld_path_split/` | ALFWorld | 39 / 18 / 134 | ALFWorld `json_2.1.1` paths | [alfworld/alfworld](https://github.com/alfworld/alfworld) | `data/alfworld_path_split` |
|
||||
|
||||
Counts are ordered as train / val / test.
|
||||
|
||||
## Direct Use
|
||||
|
||||
Only `alfworld_path_split/` can be used directly as `--split_dir` from this
|
||||
release, because the ALFWorld loader reads `gamefile` and `task_type` from the
|
||||
split items.
|
||||
|
||||
This does not mean the ALFWorld raw data is included. You still need to
|
||||
download ALFWorld separately with `alfworld-download` and set `$ALFWORLD_DATA`
|
||||
to the data root containing `json_2.1.1`.
|
||||
|
||||
The other manifest directories are lookup manifests. They intentionally omit
|
||||
full example fields such as questions, answers, contexts, images, or task
|
||||
instructions. Materialize those benchmarks into the `split_dir` paths listed
|
||||
above before running SkillOpt.
|
||||
|
||||
## Lookup Keys
|
||||
|
||||
The manifests are sufficient to locate the corresponding raw examples after
|
||||
the raw data has been downloaded or otherwise made available:
|
||||
|
||||
| Benchmark | Manifest lookup key |
|
||||
|---|---|
|
||||
| SearchQA | Match `items.json[].id` to the `key` field in `lucadiliello/searchqa`. |
|
||||
| LiveMathematicianBench | Open `source_file`, then match `no`; the manifest `id` is `<month>:<no>`. |
|
||||
| DocVQA | Match `questionId` within the official DocVQA `validation` split; `image_path` records the expected local image path. |
|
||||
| OfficeQA | Match `uid` in `officeqa_full.csv`; `source_files` and `source_docs` identify the supporting document. |
|
||||
| SpreadsheetBench | Match `id`; `spreadsheet_path` identifies the referenced spreadsheet directory. |
|
||||
| ALFWorld | Resolve `gamefile` relative to `$ALFWORLD_DATA`. |
|
||||
|
||||
## Manifest Item Examples
|
||||
|
||||
SearchQA:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "221c83e6630f4e7983da48fa28da1882"
|
||||
}
|
||||
```
|
||||
|
||||
LiveMathematicianBench:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "202602:22",
|
||||
"month": "202602",
|
||||
"no": 22,
|
||||
"paper_link": "http://arxiv.org/abs/2602.10700v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
}
|
||||
```
|
||||
|
||||
DocVQA:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "50877",
|
||||
"questionId": "50877",
|
||||
"docId": "14724",
|
||||
"image_path": "data/docvqa_images/q50877_d14724.png",
|
||||
"source_split": "validation"
|
||||
}
|
||||
```
|
||||
|
||||
OfficeQA:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "UID0002",
|
||||
"uid": "UID0002",
|
||||
"category": "easy",
|
||||
"source_files": "treasury_bulletin_1944_01.txt"
|
||||
}
|
||||
```
|
||||
|
||||
SpreadsheetBench:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "32438",
|
||||
"spreadsheet_path": "spreadsheet/32438",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
}
|
||||
```
|
||||
|
||||
ALFWorld:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "train:0000",
|
||||
"gamefile": "json_2.1.1/train/.../game.tw-pddl",
|
||||
"task_type": "look_at_obj_in_light"
|
||||
}
|
||||
```
|
||||
|
||||
## Benchmark Notes
|
||||
|
||||
### SearchQA
|
||||
|
||||
`searchqa_id_split/` is an ID-only manifest. Each released `id` exactly matches
|
||||
the `key` field in `lucadiliello/searchqa`.
|
||||
|
||||
Materialized examples must include the fields consumed by the SearchQA
|
||||
environment, including:
|
||||
|
||||
```text
|
||||
question
|
||||
context
|
||||
answers
|
||||
```
|
||||
|
||||
### LiveMathematicianBench
|
||||
|
||||
`livemathematicianbench_id_split/` was generated from these raw files:
|
||||
|
||||
```text
|
||||
data/202511/qa_202511_final.json
|
||||
data/202512/qa_202512_final.json
|
||||
data/202601/qa_202601_final.json
|
||||
data/202602/qa_202602_final.json
|
||||
```
|
||||
|
||||
The manifest stores IDs in the loader format:
|
||||
|
||||
```text
|
||||
<month>:<no>
|
||||
```
|
||||
|
||||
Materialized examples must include:
|
||||
|
||||
```text
|
||||
question
|
||||
choices
|
||||
correct_choice
|
||||
theorem_type
|
||||
theorem
|
||||
sketch
|
||||
paper_link
|
||||
```
|
||||
|
||||
### DocVQA
|
||||
|
||||
`docvqa_id_split/` records `docvqa_validation_10pct`: a 10% subset sampled from
|
||||
the official DocVQA `validation` split.
|
||||
|
||||
```text
|
||||
source_split: validation
|
||||
docvqa_validation_10pct: train=107, val=53, test=374
|
||||
```
|
||||
|
||||
Each manifest item contains question/document IDs plus image location metadata.
|
||||
Materialized examples must provide `question`, `answer` or `ground_truth`, and
|
||||
an `image_path` that resolves locally.
|
||||
|
||||
### OfficeQA
|
||||
|
||||
`officeqa_id_split/` records the split over OfficeQA Full
|
||||
(`officeqa_full.csv`). The official OfficeQA CSVs are gated on Hugging Face, so
|
||||
materialization requires authorized access.
|
||||
|
||||
Each manifest item contains `uid`, `category`, `source_files`, and
|
||||
`source_docs` hints. Materialized examples must include `question` and
|
||||
`ground_truth` or `answer`.
|
||||
|
||||
### SpreadsheetBench
|
||||
|
||||
`spreadsheetbench_id_split/` records the split over SpreadsheetBench Verified
|
||||
400, from `spreadsheetbench_verified_400.tar.gz`.
|
||||
|
||||
Each manifest item contains task identity metadata such as `id`,
|
||||
`spreadsheet_path`, and `instruction_type`. Materialization must also place the
|
||||
referenced spreadsheet directories at:
|
||||
|
||||
```text
|
||||
data/spreadsheetbench_verified_400
|
||||
```
|
||||
|
||||
### ALFWorld
|
||||
|
||||
`alfworld_path_split/` records `gamefile` paths relative to `$ALFWORLD_DATA`.
|
||||
The source payload is `json_2.1.1`, which must be downloaded separately with
|
||||
`alfworld-download`.
|
||||
|
||||
This manifest can be used directly as `--split_dir` after `$ALFWORLD_DATA`
|
||||
points to the local ALFWorld data root containing `json_2.1.1`.
|
||||
@@ -1,29 +0,0 @@
|
||||
{
|
||||
"benchmark": "ALFWorld",
|
||||
"manifest_type": "path_split",
|
||||
"source_repo": "alfworld/alfworld",
|
||||
"source_repo_type": "repository",
|
||||
"source_url": "https://github.com/alfworld/alfworld",
|
||||
"source_file": "json_2.1.1",
|
||||
"source_method": "generated by alfworld-download",
|
||||
"source_split_files": [
|
||||
"split_train.json",
|
||||
"split_val.json",
|
||||
"split_test.json"
|
||||
],
|
||||
"counts": {
|
||||
"train": 39,
|
||||
"val": 18,
|
||||
"test": 134
|
||||
},
|
||||
"item_fields": [
|
||||
"id",
|
||||
"gamefile",
|
||||
"task_type"
|
||||
],
|
||||
"path_root": "$ALFWORLD_DATA",
|
||||
"notes": [
|
||||
"This is a path manifest, not the ALFWorld game payload.",
|
||||
"The gamefile field is relative to ALFWORLD_DATA and must be expanded before direct use as split_dir data."
|
||||
]
|
||||
}
|
||||
@@ -1,672 +0,0 @@
|
||||
[
|
||||
{
|
||||
"id": "test:0000",
|
||||
"gamefile": "json_2.1.1/valid_unseen/look_at_obj_in_light-AlarmClock-None-DeskLamp-308/trial_T20190908_222917_366542/game.tw-pddl",
|
||||
"task_type": "look_at_obj_in_light"
|
||||
},
|
||||
{
|
||||
"id": "test:0001",
|
||||
"gamefile": "json_2.1.1/valid_unseen/look_at_obj_in_light-AlarmClock-None-DeskLamp-308/trial_T20190908_222933_607649/game.tw-pddl",
|
||||
"task_type": "look_at_obj_in_light"
|
||||
},
|
||||
{
|
||||
"id": "test:0002",
|
||||
"gamefile": "json_2.1.1/valid_unseen/look_at_obj_in_light-AlarmClock-None-DeskLamp-308/trial_T20190908_222951_616606/game.tw-pddl",
|
||||
"task_type": "look_at_obj_in_light"
|
||||
},
|
||||
{
|
||||
"id": "test:0003",
|
||||
"gamefile": "json_2.1.1/valid_unseen/look_at_obj_in_light-Book-None-DeskLamp-308/trial_T20190908_020029_636862/game.tw-pddl",
|
||||
"task_type": "look_at_obj_in_light"
|
||||
},
|
||||
{
|
||||
"id": "test:0004",
|
||||
"gamefile": "json_2.1.1/valid_unseen/look_at_obj_in_light-Book-None-DeskLamp-308/trial_T20190908_020048_814402/game.tw-pddl",
|
||||
"task_type": "look_at_obj_in_light"
|
||||
},
|
||||
{
|
||||
"id": "test:0005",
|
||||
"gamefile": "json_2.1.1/valid_unseen/look_at_obj_in_light-Book-None-DeskLamp-308/trial_T20190908_144951_587345/game.tw-pddl",
|
||||
"task_type": "look_at_obj_in_light"
|
||||
},
|
||||
{
|
||||
"id": "test:0006",
|
||||
"gamefile": "json_2.1.1/valid_unseen/look_at_obj_in_light-Bowl-None-DeskLamp-308/trial_T20190907_133919_856963/game.tw-pddl",
|
||||
"task_type": "look_at_obj_in_light"
|
||||
},
|
||||
{
|
||||
"id": "test:0007",
|
||||
"gamefile": "json_2.1.1/valid_unseen/look_at_obj_in_light-Bowl-None-DeskLamp-308/trial_T20190907_133935_066606/game.tw-pddl",
|
||||
"task_type": "look_at_obj_in_light"
|
||||
},
|
||||
{
|
||||
"id": "test:0008",
|
||||
"gamefile": "json_2.1.1/valid_unseen/look_at_obj_in_light-Bowl-None-DeskLamp-308/trial_T20190907_133953_562557/game.tw-pddl",
|
||||
"task_type": "look_at_obj_in_light"
|
||||
},
|
||||
{
|
||||
"id": "test:0009",
|
||||
"gamefile": "json_2.1.1/valid_unseen/look_at_obj_in_light-CD-None-DeskLamp-308/trial_T20190908_141942_810052/game.tw-pddl",
|
||||
"task_type": "look_at_obj_in_light"
|
||||
},
|
||||
{
|
||||
"id": "test:0010",
|
||||
"gamefile": "json_2.1.1/valid_unseen/look_at_obj_in_light-CD-None-DeskLamp-308/trial_T20190908_141958_463362/game.tw-pddl",
|
||||
"task_type": "look_at_obj_in_light"
|
||||
},
|
||||
{
|
||||
"id": "test:0011",
|
||||
"gamefile": "json_2.1.1/valid_unseen/look_at_obj_in_light-CD-None-DeskLamp-308/trial_T20190908_142046_281296/game.tw-pddl",
|
||||
"task_type": "look_at_obj_in_light"
|
||||
},
|
||||
{
|
||||
"id": "test:0012",
|
||||
"gamefile": "json_2.1.1/valid_unseen/look_at_obj_in_light-Mug-None-DeskLamp-308/trial_T20190908_161733_213242/game.tw-pddl",
|
||||
"task_type": "look_at_obj_in_light"
|
||||
},
|
||||
{
|
||||
"id": "test:0013",
|
||||
"gamefile": "json_2.1.1/valid_unseen/look_at_obj_in_light-Mug-None-DeskLamp-308/trial_T20190908_201421_021646/game.tw-pddl",
|
||||
"task_type": "look_at_obj_in_light"
|
||||
},
|
||||
{
|
||||
"id": "test:0014",
|
||||
"gamefile": "json_2.1.1/valid_unseen/look_at_obj_in_light-Mug-None-DeskLamp-308/trial_T20190908_201444_037645/game.tw-pddl",
|
||||
"task_type": "look_at_obj_in_light"
|
||||
},
|
||||
{
|
||||
"id": "test:0015",
|
||||
"gamefile": "json_2.1.1/valid_unseen/look_at_obj_in_light-Pencil-None-DeskLamp-308/trial_T20190908_220545_153480/game.tw-pddl",
|
||||
"task_type": "look_at_obj_in_light"
|
||||
},
|
||||
{
|
||||
"id": "test:0016",
|
||||
"gamefile": "json_2.1.1/valid_unseen/look_at_obj_in_light-Pencil-None-DeskLamp-308/trial_T20190908_220604_010430/game.tw-pddl",
|
||||
"task_type": "look_at_obj_in_light"
|
||||
},
|
||||
{
|
||||
"id": "test:0017",
|
||||
"gamefile": "json_2.1.1/valid_unseen/look_at_obj_in_light-Pencil-None-DeskLamp-308/trial_T20190908_220656_510400/game.tw-pddl",
|
||||
"task_type": "look_at_obj_in_light"
|
||||
},
|
||||
{
|
||||
"id": "test:0018",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_and_place_simple-Mug-None-Desk-308/trial_T20190908_125200_737896/game.tw-pddl",
|
||||
"task_type": "pick_and_place_simple"
|
||||
},
|
||||
{
|
||||
"id": "test:0019",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_and_place_simple-Mug-None-Desk-308/trial_T20190909_203041_433487/game.tw-pddl",
|
||||
"task_type": "pick_and_place_simple"
|
||||
},
|
||||
{
|
||||
"id": "test:0020",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_and_place_simple-Mug-None-Desk-308/trial_T20190909_210238_431966/game.tw-pddl",
|
||||
"task_type": "pick_and_place_simple"
|
||||
},
|
||||
{
|
||||
"id": "test:0021",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_and_place_simple-Pencil-None-Shelf-308/trial_T20190908_121952_610012/game.tw-pddl",
|
||||
"task_type": "pick_and_place_simple"
|
||||
},
|
||||
{
|
||||
"id": "test:0022",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_and_place_simple-Pencil-None-Shelf-308/trial_T20190908_122024_052056/game.tw-pddl",
|
||||
"task_type": "pick_and_place_simple"
|
||||
},
|
||||
{
|
||||
"id": "test:0023",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_and_place_simple-Pencil-None-Shelf-308/trial_T20190908_122154_042763/game.tw-pddl",
|
||||
"task_type": "pick_and_place_simple"
|
||||
},
|
||||
{
|
||||
"id": "test:0024",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_and_place_simple-PepperShaker-None-Drawer-10/trial_T20190906_184021_215264/game.tw-pddl",
|
||||
"task_type": "pick_and_place_simple"
|
||||
},
|
||||
{
|
||||
"id": "test:0025",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_and_place_simple-PepperShaker-None-Drawer-10/trial_T20190918_154326_823501/game.tw-pddl",
|
||||
"task_type": "pick_and_place_simple"
|
||||
},
|
||||
{
|
||||
"id": "test:0026",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_and_place_simple-PepperShaker-None-Drawer-10/trial_T20190918_154424_844749/game.tw-pddl",
|
||||
"task_type": "pick_and_place_simple"
|
||||
},
|
||||
{
|
||||
"id": "test:0027",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_and_place_simple-SaltShaker-None-Cabinet-10/trial_T20190906_191429_743650/game.tw-pddl",
|
||||
"task_type": "pick_and_place_simple"
|
||||
},
|
||||
{
|
||||
"id": "test:0028",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_and_place_simple-SaltShaker-None-Cabinet-10/trial_T20190906_191445_723170/game.tw-pddl",
|
||||
"task_type": "pick_and_place_simple"
|
||||
},
|
||||
{
|
||||
"id": "test:0029",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_and_place_simple-SaltShaker-None-Cabinet-10/trial_T20190906_191501_563086/game.tw-pddl",
|
||||
"task_type": "pick_and_place_simple"
|
||||
},
|
||||
{
|
||||
"id": "test:0030",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_and_place_simple-SaltShaker-None-Drawer-10/trial_T20190909_021613_077537/game.tw-pddl",
|
||||
"task_type": "pick_and_place_simple"
|
||||
},
|
||||
{
|
||||
"id": "test:0031",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_and_place_simple-SaltShaker-None-Drawer-10/trial_T20190909_021650_880235/game.tw-pddl",
|
||||
"task_type": "pick_and_place_simple"
|
||||
},
|
||||
{
|
||||
"id": "test:0032",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_and_place_simple-SaltShaker-None-Drawer-10/trial_T20190909_021728_339782/game.tw-pddl",
|
||||
"task_type": "pick_and_place_simple"
|
||||
},
|
||||
{
|
||||
"id": "test:0033",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_and_place_simple-SoapBottle-None-Toilet-424/trial_T20190907_004321_405868/game.tw-pddl",
|
||||
"task_type": "pick_and_place_simple"
|
||||
},
|
||||
{
|
||||
"id": "test:0034",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_and_place_simple-SoapBottle-None-Toilet-424/trial_T20190907_004351_281384/game.tw-pddl",
|
||||
"task_type": "pick_and_place_simple"
|
||||
},
|
||||
{
|
||||
"id": "test:0035",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_and_place_simple-SoapBottle-None-Toilet-424/trial_T20190907_004404_604165/game.tw-pddl",
|
||||
"task_type": "pick_and_place_simple"
|
||||
},
|
||||
{
|
||||
"id": "test:0036",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_and_place_simple-Vase-None-Safe-219/trial_T20190908_205204_244321/game.tw-pddl",
|
||||
"task_type": "pick_and_place_simple"
|
||||
},
|
||||
{
|
||||
"id": "test:0037",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_and_place_simple-Vase-None-Safe-219/trial_T20190908_205221_748352/game.tw-pddl",
|
||||
"task_type": "pick_and_place_simple"
|
||||
},
|
||||
{
|
||||
"id": "test:0038",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_and_place_simple-Vase-None-Safe-219/trial_T20190908_205246_776817/game.tw-pddl",
|
||||
"task_type": "pick_and_place_simple"
|
||||
},
|
||||
{
|
||||
"id": "test:0039",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_and_place_simple-Watch-None-Safe-219/trial_T20190907_074524_006355/game.tw-pddl",
|
||||
"task_type": "pick_and_place_simple"
|
||||
},
|
||||
{
|
||||
"id": "test:0040",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_and_place_simple-Watch-None-Safe-219/trial_T20190907_074556_124850/game.tw-pddl",
|
||||
"task_type": "pick_and_place_simple"
|
||||
},
|
||||
{
|
||||
"id": "test:0041",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_and_place_simple-Watch-None-Safe-219/trial_T20190907_074643_810052/game.tw-pddl",
|
||||
"task_type": "pick_and_place_simple"
|
||||
},
|
||||
{
|
||||
"id": "test:0042",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-Bowl-None-Cabinet-10/trial_T20190909_061130_844814/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0043",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-Bowl-None-Cabinet-10/trial_T20190909_061158_110530/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0044",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-Bowl-None-Cabinet-10/trial_T20190909_061232_368489/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0045",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-Cloth-None-Cabinet-424/trial_T20190908_022321_380927/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0046",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-Cloth-None-Cabinet-424/trial_T20190908_022436_073995/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0047",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-Cloth-None-CounterTop-424/trial_T20190908_100632_546757/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0048",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-Cloth-None-CounterTop-424/trial_T20190908_114340_674467/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0049",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-Egg-None-Microwave-10/trial_T20190909_120554_888709/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0050",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-Egg-None-Microwave-10/trial_T20190909_120632_691361/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0051",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-Egg-None-Microwave-10/trial_T20190909_120712_273910/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0052",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-Knife-None-CounterTop-10/trial_T20190909_110347_624008/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0053",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-Knife-None-CounterTop-10/trial_T20190909_110445_675754/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0054",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-Knife-None-CounterTop-10/trial_T20190909_110531_148235/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0055",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-Mug-None-CoffeeMachine-10/trial_T20190907_221208_560499/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0056",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-Mug-None-CoffeeMachine-10/trial_T20190907_221300_362511/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0057",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-Mug-None-CoffeeMachine-10/trial_T20190907_221355_558505/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0058",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-Pan-None-CounterTop-10/trial_T20190908_032434_013084/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0059",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-Pan-None-CounterTop-10/trial_T20190908_032518_891433/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0060",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-Pan-None-CounterTop-10/trial_T20190908_032543_712058/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0061",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-Plate-None-CounterTop-10/trial_T20190908_213356_017769/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0062",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-Plate-None-CounterTop-10/trial_T20190908_213420_728917/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0063",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-Plate-None-CounterTop-10/trial_T20190908_213533_897289/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0064",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-SoapBar-None-Cabinet-424/trial_T20190908_214926_337906/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0065",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-SoapBar-None-Cabinet-424/trial_T20190908_214946_567644/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0066",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-SoapBar-None-Cabinet-424/trial_T20190908_215019_162873/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0067",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-SoapBar-None-CounterTop-424/trial_T20190907_074045_109439/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0068",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-SoapBar-None-CounterTop-424/trial_T20190907_074106_050405/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0069",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-SoapBar-None-CounterTop-424/trial_T20190907_074124_966890/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0070",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-Spatula-None-Drawer-10/trial_T20190907_080730_211959/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0071",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-Spatula-None-Drawer-10/trial_T20190907_080800_275989/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0072",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-Spatula-None-Drawer-10/trial_T20190907_080825_222432/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0073",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_cool_then_place_in_recep-Bread-None-CounterTop-10/trial_T20190908_091747_866951/game.tw-pddl",
|
||||
"task_type": "pick_cool_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0074",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_cool_then_place_in_recep-Bread-None-CounterTop-10/trial_T20190908_091811_414150/game.tw-pddl",
|
||||
"task_type": "pick_cool_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0075",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_cool_then_place_in_recep-Bread-None-CounterTop-10/trial_T20190908_091835_825830/game.tw-pddl",
|
||||
"task_type": "pick_cool_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0076",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_cool_then_place_in_recep-Lettuce-None-CounterTop-10/trial_T20190909_123133_763972/game.tw-pddl",
|
||||
"task_type": "pick_cool_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0077",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_cool_then_place_in_recep-Lettuce-None-CounterTop-10/trial_T20190909_174807_646433/game.tw-pddl",
|
||||
"task_type": "pick_cool_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0078",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_cool_then_place_in_recep-Lettuce-None-CounterTop-10/trial_T20190909_174840_771703/game.tw-pddl",
|
||||
"task_type": "pick_cool_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0079",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_cool_then_place_in_recep-Mug-None-Cabinet-10/trial_T20190909_121559_082363/game.tw-pddl",
|
||||
"task_type": "pick_cool_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0080",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_cool_then_place_in_recep-Mug-None-Cabinet-10/trial_T20190909_121635_622676/game.tw-pddl",
|
||||
"task_type": "pick_cool_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0081",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_cool_then_place_in_recep-Mug-None-Cabinet-10/trial_T20190909_121710_650938/game.tw-pddl",
|
||||
"task_type": "pick_cool_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0082",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_cool_then_place_in_recep-Mug-None-CoffeeMachine-10/trial_T20190907_183715_299073/game.tw-pddl",
|
||||
"task_type": "pick_cool_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0083",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_cool_then_place_in_recep-Mug-None-CoffeeMachine-10/trial_T20190907_183807_477267/game.tw-pddl",
|
||||
"task_type": "pick_cool_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0084",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_cool_then_place_in_recep-Mug-None-CoffeeMachine-10/trial_T20190907_183853_958104/game.tw-pddl",
|
||||
"task_type": "pick_cool_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0085",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_cool_then_place_in_recep-Pan-None-CounterTop-10/trial_T20190908_114545_244903/game.tw-pddl",
|
||||
"task_type": "pick_cool_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0086",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_cool_then_place_in_recep-Pan-None-CounterTop-10/trial_T20190908_114622_738670/game.tw-pddl",
|
||||
"task_type": "pick_cool_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0087",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_cool_then_place_in_recep-Pan-None-CounterTop-10/trial_T20190908_114656_768805/game.tw-pddl",
|
||||
"task_type": "pick_cool_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0088",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_cool_then_place_in_recep-Potato-None-Microwave-10/trial_T20190907_033157_424297/game.tw-pddl",
|
||||
"task_type": "pick_cool_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0089",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_cool_then_place_in_recep-Potato-None-Microwave-10/trial_T20190907_033228_194678/game.tw-pddl",
|
||||
"task_type": "pick_cool_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0090",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_cool_then_place_in_recep-Potato-None-Microwave-10/trial_T20190907_033306_962974/game.tw-pddl",
|
||||
"task_type": "pick_cool_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0091",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_cool_then_place_in_recep-Tomato-None-Microwave-10/trial_T20190909_102608_318800/game.tw-pddl",
|
||||
"task_type": "pick_cool_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0092",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_cool_then_place_in_recep-Tomato-None-Microwave-10/trial_T20190909_102644_926781/game.tw-pddl",
|
||||
"task_type": "pick_cool_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0093",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_cool_then_place_in_recep-Tomato-None-Microwave-10/trial_T20190909_102710_795182/game.tw-pddl",
|
||||
"task_type": "pick_cool_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0094",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_heat_then_place_in_recep-Apple-None-Fridge-10/trial_T20190906_182259_116320/game.tw-pddl",
|
||||
"task_type": "pick_heat_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0095",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_heat_then_place_in_recep-Apple-None-Fridge-10/trial_T20190906_182353_418140/game.tw-pddl",
|
||||
"task_type": "pick_heat_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0096",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_heat_then_place_in_recep-Apple-None-Fridge-10/trial_T20190906_182435_622538/game.tw-pddl",
|
||||
"task_type": "pick_heat_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0097",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_heat_then_place_in_recep-Apple-None-GarbageCan-10/trial_T20190908_145050_918567/game.tw-pddl",
|
||||
"task_type": "pick_heat_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0098",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_heat_then_place_in_recep-Apple-None-GarbageCan-10/trial_T20190908_145143_820541/game.tw-pddl",
|
||||
"task_type": "pick_heat_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0099",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_heat_then_place_in_recep-Apple-None-GarbageCan-10/trial_T20190908_145356_918528/game.tw-pddl",
|
||||
"task_type": "pick_heat_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0100",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_heat_then_place_in_recep-Cup-None-Cabinet-10/trial_T20190907_083346_800823/game.tw-pddl",
|
||||
"task_type": "pick_heat_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0101",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_heat_then_place_in_recep-Cup-None-Cabinet-10/trial_T20190907_083429_887065/game.tw-pddl",
|
||||
"task_type": "pick_heat_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0102",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_heat_then_place_in_recep-Cup-None-Cabinet-10/trial_T20190907_083507_594820/game.tw-pddl",
|
||||
"task_type": "pick_heat_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0103",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_heat_then_place_in_recep-Egg-None-GarbageCan-10/trial_T20190908_113432_673307/game.tw-pddl",
|
||||
"task_type": "pick_heat_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0104",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_heat_then_place_in_recep-Egg-None-GarbageCan-10/trial_T20190908_113523_123938/game.tw-pddl",
|
||||
"task_type": "pick_heat_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0105",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_heat_then_place_in_recep-Egg-None-GarbageCan-10/trial_T20190908_113610_425142/game.tw-pddl",
|
||||
"task_type": "pick_heat_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0106",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_heat_then_place_in_recep-Mug-None-Cabinet-10/trial_T20190909_021100_341887/game.tw-pddl",
|
||||
"task_type": "pick_heat_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0107",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_heat_then_place_in_recep-Mug-None-Cabinet-10/trial_T20190909_021200_669381/game.tw-pddl",
|
||||
"task_type": "pick_heat_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0108",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_heat_then_place_in_recep-Mug-None-Cabinet-10/trial_T20190909_021247_306737/game.tw-pddl",
|
||||
"task_type": "pick_heat_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0109",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_heat_then_place_in_recep-Mug-None-CoffeeMachine-10/trial_T20190907_171806_406231/game.tw-pddl",
|
||||
"task_type": "pick_heat_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0110",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_heat_then_place_in_recep-Mug-None-CoffeeMachine-10/trial_T20190907_171850_960211/game.tw-pddl",
|
||||
"task_type": "pick_heat_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0111",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_heat_then_place_in_recep-Mug-None-CoffeeMachine-10/trial_T20190907_171933_349922/game.tw-pddl",
|
||||
"task_type": "pick_heat_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0112",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_heat_then_place_in_recep-Potato-None-GarbageCan-10/trial_T20190907_161745_664033/game.tw-pddl",
|
||||
"task_type": "pick_heat_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0113",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_heat_then_place_in_recep-Potato-None-GarbageCan-10/trial_T20190907_161853_945788/game.tw-pddl",
|
||||
"task_type": "pick_heat_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0114",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_heat_then_place_in_recep-Tomato-None-GarbageCan-10/trial_T20190908_225046_020282/game.tw-pddl",
|
||||
"task_type": "pick_heat_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0115",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_heat_then_place_in_recep-Tomato-None-GarbageCan-10/trial_T20190908_225359_617900/game.tw-pddl",
|
||||
"task_type": "pick_heat_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0116",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_heat_then_place_in_recep-Tomato-None-GarbageCan-10/trial_T20190908_225453_272533/game.tw-pddl",
|
||||
"task_type": "pick_heat_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0117",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_two_obj_and_place-CD-None-Safe-308/trial_T20190907_050942_897916/game.tw-pddl",
|
||||
"task_type": "pick_two_obj_and_place"
|
||||
},
|
||||
{
|
||||
"id": "test:0118",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_two_obj_and_place-CD-None-Safe-308/trial_T20190907_051013_060265/game.tw-pddl",
|
||||
"task_type": "pick_two_obj_and_place"
|
||||
},
|
||||
{
|
||||
"id": "test:0119",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_two_obj_and_place-CD-None-Safe-308/trial_T20190907_051056_585414/game.tw-pddl",
|
||||
"task_type": "pick_two_obj_and_place"
|
||||
},
|
||||
{
|
||||
"id": "test:0120",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_two_obj_and_place-KeyChain-None-Safe-219/trial_T20190909_011803_423115/game.tw-pddl",
|
||||
"task_type": "pick_two_obj_and_place"
|
||||
},
|
||||
{
|
||||
"id": "test:0121",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_two_obj_and_place-KeyChain-None-Safe-219/trial_T20190909_012027_782483/game.tw-pddl",
|
||||
"task_type": "pick_two_obj_and_place"
|
||||
},
|
||||
{
|
||||
"id": "test:0122",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_two_obj_and_place-PepperShaker-None-Drawer-10/trial_T20190908_010306_215435/game.tw-pddl",
|
||||
"task_type": "pick_two_obj_and_place"
|
||||
},
|
||||
{
|
||||
"id": "test:0123",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_two_obj_and_place-PepperShaker-None-Drawer-10/trial_T20190912_221016_460197/game.tw-pddl",
|
||||
"task_type": "pick_two_obj_and_place"
|
||||
},
|
||||
{
|
||||
"id": "test:0124",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_two_obj_and_place-PepperShaker-None-Drawer-10/trial_T20190912_221141_608117/game.tw-pddl",
|
||||
"task_type": "pick_two_obj_and_place"
|
||||
},
|
||||
{
|
||||
"id": "test:0125",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_two_obj_and_place-Pillow-None-Sofa-219/trial_T20190907_163240_345855/game.tw-pddl",
|
||||
"task_type": "pick_two_obj_and_place"
|
||||
},
|
||||
{
|
||||
"id": "test:0126",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_two_obj_and_place-Pillow-None-Sofa-219/trial_T20190907_163327_486300/game.tw-pddl",
|
||||
"task_type": "pick_two_obj_and_place"
|
||||
},
|
||||
{
|
||||
"id": "test:0127",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_two_obj_and_place-Pillow-None-Sofa-219/trial_T20190907_163408_914117/game.tw-pddl",
|
||||
"task_type": "pick_two_obj_and_place"
|
||||
},
|
||||
{
|
||||
"id": "test:0128",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_two_obj_and_place-SoapBar-None-Cabinet-424/trial_T20190909_081720_491733/game.tw-pddl",
|
||||
"task_type": "pick_two_obj_and_place"
|
||||
},
|
||||
{
|
||||
"id": "test:0129",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_two_obj_and_place-SoapBar-None-Cabinet-424/trial_T20190909_081746_857594/game.tw-pddl",
|
||||
"task_type": "pick_two_obj_and_place"
|
||||
},
|
||||
{
|
||||
"id": "test:0130",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_two_obj_and_place-SoapBar-None-GarbageCan-424/trial_T20190909_064053_839817/game.tw-pddl",
|
||||
"task_type": "pick_two_obj_and_place"
|
||||
},
|
||||
{
|
||||
"id": "test:0131",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_two_obj_and_place-SoapBar-None-GarbageCan-424/trial_T20190909_064221_368939/game.tw-pddl",
|
||||
"task_type": "pick_two_obj_and_place"
|
||||
},
|
||||
{
|
||||
"id": "test:0132",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_two_obj_and_place-SoapBar-None-GarbageCan-424/trial_T20190909_064309_357168/game.tw-pddl",
|
||||
"task_type": "pick_two_obj_and_place"
|
||||
},
|
||||
{
|
||||
"id": "test:0133",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_two_obj_and_place-ToiletPaper-None-Cabinet-424/trial_T20190906_202926_527010/game.tw-pddl",
|
||||
"task_type": "pick_two_obj_and_place"
|
||||
}
|
||||
]
|
||||
@@ -1,197 +0,0 @@
|
||||
[
|
||||
{
|
||||
"id": "train:0000",
|
||||
"gamefile": "json_2.1.1/train/look_at_obj_in_light-AlarmClock-None-DeskLamp-305/trial_T20190908_082736_108723/game.tw-pddl",
|
||||
"task_type": "look_at_obj_in_light"
|
||||
},
|
||||
{
|
||||
"id": "train:0001",
|
||||
"gamefile": "json_2.1.1/train/look_at_obj_in_light-CD-None-DeskLamp-304/trial_T20190907_185649_782438/game.tw-pddl",
|
||||
"task_type": "look_at_obj_in_light"
|
||||
},
|
||||
{
|
||||
"id": "train:0002",
|
||||
"gamefile": "json_2.1.1/train/look_at_obj_in_light-CD-None-DeskLamp-320/trial_T20190907_224439_174735/game.tw-pddl",
|
||||
"task_type": "look_at_obj_in_light"
|
||||
},
|
||||
{
|
||||
"id": "train:0003",
|
||||
"gamefile": "json_2.1.1/train/look_at_obj_in_light-Pillow-None-DeskLamp-316/trial_T20190908_232421_645610/game.tw-pddl",
|
||||
"task_type": "look_at_obj_in_light"
|
||||
},
|
||||
{
|
||||
"id": "train:0004",
|
||||
"gamefile": "json_2.1.1/train/look_at_obj_in_light-Statue-None-DeskLamp-319/trial_T20190907_035546_167548/game.tw-pddl",
|
||||
"task_type": "look_at_obj_in_light"
|
||||
},
|
||||
{
|
||||
"id": "train:0005",
|
||||
"gamefile": "json_2.1.1/train/pick_and_place_simple-CellPhone-None-Shelf-313/trial_T20190908_123725_452958/game.tw-pddl",
|
||||
"task_type": "pick_and_place_simple"
|
||||
},
|
||||
{
|
||||
"id": "train:0006",
|
||||
"gamefile": "json_2.1.1/train/pick_and_place_simple-Newspaper-None-Sofa-211/trial_T20190906_175004_203092/game.tw-pddl",
|
||||
"task_type": "pick_and_place_simple"
|
||||
},
|
||||
{
|
||||
"id": "train:0007",
|
||||
"gamefile": "json_2.1.1/train/pick_and_place_simple-Pencil-None-Desk-302/trial_T20190908_032836_462632/game.tw-pddl",
|
||||
"task_type": "pick_and_place_simple"
|
||||
},
|
||||
{
|
||||
"id": "train:0008",
|
||||
"gamefile": "json_2.1.1/train/pick_and_place_simple-SoapBar-None-GarbageCan-416/trial_T20190908_020839_714699/game.tw-pddl",
|
||||
"task_type": "pick_and_place_simple"
|
||||
},
|
||||
{
|
||||
"id": "train:0009",
|
||||
"gamefile": "json_2.1.1/train/pick_and_place_simple-Statue-None-CoffeeTable-222/trial_T20190907_131249_788749/game.tw-pddl",
|
||||
"task_type": "pick_and_place_simple"
|
||||
},
|
||||
{
|
||||
"id": "train:0010",
|
||||
"gamefile": "json_2.1.1/train/pick_and_place_simple-ToiletPaper-None-ToiletPaperHanger-406/trial_T20190908_122807_136741/game.tw-pddl",
|
||||
"task_type": "pick_and_place_simple"
|
||||
},
|
||||
{
|
||||
"id": "train:0011",
|
||||
"gamefile": "json_2.1.1/train/pick_and_place_simple-ToiletPaper-None-ToiletPaperHanger-415/trial_T20190908_050443_333939/game.tw-pddl",
|
||||
"task_type": "pick_and_place_simple"
|
||||
},
|
||||
{
|
||||
"id": "train:0012",
|
||||
"gamefile": "json_2.1.1/train/pick_clean_then_place_in_recep-Apple-None-DiningTable-4/trial_T20190908_104413_450768/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "train:0013",
|
||||
"gamefile": "json_2.1.1/train/pick_clean_then_place_in_recep-DishSponge-None-Shelf-20/trial_T20190907_222429_992578/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "train:0014",
|
||||
"gamefile": "json_2.1.1/train/pick_clean_then_place_in_recep-DishSponge-None-Shelf-401/trial_T20190908_072225_397518/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "train:0015",
|
||||
"gamefile": "json_2.1.1/train/pick_clean_then_place_in_recep-Kettle-None-Cabinet-2/trial_T20190909_043103_418752/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "train:0016",
|
||||
"gamefile": "json_2.1.1/train/pick_clean_then_place_in_recep-Knife-None-Drawer-22/trial_T20190907_224827_746945/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "train:0017",
|
||||
"gamefile": "json_2.1.1/train/pick_clean_then_place_in_recep-Lettuce-None-DiningTable-20/trial_T20190906_191148_519826/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "train:0018",
|
||||
"gamefile": "json_2.1.1/train/pick_clean_then_place_in_recep-Lettuce-None-Fridge-13/trial_T20190908_203022_601787/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "train:0019",
|
||||
"gamefile": "json_2.1.1/train/pick_clean_then_place_in_recep-Plate-None-Fridge-5/trial_T20190909_112954_869911/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "train:0020",
|
||||
"gamefile": "json_2.1.1/train/pick_clean_then_place_in_recep-Spoon-None-DiningTable-18/trial_T20190909_102159_277894/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "train:0021",
|
||||
"gamefile": "json_2.1.1/train/pick_cool_then_place_in_recep-Bread-None-CounterTop-1/trial_T20190908_212439_711334/game.tw-pddl",
|
||||
"task_type": "pick_cool_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "train:0022",
|
||||
"gamefile": "json_2.1.1/train/pick_cool_then_place_in_recep-Bread-None-CounterTop-15/trial_T20190909_085448_256298/game.tw-pddl",
|
||||
"task_type": "pick_cool_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "train:0023",
|
||||
"gamefile": "json_2.1.1/train/pick_cool_then_place_in_recep-Bread-None-CounterTop-16/trial_T20190908_143948_082471/game.tw-pddl",
|
||||
"task_type": "pick_cool_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "train:0024",
|
||||
"gamefile": "json_2.1.1/train/pick_cool_then_place_in_recep-Pan-None-StoveBurner-27/trial_T20190906_212619_469871/game.tw-pddl",
|
||||
"task_type": "pick_cool_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "train:0025",
|
||||
"gamefile": "json_2.1.1/train/pick_cool_then_place_in_recep-Plate-None-DiningTable-17/trial_T20190909_122939_032098/game.tw-pddl",
|
||||
"task_type": "pick_cool_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "train:0026",
|
||||
"gamefile": "json_2.1.1/train/pick_cool_then_place_in_recep-Pot-None-CounterTop-1/trial_T20190909_124252_504581/game.tw-pddl",
|
||||
"task_type": "pick_cool_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "train:0027",
|
||||
"gamefile": "json_2.1.1/train/pick_heat_then_place_in_recep-Apple-None-Fridge-20/trial_T20190908_013911_274341/game.tw-pddl",
|
||||
"task_type": "pick_heat_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "train:0028",
|
||||
"gamefile": "json_2.1.1/train/pick_heat_then_place_in_recep-Egg-None-CounterTop-12/trial_T20190908_215527_416490/game.tw-pddl",
|
||||
"task_type": "pick_heat_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "train:0029",
|
||||
"gamefile": "json_2.1.1/train/pick_heat_then_place_in_recep-Mug-None-CoffeeMachine-1/trial_T20190907_222924_821086/game.tw-pddl",
|
||||
"task_type": "pick_heat_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "train:0030",
|
||||
"gamefile": "json_2.1.1/train/pick_heat_then_place_in_recep-Mug-None-CoffeeMachine-28/trial_T20190908_062730_537428/game.tw-pddl",
|
||||
"task_type": "pick_heat_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "train:0031",
|
||||
"gamefile": "json_2.1.1/train/pick_heat_then_place_in_recep-Plate-None-Cabinet-13/trial_T20190907_062749_759882/game.tw-pddl",
|
||||
"task_type": "pick_heat_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "train:0032",
|
||||
"gamefile": "json_2.1.1/train/pick_heat_then_place_in_recep-Potato-None-Fridge-2/trial_T20190909_030845_198194/game.tw-pddl",
|
||||
"task_type": "pick_heat_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "train:0033",
|
||||
"gamefile": "json_2.1.1/train/pick_heat_then_place_in_recep-Tomato-None-CounterTop-26/trial_T20190907_005525_499114/game.tw-pddl",
|
||||
"task_type": "pick_heat_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "train:0034",
|
||||
"gamefile": "json_2.1.1/train/pick_two_obj_and_place-CD-None-Drawer-319/trial_T20190907_145515_348252/game.tw-pddl",
|
||||
"task_type": "pick_two_obj_and_place"
|
||||
},
|
||||
{
|
||||
"id": "train:0035",
|
||||
"gamefile": "json_2.1.1/train/pick_two_obj_and_place-Candle-None-Drawer-427/trial_T20190909_043917_251333/game.tw-pddl",
|
||||
"task_type": "pick_two_obj_and_place"
|
||||
},
|
||||
{
|
||||
"id": "train:0036",
|
||||
"gamefile": "json_2.1.1/train/pick_two_obj_and_place-KeyChain-None-ArmChair-222/trial_T20190909_100312_677332/game.tw-pddl",
|
||||
"task_type": "pick_two_obj_and_place"
|
||||
},
|
||||
{
|
||||
"id": "train:0037",
|
||||
"gamefile": "json_2.1.1/train/pick_two_obj_and_place-Newspaper-None-Sofa-212/trial_T20190908_112632_208041/game.tw-pddl",
|
||||
"task_type": "pick_two_obj_and_place"
|
||||
},
|
||||
{
|
||||
"id": "train:0038",
|
||||
"gamefile": "json_2.1.1/train/pick_two_obj_and_place-SaltShaker-None-SideTable-21/trial_T20190909_041626_844806/game.tw-pddl",
|
||||
"task_type": "pick_two_obj_and_place"
|
||||
}
|
||||
]
|
||||
@@ -1,92 +0,0 @@
|
||||
[
|
||||
{
|
||||
"id": "val:0000",
|
||||
"gamefile": "json_2.1.1/valid_seen/look_at_obj_in_light-AlarmClock-None-DeskLamp-323/trial_T20190909_044715_250790/game.tw-pddl",
|
||||
"task_type": "look_at_obj_in_light"
|
||||
},
|
||||
{
|
||||
"id": "val:0001",
|
||||
"gamefile": "json_2.1.1/valid_seen/look_at_obj_in_light-Bowl-None-DeskLamp-301/trial_T20190909_150719_492274/game.tw-pddl",
|
||||
"task_type": "look_at_obj_in_light"
|
||||
},
|
||||
{
|
||||
"id": "val:0002",
|
||||
"gamefile": "json_2.1.1/valid_seen/look_at_obj_in_light-Pillow-None-DeskLamp-323/trial_T20190908_053153_077977/game.tw-pddl",
|
||||
"task_type": "look_at_obj_in_light"
|
||||
},
|
||||
{
|
||||
"id": "val:0003",
|
||||
"gamefile": "json_2.1.1/valid_seen/pick_and_place_simple-Mug-None-SideTable-329/trial_T20190909_032318_169393/game.tw-pddl",
|
||||
"task_type": "pick_and_place_simple"
|
||||
},
|
||||
{
|
||||
"id": "val:0004",
|
||||
"gamefile": "json_2.1.1/valid_seen/pick_and_place_simple-Mug-None-SideTable-329/trial_T20190909_032340_274147/game.tw-pddl",
|
||||
"task_type": "pick_and_place_simple"
|
||||
},
|
||||
{
|
||||
"id": "val:0005",
|
||||
"gamefile": "json_2.1.1/valid_seen/pick_and_place_simple-Pencil-None-Desk-310/trial_T20190909_113054_894334/game.tw-pddl",
|
||||
"task_type": "pick_and_place_simple"
|
||||
},
|
||||
{
|
||||
"id": "val:0006",
|
||||
"gamefile": "json_2.1.1/valid_seen/pick_clean_then_place_in_recep-ButterKnife-None-Drawer-30/trial_T20190908_052007_212776/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "val:0007",
|
||||
"gamefile": "json_2.1.1/valid_seen/pick_clean_then_place_in_recep-ButterKnife-None-Drawer-8/trial_T20190909_124425_112757/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "val:0008",
|
||||
"gamefile": "json_2.1.1/valid_seen/pick_clean_then_place_in_recep-SoapBar-None-Cabinet-402/trial_T20190908_055221_984342/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "val:0009",
|
||||
"gamefile": "json_2.1.1/valid_seen/pick_clean_then_place_in_recep-SoapBar-None-Toilet-410/trial_T20190906_201106_979461/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "val:0010",
|
||||
"gamefile": "json_2.1.1/valid_seen/pick_cool_then_place_in_recep-Apple-None-Microwave-19/trial_T20190906_210937_878489/game.tw-pddl",
|
||||
"task_type": "pick_cool_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "val:0011",
|
||||
"gamefile": "json_2.1.1/valid_seen/pick_cool_then_place_in_recep-Plate-None-CounterTop-1/trial_T20190906_205324_559361/game.tw-pddl",
|
||||
"task_type": "pick_cool_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "val:0012",
|
||||
"gamefile": "json_2.1.1/valid_seen/pick_cool_then_place_in_recep-Tomato-None-Microwave-18/trial_T20190909_012524_159092/game.tw-pddl",
|
||||
"task_type": "pick_cool_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "val:0013",
|
||||
"gamefile": "json_2.1.1/valid_seen/pick_heat_then_place_in_recep-Apple-None-DiningTable-26/trial_T20190907_060234_011675/game.tw-pddl",
|
||||
"task_type": "pick_heat_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "val:0014",
|
||||
"gamefile": "json_2.1.1/valid_seen/pick_heat_then_place_in_recep-Tomato-None-Fridge-15/trial_T20190909_020200_054379/game.tw-pddl",
|
||||
"task_type": "pick_heat_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "val:0015",
|
||||
"gamefile": "json_2.1.1/valid_seen/pick_heat_then_place_in_recep-Tomato-None-Fridge-23/trial_T20190909_082320_103350/game.tw-pddl",
|
||||
"task_type": "pick_heat_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "val:0016",
|
||||
"gamefile": "json_2.1.1/valid_seen/pick_two_obj_and_place-Book-None-Desk-313/trial_T20190908_125930_920681/game.tw-pddl",
|
||||
"task_type": "pick_two_obj_and_place"
|
||||
},
|
||||
{
|
||||
"id": "val:0017",
|
||||
"gamefile": "json_2.1.1/valid_seen/pick_two_obj_and_place-CreditCard-None-Safe-323/trial_T20190907_001129_214240/game.tw-pddl",
|
||||
"task_type": "pick_two_obj_and_place"
|
||||
}
|
||||
]
|
||||
@@ -1,36 +0,0 @@
|
||||
{
|
||||
"benchmark": "DocVQA",
|
||||
"manifest_type": "id_split",
|
||||
"source_repo": "lmms-lab/DocVQA",
|
||||
"source_repo_type": "dataset",
|
||||
"source_url": "https://huggingface.co/datasets/lmms-lab/DocVQA",
|
||||
"source_revision": "539088ef8a8ada01ac8e2e6d4e372586748a265e",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"source_split_name": "docvqa_validation_10pct",
|
||||
"split_method": "10% subset sampled from the DocVQA validation split",
|
||||
"counts": {
|
||||
"train": 107,
|
||||
"val": 53,
|
||||
"test": 374
|
||||
},
|
||||
"item_fields": [
|
||||
"id",
|
||||
"questionId",
|
||||
"docId",
|
||||
"image_path",
|
||||
"ucsf_document_id",
|
||||
"ucsf_document_page_no",
|
||||
"topic",
|
||||
"source_dataset",
|
||||
"source_config",
|
||||
"source_split",
|
||||
"sample_seed"
|
||||
],
|
||||
"notes": [
|
||||
"This is a split manifest, not the full DocVQA payload.",
|
||||
"Materialize full CSV rows and image files before evaluation.",
|
||||
"This manifest corresponds to docvqa_validation_10pct.",
|
||||
"All released train/val/test items originate from a 10% subset of the official DocVQA validation split."
|
||||
]
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -1,691 +0,0 @@
|
||||
[
|
||||
{
|
||||
"id": "62409",
|
||||
"questionId": "62409",
|
||||
"docId": "8554",
|
||||
"image_path": "data/docvqa_images/q62409_d8554.png",
|
||||
"ucsf_document_id": "pgjw0227",
|
||||
"ucsf_document_page_no": "5",
|
||||
"topic": "table/list",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "50961",
|
||||
"questionId": "50961",
|
||||
"docId": "549",
|
||||
"image_path": "data/docvqa_images/q50961_d549.png",
|
||||
"ucsf_document_id": "qtjf0226",
|
||||
"ucsf_document_page_no": "2",
|
||||
"topic": "free_text",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "46461",
|
||||
"questionId": "46461",
|
||||
"docId": "13361",
|
||||
"image_path": "data/docvqa_images/q46461_d13361.png",
|
||||
"ucsf_document_id": "ysbw0217",
|
||||
"ucsf_document_page_no": "5",
|
||||
"topic": "layout",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "3041",
|
||||
"questionId": "3041",
|
||||
"docId": "1204",
|
||||
"image_path": "data/docvqa_images/q3041_d1204.png",
|
||||
"ucsf_document_id": "xfjv0228",
|
||||
"ucsf_document_page_no": "3",
|
||||
"topic": "form",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "41716",
|
||||
"questionId": "41716",
|
||||
"docId": "11835",
|
||||
"image_path": "data/docvqa_images/q41716_d11835.png",
|
||||
"ucsf_document_id": "qjgn0226",
|
||||
"ucsf_document_page_no": "131",
|
||||
"topic": "form",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "61123",
|
||||
"questionId": "61123",
|
||||
"docId": "7374",
|
||||
"image_path": "data/docvqa_images/q61123_d7374.png",
|
||||
"ucsf_document_id": "mldg0227",
|
||||
"ucsf_document_page_no": "5",
|
||||
"topic": "layout",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "43068",
|
||||
"questionId": "43068",
|
||||
"docId": "12393",
|
||||
"image_path": "data/docvqa_images/q43068_d12393.png",
|
||||
"ucsf_document_id": "rmwn0226",
|
||||
"ucsf_document_page_no": "52",
|
||||
"topic": "figure/diagram",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "51221",
|
||||
"questionId": "51221",
|
||||
"docId": "764",
|
||||
"image_path": "data/docvqa_images/q51221_d764.png",
|
||||
"ucsf_document_id": "kzbn0226",
|
||||
"ucsf_document_page_no": "14",
|
||||
"topic": "layout",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "6397",
|
||||
"questionId": "6397",
|
||||
"docId": "2242",
|
||||
"image_path": "data/docvqa_images/q6397_d2242.png",
|
||||
"ucsf_document_id": "jkcn0000",
|
||||
"ucsf_document_page_no": "2",
|
||||
"topic": "form",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "57428",
|
||||
"questionId": "57428",
|
||||
"docId": "4779",
|
||||
"image_path": "data/docvqa_images/q57428_d4779.png",
|
||||
"ucsf_document_id": "rnbx0223",
|
||||
"ucsf_document_page_no": "208",
|
||||
"topic": "Image/Photo",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "3135",
|
||||
"questionId": "3135",
|
||||
"docId": "1221",
|
||||
"image_path": "data/docvqa_images/q3135_d1221.png",
|
||||
"ucsf_document_id": "ngph0227",
|
||||
"ucsf_document_page_no": "5",
|
||||
"topic": "layout",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "18819",
|
||||
"questionId": "18819",
|
||||
"docId": "5749",
|
||||
"image_path": "data/docvqa_images/q18819_d5749.png",
|
||||
"ucsf_document_id": "jhfd0079",
|
||||
"ucsf_document_page_no": "9",
|
||||
"topic": "form",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "15382",
|
||||
"questionId": "15382",
|
||||
"docId": "4890",
|
||||
"image_path": "data/docvqa_images/q15382_d4890.png",
|
||||
"ucsf_document_id": "kjvw0217",
|
||||
"ucsf_document_page_no": "3",
|
||||
"topic": "table/list",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "5772",
|
||||
"questionId": "5772",
|
||||
"docId": "1940",
|
||||
"image_path": "data/docvqa_images/q5772_d1940.png",
|
||||
"ucsf_document_id": "pzyw0224",
|
||||
"ucsf_document_page_no": "10",
|
||||
"topic": "form",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "49077",
|
||||
"questionId": "49077",
|
||||
"docId": "14179",
|
||||
"image_path": "data/docvqa_images/q49077_d14179.png",
|
||||
"ucsf_document_id": "nrxb0228",
|
||||
"ucsf_document_page_no": "3",
|
||||
"topic": "layout",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "58519",
|
||||
"questionId": "58519",
|
||||
"docId": "5347",
|
||||
"image_path": "data/docvqa_images/q58519_d5347.png",
|
||||
"ucsf_document_id": "sjbw0217",
|
||||
"ucsf_document_page_no": "11",
|
||||
"topic": "table/list",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "50720",
|
||||
"questionId": "50720",
|
||||
"docId": "281",
|
||||
"image_path": "data/docvqa_images/q50720_d281.png",
|
||||
"ucsf_document_id": "nrcj0037",
|
||||
"ucsf_document_page_no": "7",
|
||||
"topic": "table/list",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "56785",
|
||||
"questionId": "56785",
|
||||
"docId": "14289",
|
||||
"image_path": "data/docvqa_images/q56785_d14289.png",
|
||||
"ucsf_document_id": "xkbv0228",
|
||||
"ucsf_document_page_no": "1",
|
||||
"topic": "layout",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "59653",
|
||||
"questionId": "59653",
|
||||
"docId": "6579",
|
||||
"image_path": "data/docvqa_images/q59653_d6579.png",
|
||||
"ucsf_document_id": "mzbx0227",
|
||||
"ucsf_document_page_no": "2",
|
||||
"topic": "layout",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "61791",
|
||||
"questionId": "61791",
|
||||
"docId": "8072",
|
||||
"image_path": "data/docvqa_images/q61791_d8072.png",
|
||||
"ucsf_document_id": "hfmf0227",
|
||||
"ucsf_document_page_no": "1",
|
||||
"topic": "layout",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "37229",
|
||||
"questionId": "37229",
|
||||
"docId": "10742",
|
||||
"image_path": "data/docvqa_images/q37229_d10742.png",
|
||||
"ucsf_document_id": "nkcd0227",
|
||||
"ucsf_document_page_no": "2",
|
||||
"topic": "free_text",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "60407",
|
||||
"questionId": "60407",
|
||||
"docId": "7135",
|
||||
"image_path": "data/docvqa_images/q60407_d7135.png",
|
||||
"ucsf_document_id": "gkpk0226",
|
||||
"ucsf_document_page_no": "1",
|
||||
"topic": "free_text",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "64420",
|
||||
"questionId": "64420",
|
||||
"docId": "10230",
|
||||
"image_path": "data/docvqa_images/q64420_d10230.png",
|
||||
"ucsf_document_id": "jnjm0223",
|
||||
"ucsf_document_page_no": "107",
|
||||
"topic": "table/list",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "47365",
|
||||
"questionId": "47365",
|
||||
"docId": "13813",
|
||||
"image_path": "data/docvqa_images/q47365_d13813.png",
|
||||
"ucsf_document_id": "nxym0227",
|
||||
"ucsf_document_page_no": "28",
|
||||
"topic": "table/list",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "47458",
|
||||
"questionId": "47458",
|
||||
"docId": "13639",
|
||||
"image_path": "data/docvqa_images/q47458_d13639.png",
|
||||
"ucsf_document_id": "skdv0228",
|
||||
"ucsf_document_page_no": "5",
|
||||
"topic": "layout",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "7621",
|
||||
"questionId": "7621",
|
||||
"docId": "2668",
|
||||
"image_path": "data/docvqa_images/q7621_d2668.png",
|
||||
"ucsf_document_id": "flxn0020",
|
||||
"ucsf_document_page_no": "1",
|
||||
"topic": "form",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "53575",
|
||||
"questionId": "53575",
|
||||
"docId": "2766",
|
||||
"image_path": "data/docvqa_images/q53575_d2766.png",
|
||||
"ucsf_document_id": "hsfn0020",
|
||||
"ucsf_document_page_no": "2",
|
||||
"topic": "free_text|table/list",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "60913",
|
||||
"questionId": "60913",
|
||||
"docId": "7349",
|
||||
"image_path": "data/docvqa_images/q60913_d7349.png",
|
||||
"ucsf_document_id": "jzhd0227",
|
||||
"ucsf_document_page_no": "61",
|
||||
"topic": "free_text",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "60454",
|
||||
"questionId": "60454",
|
||||
"docId": "7163",
|
||||
"image_path": "data/docvqa_images/q60454_d7163.png",
|
||||
"ucsf_document_id": "jgyk0226",
|
||||
"ucsf_document_page_no": "1",
|
||||
"topic": "free_text",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "57978",
|
||||
"questionId": "57978",
|
||||
"docId": "4920",
|
||||
"image_path": "data/docvqa_images/q57978_d4920.png",
|
||||
"ucsf_document_id": "lkvw0217",
|
||||
"ucsf_document_page_no": "2",
|
||||
"topic": "table/list",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "64547",
|
||||
"questionId": "64547",
|
||||
"docId": "10361",
|
||||
"image_path": "data/docvqa_images/q64547_d10361.png",
|
||||
"ucsf_document_id": "lpdl0226",
|
||||
"ucsf_document_page_no": "32",
|
||||
"topic": "table/list",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "59481",
|
||||
"questionId": "59481",
|
||||
"docId": "6243",
|
||||
"image_path": "data/docvqa_images/q59481_d6243.png",
|
||||
"ucsf_document_id": "psgv0228",
|
||||
"ucsf_document_page_no": "5",
|
||||
"topic": "free_text",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "61472",
|
||||
"questionId": "61472",
|
||||
"docId": "7757",
|
||||
"image_path": "data/docvqa_images/q61472_d7757.png",
|
||||
"ucsf_document_id": "ymkp0227",
|
||||
"ucsf_document_page_no": "13",
|
||||
"topic": "handwritten|table/list",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "5673",
|
||||
"questionId": "5673",
|
||||
"docId": "1908",
|
||||
"image_path": "data/docvqa_images/q5673_d1908.png",
|
||||
"ucsf_document_id": "lldj0224",
|
||||
"ucsf_document_page_no": "2",
|
||||
"topic": "form",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "49109",
|
||||
"questionId": "49109",
|
||||
"docId": "13644",
|
||||
"image_path": "data/docvqa_images/q49109_d13644.png",
|
||||
"ucsf_document_id": "mzdv0228",
|
||||
"ucsf_document_page_no": "1",
|
||||
"topic": "table/list",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "46123",
|
||||
"questionId": "46123",
|
||||
"docId": "13503",
|
||||
"image_path": "data/docvqa_images/q46123_d13503.png",
|
||||
"ucsf_document_id": "xmww0217",
|
||||
"ucsf_document_page_no": "17",
|
||||
"topic": "table/list",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "48158",
|
||||
"questionId": "48158",
|
||||
"docId": "13976",
|
||||
"image_path": "data/docvqa_images/q48158_d13976.png",
|
||||
"ucsf_document_id": "zqhm0227",
|
||||
"ucsf_document_page_no": "1",
|
||||
"topic": "free_text",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "1955",
|
||||
"questionId": "1955",
|
||||
"docId": "892",
|
||||
"image_path": "data/docvqa_images/q1955_d892.png",
|
||||
"ucsf_document_id": "jsbn0226",
|
||||
"ucsf_document_page_no": "2",
|
||||
"topic": "table/list",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "8127",
|
||||
"questionId": "8127",
|
||||
"docId": "2754",
|
||||
"image_path": "data/docvqa_images/q8127_d2754.png",
|
||||
"ucsf_document_id": "xtvn0020",
|
||||
"ucsf_document_page_no": "2",
|
||||
"topic": "form",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "57431",
|
||||
"questionId": "57431",
|
||||
"docId": "4779",
|
||||
"image_path": "data/docvqa_images/q57431_d4779.png",
|
||||
"ucsf_document_id": "rnbx0223",
|
||||
"ucsf_document_page_no": "208",
|
||||
"topic": "Image/Photo",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "64306",
|
||||
"questionId": "64306",
|
||||
"docId": "10149",
|
||||
"image_path": "data/docvqa_images/q64306_d10149.png",
|
||||
"ucsf_document_id": "lpjm0223",
|
||||
"ucsf_document_page_no": "23",
|
||||
"topic": "table/list",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "64887",
|
||||
"questionId": "64887",
|
||||
"docId": "9754",
|
||||
"image_path": "data/docvqa_images/q64887_d9754.png",
|
||||
"ucsf_document_id": "szpg0227",
|
||||
"ucsf_document_page_no": "9",
|
||||
"topic": "table/list",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "58680",
|
||||
"questionId": "58680",
|
||||
"docId": "5545",
|
||||
"image_path": "data/docvqa_images/q58680_d5545.png",
|
||||
"ucsf_document_id": "hhwh0078",
|
||||
"ucsf_document_page_no": "1",
|
||||
"topic": "table/list",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "5287",
|
||||
"questionId": "5287",
|
||||
"docId": "1785",
|
||||
"image_path": "data/docvqa_images/q5287_d1785.png",
|
||||
"ucsf_document_id": "mtnh0227",
|
||||
"ucsf_document_page_no": "10",
|
||||
"topic": "form",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "55471",
|
||||
"questionId": "55471",
|
||||
"docId": "4340",
|
||||
"image_path": "data/docvqa_images/q55471_d4340.png",
|
||||
"ucsf_document_id": "fsgj0223",
|
||||
"ucsf_document_page_no": "96",
|
||||
"topic": "free_text",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "53095",
|
||||
"questionId": "53095",
|
||||
"docId": "296",
|
||||
"image_path": "data/docvqa_images/q53095_d296.png",
|
||||
"ucsf_document_id": "qhxj0037",
|
||||
"ucsf_document_page_no": "3",
|
||||
"topic": "layout",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "53726",
|
||||
"questionId": "53726",
|
||||
"docId": "2008",
|
||||
"image_path": "data/docvqa_images/q53726_d2008.png",
|
||||
"ucsf_document_id": "hhnf0094",
|
||||
"ucsf_document_page_no": "5",
|
||||
"topic": "layout",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "57321",
|
||||
"questionId": "57321",
|
||||
"docId": "4722",
|
||||
"image_path": "data/docvqa_images/q57321_d4722.png",
|
||||
"ucsf_document_id": "xybx0223",
|
||||
"ucsf_document_page_no": "32",
|
||||
"topic": "table/list",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "26659",
|
||||
"questionId": "26659",
|
||||
"docId": "7470",
|
||||
"image_path": "data/docvqa_images/q26659_d7470.png",
|
||||
"ucsf_document_id": "lhmg0227",
|
||||
"ucsf_document_page_no": "1",
|
||||
"topic": "layout",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "38920",
|
||||
"questionId": "38920",
|
||||
"docId": "11157",
|
||||
"image_path": "data/docvqa_images/q38920_d11157.png",
|
||||
"ucsf_document_id": "klnf0227",
|
||||
"ucsf_document_page_no": "1",
|
||||
"topic": "table/list|layout",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "50837",
|
||||
"questionId": "50837",
|
||||
"docId": "14742",
|
||||
"image_path": "data/docvqa_images/q50837_d14742.png",
|
||||
"ucsf_document_id": "ysmc0228",
|
||||
"ucsf_document_page_no": "4",
|
||||
"topic": "table/list",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "59615",
|
||||
"questionId": "59615",
|
||||
"docId": "6569",
|
||||
"image_path": "data/docvqa_images/q59615_d6569.png",
|
||||
"ucsf_document_id": "hnnp0227",
|
||||
"ucsf_document_page_no": "45",
|
||||
"topic": "handwritten|table/list|layout",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "58687",
|
||||
"questionId": "58687",
|
||||
"docId": "5545",
|
||||
"image_path": "data/docvqa_images/q58687_d5545.png",
|
||||
"ucsf_document_id": "hhwh0078",
|
||||
"ucsf_document_page_no": "1",
|
||||
"topic": "table/list",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
}
|
||||
]
|
||||
@@ -1,34 +0,0 @@
|
||||
{
|
||||
"benchmark": "LiveMathematicianBench",
|
||||
"manifest_type": "id_split",
|
||||
"source_repo": "LiveMathematicianBench/LiveMathematicianBench",
|
||||
"source_repo_type": "dataset",
|
||||
"source_url": "https://huggingface.co/datasets/LiveMathematicianBench/LiveMathematicianBench",
|
||||
"source_revision": "b72450f6ce96c26158d64d945a5d31ef7727be41",
|
||||
"source_files": [
|
||||
"data/202511/qa_202511_final.json",
|
||||
"data/202512/qa_202512_final.json",
|
||||
"data/202601/qa_202601_final.json",
|
||||
"data/202602/qa_202602_final.json"
|
||||
],
|
||||
"split_mode": "ratio",
|
||||
"split_ratio": "2:1:7",
|
||||
"split_seed": 42,
|
||||
"counts": {
|
||||
"train": 35,
|
||||
"val": 18,
|
||||
"test": 124
|
||||
},
|
||||
"item_fields": [
|
||||
"id",
|
||||
"month",
|
||||
"no",
|
||||
"paper_link",
|
||||
"source_file"
|
||||
],
|
||||
"id_format": "<month>:<no>",
|
||||
"notes": [
|
||||
"This is an ID split manifest, not the full LiveMathematicianBench payload.",
|
||||
"Materialize full split items from the official LiveMathematicianBench raw qa_*_final.json files before evaluation."
|
||||
]
|
||||
}
|
||||
@@ -1,870 +0,0 @@
|
||||
[
|
||||
{
|
||||
"id": "202602:12",
|
||||
"month": "202602",
|
||||
"no": 12,
|
||||
"paper_link": "http://arxiv.org/abs/2602.07171v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:3",
|
||||
"month": "202601",
|
||||
"no": 3,
|
||||
"paper_link": "http://arxiv.org/abs/2601.01447v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202511:4",
|
||||
"month": "202511",
|
||||
"no": 4,
|
||||
"paper_link": "http://arxiv.org/abs/2511.23123v1",
|
||||
"source_file": "data/202511/qa_202511_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:20",
|
||||
"month": "202601",
|
||||
"no": 20,
|
||||
"paper_link": "http://arxiv.org/abs/2601.13212v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:42",
|
||||
"month": "202601",
|
||||
"no": 42,
|
||||
"paper_link": "http://arxiv.org/abs/2601.09348v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:38",
|
||||
"month": "202512",
|
||||
"no": 38,
|
||||
"paper_link": "http://arxiv.org/abs/2512.19831v2",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:4",
|
||||
"month": "202512",
|
||||
"no": 4,
|
||||
"paper_link": "http://arxiv.org/abs/2512.03141v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:4",
|
||||
"month": "202602",
|
||||
"no": 4,
|
||||
"paper_link": "http://arxiv.org/abs/2602.14368v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202511:15",
|
||||
"month": "202511",
|
||||
"no": 15,
|
||||
"paper_link": "http://arxiv.org/abs/2511.17325v1",
|
||||
"source_file": "data/202511/qa_202511_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:32",
|
||||
"month": "202602",
|
||||
"no": 32,
|
||||
"paper_link": "http://arxiv.org/abs/2602.14817v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:51",
|
||||
"month": "202512",
|
||||
"no": 51,
|
||||
"paper_link": "http://arxiv.org/abs/2512.14581v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:26",
|
||||
"month": "202512",
|
||||
"no": 26,
|
||||
"paper_link": "http://arxiv.org/abs/2512.19586v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:13",
|
||||
"month": "202601",
|
||||
"no": 13,
|
||||
"paper_link": "http://arxiv.org/abs/2601.10017v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:1",
|
||||
"month": "202602",
|
||||
"no": 1,
|
||||
"paper_link": "http://arxiv.org/abs/2602.23137v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202511:18",
|
||||
"month": "202511",
|
||||
"no": 18,
|
||||
"paper_link": "http://arxiv.org/abs/2511.10795v1",
|
||||
"source_file": "data/202511/qa_202511_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:5",
|
||||
"month": "202512",
|
||||
"no": 5,
|
||||
"paper_link": "http://arxiv.org/abs/2512.00348v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202511:19",
|
||||
"month": "202511",
|
||||
"no": 19,
|
||||
"paper_link": "http://arxiv.org/abs/2511.06951v1",
|
||||
"source_file": "data/202511/qa_202511_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:40",
|
||||
"month": "202602",
|
||||
"no": 40,
|
||||
"paper_link": "http://arxiv.org/abs/2602.20462v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:29",
|
||||
"month": "202602",
|
||||
"no": 29,
|
||||
"paper_link": "http://arxiv.org/abs/2602.10676v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:35",
|
||||
"month": "202512",
|
||||
"no": 35,
|
||||
"paper_link": "http://arxiv.org/abs/2512.08840v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:48",
|
||||
"month": "202512",
|
||||
"no": 48,
|
||||
"paper_link": "http://arxiv.org/abs/2512.03482v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:52",
|
||||
"month": "202512",
|
||||
"no": 52,
|
||||
"paper_link": "http://arxiv.org/abs/2512.11246v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:44",
|
||||
"month": "202512",
|
||||
"no": 44,
|
||||
"paper_link": "http://arxiv.org/abs/2512.10385v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202511:28",
|
||||
"month": "202511",
|
||||
"no": 28,
|
||||
"paper_link": "http://arxiv.org/abs/2511.03812v1",
|
||||
"source_file": "data/202511/qa_202511_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:43",
|
||||
"month": "202601",
|
||||
"no": 43,
|
||||
"paper_link": "http://arxiv.org/abs/2601.22555v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:9",
|
||||
"month": "202602",
|
||||
"no": 9,
|
||||
"paper_link": "http://arxiv.org/abs/2602.19882v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:23",
|
||||
"month": "202512",
|
||||
"no": 23,
|
||||
"paper_link": "http://arxiv.org/abs/2512.09180v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:21",
|
||||
"month": "202602",
|
||||
"no": 21,
|
||||
"paper_link": "http://arxiv.org/abs/2602.10509v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202511:5",
|
||||
"month": "202511",
|
||||
"no": 5,
|
||||
"paper_link": "http://arxiv.org/abs/2511.20164v1",
|
||||
"source_file": "data/202511/qa_202511_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:35",
|
||||
"month": "202601",
|
||||
"no": 35,
|
||||
"paper_link": "http://arxiv.org/abs/2601.15606v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:50",
|
||||
"month": "202602",
|
||||
"no": 50,
|
||||
"paper_link": "http://arxiv.org/abs/2602.05652v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:13",
|
||||
"month": "202512",
|
||||
"no": 13,
|
||||
"paper_link": "http://arxiv.org/abs/2512.22861v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:49",
|
||||
"month": "202602",
|
||||
"no": 49,
|
||||
"paper_link": "http://arxiv.org/abs/2602.07167v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:18",
|
||||
"month": "202602",
|
||||
"no": 18,
|
||||
"paper_link": "http://arxiv.org/abs/2602.20124v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:15",
|
||||
"month": "202601",
|
||||
"no": 15,
|
||||
"paper_link": "http://arxiv.org/abs/2601.05327v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:21",
|
||||
"month": "202601",
|
||||
"no": 21,
|
||||
"paper_link": "http://arxiv.org/abs/2601.04994v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:32",
|
||||
"month": "202601",
|
||||
"no": 32,
|
||||
"paper_link": "http://arxiv.org/abs/2601.09183v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:34",
|
||||
"month": "202602",
|
||||
"no": 34,
|
||||
"paper_link": "http://arxiv.org/abs/2602.21118v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:20",
|
||||
"month": "202602",
|
||||
"no": 20,
|
||||
"paper_link": "http://arxiv.org/abs/2602.16506v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:5",
|
||||
"month": "202602",
|
||||
"no": 5,
|
||||
"paper_link": "http://arxiv.org/abs/2602.09806v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:40",
|
||||
"month": "202512",
|
||||
"no": 40,
|
||||
"paper_link": "http://arxiv.org/abs/2512.16535v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202511:22",
|
||||
"month": "202511",
|
||||
"no": 22,
|
||||
"paper_link": "http://arxiv.org/abs/2511.07607v2",
|
||||
"source_file": "data/202511/qa_202511_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:36",
|
||||
"month": "202601",
|
||||
"no": 36,
|
||||
"paper_link": "http://arxiv.org/abs/2601.12457v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:49",
|
||||
"month": "202512",
|
||||
"no": 49,
|
||||
"paper_link": "http://arxiv.org/abs/2512.21565v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202511:10",
|
||||
"month": "202511",
|
||||
"no": 10,
|
||||
"paper_link": "http://arxiv.org/abs/2511.06484v1",
|
||||
"source_file": "data/202511/qa_202511_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:2",
|
||||
"month": "202601",
|
||||
"no": 2,
|
||||
"paper_link": "http://arxiv.org/abs/2601.07068v4",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:19",
|
||||
"month": "202602",
|
||||
"no": 19,
|
||||
"paper_link": "http://arxiv.org/abs/2602.18179v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:9",
|
||||
"month": "202601",
|
||||
"no": 9,
|
||||
"paper_link": "http://arxiv.org/abs/2601.17765v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:6",
|
||||
"month": "202512",
|
||||
"no": 6,
|
||||
"paper_link": "http://arxiv.org/abs/2512.23079v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:5",
|
||||
"month": "202601",
|
||||
"no": 5,
|
||||
"paper_link": "http://arxiv.org/abs/2601.20344v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:14",
|
||||
"month": "202602",
|
||||
"no": 14,
|
||||
"paper_link": "http://arxiv.org/abs/2602.09177v2",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:17",
|
||||
"month": "202512",
|
||||
"no": 17,
|
||||
"paper_link": "http://arxiv.org/abs/2512.11657v2",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:19",
|
||||
"month": "202512",
|
||||
"no": 19,
|
||||
"paper_link": "http://arxiv.org/abs/2512.16655v2",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:33",
|
||||
"month": "202602",
|
||||
"no": 33,
|
||||
"paper_link": "http://arxiv.org/abs/2602.13734v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:18",
|
||||
"month": "202512",
|
||||
"no": 18,
|
||||
"paper_link": "http://arxiv.org/abs/2512.22960v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:26",
|
||||
"month": "202601",
|
||||
"no": 26,
|
||||
"paper_link": "http://arxiv.org/abs/2601.06814v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:1",
|
||||
"month": "202601",
|
||||
"no": 1,
|
||||
"paper_link": "http://arxiv.org/abs/2601.18276v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:30",
|
||||
"month": "202512",
|
||||
"no": 30,
|
||||
"paper_link": "http://arxiv.org/abs/2512.07260v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:44",
|
||||
"month": "202602",
|
||||
"no": 44,
|
||||
"paper_link": "http://arxiv.org/abs/2602.01138v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:20",
|
||||
"month": "202512",
|
||||
"no": 20,
|
||||
"paper_link": "http://arxiv.org/abs/2512.14575v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202511:13",
|
||||
"month": "202511",
|
||||
"no": 13,
|
||||
"paper_link": "http://arxiv.org/abs/2511.16910v1",
|
||||
"source_file": "data/202511/qa_202511_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:30",
|
||||
"month": "202601",
|
||||
"no": 30,
|
||||
"paper_link": "http://arxiv.org/abs/2601.12140v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:40",
|
||||
"month": "202601",
|
||||
"no": 40,
|
||||
"paper_link": "http://arxiv.org/abs/2601.05146v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:29",
|
||||
"month": "202601",
|
||||
"no": 29,
|
||||
"paper_link": "http://arxiv.org/abs/2601.12846v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202511:11",
|
||||
"month": "202511",
|
||||
"no": 11,
|
||||
"paper_link": "http://arxiv.org/abs/2511.17548v1",
|
||||
"source_file": "data/202511/qa_202511_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:9",
|
||||
"month": "202512",
|
||||
"no": 9,
|
||||
"paper_link": "http://arxiv.org/abs/2512.08817v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:18",
|
||||
"month": "202601",
|
||||
"no": 18,
|
||||
"paper_link": "http://arxiv.org/abs/2601.01797v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:1",
|
||||
"month": "202512",
|
||||
"no": 1,
|
||||
"paper_link": "http://arxiv.org/abs/2512.20055v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:4",
|
||||
"month": "202601",
|
||||
"no": 4,
|
||||
"paper_link": "http://arxiv.org/abs/2601.21223v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202511:6",
|
||||
"month": "202511",
|
||||
"no": 6,
|
||||
"paper_link": "http://arxiv.org/abs/2511.14959v1",
|
||||
"source_file": "data/202511/qa_202511_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:38",
|
||||
"month": "202602",
|
||||
"no": 38,
|
||||
"paper_link": "http://arxiv.org/abs/2602.08398v2",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:10",
|
||||
"month": "202601",
|
||||
"no": 10,
|
||||
"paper_link": "http://arxiv.org/abs/2601.15524v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:11",
|
||||
"month": "202602",
|
||||
"no": 11,
|
||||
"paper_link": "http://arxiv.org/abs/2602.11045v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:45",
|
||||
"month": "202512",
|
||||
"no": 45,
|
||||
"paper_link": "http://arxiv.org/abs/2512.08395v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:12",
|
||||
"month": "202601",
|
||||
"no": 12,
|
||||
"paper_link": "http://arxiv.org/abs/2601.11877v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:47",
|
||||
"month": "202512",
|
||||
"no": 47,
|
||||
"paper_link": "http://arxiv.org/abs/2512.09683v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202511:21",
|
||||
"month": "202511",
|
||||
"no": 21,
|
||||
"paper_link": "http://arxiv.org/abs/2511.21288v1",
|
||||
"source_file": "data/202511/qa_202511_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:16",
|
||||
"month": "202601",
|
||||
"no": 16,
|
||||
"paper_link": "http://arxiv.org/abs/2601.05008v2",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:3",
|
||||
"month": "202512",
|
||||
"no": 3,
|
||||
"paper_link": "http://arxiv.org/abs/2512.13450v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:37",
|
||||
"month": "202601",
|
||||
"no": 37,
|
||||
"paper_link": "http://arxiv.org/abs/2601.09443v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202511:12",
|
||||
"month": "202511",
|
||||
"no": 12,
|
||||
"paper_link": "http://arxiv.org/abs/2511.04978v1",
|
||||
"source_file": "data/202511/qa_202511_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:39",
|
||||
"month": "202512",
|
||||
"no": 39,
|
||||
"paper_link": "http://arxiv.org/abs/2512.19003v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:8",
|
||||
"month": "202601",
|
||||
"no": 8,
|
||||
"paper_link": "http://arxiv.org/abs/2601.19754v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:11",
|
||||
"month": "202601",
|
||||
"no": 11,
|
||||
"paper_link": "http://arxiv.org/abs/2601.13552v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202511:25",
|
||||
"month": "202511",
|
||||
"no": 25,
|
||||
"paper_link": "http://arxiv.org/abs/2511.10548v1",
|
||||
"source_file": "data/202511/qa_202511_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:17",
|
||||
"month": "202601",
|
||||
"no": 17,
|
||||
"paper_link": "http://arxiv.org/abs/2601.02655v2",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:36",
|
||||
"month": "202602",
|
||||
"no": 36,
|
||||
"paper_link": "http://arxiv.org/abs/2602.13001v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:43",
|
||||
"month": "202602",
|
||||
"no": 43,
|
||||
"paper_link": "http://arxiv.org/abs/2602.06897v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:6",
|
||||
"month": "202601",
|
||||
"no": 6,
|
||||
"paper_link": "http://arxiv.org/abs/2601.04747v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:35",
|
||||
"month": "202602",
|
||||
"no": 35,
|
||||
"paper_link": "http://arxiv.org/abs/2602.20938v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:11",
|
||||
"month": "202512",
|
||||
"no": 11,
|
||||
"paper_link": "http://arxiv.org/abs/2512.03294v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:23",
|
||||
"month": "202602",
|
||||
"no": 23,
|
||||
"paper_link": "http://arxiv.org/abs/2602.09201v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:7",
|
||||
"month": "202601",
|
||||
"no": 7,
|
||||
"paper_link": "http://arxiv.org/abs/2601.02859v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:39",
|
||||
"month": "202602",
|
||||
"no": 39,
|
||||
"paper_link": "http://arxiv.org/abs/2602.21659v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:12",
|
||||
"month": "202512",
|
||||
"no": 12,
|
||||
"paper_link": "http://arxiv.org/abs/2512.00690v3",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202511:2",
|
||||
"month": "202511",
|
||||
"no": 2,
|
||||
"paper_link": "http://arxiv.org/abs/2511.19681v1",
|
||||
"source_file": "data/202511/qa_202511_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:43",
|
||||
"month": "202512",
|
||||
"no": 43,
|
||||
"paper_link": "http://arxiv.org/abs/2512.10820v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:24",
|
||||
"month": "202602",
|
||||
"no": 24,
|
||||
"paper_link": "http://arxiv.org/abs/2602.08680v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:34",
|
||||
"month": "202601",
|
||||
"no": 34,
|
||||
"paper_link": "http://arxiv.org/abs/2601.07318v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:28",
|
||||
"month": "202512",
|
||||
"no": 28,
|
||||
"paper_link": "http://arxiv.org/abs/2512.11294v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:27",
|
||||
"month": "202601",
|
||||
"no": 27,
|
||||
"paper_link": "http://arxiv.org/abs/2601.05692v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:42",
|
||||
"month": "202602",
|
||||
"no": 42,
|
||||
"paper_link": "http://arxiv.org/abs/2602.09749v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:22",
|
||||
"month": "202512",
|
||||
"no": 22,
|
||||
"paper_link": "http://arxiv.org/abs/2512.11658v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:17",
|
||||
"month": "202602",
|
||||
"no": 17,
|
||||
"paper_link": "http://arxiv.org/abs/2602.22504v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:48",
|
||||
"month": "202602",
|
||||
"no": 48,
|
||||
"paper_link": "http://arxiv.org/abs/2602.08760v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:28",
|
||||
"month": "202602",
|
||||
"no": 28,
|
||||
"paper_link": "http://arxiv.org/abs/2602.11595v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:3",
|
||||
"month": "202602",
|
||||
"no": 3,
|
||||
"paper_link": "http://arxiv.org/abs/2602.17369v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:31",
|
||||
"month": "202512",
|
||||
"no": 31,
|
||||
"paper_link": "http://arxiv.org/abs/2512.23668v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:27",
|
||||
"month": "202512",
|
||||
"no": 27,
|
||||
"paper_link": "http://arxiv.org/abs/2512.16505v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202511:24",
|
||||
"month": "202511",
|
||||
"no": 24,
|
||||
"paper_link": "http://arxiv.org/abs/2511.12549v2",
|
||||
"source_file": "data/202511/qa_202511_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202511:8",
|
||||
"month": "202511",
|
||||
"no": 8,
|
||||
"paper_link": "http://arxiv.org/abs/2511.12657v1",
|
||||
"source_file": "data/202511/qa_202511_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202511:9",
|
||||
"month": "202511",
|
||||
"no": 9,
|
||||
"paper_link": "http://arxiv.org/abs/2511.09015v1",
|
||||
"source_file": "data/202511/qa_202511_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:28",
|
||||
"month": "202601",
|
||||
"no": 28,
|
||||
"paper_link": "http://arxiv.org/abs/2601.14825v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:25",
|
||||
"month": "202602",
|
||||
"no": 25,
|
||||
"paper_link": "http://arxiv.org/abs/2602.16048v3",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202511:23",
|
||||
"month": "202511",
|
||||
"no": 23,
|
||||
"paper_link": "http://arxiv.org/abs/2511.06595v1",
|
||||
"source_file": "data/202511/qa_202511_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:13",
|
||||
"month": "202602",
|
||||
"no": 13,
|
||||
"paper_link": "http://arxiv.org/abs/2602.12261v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202511:27",
|
||||
"month": "202511",
|
||||
"no": 27,
|
||||
"paper_link": "http://arxiv.org/abs/2511.04407v1",
|
||||
"source_file": "data/202511/qa_202511_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:7",
|
||||
"month": "202512",
|
||||
"no": 7,
|
||||
"paper_link": "http://arxiv.org/abs/2512.09490v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:29",
|
||||
"month": "202512",
|
||||
"no": 29,
|
||||
"paper_link": "http://arxiv.org/abs/2512.08562v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:34",
|
||||
"month": "202512",
|
||||
"no": 34,
|
||||
"paper_link": "http://arxiv.org/abs/2512.09598v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:42",
|
||||
"month": "202512",
|
||||
"no": 42,
|
||||
"paper_link": "http://arxiv.org/abs/2512.10845v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202511:7",
|
||||
"month": "202511",
|
||||
"no": 7,
|
||||
"paper_link": "http://arxiv.org/abs/2511.13976v1",
|
||||
"source_file": "data/202511/qa_202511_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202511:29",
|
||||
"month": "202511",
|
||||
"no": 29,
|
||||
"paper_link": "http://arxiv.org/abs/2511.03722v1",
|
||||
"source_file": "data/202511/qa_202511_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:37",
|
||||
"month": "202602",
|
||||
"no": 37,
|
||||
"paper_link": "http://arxiv.org/abs/2602.08644v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
}
|
||||
]
|
||||
@@ -1,247 +0,0 @@
|
||||
[
|
||||
{
|
||||
"id": "202602:22",
|
||||
"month": "202602",
|
||||
"no": 22,
|
||||
"paper_link": "http://arxiv.org/abs/2602.10700v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:8",
|
||||
"month": "202512",
|
||||
"no": 8,
|
||||
"paper_link": "http://arxiv.org/abs/2512.08863v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202511:16",
|
||||
"month": "202511",
|
||||
"no": 16,
|
||||
"paper_link": "http://arxiv.org/abs/2511.15668v1",
|
||||
"source_file": "data/202511/qa_202511_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:44",
|
||||
"month": "202601",
|
||||
"no": 44,
|
||||
"paper_link": "http://arxiv.org/abs/2601.21267v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202511:14",
|
||||
"month": "202511",
|
||||
"no": 14,
|
||||
"paper_link": "http://arxiv.org/abs/2511.13447v1",
|
||||
"source_file": "data/202511/qa_202511_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:30",
|
||||
"month": "202602",
|
||||
"no": 30,
|
||||
"paper_link": "http://arxiv.org/abs/2602.16692v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:2",
|
||||
"month": "202602",
|
||||
"no": 2,
|
||||
"paper_link": "http://arxiv.org/abs/2602.22933v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:41",
|
||||
"month": "202601",
|
||||
"no": 41,
|
||||
"paper_link": "http://arxiv.org/abs/2601.01164v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:23",
|
||||
"month": "202601",
|
||||
"no": 23,
|
||||
"paper_link": "http://arxiv.org/abs/2601.02528v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202511:20",
|
||||
"month": "202511",
|
||||
"no": 20,
|
||||
"paper_link": "http://arxiv.org/abs/2511.02963v1",
|
||||
"source_file": "data/202511/qa_202511_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:22",
|
||||
"month": "202601",
|
||||
"no": 22,
|
||||
"paper_link": "http://arxiv.org/abs/2601.03984v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:14",
|
||||
"month": "202512",
|
||||
"no": 14,
|
||||
"paper_link": "http://arxiv.org/abs/2512.22459v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202511:26",
|
||||
"month": "202511",
|
||||
"no": 26,
|
||||
"paper_link": "http://arxiv.org/abs/2511.07817v1",
|
||||
"source_file": "data/202511/qa_202511_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202511:3",
|
||||
"month": "202511",
|
||||
"no": 3,
|
||||
"paper_link": "http://arxiv.org/abs/2511.11409v1",
|
||||
"source_file": "data/202511/qa_202511_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:33",
|
||||
"month": "202601",
|
||||
"no": 33,
|
||||
"paper_link": "http://arxiv.org/abs/2601.07747v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:7",
|
||||
"month": "202602",
|
||||
"no": 7,
|
||||
"paper_link": "http://arxiv.org/abs/2602.22912v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:27",
|
||||
"month": "202602",
|
||||
"no": 27,
|
||||
"paper_link": "http://arxiv.org/abs/2602.13968v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:31",
|
||||
"month": "202602",
|
||||
"no": 31,
|
||||
"paper_link": "http://arxiv.org/abs/2602.15528v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:41",
|
||||
"month": "202602",
|
||||
"no": 41,
|
||||
"paper_link": "http://arxiv.org/abs/2602.10707v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:25",
|
||||
"month": "202512",
|
||||
"no": 25,
|
||||
"paper_link": "http://arxiv.org/abs/2512.04531v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:10",
|
||||
"month": "202602",
|
||||
"no": 10,
|
||||
"paper_link": "http://arxiv.org/abs/2602.17863v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:16",
|
||||
"month": "202602",
|
||||
"no": 16,
|
||||
"paper_link": "http://arxiv.org/abs/2602.02723v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:16",
|
||||
"month": "202512",
|
||||
"no": 16,
|
||||
"paper_link": "http://arxiv.org/abs/2512.11601v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:2",
|
||||
"month": "202512",
|
||||
"no": 2,
|
||||
"paper_link": "http://arxiv.org/abs/2512.16120v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:24",
|
||||
"month": "202512",
|
||||
"no": 24,
|
||||
"paper_link": "http://arxiv.org/abs/2512.08391v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:32",
|
||||
"month": "202512",
|
||||
"no": 32,
|
||||
"paper_link": "http://arxiv.org/abs/2512.23224v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:47",
|
||||
"month": "202602",
|
||||
"no": 47,
|
||||
"paper_link": "http://arxiv.org/abs/2602.10391v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:46",
|
||||
"month": "202602",
|
||||
"no": 46,
|
||||
"paper_link": "http://arxiv.org/abs/2602.13727v2",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:21",
|
||||
"month": "202512",
|
||||
"no": 21,
|
||||
"paper_link": "http://arxiv.org/abs/2512.12835v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:33",
|
||||
"month": "202512",
|
||||
"no": 33,
|
||||
"paper_link": "http://arxiv.org/abs/2512.19500v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:45",
|
||||
"month": "202602",
|
||||
"no": 45,
|
||||
"paper_link": "http://arxiv.org/abs/2602.23912v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:26",
|
||||
"month": "202602",
|
||||
"no": 26,
|
||||
"paper_link": "http://arxiv.org/abs/2602.14658v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:41",
|
||||
"month": "202512",
|
||||
"no": 41,
|
||||
"paper_link": "http://arxiv.org/abs/2512.15177v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:38",
|
||||
"month": "202601",
|
||||
"no": 38,
|
||||
"paper_link": "http://arxiv.org/abs/2601.07817v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:14",
|
||||
"month": "202601",
|
||||
"no": 14,
|
||||
"paper_link": "http://arxiv.org/abs/2601.08704v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
}
|
||||
]
|
||||
@@ -1,128 +0,0 @@
|
||||
[
|
||||
{
|
||||
"id": "202602:8",
|
||||
"month": "202602",
|
||||
"no": 8,
|
||||
"paper_link": "http://arxiv.org/abs/2602.19529v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:50",
|
||||
"month": "202512",
|
||||
"no": 50,
|
||||
"paper_link": "http://arxiv.org/abs/2512.15277v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:36",
|
||||
"month": "202512",
|
||||
"no": 36,
|
||||
"paper_link": "http://arxiv.org/abs/2512.06696v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202511:1",
|
||||
"month": "202511",
|
||||
"no": 1,
|
||||
"paper_link": "http://arxiv.org/abs/2511.04651v1",
|
||||
"source_file": "data/202511/qa_202511_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:31",
|
||||
"month": "202601",
|
||||
"no": 31,
|
||||
"paper_link": "http://arxiv.org/abs/2601.10298v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202511:17",
|
||||
"month": "202511",
|
||||
"no": 17,
|
||||
"paper_link": "http://arxiv.org/abs/2511.13215v1",
|
||||
"source_file": "data/202511/qa_202511_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:37",
|
||||
"month": "202512",
|
||||
"no": 37,
|
||||
"paper_link": "http://arxiv.org/abs/2512.20498v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:39",
|
||||
"month": "202601",
|
||||
"no": 39,
|
||||
"paper_link": "http://arxiv.org/abs/2601.06601v2",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:25",
|
||||
"month": "202601",
|
||||
"no": 25,
|
||||
"paper_link": "http://arxiv.org/abs/2601.10996v3",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:24",
|
||||
"month": "202601",
|
||||
"no": 24,
|
||||
"paper_link": "http://arxiv.org/abs/2601.12250v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:45",
|
||||
"month": "202601",
|
||||
"no": 45,
|
||||
"paper_link": "http://arxiv.org/abs/2601.12113v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:19",
|
||||
"month": "202601",
|
||||
"no": 19,
|
||||
"paper_link": "http://arxiv.org/abs/2601.00779v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:10",
|
||||
"month": "202512",
|
||||
"no": 10,
|
||||
"paper_link": "http://arxiv.org/abs/2512.07073v2",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:46",
|
||||
"month": "202601",
|
||||
"no": 46,
|
||||
"paper_link": "http://arxiv.org/abs/2601.07793v2",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:15",
|
||||
"month": "202512",
|
||||
"no": 15,
|
||||
"paper_link": "http://arxiv.org/abs/2512.16165v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:15",
|
||||
"month": "202602",
|
||||
"no": 15,
|
||||
"paper_link": "http://arxiv.org/abs/2602.05303v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:6",
|
||||
"month": "202602",
|
||||
"no": 6,
|
||||
"paper_link": "http://arxiv.org/abs/2602.01571v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:46",
|
||||
"month": "202512",
|
||||
"no": 46,
|
||||
"paper_link": "http://arxiv.org/abs/2512.05945v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
}
|
||||
]
|
||||
@@ -1,27 +0,0 @@
|
||||
{
|
||||
"benchmark": "OfficeQA",
|
||||
"manifest_type": "id_split",
|
||||
"source_repo": "databricks/officeqa",
|
||||
"source_repo_type": "dataset",
|
||||
"source_url": "https://huggingface.co/datasets/databricks/officeqa",
|
||||
"source_revision": "8ecbf18d3833daf4750a903d14963e4c4c1d4cd8",
|
||||
"source_file": "officeqa_full.csv",
|
||||
"source_split_name": "officeqa_split",
|
||||
"counts": {
|
||||
"train": 50,
|
||||
"val": 24,
|
||||
"test": 172
|
||||
},
|
||||
"item_fields": [
|
||||
"id",
|
||||
"uid",
|
||||
"category",
|
||||
"source_files",
|
||||
"source_docs",
|
||||
"source_split"
|
||||
],
|
||||
"notes": [
|
||||
"This is a split manifest, not the full OfficeQA payload.",
|
||||
"The official OfficeQA CSV is gated on Hugging Face; materialization requires authorized access."
|
||||
]
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,402 +0,0 @@
|
||||
[
|
||||
{
|
||||
"id": "UID0002",
|
||||
"uid": "UID0002",
|
||||
"category": "easy",
|
||||
"source_files": "treasury_bulletin_1944_01.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/january-1944-6565?page=18",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0007",
|
||||
"uid": "UID0007",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_1950_02.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/february-1950-6637?page=15",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0014",
|
||||
"uid": "UID0014",
|
||||
"category": "easy",
|
||||
"source_files": "treasury_bulletin_1942_07.txt\r\ntreasury_bulletin_2001_12.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/july-1942-6547?page=76\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/december-2001-7108?page=17&deep=true",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0017",
|
||||
"uid": "UID0017",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_1982_08.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/august-1982-7028?page=13",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0018",
|
||||
"uid": "UID0018",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_1985_03.txt\r\ntreasury_bulletin_1986_03.txt\r\ntreasury_bulletin_1987_03.txt\r\ntreasury_bulletin_1988_03.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/march-1985-7040?page=22\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/march-1986-7045?page=26\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/march-1987-7049?page=24\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/march-1988-7052?page=36",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0019",
|
||||
"uid": "UID0019",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_2016_09.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/september-2016-533966?page=54\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/september-2016-533966?page=58",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0026",
|
||||
"uid": "UID0026",
|
||||
"category": "easy",
|
||||
"source_files": "treasury_bulletin_1963_01.txt\r\ntreasury_bulletin_1962_01.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/january-1963-6793?page=88\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/january-1962-6781?page=82&deep=true",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0028",
|
||||
"uid": "UID0028",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_1970_06.txt\r\ntreasury_bulletin_1964_12.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/june-1970-6882?page=89&deep=true\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/december-1964-6816?page=25&deep=true",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0030",
|
||||
"uid": "UID0030",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_1990_09.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/september-1990-7062?page=19&deep=true",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0031",
|
||||
"uid": "UID0031",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_1992_03.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/march-1992-7068?page=158&deep=true",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0033",
|
||||
"uid": "UID0033",
|
||||
"category": "easy",
|
||||
"source_files": "treasury_bulletin_1977_04.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/april-1977-6964?page=9",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0034",
|
||||
"uid": "UID0034",
|
||||
"category": "easy",
|
||||
"source_files": "treasury_bulletin_1992_06.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/june-1992-7069?page=32",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0044",
|
||||
"uid": "UID0044",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_1939_04.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/april-1939-6506?page=61",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0046",
|
||||
"uid": "UID0046",
|
||||
"category": "easy",
|
||||
"source_files": "treasury_bulletin_1988_09.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/september-1988-7054?page=37",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0049",
|
||||
"uid": "UID0049",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_1942_02.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/february-1942-6542?page=19&deep=true",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0056",
|
||||
"uid": "UID0056",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_1991_09.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/september-1991-7066?page=30&deep=true",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0063",
|
||||
"uid": "UID0063",
|
||||
"category": "easy",
|
||||
"source_files": "treasury_bulletin_1990_06.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/june-1990-7061?page=127",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0065",
|
||||
"uid": "UID0065",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_1998_06.txt\r\ntreasury_bulletin_1995_12.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/june-1998-7094?page=7\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/december-1995-7084?page=16",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0073",
|
||||
"uid": "UID0073",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_1982_03.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/march-1982-7023?page=24",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0079",
|
||||
"uid": "UID0079",
|
||||
"category": "easy",
|
||||
"source_files": "treasury_bulletin_2011_12.txt\r\ntreasury_bulletin_2013_12.txt\r\ntreasury_bulletin_2015_12.txt\r\ntreasury_bulletin_2017_12.txt\r\ntreasury_bulletin_2019_12.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/december-2011-7149?page=25\r\n \r\n https://fraser.stlouisfed.org/title/treasury-bulletin-407/december-2013-7155?page=24\r\n \r\n https://fraser.stlouisfed.org/title/treasury-bulletin-407/december-2015-519209?page=23\r\n \r\n https://fraser.stlouisfed.org/title/treasury-bulletin-407/december-2017-575188?page=23\r\n \r\n https://fraser.stlouisfed.org/title/treasury-bulletin-407/december-2019-584842?page=22",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0083",
|
||||
"uid": "UID0083",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_1981_12.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/december-1981-7020?page=24",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0085",
|
||||
"uid": "UID0085",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_2019_12.txt\r\ntreasury_bulletin_2018_12.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/december-2019-584842?page=23\r\n\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/december-2018-581283?page=22",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0087",
|
||||
"uid": "UID0087",
|
||||
"category": "easy",
|
||||
"source_files": "treasury_bulletin_2013_12.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/december-2013-7155?page=17",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0092",
|
||||
"uid": "UID0092",
|
||||
"category": "easy",
|
||||
"source_files": "treasury_bulletin_1987_12.txt\r\ntreasury_bulletin_1992_12.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/december-1987-7051?page=69\r\n\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/december-1992-7071?page=84",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0098",
|
||||
"uid": "UID0098",
|
||||
"category": "easy",
|
||||
"source_files": "treasury_bulletin_2020_12.txt\r\ntreasury_bulletin_2024_12.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/december-2020-598551?page=21\r\n \r\n https://fraser.stlouisfed.org/title/treasury-bulletin-407/december-2024-679984?page=22",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0101",
|
||||
"uid": "UID0101",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_2011_12.txt\r\ntreasury_bulletin_2019_12.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/december-2011-7149?page=25\r\n \r\n https://fraser.stlouisfed.org/title/treasury-bulletin-407/december-2019-584842?page=22",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0110",
|
||||
"uid": "UID0110",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_2020_03.txt\r\ntreasury_bulletin_2016_03.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/march-2020-587316?page=10\r\n \r\n https://fraser.stlouisfed.org/title/treasury-bulletin-407/march-2016-527290?page=9",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0115",
|
||||
"uid": "UID0115",
|
||||
"category": "easy",
|
||||
"source_files": "treasury_bulletin_1980_02.txt\r\ntreasury_bulletin_1981_02.txt\r\ntreasury_bulletin_1982_02.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/february-1980-6998?page=27\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/february-1981-7010?page=38\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/february-1982-7022?page=31",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0122",
|
||||
"uid": "UID0122",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_2001_03.txt\r\ntreasury_bulletin_2002_03.txt\r\ntreasury_bulletin_2003_03.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/march-2001-7105?page=112\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/march-2002-7110?page=115\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/march-2003-7109?page=113",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0123",
|
||||
"uid": "UID0123",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_1941_12.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/december-1941-6540?page=91",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0133",
|
||||
"uid": "UID0133",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_2004_09.txt\r\ntreasury_bulletin_2013_12.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/september-2004-7119?page=63\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/december-2013-7155?page=68",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0141",
|
||||
"uid": "UID0141",
|
||||
"category": "easy",
|
||||
"source_files": "treasury_bulletin_1962_04.txt\r\ntreasury_bulletin_1963_04.txt\r\ntreasury_bulletin_1964_04.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/april-1962-6784?page=75\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/april-1963-6796?page=79\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/april-1964-6808?page=82",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0144",
|
||||
"uid": "UID0144",
|
||||
"category": "easy",
|
||||
"source_files": "treasury_bulletin_1980_11.txt\r\ntreasury_bulletin_1981_11.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/november-1980-7007?page=76\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/november-1981-7019?page=67",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0145",
|
||||
"uid": "UID0145",
|
||||
"category": "easy",
|
||||
"source_files": "treasury_bulletin_1943_01.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/january-1943-6553?page=41",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0150",
|
||||
"uid": "UID0150",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_1972_04.txt\r\ntreasury_bulletin_1973_04.txt\r\ntreasury_bulletin_1974_04.txt\r\ntreasury_bulletin_1975_04.txt\r\ntreasury_bulletin_1976_04.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/april-1972-6905?page=89\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/april-1973-6916?page=91\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/april-1974-6927?page=88\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/april-1975-6940?page=88\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/april-1976-6952?page=104",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0151",
|
||||
"uid": "UID0151",
|
||||
"category": "easy",
|
||||
"source_files": "treasury_bulletin_1953_03.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/march-1953-6674?page=54",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0162",
|
||||
"uid": "UID0162",
|
||||
"category": "easy",
|
||||
"source_files": "treasury_bulletin_2011_06.txt\r\ntreasury_bulletin_2012_06.txt\r\ntreasury_bulletin_2013_06.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/june-2011-7129?page=105\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/june-2012-7151?page=105\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/june-2013-7153?page=104",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0163",
|
||||
"uid": "UID0163",
|
||||
"category": "easy",
|
||||
"source_files": "treasury_bulletin_1981_12.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/december-1981-7020?page=28",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0165",
|
||||
"uid": "UID0165",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_2010_03.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/march-2010-7143?page=49",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0166",
|
||||
"uid": "UID0166",
|
||||
"category": "easy",
|
||||
"source_files": "treasury_bulletin_1943_03.txt\r\ntreasury_bulletin_1944_03.txt\r\ntreasury_bulletin_1945_03.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/march-1943-6555?page=68\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/march-1944-6567?page=83\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/march-1945-6577?page=71",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0169",
|
||||
"uid": "UID0169",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_1982_03.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/march-1982-7023?page=73",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0189",
|
||||
"uid": "UID0189",
|
||||
"category": "easy",
|
||||
"source_files": "treasury_bulletin_1970_08.txt\r\ntreasury_bulletin_1970_09.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/august-1970-6884?page=70\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/september-1970-6885?page=70",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0195",
|
||||
"uid": "UID0195",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_1956_08.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/august-1956-6715?page=59",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0202",
|
||||
"uid": "UID0202",
|
||||
"category": "easy",
|
||||
"source_files": "treasury_bulletin_1939_07.txt\r\ntreasury_bulletin_1939_08.txt\r\ntreasury_bulletin_1939_09.txt\r\ntreasury_bulletin_1939_10.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/july-1939-6509?page=99\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/august-1939-6510?page=107\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/september-1939-6511?page=60\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/october-1939-6520?page=62",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0212",
|
||||
"uid": "UID0212",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_1964_01.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/january-1964-6805?page=99",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0222",
|
||||
"uid": "UID0222",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_2001_06.txt\r\ntreasury_bulletin_2006_06.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/june-2001-7106?page=50\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/june-2006-7126?page=50",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0228",
|
||||
"uid": "UID0228",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_1956_03.txt\r\ntreasury_bulletin_1956_04.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/march-1956-6710?page=22\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/april-1956-6711?page=22",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0229",
|
||||
"uid": "UID0229",
|
||||
"category": "easy",
|
||||
"source_files": "treasury_bulletin_2005_03.txt\r\ntreasury_bulletin_2006_03.txt\r\ntreasury_bulletin_2007_03.txt\r\ntreasury_bulletin_2008_03.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/march-2005-7121?page=109\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/march-2006-7125?page=106\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/march-2007-7130?page=109\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/march-2008-7134?page=107",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0238",
|
||||
"uid": "UID0238",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_1982_03.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/march-1982-7023?page=44",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0241",
|
||||
"uid": "UID0241",
|
||||
"category": "easy",
|
||||
"source_files": "treasury_bulletin_1963_06.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/june-1963-6798?page=13",
|
||||
"source_split": "train"
|
||||
}
|
||||
]
|
||||
@@ -1,194 +0,0 @@
|
||||
[
|
||||
{
|
||||
"id": "UID0001",
|
||||
"uid": "UID0001",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_1941_01.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/january-1941-6529?page=15",
|
||||
"source_split": "val"
|
||||
},
|
||||
{
|
||||
"id": "UID0027",
|
||||
"uid": "UID0027",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_1970_06.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/june-1970-6882?page=89&deep=true",
|
||||
"source_split": "val"
|
||||
},
|
||||
{
|
||||
"id": "UID0039",
|
||||
"uid": "UID0039",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_2004_03.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/march-2004-7117?page=20\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/march-2004-7117?page=21&deep=true",
|
||||
"source_split": "val"
|
||||
},
|
||||
{
|
||||
"id": "UID0041",
|
||||
"uid": "UID0041",
|
||||
"category": "easy",
|
||||
"source_files": "treasury_bulletin_1970_10.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/october-1970-6886?page=35",
|
||||
"source_split": "val"
|
||||
},
|
||||
{
|
||||
"id": "UID0052",
|
||||
"uid": "UID0052",
|
||||
"category": "easy",
|
||||
"source_files": "treasury_bulletin_2000_06.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/june-2000-7102?page=56",
|
||||
"source_split": "val"
|
||||
},
|
||||
{
|
||||
"id": "UID0070",
|
||||
"uid": "UID0070",
|
||||
"category": "easy",
|
||||
"source_files": "treasury_bulletin_1939_01.txt\r\ntreasury_bulletin_1939_02.txt\r\ntreasury_bulletin_1939_03.txt\r\ntreasury_bulletin_1939_04.txt\r\ntreasury_bulletin_1939_05.txt\r\ntreasury_bulletin_1939_06.txt\r\ntreasury_bulletin_1939_07.txt\r\ntreasury_bulletin_1939_08.txt\r\ntreasury_bulletin_1939_09.txt\r\ntreasury_bulletin_1939_10.txt\r\ntreasury_bulletin_1939_11.txt\r\ntreasury_bulletin_1939_12.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/january-1939-6518?page=81\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/february-1939-6505?page=111\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/march-1939-6519?page=117\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/april-1939-6506?page=95\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/may-1939-6507?page=109\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/june-1939-6508?page=117\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/july-1939-6509?page=109\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/august-1939-6510?page=117\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/september-1939-6511?page=66&deep=true \r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/october-1939-6520?page=68\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/november-1939-6512?page=70\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/december-1939-6513?page=72",
|
||||
"source_split": "val"
|
||||
},
|
||||
{
|
||||
"id": "UID0072",
|
||||
"uid": "UID0072",
|
||||
"category": "easy",
|
||||
"source_files": "treasury_bulletin_2011_12.txt\r\ntreasury_bulletin_2016_12.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/december-2011-7149?page=58\r\n\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/december-2016-535293?page=57",
|
||||
"source_split": "val"
|
||||
},
|
||||
{
|
||||
"id": "UID0086",
|
||||
"uid": "UID0086",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_2022_12.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/december-2022-627778?page=86",
|
||||
"source_split": "val"
|
||||
},
|
||||
{
|
||||
"id": "UID0091",
|
||||
"uid": "UID0091",
|
||||
"category": "easy",
|
||||
"source_files": "treasury_bulletin_1940_12.txt\r\ntreasury_bulletin_1941_12.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/december-1940-6528?page=21\r\n\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/december-1941-6540?page=64",
|
||||
"source_split": "val"
|
||||
},
|
||||
{
|
||||
"id": "UID0109",
|
||||
"uid": "UID0109",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_2015_12.txt\r\ntreasury_bulletin_2020_12.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/december-2015-519209?page=21\r\n \r\n https://fraser.stlouisfed.org/title/treasury-bulletin-407/december-2020-598551?page=24",
|
||||
"source_split": "val"
|
||||
},
|
||||
{
|
||||
"id": "UID0142",
|
||||
"uid": "UID0142",
|
||||
"category": "easy",
|
||||
"source_files": "treasury_bulletin_1944_03.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/march-1944-6567?page=93",
|
||||
"source_split": "val"
|
||||
},
|
||||
{
|
||||
"id": "UID0154",
|
||||
"uid": "UID0154",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_1977_03.txt\r\ntreasury_bulletin_1978_03.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/march-1977-6963?page=83\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/march-1978-6975?page=84",
|
||||
"source_split": "val"
|
||||
},
|
||||
{
|
||||
"id": "UID0159",
|
||||
"uid": "UID0159",
|
||||
"category": "easy",
|
||||
"source_files": "treasury_bulletin_2000_09.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/september-2000-7103?page=109",
|
||||
"source_split": "val"
|
||||
},
|
||||
{
|
||||
"id": "UID0161",
|
||||
"uid": "UID0161",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_1980_03.txt\r\ntreasury_bulletin_1985_03.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/march-1980-6999?page=88\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/march-1985-7040?page=48",
|
||||
"source_split": "val"
|
||||
},
|
||||
{
|
||||
"id": "UID0170",
|
||||
"uid": "UID0170",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_1960_03.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/march-1960-6759?page=64",
|
||||
"source_split": "val"
|
||||
},
|
||||
{
|
||||
"id": "UID0190",
|
||||
"uid": "UID0190",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_1939_10.txt\r\ntreasury_bulletin_1939_11.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/october-1939-6520?page=14\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/november-1939-6512?page=14",
|
||||
"source_split": "val"
|
||||
},
|
||||
{
|
||||
"id": "UID0213",
|
||||
"uid": "UID0213",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_1947_04.txt\r\ntreasury_bulletin_1948_04.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/april-1947-6603?page=28\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/april-1948-6615?page=18",
|
||||
"source_split": "val"
|
||||
},
|
||||
{
|
||||
"id": "UID0217",
|
||||
"uid": "UID0217",
|
||||
"category": "easy",
|
||||
"source_files": "treasury_bulletin_1963_10.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/october-1963-6802?page=15",
|
||||
"source_split": "val"
|
||||
},
|
||||
{
|
||||
"id": "UID0220",
|
||||
"uid": "UID0220",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_1939_02.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/february-1939-6505?page=25",
|
||||
"source_split": "val"
|
||||
},
|
||||
{
|
||||
"id": "UID0233",
|
||||
"uid": "UID0233",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_1948_04.txt\r\ntreasury_bulletin_1958_04.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/april-1948-6615?page=42\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/april-1958-6735?page=54",
|
||||
"source_split": "val"
|
||||
},
|
||||
{
|
||||
"id": "UID0234",
|
||||
"uid": "UID0234",
|
||||
"category": "easy",
|
||||
"source_files": "treasury_bulletin_1958_01.txt\r\ntreasury_bulletin_1958_02.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/january-1958-6732?page=28\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/february-1958-6733?page=32",
|
||||
"source_split": "val"
|
||||
},
|
||||
{
|
||||
"id": "UID0235",
|
||||
"uid": "UID0235",
|
||||
"category": "easy",
|
||||
"source_files": "treasury_bulletin_1948_04.txt\r\ntreasury_bulletin_1948_05.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/april-1948-6615?page=27\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/may-1948-6616?page=27",
|
||||
"source_split": "val"
|
||||
},
|
||||
{
|
||||
"id": "UID0239",
|
||||
"uid": "UID0239",
|
||||
"category": "easy",
|
||||
"source_files": "treasury_bulletin_1953_01.txt\r\ntreasury_bulletin_1954_01.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/january-1953-6672?page=62\r\nhttp://fraser.stlouisfed.org/title/treasury-bulletin-407/january-1954-6684?page=51",
|
||||
"source_split": "val"
|
||||
},
|
||||
{
|
||||
"id": "UID0240",
|
||||
"uid": "UID0240",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_1957_12.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/december-1957-6731?page=26",
|
||||
"source_split": "val"
|
||||
}
|
||||
]
|
||||
@@ -1,21 +0,0 @@
|
||||
{
|
||||
"benchmark": "SearchQA",
|
||||
"manifest_type": "id_split",
|
||||
"source_repo": "lucadiliello/searchqa",
|
||||
"source_repo_type": "dataset",
|
||||
"source_url": "https://huggingface.co/datasets/lucadiliello/searchqa",
|
||||
"source_id_field": "key",
|
||||
"counts": {
|
||||
"train": 400,
|
||||
"val": 200,
|
||||
"test": 1400
|
||||
},
|
||||
"item_fields": [
|
||||
"id"
|
||||
],
|
||||
"notes": [
|
||||
"This is a split manifest, not the full SearchQA payload.",
|
||||
"Materialize full split items from lucadiliello/searchqa before evaluation.",
|
||||
"The IDs in items.json exactly match the key field in lucadiliello/searchqa."
|
||||
]
|
||||
}
|
||||
@@ -1,24 +0,0 @@
|
||||
{
|
||||
"benchmark": "SpreadsheetBench",
|
||||
"manifest_type": "id_split",
|
||||
"source_repo": "KAKA22/SpreadsheetBench",
|
||||
"source_repo_type": "dataset",
|
||||
"source_url": "https://huggingface.co/datasets/KAKA22/SpreadsheetBench",
|
||||
"source_revision": "ab0b742b0fc95b946f212d80ac7771b5531272e4",
|
||||
"source_file": "spreadsheetbench_verified_400.tar.gz",
|
||||
"source_split_name": "spreadsheetbench_split",
|
||||
"counts": {
|
||||
"train": 80,
|
||||
"val": 40,
|
||||
"test": 280
|
||||
},
|
||||
"item_fields": [
|
||||
"id",
|
||||
"spreadsheet_path",
|
||||
"instruction_type"
|
||||
],
|
||||
"notes": [
|
||||
"This is a split manifest, not the full SpreadsheetBench payload.",
|
||||
"Materialize full task JSON rows plus spreadsheet files from SpreadsheetBench Verified 400 before evaluation."
|
||||
]
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,402 +0,0 @@
|
||||
[
|
||||
{
|
||||
"id": "32438",
|
||||
"spreadsheet_path": "spreadsheet/32438",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "398-14",
|
||||
"spreadsheet_path": "spreadsheet/398-14",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "47766",
|
||||
"spreadsheet_path": "spreadsheet/47766",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "48365",
|
||||
"spreadsheet_path": "spreadsheet/48365",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "32255",
|
||||
"spreadsheet_path": "spreadsheet/32255",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "10747",
|
||||
"spreadsheet_path": "spreadsheet/10747",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "50916",
|
||||
"spreadsheet_path": "spreadsheet/50916",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "577-40",
|
||||
"spreadsheet_path": "spreadsheet/577-40",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "35742",
|
||||
"spreadsheet_path": "spreadsheet/35742",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "46121",
|
||||
"spreadsheet_path": "spreadsheet/46121",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "51090",
|
||||
"spreadsheet_path": "spreadsheet/51090",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "51249",
|
||||
"spreadsheet_path": "spreadsheet/51249",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "82-30",
|
||||
"spreadsheet_path": "spreadsheet/82-30",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "56274",
|
||||
"spreadsheet_path": "spreadsheet/56274",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "57445",
|
||||
"spreadsheet_path": "spreadsheet/57445",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "46646",
|
||||
"spreadsheet_path": "spreadsheet/46646",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "105-24",
|
||||
"spreadsheet_path": "spreadsheet/105-24",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "6239",
|
||||
"spreadsheet_path": "spreadsheet/6239",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "414-20",
|
||||
"spreadsheet_path": "spreadsheet/414-20",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "165-23",
|
||||
"spreadsheet_path": "spreadsheet/165-23",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "40892",
|
||||
"spreadsheet_path": "spreadsheet/40892",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "48745",
|
||||
"spreadsheet_path": "spreadsheet/48745",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "32612",
|
||||
"spreadsheet_path": "spreadsheet/32612",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "325-44",
|
||||
"spreadsheet_path": "spreadsheet/325-44",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "262-17",
|
||||
"spreadsheet_path": "spreadsheet/262-17",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "141-20",
|
||||
"spreadsheet_path": "spreadsheet/141-20",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "52216",
|
||||
"spreadsheet_path": "spreadsheet/52216",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "22-47",
|
||||
"spreadsheet_path": "spreadsheet/22-47",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "55421",
|
||||
"spreadsheet_path": "spreadsheet/55421",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "56427",
|
||||
"spreadsheet_path": "spreadsheet/56427",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "36097",
|
||||
"spreadsheet_path": "spreadsheet/36097",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "32902",
|
||||
"spreadsheet_path": "spreadsheet/32902",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "32023",
|
||||
"spreadsheet_path": "spreadsheet/32023",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "1818",
|
||||
"spreadsheet_path": "spreadsheet/1818",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "170-13",
|
||||
"spreadsheet_path": "spreadsheet/170-13",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "66-24",
|
||||
"spreadsheet_path": "spreadsheet/66-24",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "58949",
|
||||
"spreadsheet_path": "spreadsheet/58949",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "42354",
|
||||
"spreadsheet_path": "spreadsheet/42354",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "194-19",
|
||||
"spreadsheet_path": "spreadsheet/194-19",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "31915",
|
||||
"spreadsheet_path": "spreadsheet/31915",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "58499",
|
||||
"spreadsheet_path": "spreadsheet/58499",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "45372",
|
||||
"spreadsheet_path": "spreadsheet/45372",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "11842",
|
||||
"spreadsheet_path": "spreadsheet/11842",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "57558",
|
||||
"spreadsheet_path": "spreadsheet/57558",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "472-15",
|
||||
"spreadsheet_path": "spreadsheet/472-15",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "55060",
|
||||
"spreadsheet_path": "spreadsheet/55060",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "31011",
|
||||
"spreadsheet_path": "spreadsheet/31011",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "408-39",
|
||||
"spreadsheet_path": "spreadsheet/408-39",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "54085",
|
||||
"spreadsheet_path": "spreadsheet/54085",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "39903",
|
||||
"spreadsheet_path": "spreadsheet/39903",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "48983",
|
||||
"spreadsheet_path": "spreadsheet/48983",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "108-24",
|
||||
"spreadsheet_path": "spreadsheet/108-24",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "58484",
|
||||
"spreadsheet_path": "spreadsheet/58484",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "118-50",
|
||||
"spreadsheet_path": "spreadsheet/118-50",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "10452",
|
||||
"spreadsheet_path": "spreadsheet/10452",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "39931",
|
||||
"spreadsheet_path": "spreadsheet/39931",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "3413",
|
||||
"spreadsheet_path": "spreadsheet/3413",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "247-24",
|
||||
"spreadsheet_path": "spreadsheet/247-24",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "56786",
|
||||
"spreadsheet_path": "spreadsheet/56786",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "55965",
|
||||
"spreadsheet_path": "spreadsheet/55965",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "379-36",
|
||||
"spreadsheet_path": "spreadsheet/379-36",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "58109",
|
||||
"spreadsheet_path": "spreadsheet/58109",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "433-47",
|
||||
"spreadsheet_path": "spreadsheet/433-47",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "192-22",
|
||||
"spreadsheet_path": "spreadsheet/192-22",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "49333",
|
||||
"spreadsheet_path": "spreadsheet/49333",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "493-18",
|
||||
"spreadsheet_path": "spreadsheet/493-18",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "54638",
|
||||
"spreadsheet_path": "spreadsheet/54638",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "34033",
|
||||
"spreadsheet_path": "spreadsheet/34033",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "30930",
|
||||
"spreadsheet_path": "spreadsheet/30930",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "585-41",
|
||||
"spreadsheet_path": "spreadsheet/585-41",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "32337",
|
||||
"spreadsheet_path": "spreadsheet/32337",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "55427",
|
||||
"spreadsheet_path": "spreadsheet/55427",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "263-1",
|
||||
"spreadsheet_path": "spreadsheet/263-1",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "254-34",
|
||||
"spreadsheet_path": "spreadsheet/254-34",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "57113",
|
||||
"spreadsheet_path": "spreadsheet/57113",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "57743",
|
||||
"spreadsheet_path": "spreadsheet/57743",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "43589",
|
||||
"spreadsheet_path": "spreadsheet/43589",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "250-20",
|
||||
"spreadsheet_path": "spreadsheet/250-20",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "48080",
|
||||
"spreadsheet_path": "spreadsheet/48080",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "370-43",
|
||||
"spreadsheet_path": "spreadsheet/370-43",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
}
|
||||
]
|
||||
@@ -1,202 +0,0 @@
|
||||
[
|
||||
{
|
||||
"id": "45635",
|
||||
"spreadsheet_path": "spreadsheet/45635",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "560-12",
|
||||
"spreadsheet_path": "spreadsheet/560-12",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "55049",
|
||||
"spreadsheet_path": "spreadsheet/55049",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "9569",
|
||||
"spreadsheet_path": "spreadsheet/9569",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "7902",
|
||||
"spreadsheet_path": "spreadsheet/7902",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "227-40",
|
||||
"spreadsheet_path": "spreadsheet/227-40",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "463-17",
|
||||
"spreadsheet_path": "spreadsheet/463-17",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "54144",
|
||||
"spreadsheet_path": "spreadsheet/54144",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "80-42",
|
||||
"spreadsheet_path": "spreadsheet/80-42",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "2768",
|
||||
"spreadsheet_path": "spreadsheet/2768",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "37456",
|
||||
"spreadsheet_path": "spreadsheet/37456",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "12864",
|
||||
"spreadsheet_path": "spreadsheet/12864",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "55979",
|
||||
"spreadsheet_path": "spreadsheet/55979",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "48620",
|
||||
"spreadsheet_path": "spreadsheet/48620",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "48588",
|
||||
"spreadsheet_path": "spreadsheet/48588",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "395-36",
|
||||
"spreadsheet_path": "spreadsheet/395-36",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "382-10",
|
||||
"spreadsheet_path": "spreadsheet/382-10",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "59595",
|
||||
"spreadsheet_path": "spreadsheet/59595",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "53383",
|
||||
"spreadsheet_path": "spreadsheet/53383",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "48921",
|
||||
"spreadsheet_path": "spreadsheet/48921",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "416-15",
|
||||
"spreadsheet_path": "spreadsheet/416-15",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "47798",
|
||||
"spreadsheet_path": "spreadsheet/47798",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "56563",
|
||||
"spreadsheet_path": "spreadsheet/56563",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "46897",
|
||||
"spreadsheet_path": "spreadsheet/46897",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "9726",
|
||||
"spreadsheet_path": "spreadsheet/9726",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "50768",
|
||||
"spreadsheet_path": "spreadsheet/50768",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "51-12",
|
||||
"spreadsheet_path": "spreadsheet/51-12",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "31628",
|
||||
"spreadsheet_path": "spreadsheet/31628",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "39046",
|
||||
"spreadsheet_path": "spreadsheet/39046",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "8942",
|
||||
"spreadsheet_path": "spreadsheet/8942",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "48527",
|
||||
"spreadsheet_path": "spreadsheet/48527",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "59196",
|
||||
"spreadsheet_path": "spreadsheet/59196",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "6698",
|
||||
"spreadsheet_path": "spreadsheet/6698",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "43436",
|
||||
"spreadsheet_path": "spreadsheet/43436",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "38462",
|
||||
"spreadsheet_path": "spreadsheet/38462",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "402-43",
|
||||
"spreadsheet_path": "spreadsheet/402-43",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "267-18",
|
||||
"spreadsheet_path": "spreadsheet/267-18",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "37378",
|
||||
"spreadsheet_path": "spreadsheet/37378",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "53647",
|
||||
"spreadsheet_path": "spreadsheet/53647",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "142-12",
|
||||
"spreadsheet_path": "spreadsheet/142-12",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
}
|
||||
]
|
||||
@@ -25,10 +25,10 @@ Open an issue with:
|
||||
See [Add a New Benchmark](guide/new-benchmark.md) for the implementation guide.
|
||||
|
||||
**Checklist:**
|
||||
- [ ] Data loader in `skillopt/envs/<benchmark>/dataloader.py`
|
||||
- [ ] Environment adapter in `skillopt/envs/<benchmark>/adapter.py`
|
||||
- [ ] Data loader in `skillopt/envs/<benchmark>/loader.py`
|
||||
- [ ] Environment adapter in `skillopt/envs/<benchmark>/env.py`
|
||||
- [ ] Config file in `configs/<benchmark>/default.yaml`
|
||||
- [ ] Registration in `scripts/train.py` (`_ENV_REGISTRY`)
|
||||
- [ ] Registration in `skillopt/envs/__init__.py`
|
||||
- [ ] Documentation page in `docs/`
|
||||
|
||||
### 🤖 New Model Backend
|
||||
|
||||
+107
-56
@@ -6,10 +6,9 @@ Extend SkillOpt with your own benchmark in ~100 lines of code.
|
||||
|
||||
To add a benchmark, you need:
|
||||
|
||||
1. **Data Loader** — Subclass `SplitDataLoader` to load your split data
|
||||
2. **Environment Adapter** — Subclass `EnvAdapter` and implement rollout/reflect hooks
|
||||
1. **Data Loader** — Loads and splits your dataset
|
||||
2. **Environment Adapter** — Executes tasks and returns scores
|
||||
3. **Config** — YAML configuration file
|
||||
4. **Registration** — Add your adapter to the train script registry
|
||||
|
||||
## Step 1: Create the Benchmark Package
|
||||
|
||||
@@ -20,73 +19,126 @@ touch skillopt/envs/my_benchmark/__init__.py
|
||||
|
||||
## Step 2: Implement the Data Loader
|
||||
|
||||
Create `skillopt/envs/my_benchmark/dataloader.py`:
|
||||
Create `skillopt/envs/my_benchmark/loader.py`:
|
||||
|
||||
```python
|
||||
from skillopt.datasets.base import SplitDataLoader
|
||||
from skillopt.data.base import DataLoader, DataItem
|
||||
|
||||
|
||||
class MyBenchmarkDataLoader(SplitDataLoader):
|
||||
"""Load benchmark items from raw data and/or split directories."""
|
||||
|
||||
def load_raw_items(self, data_path: str) -> list[dict]:
|
||||
# For ratio mode, parse your source dataset from data_path.
|
||||
# Return list[dict] where each item has at least a unique, deterministic "id".
|
||||
return super().load_raw_items(data_path)
|
||||
|
||||
def load_split_items(self, split_path: str) -> list[dict]:
|
||||
# For split_dir mode, parse one split directory.
|
||||
return super().load_split_items(split_path)
|
||||
class MyBenchmarkDataLoader(DataLoader):
|
||||
"""Load and split your benchmark data."""
|
||||
|
||||
def __init__(self, data_dir: str, **kwargs):
|
||||
super().__init__(**kwargs)
|
||||
self.data_dir = data_dir
|
||||
|
||||
def setup(self, cfg: dict):
|
||||
"""Initialize splits based on config."""
|
||||
self.split_mode = cfg.get('split_mode', 'ratio')
|
||||
# Load your data here
|
||||
self.items = self._load_items()
|
||||
self._create_splits(cfg)
|
||||
|
||||
def _load_items(self) -> list[DataItem]:
|
||||
"""Load raw data into DataItem objects."""
|
||||
items = []
|
||||
# TODO: Load your data
|
||||
for entry in your_data:
|
||||
items.append(DataItem(
|
||||
id=entry['id'],
|
||||
input=entry['question'],
|
||||
ground_truth=entry['answer'],
|
||||
metadata=entry.get('metadata', {})
|
||||
))
|
||||
return items
|
||||
|
||||
def get_split_items(self, split: str) -> list[DataItem]:
|
||||
"""Return items for a given split (train/valid/test)."""
|
||||
return self.splits[split]
|
||||
```
|
||||
|
||||
## Step 3: Implement the Environment Adapter
|
||||
|
||||
Create `skillopt/envs/my_benchmark/adapter.py`:
|
||||
Create `skillopt/envs/my_benchmark/env.py`:
|
||||
|
||||
```python
|
||||
from skillopt.envs.base import EnvAdapter
|
||||
from skillopt.envs.my_benchmark.dataloader import MyBenchmarkDataLoader
|
||||
from skillopt.envs.base import EnvAdapter, TaskResult
|
||||
|
||||
class MyBenchmarkAdapter(EnvAdapter):
|
||||
def __init__(self, split_dir: str = "", data_path: str = "", **kwargs):
|
||||
self.dataloader = MyBenchmarkDataLoader(split_dir=split_dir, data_path=data_path, **kwargs)
|
||||
|
||||
def setup(self, cfg: dict) -> None:
|
||||
super().setup(cfg)
|
||||
self.dataloader.setup(cfg)
|
||||
|
||||
def get_dataloader(self):
|
||||
return self.dataloader
|
||||
|
||||
def build_train_env(self, batch_size: int, seed: int, **kwargs):
|
||||
return self.dataloader.build_train_batch(batch_size=batch_size, seed=seed, **kwargs).payload
|
||||
|
||||
def build_eval_env(self, env_num: int, split: str, seed: int, **kwargs):
|
||||
return self.dataloader.build_eval_batch(env_num=env_num, split=split, seed=seed, **kwargs).payload
|
||||
|
||||
def rollout(self, env_manager, skill_content: str, out_dir: str, **kwargs) -> list[dict]:
|
||||
# env_manager is the payload returned by build_train_env/build_eval_env
|
||||
# (commonly list[dict] task items).
|
||||
# Run target model on each item and return list[dict].
|
||||
# Required keys per row: "id", "hard" (0/1), "soft" (0.0-1.0)
|
||||
raise NotImplementedError
|
||||
|
||||
def reflect(self, results: list[dict], skill_content: str, out_dir: str, **kwargs) -> list[dict | None]:
|
||||
# Convert failure/success analysis into RawPatch-like dicts.
|
||||
raise NotImplementedError
|
||||
|
||||
def get_task_types(self) -> list[str]:
|
||||
return ["my_benchmark"]
|
||||
class MyBenchmarkEnv(EnvAdapter):
|
||||
"""Execute tasks and evaluate results."""
|
||||
|
||||
def __init__(self, cfg: dict):
|
||||
super().__init__(cfg)
|
||||
|
||||
async def execute(self, item: DataItem, skill: str, model) -> TaskResult:
|
||||
"""
|
||||
Execute a single task.
|
||||
|
||||
Args:
|
||||
item: The data item to process
|
||||
skill: Current skill document content
|
||||
model: The target model instance
|
||||
|
||||
Returns:
|
||||
TaskResult with prediction, score, and trajectory
|
||||
"""
|
||||
# Build prompt with skill document
|
||||
prompt = self.build_prompt(item, skill)
|
||||
|
||||
# Get model response
|
||||
response = await model.generate(prompt)
|
||||
|
||||
# Extract prediction
|
||||
prediction = self.parse_response(response)
|
||||
|
||||
# Score against ground truth
|
||||
score = self.evaluate(prediction, item.ground_truth)
|
||||
|
||||
return TaskResult(
|
||||
item_id=item.id,
|
||||
prediction=prediction,
|
||||
score=score,
|
||||
trajectory=[
|
||||
{"role": "system", "content": skill},
|
||||
{"role": "user", "content": item.input},
|
||||
{"role": "assistant", "content": response}
|
||||
]
|
||||
)
|
||||
|
||||
def evaluate(self, prediction: str, ground_truth: str) -> float:
|
||||
"""
|
||||
Score a prediction against ground truth.
|
||||
|
||||
Returns:
|
||||
Float between 0.0 and 1.0
|
||||
"""
|
||||
# TODO: Implement your scoring logic
|
||||
# Examples: exact match, F1, ANLS, etc.
|
||||
return float(prediction.strip() == ground_truth.strip())
|
||||
|
||||
def build_prompt(self, item, skill: str) -> str:
|
||||
"""Combine skill document with task input."""
|
||||
return f"{skill}\n\n---\n\nQuestion: {item.input}"
|
||||
|
||||
def parse_response(self, response: str) -> str:
|
||||
"""Extract the answer from model response."""
|
||||
return response.strip()
|
||||
```
|
||||
|
||||
## Step 4: Register the Benchmark
|
||||
|
||||
Add your adapter to `_register_builtins()` in `scripts/train.py`:
|
||||
Add to `skillopt/envs/__init__.py`:
|
||||
|
||||
```python
|
||||
from skillopt.envs.my_benchmark.adapter import MyBenchmarkAdapter
|
||||
from .my_benchmark.env import MyBenchmarkEnv
|
||||
from .my_benchmark.loader import MyBenchmarkDataLoader
|
||||
|
||||
_ENV_REGISTRY["my_benchmark"] = MyBenchmarkAdapter
|
||||
BENCHMARK_REGISTRY = {
|
||||
# ... existing benchmarks ...
|
||||
'my_benchmark': {
|
||||
'env': MyBenchmarkEnv,
|
||||
'loader': MyBenchmarkDataLoader,
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
## Step 5: Create Config
|
||||
@@ -94,7 +146,7 @@ _ENV_REGISTRY["my_benchmark"] = MyBenchmarkAdapter
|
||||
Create `configs/my_benchmark/default.yaml`:
|
||||
|
||||
```yaml
|
||||
_base_: ../_base_/default.yaml
|
||||
_base_: ['../_base_/default.yaml']
|
||||
|
||||
env:
|
||||
name: my_benchmark
|
||||
@@ -126,5 +178,4 @@ python scripts/train.py --config configs/my_benchmark/default.yaml
|
||||
|
||||
!!! tip
|
||||
- Use a small `batch_size` (10-20) for initial testing
|
||||
- Start from `skillopt/envs/_template/` and adapt from there
|
||||
- Use an existing adapter (for example `skillopt/envs/officeqa/adapter.py`) as a concrete reference
|
||||
- The `evaluate()` method is critical — a noisy metric will confuse the optimizer
|
||||
|
||||
@@ -0,0 +1,911 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>SkillOpt — Documentation & Reproduction Guide</title>
|
||||
<meta name="description" content="Complete documentation and reproduction guide for SkillOpt: installation, data preparation, training, configuration reference, framework internals, and API reference.">
|
||||
<link rel="icon" type="image/svg+xml" href="data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 23 23'%3E%3Crect width='10' height='10' fill='%23F25022'/%3E%3Crect x='13' width='10' height='10' fill='%237FBA00'/%3E%3Crect y='13' width='10' height='10' fill='%2300A4EF'/%3E%3Crect x='13' y='13' width='10' height='10' fill='%23FFB900'/%3E%3C/svg%3E">
|
||||
<style>
|
||||
:root {
|
||||
--bg: #ffffff;
|
||||
--bg-soft: #f7f8fb;
|
||||
--sidebar-bg: #fbfcfe;
|
||||
--ink: #1f2733;
|
||||
--muted: #5b6675;
|
||||
--quiet: #8a94a3;
|
||||
--line: #e6e9ef;
|
||||
--line-strong: #d3d9e3;
|
||||
--brand: #4f46e5;
|
||||
--brand-soft: #eef0fe;
|
||||
--accent: #0ea5e9;
|
||||
--green: #16a34a;
|
||||
--amber: #d97706;
|
||||
--red: #dc2626;
|
||||
--code-bg: #0f172a;
|
||||
--code-ink: #e2e8f0;
|
||||
--inline-code-bg: #eef1f6;
|
||||
--inline-code-ink: #b3146b;
|
||||
--sidebar-w: 300px;
|
||||
--toc-w: 220px;
|
||||
--mono: "SFMono-Regular", "JetBrains Mono", Consolas, "Liberation Mono", monospace;
|
||||
--sans: "Inter", -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif;
|
||||
}
|
||||
* { box-sizing: border-box; }
|
||||
html { scroll-behavior: smooth; }
|
||||
body {
|
||||
margin: 0;
|
||||
font-family: var(--sans);
|
||||
color: var(--ink);
|
||||
background: var(--bg);
|
||||
font-size: 15px;
|
||||
line-height: 1.65;
|
||||
-webkit-font-smoothing: antialiased;
|
||||
}
|
||||
|
||||
/* ── Top bar ─────────────────────────────────────────── */
|
||||
header.topbar {
|
||||
position: sticky; top: 0; z-index: 40;
|
||||
height: 56px;
|
||||
display: flex; align-items: center; gap: 14px;
|
||||
padding: 0 20px;
|
||||
background: rgba(255,255,255,0.92);
|
||||
backdrop-filter: blur(8px);
|
||||
border-bottom: 1px solid var(--line);
|
||||
}
|
||||
.topbar .logo { width: 22px; height: 22px; flex: none; }
|
||||
.topbar .brand { font-weight: 700; font-size: 16px; letter-spacing: -0.01em; }
|
||||
.topbar .brand span { color: var(--brand); }
|
||||
.topbar .tag { color: var(--quiet); font-size: 13px; border-left: 1px solid var(--line-strong); padding-left: 14px; }
|
||||
.topbar .spacer { flex: 1; }
|
||||
.topbar a.gh {
|
||||
display: inline-flex; align-items: center; gap: 6px;
|
||||
font-size: 13px; font-weight: 600; color: var(--muted);
|
||||
text-decoration: none; padding: 6px 12px; border: 1px solid var(--line-strong);
|
||||
border-radius: 8px;
|
||||
}
|
||||
.topbar a.gh:hover { color: var(--brand); border-color: var(--brand); }
|
||||
#menuBtn {
|
||||
display: none; background: none; border: 1px solid var(--line-strong);
|
||||
border-radius: 8px; width: 38px; height: 34px; cursor: pointer; font-size: 18px; color: var(--muted);
|
||||
}
|
||||
|
||||
/* ── Layout ──────────────────────────────────────────── */
|
||||
.layout { display: flex; align-items: flex-start; }
|
||||
|
||||
/* ── Sidebar (left nav) ──────────────────────────────── */
|
||||
nav.sidebar {
|
||||
position: sticky; top: 56px;
|
||||
width: var(--sidebar-w); flex: none;
|
||||
height: calc(100vh - 56px);
|
||||
overflow-y: auto;
|
||||
background: var(--sidebar-bg);
|
||||
border-right: 1px solid var(--line);
|
||||
padding: 22px 14px 60px 20px;
|
||||
}
|
||||
nav.sidebar .group { margin-bottom: 22px; }
|
||||
nav.sidebar .group > .glabel {
|
||||
display: flex; align-items: center; gap: 8px;
|
||||
font-size: 11.5px; font-weight: 700; text-transform: uppercase;
|
||||
letter-spacing: 0.07em; color: var(--quiet);
|
||||
margin: 0 0 8px 2px;
|
||||
}
|
||||
nav.sidebar .group > .glabel .num {
|
||||
display: inline-flex; align-items: center; justify-content: center;
|
||||
width: 18px; height: 18px; border-radius: 5px;
|
||||
background: var(--brand-soft); color: var(--brand);
|
||||
font-size: 11px; font-weight: 700;
|
||||
}
|
||||
nav.sidebar a {
|
||||
display: block; text-decoration: none;
|
||||
color: var(--muted); font-size: 13.5px;
|
||||
padding: 5px 10px; border-radius: 7px; margin: 1px 0;
|
||||
border-left: 2px solid transparent;
|
||||
}
|
||||
nav.sidebar a:hover { background: #eef1f6; color: var(--ink); }
|
||||
nav.sidebar a.active {
|
||||
color: var(--brand); background: var(--brand-soft);
|
||||
border-left-color: var(--brand); font-weight: 600;
|
||||
}
|
||||
|
||||
/* ── Content ─────────────────────────────────────────── */
|
||||
main.content {
|
||||
flex: 1; min-width: 0;
|
||||
padding: 38px 46px 120px;
|
||||
max-width: 900px;
|
||||
}
|
||||
main.content section { scroll-margin-top: 72px; }
|
||||
main h1 { font-size: 30px; line-height: 1.2; letter-spacing: -0.02em; margin: 0 0 8px; }
|
||||
main h2 {
|
||||
font-size: 23px; letter-spacing: -0.015em; margin: 52px 0 14px;
|
||||
padding-bottom: 10px; border-bottom: 1px solid var(--line);
|
||||
}
|
||||
main section:first-of-type h2 { margin-top: 8px; }
|
||||
main h3 { font-size: 17.5px; margin: 30px 0 10px; letter-spacing: -0.01em; }
|
||||
main h4 { font-size: 15px; margin: 22px 0 8px; color: var(--ink); }
|
||||
main p { margin: 12px 0; color: #2c3645; }
|
||||
main ul, main ol { margin: 12px 0; padding-left: 22px; }
|
||||
main li { margin: 5px 0; }
|
||||
main a { color: var(--brand); text-decoration: none; }
|
||||
main a:hover { text-decoration: underline; }
|
||||
.lead { font-size: 16.5px; color: var(--muted); margin: 6px 0 4px; }
|
||||
.eyebrow { color: var(--brand); font-weight: 700; font-size: 12.5px; letter-spacing: 0.08em; text-transform: uppercase; }
|
||||
|
||||
/* code */
|
||||
code {
|
||||
font-family: var(--mono); font-size: 0.86em;
|
||||
background: var(--inline-code-bg); color: var(--inline-code-ink);
|
||||
padding: 2px 6px; border-radius: 5px;
|
||||
}
|
||||
pre {
|
||||
background: var(--code-bg); color: var(--code-ink);
|
||||
border-radius: 12px; padding: 16px 18px; overflow-x: auto;
|
||||
font-family: var(--mono); font-size: 13px; line-height: 1.6;
|
||||
margin: 14px 0; border: 1px solid #1e293b;
|
||||
}
|
||||
pre code { background: none; color: inherit; padding: 0; font-size: inherit; }
|
||||
.tok-c { color: #7c8aa5; } /* comment */
|
||||
.tok-k { color: #c4b5fd; } /* keyword */
|
||||
.tok-s { color: #86efac; } /* string */
|
||||
.tok-f { color: #93c5fd; } /* flag/path */
|
||||
.tok-n { color: #fca5a5; } /* number/value */
|
||||
|
||||
/* tables */
|
||||
.table-wrap { overflow-x: auto; margin: 16px 0; border: 1px solid var(--line); border-radius: 12px; }
|
||||
table { border-collapse: collapse; width: 100%; font-size: 13.5px; }
|
||||
th, td { text-align: left; padding: 9px 13px; border-bottom: 1px solid var(--line); vertical-align: top; }
|
||||
thead th { background: var(--bg-soft); font-weight: 700; color: var(--ink); white-space: nowrap; }
|
||||
tbody tr:last-child td { border-bottom: none; }
|
||||
td code { white-space: nowrap; }
|
||||
td.def { color: var(--muted); font-family: var(--mono); font-size: 12px; }
|
||||
|
||||
/* callouts */
|
||||
.note { border-radius: 10px; padding: 12px 16px; margin: 16px 0; border: 1px solid; font-size: 14px; }
|
||||
.note p { margin: 4px 0; }
|
||||
.note .nh { font-weight: 700; display: block; margin-bottom: 2px; }
|
||||
.note.info { background: #eff6ff; border-color: #bfdbfe; }
|
||||
.note.info .nh { color: #1d4ed8; }
|
||||
.note.tip { background: #ecfdf5; border-color: #a7f3d0; }
|
||||
.note.tip .nh { color: #047857; }
|
||||
.note.warn { background: #fffbeb; border-color: #fde68a; }
|
||||
.note.warn .nh { color: #b45309; }
|
||||
|
||||
.pill { display:inline-block; font-size: 11px; font-weight:700; padding: 1px 8px; border-radius: 999px; vertical-align: middle; }
|
||||
.pill.def { background:#eef2ff; color:#4338ca; }
|
||||
.pill.opt { background:#f1f5f9; color:#475569; }
|
||||
|
||||
/* card grid */
|
||||
.cards { display: grid; grid-template-columns: repeat(auto-fit, minmax(230px,1fr)); gap: 14px; margin: 18px 0; }
|
||||
.card { border: 1px solid var(--line); border-radius: 12px; padding: 16px; background: var(--bg-soft); }
|
||||
.card h4 { margin: 0 0 6px; font-size: 14.5px; }
|
||||
.card p { margin: 0; font-size: 13px; color: var(--muted); }
|
||||
|
||||
/* anchor link on hover */
|
||||
.anchor { color: var(--quiet); text-decoration: none; font-weight: 400; opacity: 0; margin-left: 8px; font-size: 0.8em; }
|
||||
h2:hover .anchor, h3:hover .anchor { opacity: 1; }
|
||||
|
||||
/* ── Right TOC ───────────────────────────────────────── */
|
||||
aside.toc {
|
||||
position: sticky; top: 56px;
|
||||
width: var(--toc-w); flex: none;
|
||||
height: calc(100vh - 56px); overflow-y: auto;
|
||||
padding: 38px 18px; border-left: 1px solid var(--line);
|
||||
}
|
||||
aside.toc .tl { font-size: 11.5px; font-weight: 700; text-transform: uppercase; letter-spacing: 0.07em; color: var(--quiet); margin-bottom: 10px; }
|
||||
aside.toc a { display: block; color: var(--muted); text-decoration: none; font-size: 12.5px; padding: 4px 8px; border-left: 2px solid var(--line); line-height: 1.45; }
|
||||
aside.toc a:hover { color: var(--ink); }
|
||||
aside.toc a.active { color: var(--brand); border-left-color: var(--brand); font-weight: 600; }
|
||||
|
||||
.footer-note { margin-top: 60px; padding-top: 20px; border-top: 1px solid var(--line); color: var(--quiet); font-size: 13px; }
|
||||
|
||||
/* responsive */
|
||||
@media (max-width: 1180px) { aside.toc { display: none; } }
|
||||
@media (max-width: 860px) {
|
||||
#menuBtn { display: inline-block; }
|
||||
nav.sidebar {
|
||||
position: fixed; left: 0; top: 56px; z-index: 35;
|
||||
transform: translateX(-100%); transition: transform 0.22s ease;
|
||||
box-shadow: 0 16px 40px rgba(15,23,42,0.18);
|
||||
}
|
||||
nav.sidebar.open { transform: translateX(0); }
|
||||
main.content { padding: 28px 20px 100px; }
|
||||
.topbar .tag { display: none; }
|
||||
}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
|
||||
<header class="topbar">
|
||||
<button id="menuBtn" aria-label="Toggle navigation">☰</button>
|
||||
<svg class="logo" viewBox="0 0 23 23"><rect width="10" height="10" fill="#F25022"/><rect x="13" width="10" height="10" fill="#7FBA00"/><rect y="13" width="10" height="10" fill="#00A4EF"/><rect x="13" y="13" width="10" height="10" fill="#FFB900"/></svg>
|
||||
<span class="brand">Skill<span>Opt</span></span>
|
||||
<span class="tag">Documentation & Reproduction Guide</span>
|
||||
<span class="spacer"></span>
|
||||
<a class="gh" href="https://github.com/microsoft/SkillOpt" target="_blank" rel="noopener">GitHub ↗</a>
|
||||
<a class="gh" href="https://arxiv.org/abs/2605.23904" target="_blank" rel="noopener">Paper ↗</a>
|
||||
</header>
|
||||
|
||||
<div class="layout">
|
||||
|
||||
<!-- ───────────── LEFT NAV ───────────── -->
|
||||
<nav class="sidebar" id="sidebar">
|
||||
<div class="group">
|
||||
<div class="glabel"><span class="num">1</span> Overview</div>
|
||||
<a href="#what-is">What is SkillOpt</a>
|
||||
<a href="#analogy">DL ↔ SkillOpt analogy</a>
|
||||
<a href="#features">Key features</a>
|
||||
<a href="#layout">Repository layout</a>
|
||||
</div>
|
||||
<div class="group">
|
||||
<div class="glabel"><span class="num">2</span> Installation</div>
|
||||
<a href="#requirements">Requirements</a>
|
||||
<a href="#install">Install the package</a>
|
||||
<a href="#credentials">Configure credentials</a>
|
||||
<a href="#verify">Verify installation</a>
|
||||
</div>
|
||||
<div class="group">
|
||||
<div class="glabel"><span class="num">3</span> Data Preparation</div>
|
||||
<a href="#split-dir">Split directory format</a>
|
||||
<a href="#item-schema">Item JSON schema</a>
|
||||
<a href="#split-modes">Split modes</a>
|
||||
</div>
|
||||
<div class="group">
|
||||
<div class="glabel"><span class="num">4</span> Quick Start</div>
|
||||
<a href="#train">Train a skill</a>
|
||||
<a href="#eval">Evaluate a skill</a>
|
||||
<a href="#outputs">Output structure</a>
|
||||
<a href="#resume">Auto-resume</a>
|
||||
</div>
|
||||
<div class="group">
|
||||
<div class="glabel"><span class="num">5</span> How It Works</div>
|
||||
<a href="#loop">The training loop</a>
|
||||
<a href="#stages">The six per-step stages</a>
|
||||
<a href="#gate">Validation gate</a>
|
||||
<a href="#slow-update">Slow update (momentum)</a>
|
||||
<a href="#meta-skill">Meta skill (memory)</a>
|
||||
<a href="#skill-doc">Skill document anatomy</a>
|
||||
</div>
|
||||
<div class="group">
|
||||
<div class="glabel"><span class="num">6</span> Configuration</div>
|
||||
<a href="#config-system">Config system</a>
|
||||
<a href="#cfg-model">model.*</a>
|
||||
<a href="#cfg-train">train.*</a>
|
||||
<a href="#cfg-gradient">gradient.*</a>
|
||||
<a href="#cfg-optimizer">optimizer.*</a>
|
||||
<a href="#cfg-evaluation">evaluation.*</a>
|
||||
<a href="#cfg-env">env.*</a>
|
||||
</div>
|
||||
<div class="group">
|
||||
<div class="glabel"><span class="num">7</span> Benchmarks</div>
|
||||
<a href="#bench-list">Supported benchmarks</a>
|
||||
<a href="#bench-new">Add a new benchmark</a>
|
||||
</div>
|
||||
<div class="group">
|
||||
<div class="glabel"><span class="num">8</span> API Reference</div>
|
||||
<a href="#module-map">Module map</a>
|
||||
<a href="#functions">Core functions</a>
|
||||
<a href="#cli">CLI scripts</a>
|
||||
<a href="#webui">WebUI</a>
|
||||
</div>
|
||||
</nav>
|
||||
|
||||
<!-- ───────────── MAIN CONTENT ───────────── -->
|
||||
<main class="content">
|
||||
|
||||
<span class="eyebrow">Microsoft Research</span>
|
||||
<h1>SkillOpt Documentation & Reproduction Guide</h1>
|
||||
<p class="lead">Train agent skills like you train neural networks — with epochs, (mini-)batch size, learning rates, and validation gates — but without touching any model weights.</p>
|
||||
<p>This guide walks you from a clean checkout to a reproduced result and a full reference for every configuration knob and core function. It is generated from, and kept consistent with, the current state of the codebase.</p>
|
||||
|
||||
<!-- ===================== 1. OVERVIEW ===================== -->
|
||||
<section id="what-is">
|
||||
<h2>1.1 What is SkillOpt <a class="anchor" href="#what-is">#</a></h2>
|
||||
<p><strong>SkillOpt</strong> is a text-space optimizer that improves a <em>frozen</em> language agent by iteratively editing a natural-language <strong>skill document</strong> — never the model weights. The skill document is a Markdown file that conditions a target model as it executes tasks. SkillOpt treats this document as the "weights" and runs a training loop that mirrors deep-learning training: rollout (forward pass), reflect (backward pass / gradients), select & apply edits (optimizer step), and a validation gate (accept/reject).</p>
|
||||
<p>Two roles split every model call:</p>
|
||||
<ul>
|
||||
<li><strong>Target</strong> — executes tasks using the current skill document (the agent being improved).</li>
|
||||
<li><strong>Optimizer</strong> — analyzes the target's trajectories and proposes edits to the skill document.</li>
|
||||
</ul>
|
||||
<p>The same loop drives six benchmarks out of the box (QA, document QA, embodied agents, math, spreadsheet code generation, and tool-augmented QA).</p>
|
||||
</section>
|
||||
|
||||
<section id="analogy">
|
||||
<h2>1.2 Deep-Learning ↔ SkillOpt Analogy <a class="anchor" href="#analogy">#</a></h2>
|
||||
<p>Every concept below maps to a concrete code construct, so deep-learning intuitions transfer directly to hyperparameter tuning.</p>
|
||||
<div class="table-wrap">
|
||||
<table>
|
||||
<thead><tr><th>Deep learning</th><th>SkillOpt</th><th>Where it lives</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td>Model weights</td><td>Skill document (Markdown)</td><td><code>skillopt/optimizer/skill.py</code></td></tr>
|
||||
<tr><td>Forward pass</td><td>Rollout — target runs tasks</td><td><code>envs/<bench>/rollout.py</code></td></tr>
|
||||
<tr><td>Loss / score</td><td>Task evaluator</td><td><code>envs/<bench>/evaluator.py</code></td></tr>
|
||||
<tr><td>Backprop / gradients</td><td>Reflect → edit patches</td><td><code>gradient/reflect.py</code></td></tr>
|
||||
<tr><td>Gradient aggregation</td><td>Hierarchical patch merge</td><td><code>gradient/aggregate.py</code></td></tr>
|
||||
<tr><td>Gradient clipping</td><td>Rank & select top-k edits</td><td><code>optimizer/clip.py</code></td></tr>
|
||||
<tr><td>Learning rate</td><td><code>optimizer.learning_rate</code> (edits/step)</td><td><code>optimizer/scheduler.py</code></td></tr>
|
||||
<tr><td>LR scheduler</td><td><code>lr_scheduler</code> (cosine/linear/…)</td><td><code>optimizer/scheduler.py</code></td></tr>
|
||||
<tr><td>Optimizer step</td><td>Apply patches to the document</td><td><code>optimizer/skill.py</code></td></tr>
|
||||
<tr><td>Validation set</td><td>Selection split (<code>valid_seen</code>)</td><td><code>evaluation/gate.py</code></td></tr>
|
||||
<tr><td>Early stopping / accept</td><td>Validation gate</td><td><code>evaluation/gate.py</code></td></tr>
|
||||
<tr><td>Momentum</td><td>Slow update (epoch boundary)</td><td><code>optimizer/slow_update.py</code></td></tr>
|
||||
<tr><td>Meta-learning</td><td>Meta skill (cross-epoch memory)</td><td><code>optimizer/meta_skill.py</code></td></tr>
|
||||
<tr><td>Batch / minibatch</td><td><code>batch_size</code> / <code>minibatch_size</code></td><td><code>engine/trainer.py</code></td></tr>
|
||||
<tr><td>Epoch</td><td>Epoch (+ slow update & meta skill)</td><td><code>engine/trainer.py</code></td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
<div class="note tip"><span class="nh">What transfers from DL</span>
|
||||
<p>Cosine schedule tends to beat constant; moderate learning rates (≈4–16 edits/step) beat very high/low; slow update curbs cross-epoch forgetting; meta-skill memory improves reflection quality. Conversely, bigger rollout batches and many epochs show diminishing returns — skills converge in ~2–4 epochs.</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section id="features">
|
||||
<h2>1.3 Key Features <a class="anchor" href="#features">#</a></h2>
|
||||
<div class="cards">
|
||||
<div class="card"><h4>Validation gating</h4><p>Every candidate skill is scored on a held-out selection split and only accepted if it beats the current/best skill.</p></div>
|
||||
<div class="card"><h4>Slow update</h4><p>Epoch-boundary longitudinal comparison writes guidance into a protected region — momentum against forgetting. Force-injected or selection-gated.</p></div>
|
||||
<div class="card"><h4>Meta skill</h4><p>Optimizer-side memory that reflects on what worked across epochs and feeds back into reflection.</p></div>
|
||||
<div class="card"><h4>Pluggable backends</h4><p>OpenAI / Azure OpenAI, Anthropic Claude, local Qwen (vLLM), plus Codex/Claude-Code exec backends for the target.</p></div>
|
||||
<div class="card"><h4>Six benchmarks</h4><p>SearchQA, DocVQA, ALFWorld, LiveMathematicianBench, SpreadsheetBench, OfficeQA — each a self-contained env module.</p></div>
|
||||
<div class="card"><h4>Auto-resume</h4><p>Every run is checkpointed step-by-step; re-running the same command continues from the last completed step.</p></div>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section id="layout">
|
||||
<h2>1.4 Repository Layout <a class="anchor" href="#layout">#</a></h2>
|
||||
<pre><code><span class="tok-c"># top level</span>
|
||||
configs/ <span class="tok-c"># YAML configs (_base_ + per-benchmark)</span>
|
||||
scripts/ <span class="tok-c"># train.py, eval_only.py CLIs</span>
|
||||
ckpt/ <span class="tok-c"># packaged reference skills (e.g. gpt5.5_skill.md)</span>
|
||||
docs/ <span class="tok-c"># this guide + mkdocs sources</span>
|
||||
skillopt/ <span class="tok-c"># the package</span>
|
||||
├─ config.py <span class="tok-c"># YAML loading, _base_ inheritance, flatten</span>
|
||||
├─ engine/trainer.py<span class="tok-c"># the training loop (ReflACTTrainer)</span>
|
||||
├─ gradient/ <span class="tok-c"># reflect.py (analyst), aggregate.py (merge)</span>
|
||||
├─ optimizer/ <span class="tok-c"># skill edits, scheduler, clip, slow_update, meta_skill</span>
|
||||
├─ evaluation/ <span class="tok-c"># gate.py (accept/reject logic)</span>
|
||||
├─ model/ <span class="tok-c"># backend clients + routing</span>
|
||||
└─ envs/<benchmark>/ <span class="tok-c"># adapter, dataloader, rollout, evaluator, reflect</span></code></pre>
|
||||
</section>
|
||||
|
||||
<!-- ===================== 2. INSTALLATION ===================== -->
|
||||
<section id="requirements">
|
||||
<h2>2.1 Requirements <a class="anchor" href="#requirements">#</a></h2>
|
||||
<ul>
|
||||
<li>Python ≥ 3.10</li>
|
||||
<li>Credentials for at least one model backend (Azure OpenAI, OpenAI-compatible, Anthropic, or a local Qwen server)</li>
|
||||
<li>Benchmark datasets are <strong>not</strong> bundled — prepare your own splits (see §3)</li>
|
||||
</ul>
|
||||
</section>
|
||||
|
||||
<section id="install">
|
||||
<h2>2.2 Install the Package <a class="anchor" href="#install">#</a></h2>
|
||||
<pre><code><span class="tok-k">git</span> clone https://github.com/microsoft/SkillOpt.git
|
||||
<span class="tok-k">cd</span> SkillOpt
|
||||
<span class="tok-k">pip</span> install -e .
|
||||
|
||||
<span class="tok-c"># Optional extras (install only what you need):</span>
|
||||
<span class="tok-k">pip</span> install -e <span class="tok-s">".[alfworld]"</span> <span class="tok-c"># ALFWorld benchmark</span>
|
||||
<span class="tok-k">pip</span> install -e <span class="tok-s">".[claude]"</span> <span class="tok-c"># Anthropic Claude backend</span>
|
||||
<span class="tok-k">pip</span> install -e <span class="tok-s">".[qwen]"</span> <span class="tok-c"># local Qwen backend</span>
|
||||
<span class="tok-k">pip</span> install -e <span class="tok-s">".[webui]"</span> <span class="tok-c"># monitoring dashboard</span>
|
||||
|
||||
<span class="tok-c"># ALFWorld also needs its data assets:</span>
|
||||
<span class="tok-k">alfworld-download</span></code></pre>
|
||||
</section>
|
||||
|
||||
<section id="credentials">
|
||||
<h2>2.3 Configure Credentials <a class="anchor" href="#credentials">#</a></h2>
|
||||
<p>Copy the template and fill in whichever backend you will use:</p>
|
||||
<pre><code><span class="tok-k">cp</span> .env.example .env
|
||||
<span class="tok-c"># edit .env, then:</span>
|
||||
<span class="tok-k">set</span> -a; <span class="tok-k">source</span> .env; <span class="tok-k">set</span> +a</code></pre>
|
||||
<div class="note info"><span class="nh">One env-var family for all OpenAI modes</span>
|
||||
<p>SkillOpt reuses the <code>AZURE_OPENAI_*</code> variable names even for plain OpenAI — there is no separate <code>OPENAI_API_KEY</code> knob. <code>AZURE_OPENAI_ENDPOINT</code> is required for every OpenAI auth mode.</p>
|
||||
</div>
|
||||
<h4>Azure OpenAI (default)</h4>
|
||||
<pre><code><span class="tok-k">export</span> AZURE_OPENAI_ENDPOINT=<span class="tok-s">"https://your-resource.openai.azure.com/"</span>
|
||||
<span class="tok-k">export</span> AZURE_OPENAI_API_VERSION=<span class="tok-s">"2024-12-01-preview"</span>
|
||||
<span class="tok-c"># Auth option 1 — API key:</span>
|
||||
<span class="tok-k">export</span> AZURE_OPENAI_API_KEY=<span class="tok-s">"your-key"</span>
|
||||
<span class="tok-c"># Auth option 2 — Azure CLI (no key; recommended on Azure VMs):</span>
|
||||
<span class="tok-k">export</span> AZURE_OPENAI_AUTH_MODE=azure_cli
|
||||
<span class="tok-c"># Auth option 3 — Managed Identity:</span>
|
||||
<span class="tok-k">export</span> AZURE_OPENAI_AUTH_MODE=managed_identity
|
||||
<span class="tok-k">export</span> AZURE_OPENAI_MANAGED_IDENTITY_CLIENT_ID=<span class="tok-s">"your-client-id"</span></code></pre>
|
||||
<h4>OpenAI-compatible endpoint</h4>
|
||||
<pre><code><span class="tok-k">export</span> AZURE_OPENAI_ENDPOINT=<span class="tok-s">"https://api.openai.com/v1"</span>
|
||||
<span class="tok-k">export</span> AZURE_OPENAI_API_KEY=<span class="tok-s">"sk-..."</span>
|
||||
<span class="tok-k">export</span> AZURE_OPENAI_AUTH_MODE=openai_compatible</code></pre>
|
||||
<h4>Anthropic Claude / local Qwen</h4>
|
||||
<pre><code><span class="tok-k">export</span> ANTHROPIC_API_KEY=<span class="tok-s">"sk-ant-..."</span> <span class="tok-c"># claude_chat backend</span>
|
||||
|
||||
<span class="tok-k">export</span> QWEN_CHAT_BASE_URL=<span class="tok-s">"http://localhost:8000/v1"</span> <span class="tok-c"># local vLLM</span>
|
||||
<span class="tok-k">export</span> QWEN_CHAT_MODEL=<span class="tok-s">"Qwen/Qwen3.5-4B"</span></code></pre>
|
||||
</section>
|
||||
|
||||
<section id="verify">
|
||||
<h2>2.4 Verify Installation <a class="anchor" href="#verify">#</a></h2>
|
||||
<pre><code><span class="tok-k">python</span> -c <span class="tok-s">"import skillopt; print('SkillOpt ready!')"</span></code></pre>
|
||||
</section>
|
||||
|
||||
<!-- ===================== 3. DATA ===================== -->
|
||||
<section id="split-dir">
|
||||
<h2>3.1 Split Directory Format <a class="anchor" href="#split-dir">#</a></h2>
|
||||
<p>With <code>env.split_mode: split_dir</code> (the recommended, deterministic mode), SkillOpt reads a directory containing <code>train/</code>, <code>val/</code>, and <code>test/</code> subfolders, each holding a JSON array of task items:</p>
|
||||
<pre><code>data/my_split/
|
||||
├─ train/items.json <span class="tok-c"># used for rollout (the "train split")</span>
|
||||
├─ val/items.json <span class="tok-c"># selection split → validation gate (valid_seen)</span>
|
||||
└─ test/items.json <span class="tok-c"># held-out final eval (valid_unseen)</span></code></pre>
|
||||
<div class="note info"><span class="nh">Split naming</span>
|
||||
<p>Internally the splits are referred to as <code>train</code>, <code>valid_seen</code> (validation/selection), and <code>valid_unseen</code> (test). The <code>--split</code> flag of <code>eval_only.py</code> uses these names.</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section id="item-schema">
|
||||
<h2>3.2 Item JSON Schema <a class="anchor" href="#item-schema">#</a></h2>
|
||||
<p>Required fields depend on the benchmark; consult <code>skillopt/envs/<benchmark>/dataloader.py</code> for the exact contract. A SearchQA item, for example:</p>
|
||||
<pre><code>[
|
||||
{
|
||||
<span class="tok-f">"id"</span>: <span class="tok-s">"unique_item_id"</span>,
|
||||
<span class="tok-f">"question"</span>: <span class="tok-s">"Who wrote the novel ..."</span>,
|
||||
<span class="tok-f">"context"</span>: <span class="tok-s">"[DOC] relevant passage text ..."</span>,
|
||||
<span class="tok-f">"answers"</span>: [<span class="tok-s">"expected answer"</span>]
|
||||
}
|
||||
]</code></pre>
|
||||
<div class="note warn"><span class="nh">Datasets not included</span>
|
||||
<p>This repository ships no benchmark data. Prepare your own splits in the format above before training.</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section id="split-modes">
|
||||
<h2>3.3 Split Modes <a class="anchor" href="#split-modes">#</a></h2>
|
||||
<div class="table-wrap"><table>
|
||||
<thead><tr><th><code>env.split_mode</code></th><th>Behavior</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td><code>split_dir</code></td><td>Use a pre-built directory with explicit <code>train/val/test</code> folders (set <code>env.split_dir</code>). Deterministic and reproducible.</td></tr>
|
||||
<tr><td><code>ratio</code></td><td>Build a deterministic split on the fly from a single <code>env.data_path</code>, using <code>split_seed</code> (and a train:val:test ratio). Convenient for quick experiments.</td></tr>
|
||||
</tbody>
|
||||
</table></div>
|
||||
</section>
|
||||
|
||||
<!-- ===================== 4. QUICK START ===================== -->
|
||||
<section id="train">
|
||||
<h2>4.1 Train a Skill <a class="anchor" href="#train">#</a></h2>
|
||||
<pre><code><span class="tok-c"># Minimal SearchQA run</span>
|
||||
<span class="tok-k">python</span> scripts/train.py \
|
||||
<span class="tok-f">--config</span> configs/searchqa/default.yaml \
|
||||
<span class="tok-f">--split_dir</span> /path/to/your/searchqa_split \
|
||||
<span class="tok-f">--azure_openai_endpoint</span> https://your-resource.openai.azure.com/ \
|
||||
<span class="tok-f">--optimizer_model</span> gpt-5.5 \
|
||||
<span class="tok-f">--target_model</span> gpt-5.5</code></pre>
|
||||
<p>Swap the config for another benchmark (e.g. <code>configs/livemathematicianbench/default.yaml</code>, <code>configs/alfworld/default.yaml</code>). Common CLI arguments:</p>
|
||||
<div class="table-wrap"><table>
|
||||
<thead><tr><th>Argument</th><th>Description</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td><code>--config</code></td><td>Benchmark config YAML (required)</td></tr>
|
||||
<tr><td><code>--split_dir</code></td><td>Path to the data split directory</td></tr>
|
||||
<tr><td><code>--azure_openai_endpoint</code></td><td>Azure OpenAI endpoint URL</td></tr>
|
||||
<tr><td><code>--optimizer_model</code> / <code>--target_model</code></td><td>Deployment names for optimizer / target</td></tr>
|
||||
<tr><td><code>--num_epochs</code> / <code>--batch_size</code></td><td>Epochs and rollout batch size</td></tr>
|
||||
<tr><td><code>--out_root</code></td><td>Output directory</td></tr>
|
||||
<tr><td><code>--cfg-options k=v ...</code></td><td>Override any config key (see §6.1)</td></tr>
|
||||
</tbody>
|
||||
</table></div>
|
||||
</section>
|
||||
|
||||
<section id="eval">
|
||||
<h2>4.2 Evaluate a Skill <a class="anchor" href="#eval">#</a></h2>
|
||||
<p>Evaluate any skill document (a packaged reference skill, or a trained run's <code>best_skill.md</code>) without training:</p>
|
||||
<pre><code><span class="tok-c"># Evaluate the packaged GPT-5.5 SearchQA skill on the test split</span>
|
||||
<span class="tok-k">python</span> scripts/eval_only.py \
|
||||
<span class="tok-f">--config</span> configs/searchqa/default.yaml \
|
||||
<span class="tok-f">--skill</span> ckpt/searchqa/gpt5.5_skill.md \
|
||||
<span class="tok-f">--split</span> valid_unseen \
|
||||
<span class="tok-f">--split_dir</span> /path/to/searchqa_split \
|
||||
<span class="tok-f">--azure_openai_endpoint</span> https://your-resource.openai.azure.com/</code></pre>
|
||||
<div class="table-wrap"><table>
|
||||
<thead><tr><th><code>--split</code></th><th>Meaning</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td><code>valid_unseen</code></td><td>Test set (held-out)</td></tr>
|
||||
<tr><td><code>valid_seen</code></td><td>Validation / selection set</td></tr>
|
||||
<tr><td><code>train</code></td><td>Training set</td></tr>
|
||||
<tr><td><code>all</code></td><td>All splits combined (default)</td></tr>
|
||||
</tbody>
|
||||
</table></div>
|
||||
</section>
|
||||
|
||||
<section id="outputs">
|
||||
<h2>4.3 Output Structure <a class="anchor" href="#outputs">#</a></h2>
|
||||
<pre><code>outputs/<run_name>/
|
||||
├─ config.json <span class="tok-c"># flattened runtime config</span>
|
||||
├─ history.json <span class="tok-c"># per-step training history</span>
|
||||
├─ runtime_state.json <span class="tok-c"># resume checkpoint</span>
|
||||
├─ best_skill.md <span class="tok-c"># best validated skill document</span>
|
||||
├─ skills/skill_vXXXX.md<span class="tok-c"># skill snapshot per step</span>
|
||||
├─ steps/step_XXXX/ <span class="tok-c"># per-step artifacts (patches, evals)</span>
|
||||
├─ slow_update/epoch_XX/<span class="tok-c"># slow-update logs & rollouts</span>
|
||||
└─ meta_skill/epoch_XX/ <span class="tok-c"># meta-skill logs</span></code></pre>
|
||||
</section>
|
||||
|
||||
<section id="resume">
|
||||
<h2>4.4 Auto-Resume <a class="anchor" href="#resume">#</a></h2>
|
||||
<p>Each completed step persists its state to <code>runtime_state.json</code> and a <code>steps/step_XXXX/</code> directory. Re-running the <em>same command</em> against the same <code>out_root</code> detects finished work and continues from the last completed step — including epoch-boundary slow-update and meta-skill stages.</p>
|
||||
</section>
|
||||
|
||||
<!-- ===================== 5. HOW IT WORKS ===================== -->
|
||||
<section id="loop">
|
||||
<h2>5.1 The Training Loop <a class="anchor" href="#loop">#</a></h2>
|
||||
<p>The loop lives in <code>ReflACTTrainer</code> (<code>skillopt/engine/trainer.py</code>). Each epoch runs a series of optimization steps over rollout batches, then performs two epoch-boundary stages.</p>
|
||||
<pre><code><span class="tok-k">for</span> epoch <span class="tok-k">in</span> epochs:
|
||||
<span class="tok-k">for</span> step <span class="tok-k">in</span> steps:
|
||||
1. Rollout <span class="tok-c"># target executes a batch of tasks</span>
|
||||
2. Reflect <span class="tok-c"># optimizer analyzes trajectories → edit patches</span>
|
||||
3. Aggregate <span class="tok-c"># hierarchically merge similar patches</span>
|
||||
4. Select <span class="tok-c"># rank & clip edits to the learning rate</span>
|
||||
5. Update <span class="tok-c"># apply patches → candidate skill</span>
|
||||
6. Gate <span class="tok-c"># score on selection split → accept / reject</span>
|
||||
|
||||
<span class="tok-c"># epoch boundary (from epoch 2 onward)</span>
|
||||
Slow update <span class="tok-c"># longitudinal comparison → protected guidance</span>
|
||||
Meta skill <span class="tok-c"># cross-epoch optimizer memory</span></code></pre>
|
||||
</section>
|
||||
|
||||
<section id="stages">
|
||||
<h2>5.2 The Six Per-Step Stages <a class="anchor" href="#stages">#</a></h2>
|
||||
<div class="table-wrap"><table>
|
||||
<thead><tr><th>Stage</th><th>What happens</th><th>Source</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td><strong>1. Rollout</strong></td><td>The target model runs each task in the batch with the current skill as context, producing trajectories and scores.</td><td><code>envs/<b>/rollout.py</code></td></tr>
|
||||
<tr><td><strong>2. Reflect</strong></td><td>The optimizer runs an error analyst (and optional success analyst) over minibatches of trajectories, emitting structured edit patches. Runs in parallel across <code>analyst_workers</code>.</td><td><code>gradient/reflect.py</code></td></tr>
|
||||
<tr><td><strong>3. Aggregate</strong></td><td>Semantically similar patches are merged hierarchically to remove redundancy.</td><td><code>gradient/aggregate.py</code> → <code>merge_patches</code></td></tr>
|
||||
<tr><td><strong>4. Select</strong></td><td>Patches are ranked and clipped to the current learning rate (max edits this step), set by the scheduler.</td><td><code>optimizer/clip.py</code> → <code>rank_and_select</code></td></tr>
|
||||
<tr><td><strong>5. Update</strong></td><td>Selected edits are applied to the skill document, producing a candidate skill (patch / rewrite modes).</td><td><code>optimizer/skill.py</code>, <code>update_modes.py</code></td></tr>
|
||||
<tr><td><strong>6. Gate</strong></td><td>The candidate is scored on the selection split and accepted only if it improves (see §5.3).</td><td><code>evaluation/gate.py</code> → <code>evaluate_gate</code></td></tr>
|
||||
</tbody>
|
||||
</table></div>
|
||||
</section>
|
||||
|
||||
<section id="gate">
|
||||
<h2>5.3 Validation Gate <a class="anchor" href="#gate">#</a></h2>
|
||||
<p><code>evaluate_gate</code> is a pure decision function. It compares the candidate's selection-set score against the <em>current</em> and <em>best</em> skills:</p>
|
||||
<ul>
|
||||
<li><strong>accept_new_best</strong> — candidate > current <em>and</em> candidate > best → becomes both current and best.</li>
|
||||
<li><strong>accept</strong> — candidate > current but ≤ best → becomes current only.</li>
|
||||
<li><strong>reject</strong> — candidate ≤ current → discarded; current/best unchanged.</li>
|
||||
</ul>
|
||||
<p>The comparison metric is configurable via <code>evaluation.gate_metric</code>:</p>
|
||||
<div class="table-wrap"><table>
|
||||
<thead><tr><th>Metric</th><th>Score used</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td><code>hard</code> <span class="pill def">default</span></td><td>Exact-match / discrete score</td></tr>
|
||||
<tr><td><code>soft</code></td><td>Partial-credit / continuous score</td></tr>
|
||||
<tr><td><code>mixed</code></td><td>Weighted blend, controlled by <code>gate_mixed_weight</code></td></tr>
|
||||
</tbody>
|
||||
</table></div>
|
||||
<div class="note info"><span class="nh">When to use soft/mixed</span>
|
||||
<p>The <code>soft</code>/<code>mixed</code> metrics (contributed config <code>configs/examples/soft_gate.yaml</code>) help when the selection split is small and rewards are continuous, where a discrete hard gate may reject every candidate and stall training. Paper numbers use the default <code>hard</code> gate.</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section id="slow-update">
|
||||
<h2>5.4 Slow Update (Momentum) <a class="anchor" href="#slow-update">#</a></h2>
|
||||
<p>At each epoch boundary (from epoch 2), the slow update rolls out both the <em>previous</em> epoch's skill and the <em>current</em> skill on the same sampled tasks, categorizes items (improved / regressed / persistent-fail / stable-success), and asks the optimizer to write a free-form <strong>guidance</strong> block. This guidance lands in a <strong>protected region</strong> of the skill that step-level edits cannot touch — only the slow update overwrites it. It is SkillOpt's analogue of momentum, countering cross-epoch forgetting.</p>
|
||||
<p>Acceptance has two modes, selected by <code>optimizer.slow_update_gate_with_selection</code>:</p>
|
||||
<div class="table-wrap"><table>
|
||||
<thead><tr><th>Mode</th><th>Behavior</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td><code>false</code> <span class="pill def">default</span> — force-injected</td><td>Guidance is injected into both current and best skills unconditionally. The longitudinal guidance always persists; it is not gated by step-level selection scores.</td></tr>
|
||||
<tr><td><code>true</code> — gated</td><td>The slow-update candidate is scored on the selection split and accepted/rejected through the same validation gate as step-level updates.</td></tr>
|
||||
</tbody>
|
||||
</table></div>
|
||||
</section>
|
||||
|
||||
<section id="meta-skill">
|
||||
<h2>5.5 Meta Skill (Optimizer Memory) <a class="anchor" href="#meta-skill">#</a></h2>
|
||||
<p>The meta skill is <strong>optimizer-side memory</strong> — it never modifies the target skill document. At the end of each epoch (skipped for epoch 1), the optimizer compares the previous and current epoch's last-step skills on the same sampled tasks and writes a compact, evidence-based reflection on what kind of edits helped or hurt. That memory is then injected as extra context into the next epoch's reflect / merge / learning-rate / ranking stages, so the optimizer accumulates strategy across the run.</p>
|
||||
</section>
|
||||
|
||||
<section id="skill-doc">
|
||||
<h2>5.6 Skill Document Anatomy <a class="anchor" href="#skill-doc">#</a></h2>
|
||||
<p>A skill document is plain Markdown. Initial skills can be empty (learn from scratch) or seeded with domain knowledge via <code>env.skill_init</code>. During training the document accrues rules, patterns, and edge-case handling through accepted edit patches. A dedicated protected region holds the slow-update guidance, delimited by HTML-comment markers:</p>
|
||||
<pre><code><span class="tok-c"># Question Answering Skill</span>
|
||||
|
||||
<span class="tok-c">## Learned rules ...</span>
|
||||
- When the context contains multiple candidates, prefer ...
|
||||
|
||||
<span class="tok-c"><!-- SLOW_UPDATE_START --></span>
|
||||
<span class="tok-c"># (epoch-level longitudinal guidance — only the slow update writes here)</span>
|
||||
<span class="tok-c"><!-- SLOW_UPDATE_END --></span></code></pre>
|
||||
<p>Helpers in <code>optimizer/slow_update.py</code> manage this region: <code>inject_empty_slow_update_field</code> (placeholder at epoch 1), <code>extract_slow_update_field</code> (read), and <code>replace_slow_update_field</code> (overwrite). Step-level edits are blocked from modifying anything inside the markers.</p>
|
||||
</section>
|
||||
|
||||
<!-- ===================== 6. CONFIGURATION ===================== -->
|
||||
<section id="config-system">
|
||||
<h2>6.1 Configuration System <a class="anchor" href="#config-system">#</a></h2>
|
||||
<p>Configs are <strong>structured YAML</strong> with section blocks (<code>model</code>, <code>train</code>, <code>gradient</code>, <code>optimizer</code>, <code>evaluation</code>, <code>env</code>) and <code>_base_</code> inheritance. A benchmark config inherits the shared defaults and overrides only what differs:</p>
|
||||
<pre><code><span class="tok-c"># configs/searchqa/default.yaml</span>
|
||||
<span class="tok-f">_base_</span>: ../_base_/default.yaml
|
||||
<span class="tok-f">train</span>:
|
||||
<span class="tok-f">train_size</span>: <span class="tok-n">400</span>
|
||||
<span class="tok-f">batch_size</span>: <span class="tok-n">40</span>
|
||||
<span class="tok-f">optimizer</span>:
|
||||
<span class="tok-f">learning_rate</span>: <span class="tok-n">4</span>
|
||||
<span class="tok-f">env</span>:
|
||||
<span class="tok-f">name</span>: searchqa
|
||||
<span class="tok-f">split_dir</span>: data/searchqa_split</code></pre>
|
||||
<p>Override any key at the command line without editing files:</p>
|
||||
<pre><code><span class="tok-k">python</span> scripts/train.py --config configs/searchqa/default.yaml \
|
||||
<span class="tok-f">--cfg-options</span> optimizer.learning_rate=<span class="tok-n">16</span> optimizer.lr_scheduler=linear</code></pre>
|
||||
<div class="note info"><span class="nh">Reading the tables below</span>
|
||||
<p>Each section lists the key (relative to its YAML block), type, default (from <code>configs/_base_/default.yaml</code>), allowed values, and meaning. Defaults shown are the shipped base defaults.</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section id="cfg-model">
|
||||
<h2>6.2 <code>model.*</code> <a class="anchor" href="#cfg-model">#</a></h2>
|
||||
<div class="table-wrap"><table>
|
||||
<thead><tr><th>Key</th><th>Type</th><th>Default</th><th>Description / options</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td><code>backend</code></td><td>str</td><td class="def">azure_openai</td><td>High-level backend label for the run.</td></tr>
|
||||
<tr><td><code>optimizer</code></td><td>str</td><td class="def">gpt-5.5</td><td>Optimizer model deployment (writes skill edits).</td></tr>
|
||||
<tr><td><code>target</code></td><td>str</td><td class="def">gpt-5.5</td><td>Target model deployment (executes tasks).</td></tr>
|
||||
<tr><td><code>optimizer_backend</code></td><td>str</td><td class="def">openai_chat</td><td>Client path for the optimizer: <code>openai_chat</code> or <code>claude_chat</code>.</td></tr>
|
||||
<tr><td><code>target_backend</code></td><td>str</td><td class="def">openai_chat</td><td>Client path for the target: <code>openai_chat</code> / <code>claude_chat</code> / <code>qwen_chat</code> / <code>codex_exec</code> / <code>claude_code_exec</code>.</td></tr>
|
||||
<tr><td><code>reasoning_effort</code></td><td>str</td><td class="def">medium</td><td><code>low</code> / <code>medium</code> / <code>high</code> / <code>xhigh</code> / <code>max</code> (or empty).</td></tr>
|
||||
<tr><td><code>rewrite_reasoning_effort</code></td><td>str</td><td class="def">""</td><td>Override effort for full-rewrite calls (empty = inherit).</td></tr>
|
||||
<tr><td><code>rewrite_max_completion_tokens</code></td><td>int</td><td class="def">64000</td><td>Token cap for full-rewrite optimizer calls.</td></tr>
|
||||
<tr><td><code>azure_openai_endpoint</code></td><td>str</td><td class="def">""</td><td>Azure resource URL (or via <code>AZURE_OPENAI_ENDPOINT</code>).</td></tr>
|
||||
<tr><td><code>azure_openai_api_version</code></td><td>str</td><td class="def">2024-12-01-preview</td><td>Azure API version header.</td></tr>
|
||||
<tr><td><code>azure_openai_auth_mode</code></td><td>str</td><td class="def">""</td><td><code>api_key</code> / <code>azure_cli</code> / <code>managed_identity</code> / <code>openai_compatible</code> (empty → env default).</td></tr>
|
||||
</tbody>
|
||||
</table></div>
|
||||
<div class="note info"><span class="nh">Separate optimizer / target endpoints</span>
|
||||
<p>Every <code>azure_openai_*</code> key also has <code>optimizer_azure_openai_*</code> and <code>target_azure_openai_*</code> variants, letting you point the optimizer and target at different Azure resources. Exec backends (<code>codex_exec</code>, <code>claude_code_exec</code>) add their own <code>codex_exec_*</code> / <code>claude_code_exec_*</code> knobs (sandbox, reasoning effort, SDK mode, etc.).</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section id="cfg-train">
|
||||
<h2>6.3 <code>train.*</code> <a class="anchor" href="#cfg-train">#</a></h2>
|
||||
<div class="table-wrap"><table>
|
||||
<thead><tr><th>Key</th><th>Type</th><th>Default</th><th>DL analogy</th><th>Description</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td><code>num_epochs</code></td><td>int</td><td class="def">4</td><td>Epochs</td><td>Number of training epochs.</td></tr>
|
||||
<tr><td><code>train_size</code></td><td>int</td><td class="def">0</td><td>Train-set size</td><td>0 = derive from the dataset split. (Fixed by split size when using <code>split_dir</code>.)</td></tr>
|
||||
<tr><td><code>batch_size</code></td><td>int</td><td class="def">40</td><td>Batch size</td><td>Tasks rolled out per optimization step.</td></tr>
|
||||
<tr><td><code>accumulation</code></td><td>int</td><td class="def">1</td><td>Grad accumulation</td><td>Accumulation rounds per step.</td></tr>
|
||||
<tr><td><code>seed</code></td><td>int</td><td class="def">42</td><td>Random seed</td><td>Reproducibility seed.</td></tr>
|
||||
</tbody>
|
||||
</table></div>
|
||||
</section>
|
||||
|
||||
<section id="cfg-gradient">
|
||||
<h2>6.4 <code>gradient.*</code> <a class="anchor" href="#cfg-gradient">#</a></h2>
|
||||
<div class="table-wrap"><table>
|
||||
<thead><tr><th>Key</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td><code>minibatch_size</code></td><td>int</td><td class="def">8</td><td>Trajectories per reflect minibatch.</td></tr>
|
||||
<tr><td><code>merge_batch_size</code></td><td>int</td><td class="def">8</td><td>Patches per merge batch during aggregation.</td></tr>
|
||||
<tr><td><code>analyst_workers</code></td><td>int</td><td class="def">16</td><td>Parallel reflection workers (data parallelism).</td></tr>
|
||||
<tr><td><code>max_analyst_rounds</code></td><td>int</td><td class="def">3</td><td>Max rounds of analyst reflection per step.</td></tr>
|
||||
<tr><td><code>failure_only</code></td><td>bool</td><td class="def">false</td><td>Reflect only on failed trajectories when true.</td></tr>
|
||||
</tbody>
|
||||
</table></div>
|
||||
</section>
|
||||
|
||||
<section id="cfg-optimizer">
|
||||
<h2>6.5 <code>optimizer.*</code> <a class="anchor" href="#cfg-optimizer">#</a></h2>
|
||||
<div class="table-wrap"><table>
|
||||
<thead><tr><th>Key</th><th>Type</th><th>Default</th><th>DL analogy</th><th>Description / options</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td><code>learning_rate</code></td><td>int</td><td class="def">4</td><td>Learning rate</td><td>Max edit patches applied per step (the "edit budget").</td></tr>
|
||||
<tr><td><code>min_learning_rate</code></td><td>int</td><td class="def">2</td><td>Min LR</td><td>Floor edit budget for decaying schedulers.</td></tr>
|
||||
<tr><td><code>lr_scheduler</code></td><td>str</td><td class="def">cosine</td><td>LR schedule</td><td><code>constant</code> / <code>linear</code> / <code>cosine</code> / <code>autonomous</code>.</td></tr>
|
||||
<tr><td><code>lr_control_mode</code></td><td>str</td><td class="def">fixed</td><td>—</td><td><code>fixed</code> / <code>autonomous</code> / <code>none</code>.</td></tr>
|
||||
<tr><td><code>skill_update_mode</code></td><td>str</td><td class="def">patch</td><td>—</td><td><code>patch</code> / <code>rewrite_from_suggestions</code> / <code>full_rewrite_minibatch</code>.</td></tr>
|
||||
<tr><td><code>use_slow_update</code></td><td>bool</td><td class="def">true</td><td>Momentum</td><td>Enable epoch-boundary slow update.</td></tr>
|
||||
<tr><td><code>slow_update_samples</code></td><td>int</td><td class="def">20</td><td>—</td><td>Tasks sampled for the longitudinal comparison.</td></tr>
|
||||
<tr><td><code>slow_update_gate_with_selection</code></td><td>bool</td><td class="def">false</td><td>—</td><td><code>false</code> = force-inject guidance; <code>true</code> = gate it on the selection split (see §5.4).</td></tr>
|
||||
<tr><td><code>longitudinal_pair_policy</code></td><td>str</td><td class="def">mixed</td><td>—</td><td><code>mixed</code> / <code>changed</code> / <code>unchanged</code> — which comparison pairs to keep.</td></tr>
|
||||
<tr><td><code>use_meta_skill</code></td><td>bool</td><td class="def">true</td><td>Meta-learning</td><td>Enable cross-epoch optimizer memory.</td></tr>
|
||||
</tbody>
|
||||
</table></div>
|
||||
</section>
|
||||
|
||||
<section id="cfg-evaluation">
|
||||
<h2>6.6 <code>evaluation.*</code> <a class="anchor" href="#cfg-evaluation">#</a></h2>
|
||||
<div class="table-wrap"><table>
|
||||
<thead><tr><th>Key</th><th>Type</th><th>Default</th><th>Description / options</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td><code>use_gate</code></td><td>bool</td><td class="def">true</td><td>Validation gating is mandatory in this branch (must remain <code>true</code>).</td></tr>
|
||||
<tr><td><code>gate_metric</code></td><td>str</td><td class="def">hard</td><td><code>hard</code> / <code>soft</code> / <code>mixed</code> — score used by the gate (see §5.3).</td></tr>
|
||||
<tr><td><code>gate_mixed_weight</code></td><td>float</td><td class="def">0.5</td><td>Weight on the soft score when <code>gate_metric = mixed</code>.</td></tr>
|
||||
<tr><td><code>sel_env_num</code></td><td>int</td><td class="def">0</td><td>Selection-split eval size (0 = use full split).</td></tr>
|
||||
<tr><td><code>test_env_num</code></td><td>int</td><td class="def">0</td><td>Test-split eval size (0 = use full split).</td></tr>
|
||||
<tr><td><code>eval_test</code></td><td>bool</td><td class="def">true</td><td>Run a final test evaluation after training.</td></tr>
|
||||
</tbody>
|
||||
</table></div>
|
||||
<div class="note warn"><span class="nh">Gate is required</span>
|
||||
<p>Setting <code>evaluation.use_gate: false</code> raises an error — validation gating cannot be disabled in this branch.</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section id="cfg-env">
|
||||
<h2>6.7 <code>env.*</code> <a class="anchor" href="#cfg-env">#</a></h2>
|
||||
<div class="table-wrap"><table>
|
||||
<thead><tr><th>Key</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td><code>name</code></td><td>str</td><td class="def">""</td><td>Benchmark name (<code>searchqa</code>, <code>docvqa</code>, <code>alfworld</code>, …). Selects the env module.</td></tr>
|
||||
<tr><td><code>skill_init</code></td><td>str</td><td class="def">""</td><td>Path to a seed skill (empty = start from scratch).</td></tr>
|
||||
<tr><td><code>split_mode</code></td><td>str</td><td class="def">ratio</td><td><code>ratio</code> or <code>split_dir</code> (see §3.3).</td></tr>
|
||||
<tr><td><code>split_dir</code></td><td>str</td><td class="def">""</td><td>Pre-split directory (when <code>split_mode = split_dir</code>).</td></tr>
|
||||
<tr><td><code>data_path</code></td><td>str</td><td class="def">""</td><td>Single dataset path (when <code>split_mode = ratio</code>).</td></tr>
|
||||
<tr><td><code>split_seed</code></td><td>int</td><td class="def">42</td><td>Seed for deterministic ratio splitting.</td></tr>
|
||||
<tr><td><code>exec_timeout</code></td><td>int</td><td class="def">120</td><td>Per-task target/code-agent timeout (seconds).</td></tr>
|
||||
<tr><td><code>out_root</code></td><td>str</td><td class="def">""</td><td>Output directory for the run.</td></tr>
|
||||
</tbody>
|
||||
</table></div>
|
||||
<div class="note info"><span class="nh">Benchmark-specific env keys</span>
|
||||
<p>Env blocks may carry extra benchmark-specific keys (e.g. <code>max_turns</code>, <code>workers</code>, <code>max_completion_tokens</code>, <code>limit</code>). Unmapped env keys are passed straight through to the benchmark adapter — check the relevant <code>configs/<benchmark>/default.yaml</code>.</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<!-- ===================== 7. BENCHMARKS ===================== -->
|
||||
<section id="bench-list">
|
||||
<h2>7.1 Supported Benchmarks <a class="anchor" href="#bench-list">#</a></h2>
|
||||
<div class="table-wrap"><table>
|
||||
<thead><tr><th>Benchmark</th><th>Type</th><th>Config</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td>SearchQA</td><td>Question answering</td><td><code>configs/searchqa/default.yaml</code></td></tr>
|
||||
<tr><td>DocVQA</td><td>Document QA</td><td><code>configs/docvqa/default.yaml</code></td></tr>
|
||||
<tr><td>ALFWorld</td><td>Embodied agent</td><td><code>configs/alfworld/default.yaml</code></td></tr>
|
||||
<tr><td>LiveMathematicianBench</td><td>Math reasoning</td><td><code>configs/livemathematicianbench/default.yaml</code></td></tr>
|
||||
<tr><td>SpreadsheetBench</td><td>Spreadsheet code generation</td><td><code>configs/spreadsheetbench/default.yaml</code></td></tr>
|
||||
<tr><td>OfficeQA</td><td>Tool-augmented QA</td><td><code>configs/officeqa/default.yaml</code></td></tr>
|
||||
</tbody>
|
||||
</table></div>
|
||||
<p>Each benchmark is a self-contained module under <code>skillopt/envs/<benchmark>/</code> with an <code>adapter.py</code>, <code>dataloader.py</code>, <code>rollout.py</code>, and <code>evaluator.py</code> (some add a custom <code>reflect.py</code>). Packaged reference skills live in <code>ckpt/<benchmark>/</code>.</p>
|
||||
</section>
|
||||
|
||||
<section id="bench-new">
|
||||
<h2>7.2 Add a New Benchmark <a class="anchor" href="#bench-new">#</a></h2>
|
||||
<p>Use <code>skillopt/envs/_template/</code> as a starting point. At minimum, implement:</p>
|
||||
<ol>
|
||||
<li><strong>Dataloader</strong> — read your item JSON into the framework's item dicts (<code>dataloader.py</code>).</li>
|
||||
<li><strong>Rollout</strong> — run the target on one item with the current skill and return a trajectory + score (<code>rollout.py</code>).</li>
|
||||
<li><strong>Evaluator</strong> — score predictions against ground truth (<code>evaluator.py</code>).</li>
|
||||
<li><strong>Adapter</strong> — wire the above into the trainer's expected interface and register the env name (<code>adapter.py</code>).</li>
|
||||
</ol>
|
||||
<p>Then add a <code>configs/<name>/default.yaml</code> inheriting <code>_base_/default.yaml</code> and set <code>env.name</code> to your new benchmark.</p>
|
||||
</section>
|
||||
|
||||
<!-- ===================== 8. API REFERENCE ===================== -->
|
||||
<section id="module-map">
|
||||
<h2>8.1 Module Map <a class="anchor" href="#module-map">#</a></h2>
|
||||
<div class="table-wrap"><table>
|
||||
<thead><tr><th>Module</th><th>Responsibility</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td><code>skillopt/config.py</code></td><td>Load structured YAML, resolve <code>_base_</code> inheritance, flatten to the trainer's flat dict, apply CLI overrides.</td></tr>
|
||||
<tr><td><code>skillopt/engine/trainer.py</code></td><td><code>ReflACTTrainer</code> — orchestrates the whole loop, gating, slow update, meta skill, resume, and artifact writing.</td></tr>
|
||||
<tr><td><code>skillopt/gradient/</code></td><td>Reflection ("backward pass"): <code>reflect.py</code> analysts, <code>aggregate.py</code> patch merging.</td></tr>
|
||||
<tr><td><code>skillopt/optimizer/</code></td><td>The "optimizer": edit application, learning-rate scheduling, edit selection, slow update, meta skill, rewrite modes.</td></tr>
|
||||
<tr><td><code>skillopt/evaluation/gate.py</code></td><td>Pure accept/reject decision and metric selection.</td></tr>
|
||||
<tr><td><code>skillopt/model/</code></td><td>Backend clients (OpenAI/Azure, Claude, Qwen, Codex/Claude-Code exec) and routing.</td></tr>
|
||||
<tr><td><code>skillopt/envs/<b>/</code></td><td>Per-benchmark dataloader, rollout, evaluator, adapter.</td></tr>
|
||||
</tbody>
|
||||
</table></div>
|
||||
</section>
|
||||
|
||||
<section id="functions">
|
||||
<h2>8.2 Core Functions <a class="anchor" href="#functions">#</a></h2>
|
||||
<div class="table-wrap"><table>
|
||||
<thead><tr><th>Function</th><th>File</th><th>Purpose</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td><code>load_config</code> / <code>flatten_config</code> / <code>apply_overrides</code></td><td><code>config.py</code></td><td>Load YAML with inheritance; flatten sections; apply <code>key=value</code> overrides.</td></tr>
|
||||
<tr><td><code>run_minibatch_reflect</code></td><td><code>gradient/reflect.py</code></td><td>Run error/success analysts over trajectory minibatches → edit patches.</td></tr>
|
||||
<tr><td><code>merge_patches</code></td><td><code>gradient/aggregate.py</code></td><td>Hierarchically merge semantically similar patches.</td></tr>
|
||||
<tr><td><code>rank_and_select</code></td><td><code>optimizer/clip.py</code></td><td>Rank edits and clip to the learning-rate budget.</td></tr>
|
||||
<tr><td><code>build_scheduler</code></td><td><code>optimizer/scheduler.py</code></td><td>Construct the LR (edit-budget) scheduler: constant/linear/cosine/autonomous.</td></tr>
|
||||
<tr><td><code>decide_autonomous_learning_rate</code></td><td><code>optimizer/lr_autonomous.py</code></td><td>Let the optimizer pick the next learning rate (autonomous mode).</td></tr>
|
||||
<tr><td><code>apply_patch</code> / <code>apply_edit</code></td><td><code>optimizer/skill.py</code></td><td>Apply edits to the skill document (respecting the protected region).</td></tr>
|
||||
<tr><td><code>rewrite_skill_from_suggestions</code></td><td><code>optimizer/rewrite.py</code></td><td>Full-rewrite update mode from accumulated suggestions.</td></tr>
|
||||
<tr><td><code>evaluate_gate</code> / <code>select_gate_score</code></td><td><code>evaluation/gate.py</code></td><td>Accept/reject decision; compute hard/soft/mixed score.</td></tr>
|
||||
<tr><td><code>run_slow_update</code></td><td><code>optimizer/slow_update.py</code></td><td>Produce epoch-boundary longitudinal guidance.</td></tr>
|
||||
<tr><td><code>replace_slow_update_field</code> / <code>extract_slow_update_field</code></td><td><code>optimizer/slow_update.py</code></td><td>Read/overwrite the protected guidance region.</td></tr>
|
||||
<tr><td><code>run_meta_skill</code> / <code>format_meta_skill_context</code></td><td><code>optimizer/meta_skill.py</code></td><td>Generate cross-epoch optimizer memory and render it into reflection context.</td></tr>
|
||||
</tbody>
|
||||
</table></div>
|
||||
</section>
|
||||
|
||||
<section id="cli">
|
||||
<h2>8.3 CLI Scripts <a class="anchor" href="#cli">#</a></h2>
|
||||
<h4>scripts/train.py</h4>
|
||||
<p>Runs a full training loop. Required: <code>--config</code>. Override config via <code>--cfg-options section.key=value …</code> or legacy flat flags (<code>--num_epochs</code>, <code>--batch_size</code>, <code>--optimizer_model</code>, <code>--target_model</code>, <code>--lr_scheduler</code>, <code>--edit_budget</code>, <code>--split_dir</code>, …).</p>
|
||||
<h4>scripts/eval_only.py</h4>
|
||||
<p>Evaluates a skill document without training. Required: <code>--config</code> and <code>--skill</code>. Use <code>--split</code> to choose <code>train</code> / <code>valid_seen</code> / <code>valid_unseen</code> / <code>all</code>.</p>
|
||||
<pre><code><span class="tok-k">python</span> scripts/eval_only.py \
|
||||
--config configs/searchqa/default.yaml \
|
||||
--skill outputs/my_run/best_skill.md \
|
||||
--split valid_unseen</code></pre>
|
||||
</section>
|
||||
|
||||
<section id="webui">
|
||||
<h2>8.4 WebUI <a class="anchor" href="#webui">#</a></h2>
|
||||
<p>An optional Gradio dashboard to configure parameters and monitor runs:</p>
|
||||
<pre><code><span class="tok-k">pip</span> install -e <span class="tok-s">".[webui]"</span>
|
||||
<span class="tok-k">python</span> -m skillopt_webui.app <span class="tok-c"># http://localhost:7860</span>
|
||||
<span class="tok-k">python</span> -m skillopt_webui.app --share <span class="tok-c"># public share link</span></code></pre>
|
||||
<div class="table-wrap"><table>
|
||||
<thead><tr><th>Flag</th><th>Default</th><th>Description</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td><code>--port</code></td><td class="def">7860</td><td>Server port.</td></tr>
|
||||
<tr><td><code>--host</code></td><td class="def">0.0.0.0</td><td>Bind address.</td></tr>
|
||||
<tr><td><code>--share</code></td><td class="def">off</td><td>Create a public Gradio share link.</td></tr>
|
||||
</tbody>
|
||||
</table></div>
|
||||
|
||||
<div class="footer-note">
|
||||
SkillOpt — Executive Strategy for Self-Evolving Agent Skills ·
|
||||
<a href="https://github.com/microsoft/SkillOpt">github.com/microsoft/SkillOpt</a> ·
|
||||
<a href="https://arxiv.org/abs/2605.23904">arXiv:2605.23904</a><br>
|
||||
This guide reflects the current configuration defaults in <code>configs/_base_/default.yaml</code>. When in doubt, the code is the source of truth.
|
||||
</div>
|
||||
</section>
|
||||
|
||||
</main>
|
||||
|
||||
<!-- ───────────── RIGHT TOC ───────────── -->
|
||||
<aside class="toc" id="toc">
|
||||
<div class="tl">On this page</div>
|
||||
<div id="tocLinks"></div>
|
||||
</aside>
|
||||
|
||||
</div>
|
||||
|
||||
<script>
|
||||
(function () {
|
||||
// Build right-hand "On this page" from <h2> elements
|
||||
var sections = Array.prototype.slice.call(document.querySelectorAll('main.content section[id]'));
|
||||
var tocLinks = document.getElementById('tocLinks');
|
||||
var h2s = Array.prototype.slice.call(document.querySelectorAll('main.content h2'));
|
||||
h2s.forEach(function (h) {
|
||||
var sec = h.closest('section');
|
||||
if (!sec || !sec.id) return;
|
||||
var a = document.createElement('a');
|
||||
a.href = '#' + sec.id;
|
||||
a.textContent = h.textContent.replace(/#$/, '').trim();
|
||||
a.dataset.target = sec.id;
|
||||
tocLinks.appendChild(a);
|
||||
});
|
||||
|
||||
var sideLinks = Array.prototype.slice.call(document.querySelectorAll('nav.sidebar a'));
|
||||
var tocAnchors = Array.prototype.slice.call(tocLinks.querySelectorAll('a'));
|
||||
|
||||
function setActive(id) {
|
||||
sideLinks.forEach(function (a) {
|
||||
a.classList.toggle('active', a.getAttribute('href') === '#' + id);
|
||||
});
|
||||
tocAnchors.forEach(function (a) {
|
||||
a.classList.toggle('active', a.dataset.target === id);
|
||||
});
|
||||
}
|
||||
|
||||
// Scroll spy
|
||||
var observer = new IntersectionObserver(function (entries) {
|
||||
entries.forEach(function (e) {
|
||||
if (e.isIntersecting) setActive(e.target.id);
|
||||
});
|
||||
}, { rootMargin: '-64px 0px -75% 0px', threshold: 0 });
|
||||
sections.forEach(function (s) { observer.observe(s); });
|
||||
|
||||
// Mobile sidebar toggle
|
||||
var btn = document.getElementById('menuBtn');
|
||||
var sidebar = document.getElementById('sidebar');
|
||||
btn.addEventListener('click', function () { sidebar.classList.toggle('open'); });
|
||||
sideLinks.forEach(function (a) {
|
||||
a.addEventListener('click', function () { sidebar.classList.remove('open'); });
|
||||
});
|
||||
})();
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
+50
-44
@@ -4,72 +4,78 @@
|
||||
|
||||
### `EnvAdapter`
|
||||
|
||||
Abstract base class for benchmark environments (`skillopt/envs/base.py`).
|
||||
Abstract base class for benchmark environments.
|
||||
|
||||
```python
|
||||
class EnvAdapter(ABC):
|
||||
async def execute(self, item, skill, model) -> TaskResult
|
||||
def evaluate(self, prediction, ground_truth) -> float
|
||||
def build_prompt(self, item, skill) -> str
|
||||
```
|
||||
|
||||
### `DataLoader`
|
||||
|
||||
Abstract base class for data loading and splitting.
|
||||
|
||||
```python
|
||||
class DataLoader(ABC):
|
||||
def setup(self, cfg: dict) -> None
|
||||
def get_dataloader(self) -> BaseDataLoader | None
|
||||
def build_train_env(self, batch_size: int, seed: int, **kwargs)
|
||||
def build_eval_env(self, env_num: int, split: str, seed: int, **kwargs)
|
||||
def rollout(self, env_manager, skill_content: str, out_dir: str, **kwargs) -> list[dict]
|
||||
def reflect(self, results: list[dict], skill_content: str, out_dir: str, **kwargs) -> list[dict | None]
|
||||
def get_task_types(self) -> list[str]
|
||||
def get_split_items(self, split: str) -> list[DataItem]
|
||||
```
|
||||
|
||||
The rollout contract expects result rows with at least:
|
||||
### `ModelBackend`
|
||||
|
||||
Abstract base class for LLM backends.
|
||||
|
||||
```python
|
||||
{"id": str, "hard": int, "soft": float}
|
||||
class ModelBackend(ABC):
|
||||
async def generate(self, messages, **kwargs) -> ModelResponse
|
||||
async def generate_with_tools(self, messages, tools, **kwargs) -> ModelResponse
|
||||
```
|
||||
|
||||
### `BaseDataLoader` / `SplitDataLoader`
|
||||
### `Trainer`
|
||||
|
||||
Data loader abstractions (`skillopt/datasets/base.py`).
|
||||
Main training loop orchestrator.
|
||||
|
||||
```python
|
||||
class BaseDataLoader(ABC):
|
||||
def setup(self, cfg: dict) -> None
|
||||
def build_train_batch(self, batch_size: int, seed: int, **kwargs) -> BatchSpec
|
||||
def build_eval_batch(self, env_num: int, split: str, seed: int, **kwargs) -> BatchSpec
|
||||
|
||||
class SplitDataLoader(BaseDataLoader):
|
||||
def load_raw_items(self, data_path: str) -> list[dict]
|
||||
def load_split_items(self, split_path: str) -> list[dict]
|
||||
def get_split_items(self, split: str) -> list[dict]
|
||||
class Trainer:
|
||||
def __init__(self, cfg: dict)
|
||||
async def train(self) -> TrainResult
|
||||
async def evaluate(self, skill: str, split: str) -> EvalResult
|
||||
```
|
||||
|
||||
### `BatchSpec`
|
||||
## Data Classes
|
||||
|
||||
Represents one concrete batch request.
|
||||
|
||||
```python
|
||||
@dataclass(slots=True)
|
||||
class BatchSpec:
|
||||
phase: str
|
||||
split: str
|
||||
seed: int
|
||||
batch_size: int
|
||||
payload: object | None = None
|
||||
metadata: dict[str, Any] = field(default_factory=dict)
|
||||
```
|
||||
|
||||
### `RolloutResult` / `RawPatch`
|
||||
|
||||
Typed helpers for stage I/O in `skillopt/types.py`.
|
||||
### `DataItem`
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class RolloutResult:
|
||||
class DataItem:
|
||||
id: str
|
||||
hard: int
|
||||
soft: float
|
||||
# optional benchmark-specific fields
|
||||
input: str
|
||||
ground_truth: str
|
||||
metadata: dict = field(default_factory=dict)
|
||||
```
|
||||
|
||||
### `TaskResult`
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class RawPatch:
|
||||
patch: Patch
|
||||
source_type: Literal["failure", "success"] = "failure"
|
||||
class TaskResult:
|
||||
item_id: str
|
||||
prediction: str
|
||||
score: float
|
||||
trajectory: list[dict]
|
||||
```
|
||||
|
||||
### `ModelResponse`
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class ModelResponse:
|
||||
content: str
|
||||
usage: dict
|
||||
model: str
|
||||
```
|
||||
|
||||
For detailed source code, see the [`skillopt/`](https://github.com/microsoft/SkillOpt/tree/main/skillopt) directory.
|
||||
|
||||
@@ -10,12 +10,6 @@ Complete reference for all SkillOpt configuration parameters.
|
||||
| `model.optimizer` | str | `gpt-5.5` | Optimizer model (for reflection & slow update) |
|
||||
| `model.target` | str | `gpt-5.5` | Target model (for rollout execution) |
|
||||
| `model.reasoning_effort` | str | `medium` | Reasoning effort level |
|
||||
| `model.optimizer_backend` | str | `openai_chat` | Optimizer backend: `openai_chat` / `claude_chat` / `qwen_chat` / `minimax_chat` |
|
||||
| `model.target_backend` | str | `openai_chat` | Target backend: chat backends plus execution harnesses |
|
||||
| `model.qwen_chat_base_url` | str | `http://localhost:8000/v1` | Shared Qwen/vLLM OpenAI-compatible endpoint |
|
||||
| `model.qwen_chat_enable_thinking` | bool | `false` | Shared Qwen thinking flag |
|
||||
| `model.optimizer_qwen_chat_base_url` | str | — | Optimizer-specific Qwen/vLLM endpoint; overrides shared `qwen_chat_base_url` |
|
||||
| `model.target_qwen_chat_base_url` | str | — | Target-specific Qwen/vLLM endpoint; overrides shared `qwen_chat_base_url` |
|
||||
|
||||
## Training (`train`)
|
||||
|
||||
@@ -76,10 +70,3 @@ Complete reference for all SkillOpt configuration parameters.
|
||||
| `AZURE_OPENAI_API_KEY` / `model.azure_openai_api_key` | Azure API key |
|
||||
| `OPENAI_API_KEY` | OpenAI API key (for `openai_chat` backend) |
|
||||
| `ANTHROPIC_API_KEY` | Anthropic API key (for `claude_code_exec` backend) |
|
||||
| `QWEN_CHAT_BASE_URL` | Shared local vLLM endpoint for `qwen_chat` |
|
||||
| `QWEN_CHAT_MODEL` | Shared served model name for `qwen_chat` |
|
||||
| `QWEN_CHAT_API_KEY` | Optional API key for the shared Qwen endpoint |
|
||||
| `OPTIMIZER_QWEN_CHAT_BASE_URL` | Optimizer-specific local vLLM endpoint |
|
||||
| `OPTIMIZER_QWEN_CHAT_MODEL` | Optimizer-specific served model name |
|
||||
| `TARGET_QWEN_CHAT_BASE_URL` | Target-specific local vLLM endpoint |
|
||||
| `TARGET_QWEN_CHAT_MODEL` | Target-specific served model name |
|
||||
|
||||
+1
-55
@@ -137,7 +137,7 @@ def parse_args() -> argparse.Namespace:
|
||||
# Legacy flat CLI overrides (still work, prefer --cfg-options for new usage)
|
||||
p.add_argument("--env", type=str)
|
||||
p.add_argument("--backend", type=str,
|
||||
choices=["azure_openai", "codex", "codex_exec", "claude", "claude_chat", "claude_code_exec", "qwen", "qwen_chat", "minimax", "minimax_chat"])
|
||||
choices=["azure_openai", "codex", "codex_exec", "claude", "claude_chat", "claude_code_exec", "qwen", "qwen_chat"])
|
||||
p.add_argument("--optimizer_model", type=str)
|
||||
p.add_argument("--target_model", type=str)
|
||||
p.add_argument("--optimizer_backend", type=str)
|
||||
@@ -173,24 +173,6 @@ def parse_args() -> argparse.Namespace:
|
||||
p.add_argument("--qwen_chat_timeout_seconds", type=float)
|
||||
p.add_argument("--qwen_chat_max_tokens", type=int)
|
||||
p.add_argument("--qwen_chat_enable_thinking", type=_BOOL)
|
||||
p.add_argument("--optimizer_qwen_chat_base_url", type=str)
|
||||
p.add_argument("--optimizer_qwen_chat_api_key", type=str)
|
||||
p.add_argument("--optimizer_qwen_chat_temperature", type=float)
|
||||
p.add_argument("--optimizer_qwen_chat_timeout_seconds", type=float)
|
||||
p.add_argument("--optimizer_qwen_chat_max_tokens", type=int)
|
||||
p.add_argument("--optimizer_qwen_chat_enable_thinking", type=_BOOL)
|
||||
p.add_argument("--target_qwen_chat_base_url", type=str)
|
||||
p.add_argument("--target_qwen_chat_api_key", type=str)
|
||||
p.add_argument("--target_qwen_chat_temperature", type=float)
|
||||
p.add_argument("--target_qwen_chat_timeout_seconds", type=float)
|
||||
p.add_argument("--target_qwen_chat_max_tokens", type=int)
|
||||
p.add_argument("--target_qwen_chat_enable_thinking", type=_BOOL)
|
||||
p.add_argument("--minimax_base_url", type=str)
|
||||
p.add_argument("--minimax_api_key", type=str)
|
||||
p.add_argument("--minimax_model", type=str)
|
||||
p.add_argument("--minimax_temperature", type=float)
|
||||
p.add_argument("--minimax_max_tokens", type=int)
|
||||
p.add_argument("--minimax_enable_thinking", type=_BOOL)
|
||||
p.add_argument("--codex_exec_path", type=str)
|
||||
p.add_argument("--codex_exec_sandbox", type=str)
|
||||
p.add_argument("--codex_exec_profile", type=str)
|
||||
@@ -307,24 +289,6 @@ _LEGACY_TO_STRUCTURED: dict[str, str] = {
|
||||
"qwen_chat_timeout_seconds": "model.qwen_chat_timeout_seconds",
|
||||
"qwen_chat_max_tokens": "model.qwen_chat_max_tokens",
|
||||
"qwen_chat_enable_thinking": "model.qwen_chat_enable_thinking",
|
||||
"optimizer_qwen_chat_base_url": "model.optimizer_qwen_chat_base_url",
|
||||
"optimizer_qwen_chat_api_key": "model.optimizer_qwen_chat_api_key",
|
||||
"optimizer_qwen_chat_temperature": "model.optimizer_qwen_chat_temperature",
|
||||
"optimizer_qwen_chat_timeout_seconds": "model.optimizer_qwen_chat_timeout_seconds",
|
||||
"optimizer_qwen_chat_max_tokens": "model.optimizer_qwen_chat_max_tokens",
|
||||
"optimizer_qwen_chat_enable_thinking": "model.optimizer_qwen_chat_enable_thinking",
|
||||
"target_qwen_chat_base_url": "model.target_qwen_chat_base_url",
|
||||
"target_qwen_chat_api_key": "model.target_qwen_chat_api_key",
|
||||
"target_qwen_chat_temperature": "model.target_qwen_chat_temperature",
|
||||
"target_qwen_chat_timeout_seconds": "model.target_qwen_chat_timeout_seconds",
|
||||
"target_qwen_chat_max_tokens": "model.target_qwen_chat_max_tokens",
|
||||
"target_qwen_chat_enable_thinking": "model.target_qwen_chat_enable_thinking",
|
||||
"minimax_base_url": "model.minimax_base_url",
|
||||
"minimax_api_key": "model.minimax_api_key",
|
||||
"minimax_model": "model.minimax_model",
|
||||
"minimax_temperature": "model.minimax_temperature",
|
||||
"minimax_max_tokens": "model.minimax_max_tokens",
|
||||
"minimax_enable_thinking": "model.minimax_enable_thinking",
|
||||
"codex_exec_path": "model.codex_exec_path",
|
||||
"codex_exec_sandbox": "model.codex_exec_sandbox",
|
||||
"codex_exec_profile": "model.codex_exec_profile",
|
||||
@@ -439,9 +403,6 @@ def load_config(args: argparse.Namespace) -> dict:
|
||||
elif backend in {"qwen", "qwen_chat"}:
|
||||
flat.setdefault("optimizer_backend", "openai_chat")
|
||||
flat.setdefault("target_backend", "qwen_chat")
|
||||
elif backend in {"minimax", "minimax_chat"}:
|
||||
flat.setdefault("optimizer_backend", "openai_chat")
|
||||
flat.setdefault("target_backend", "minimax_chat")
|
||||
else:
|
||||
flat.setdefault("optimizer_backend", "openai_chat")
|
||||
flat.setdefault("target_backend", "openai_chat")
|
||||
@@ -455,12 +416,6 @@ def load_config(args: argparse.Namespace) -> dict:
|
||||
and not _has_model_override("model.optimizer", "optimizer_model")
|
||||
):
|
||||
flat["optimizer_model"] = default_model_for_backend("claude_chat")
|
||||
if flat.get("optimizer_backend") == "qwen_chat":
|
||||
if (
|
||||
str(flat.get("optimizer_model", "") or "").strip() in _OPENAI_DEFAULT_MODEL_SENTINELS
|
||||
and not _has_model_override("model.optimizer", "optimizer_model")
|
||||
):
|
||||
flat["optimizer_model"] = default_model_for_backend("qwen_chat")
|
||||
if flat.get("target_backend") == "claude_chat":
|
||||
if (
|
||||
str(flat.get("target_model", "") or "").strip() in _OPENAI_DEFAULT_MODEL_SENTINELS
|
||||
@@ -479,15 +434,6 @@ def load_config(args: argparse.Namespace) -> dict:
|
||||
and not _has_model_override("model.target", "target_model")
|
||||
):
|
||||
flat["target_model"] = default_model_for_backend("qwen_chat")
|
||||
if flat.get("target_backend") == "minimax_chat":
|
||||
if (
|
||||
str(flat.get("target_model", "") or "").strip() in _OPENAI_DEFAULT_MODEL_SENTINELS
|
||||
and not _has_model_override("model.target", "target_model")
|
||||
):
|
||||
flat["target_model"] = (
|
||||
flat.get("minimax_model")
|
||||
or default_model_for_backend("minimax_chat")
|
||||
)
|
||||
|
||||
# Auto-generate output root
|
||||
if not flat.get("out_root"):
|
||||
|
||||
@@ -79,24 +79,6 @@ _FLATTEN_MAP: dict[str, str] = {
|
||||
"model.qwen_chat_timeout_seconds": "qwen_chat_timeout_seconds",
|
||||
"model.qwen_chat_max_tokens": "qwen_chat_max_tokens",
|
||||
"model.qwen_chat_enable_thinking": "qwen_chat_enable_thinking",
|
||||
"model.optimizer_qwen_chat_base_url": "optimizer_qwen_chat_base_url",
|
||||
"model.optimizer_qwen_chat_api_key": "optimizer_qwen_chat_api_key",
|
||||
"model.optimizer_qwen_chat_temperature": "optimizer_qwen_chat_temperature",
|
||||
"model.optimizer_qwen_chat_timeout_seconds": "optimizer_qwen_chat_timeout_seconds",
|
||||
"model.optimizer_qwen_chat_max_tokens": "optimizer_qwen_chat_max_tokens",
|
||||
"model.optimizer_qwen_chat_enable_thinking": "optimizer_qwen_chat_enable_thinking",
|
||||
"model.target_qwen_chat_base_url": "target_qwen_chat_base_url",
|
||||
"model.target_qwen_chat_api_key": "target_qwen_chat_api_key",
|
||||
"model.target_qwen_chat_temperature": "target_qwen_chat_temperature",
|
||||
"model.target_qwen_chat_timeout_seconds": "target_qwen_chat_timeout_seconds",
|
||||
"model.target_qwen_chat_max_tokens": "target_qwen_chat_max_tokens",
|
||||
"model.target_qwen_chat_enable_thinking": "target_qwen_chat_enable_thinking",
|
||||
"model.minimax_base_url": "minimax_base_url",
|
||||
"model.minimax_api_key": "minimax_api_key",
|
||||
"model.minimax_model": "minimax_model",
|
||||
"model.minimax_temperature": "minimax_temperature",
|
||||
"model.minimax_max_tokens": "minimax_max_tokens",
|
||||
"model.minimax_enable_thinking": "minimax_enable_thinking",
|
||||
"train.num_epochs": "num_epochs",
|
||||
"train.train_size": "train_size",
|
||||
"train.steps_per_epoch": "steps_per_epoch",
|
||||
|
||||
@@ -51,7 +51,6 @@ from skillopt.model import (
|
||||
configure_azure_openai,
|
||||
configure_claude_code_exec,
|
||||
configure_codex_exec,
|
||||
configure_minimax_chat,
|
||||
configure_qwen_chat,
|
||||
get_token_summary,
|
||||
reset_token_tracker,
|
||||
@@ -629,36 +628,14 @@ class ReflACTTrainer:
|
||||
effort=cfg.get("claude_code_exec_effort", cfg.get("reasoning_effort", "medium")),
|
||||
max_thinking_tokens=cfg.get("claude_code_exec_max_thinking_tokens", 16384),
|
||||
)
|
||||
configure_qwen_chat(
|
||||
base_url=cfg.get("qwen_chat_base_url") or None,
|
||||
api_key=cfg.get("qwen_chat_api_key") or None,
|
||||
temperature=cfg.get("qwen_chat_temperature"),
|
||||
timeout_seconds=cfg.get("qwen_chat_timeout_seconds"),
|
||||
max_tokens=cfg.get("qwen_chat_max_tokens"),
|
||||
enable_thinking=cfg.get("qwen_chat_enable_thinking"),
|
||||
optimizer_base_url=cfg.get("optimizer_qwen_chat_base_url") or None,
|
||||
optimizer_api_key=cfg.get("optimizer_qwen_chat_api_key") or None,
|
||||
optimizer_temperature=cfg.get("optimizer_qwen_chat_temperature"),
|
||||
optimizer_timeout_seconds=cfg.get("optimizer_qwen_chat_timeout_seconds"),
|
||||
optimizer_max_tokens=cfg.get("optimizer_qwen_chat_max_tokens"),
|
||||
optimizer_enable_thinking=cfg.get("optimizer_qwen_chat_enable_thinking"),
|
||||
target_base_url=cfg.get("target_qwen_chat_base_url") or None,
|
||||
target_api_key=cfg.get("target_qwen_chat_api_key") or None,
|
||||
target_temperature=cfg.get("target_qwen_chat_temperature"),
|
||||
target_timeout_seconds=cfg.get("target_qwen_chat_timeout_seconds"),
|
||||
target_max_tokens=cfg.get("target_qwen_chat_max_tokens"),
|
||||
target_enable_thinking=cfg.get("target_qwen_chat_enable_thinking"),
|
||||
)
|
||||
configure_minimax_chat(
|
||||
base_url=cfg.get("minimax_base_url") or None,
|
||||
api_key=cfg.get("minimax_api_key") or None,
|
||||
temperature=cfg.get("minimax_temperature"),
|
||||
max_tokens=cfg.get("minimax_max_tokens"),
|
||||
enable_thinking=cfg.get("minimax_enable_thinking"),
|
||||
configure_qwen_chat(
|
||||
base_url=cfg.get("qwen_chat_base_url") or None,
|
||||
api_key=cfg.get("qwen_chat_api_key") or None,
|
||||
temperature=cfg.get("qwen_chat_temperature"),
|
||||
timeout_seconds=cfg.get("qwen_chat_timeout_seconds"),
|
||||
max_tokens=cfg.get("qwen_chat_max_tokens"),
|
||||
enable_thinking=cfg.get("qwen_chat_enable_thinking"),
|
||||
)
|
||||
minimax_model_cfg = cfg.get("minimax_model")
|
||||
if minimax_model_cfg and cfg.get("target_backend") == "minimax_chat":
|
||||
set_target_deployment(str(minimax_model_cfg))
|
||||
os.environ["REFLACT_CODEX_TRACE_TO_OPTIMIZER"] = (
|
||||
"1"
|
||||
if target_backend == "codex_exec" and cfg.get("codex_trace_to_optimizer", False)
|
||||
|
||||
@@ -13,7 +13,7 @@ This directory provides scaffold files for adding a new benchmark to SkillOpt.
|
||||
1. Copy this directory: `cp -r skillopt/envs/_template skillopt/envs/your_benchmark`
|
||||
2. Rename files: remove `_template` suffix
|
||||
3. Implement the `TODO` sections
|
||||
4. Register your adapter in `_ENV_REGISTRY` inside `scripts/train.py`
|
||||
4. Register in `skillopt/envs/__init__.py`
|
||||
5. Create config at `configs/your_benchmark/default.yaml`
|
||||
|
||||
See the [documentation](../../docs/guide/new-benchmark.md) for the full guide.
|
||||
|
||||
@@ -5,11 +5,11 @@
|
||||
# and customize the values below.
|
||||
|
||||
# Inherit global defaults
|
||||
_base_: ../_base_/default.yaml
|
||||
_base_: ['../_base_/default.yaml']
|
||||
|
||||
# ── Environment ──────────────────────────────────
|
||||
env:
|
||||
name: your_benchmark # Must match _ENV_REGISTRY key in scripts/train.py
|
||||
name: your_benchmark # Must match registry key
|
||||
data_path: data/your_benchmark # Path to your data
|
||||
split_mode: ratio # "ratio" or "split_dir"
|
||||
split_ratio: "2:1:7" # train:val:test
|
||||
|
||||
@@ -4,80 +4,89 @@ Benchmark Environment Template
|
||||
Copy this file and implement the TODO sections to add a new benchmark.
|
||||
|
||||
The EnvAdapter is responsible for:
|
||||
1. Building train/eval environment payloads
|
||||
2. Running rollout and returning scored result rows
|
||||
3. Reflecting on results and returning patch candidates
|
||||
1. Executing tasks using the target model + current skill document
|
||||
2. Evaluating predictions against ground truth
|
||||
3. Returning structured results for the training loop
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from skillopt.datasets.base import BatchSpec
|
||||
from skillopt.envs._template.loader_template import TemplateBenchmarkDataLoader
|
||||
from skillopt.envs.base import EnvAdapter
|
||||
|
||||
|
||||
class TemplateBenchmarkAdapter(EnvAdapter):
|
||||
class TemplateBenchmarkEnv(EnvAdapter):
|
||||
"""
|
||||
Environment adapter for <Your Benchmark Name>.
|
||||
|
||||
|
||||
Rename this class and implement the abstract methods below.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
split_dir: str = "",
|
||||
data_path: str = "",
|
||||
split_mode: str = "ratio",
|
||||
split_ratio: str = "2:1:7",
|
||||
split_seed: int = 42,
|
||||
split_output_dir: str = "",
|
||||
seed: int = 42,
|
||||
limit: int = 0,
|
||||
**kwargs,
|
||||
) -> None:
|
||||
self.dataloader = TemplateBenchmarkDataLoader(
|
||||
split_dir=split_dir,
|
||||
data_path=data_path,
|
||||
split_mode=split_mode,
|
||||
split_ratio=split_ratio,
|
||||
split_seed=split_seed,
|
||||
split_output_dir=split_output_dir,
|
||||
seed=seed,
|
||||
limit=limit,
|
||||
)
|
||||
# TODO: initialize runtime options, e.g.
|
||||
# self.max_retries = int(kwargs.get("max_retries", 3))
|
||||
# self.timeout_s = int(kwargs.get("timeout_s", 120))
|
||||
def __init__(self, cfg: dict):
|
||||
super().__init__(cfg)
|
||||
# TODO: Initialize benchmark-specific state
|
||||
# Example: self.tools = load_tools(cfg)
|
||||
|
||||
def setup(self, cfg: dict) -> None:
|
||||
super().setup(cfg)
|
||||
self.dataloader.setup(cfg)
|
||||
|
||||
def get_dataloader(self):
|
||||
return self.dataloader
|
||||
|
||||
def build_env_from_batch(self, batch: BatchSpec, **kwargs):
|
||||
return list(batch.payload or [])
|
||||
|
||||
def build_train_env(self, batch_size: int, seed: int, **kwargs):
|
||||
batch = self.dataloader.build_train_batch(batch_size=batch_size, seed=seed, **kwargs)
|
||||
return self.build_env_from_batch(batch, **kwargs)
|
||||
|
||||
def build_eval_env(self, env_num: int, split: str, seed: int, **kwargs):
|
||||
batch = self.dataloader.build_eval_batch(env_num=env_num, split=split, seed=seed, **kwargs)
|
||||
return self.build_env_from_batch(batch, **kwargs)
|
||||
|
||||
def rollout(self, env_manager, skill_content: str, out_dir: str, **kwargs) -> list[dict]:
|
||||
async def execute(self, item, skill: str, model):
|
||||
"""
|
||||
Run one batch and return list[dict] with at least:
|
||||
{"id": str, "hard": int, "soft": float}
|
||||
"""
|
||||
raise NotImplementedError("Implement rollout() for your benchmark")
|
||||
Execute a single task with the target model.
|
||||
|
||||
def reflect(self, results: list[dict], skill_content: str, out_dir: str, **kwargs) -> list[dict | None]:
|
||||
"""
|
||||
Reflect on rollout results and return patch dicts (or None entries).
|
||||
"""
|
||||
raise NotImplementedError("Implement reflect() for your benchmark")
|
||||
Args:
|
||||
item: DataItem with .id, .input, .ground_truth, .metadata
|
||||
skill: Current skill document content (Markdown string)
|
||||
model: Target model backend instance
|
||||
|
||||
def get_task_types(self) -> list[str]:
|
||||
return ["your_benchmark"]
|
||||
Returns:
|
||||
TaskResult with prediction, score, and trajectory
|
||||
"""
|
||||
# Step 1: Build the prompt combining skill + task input
|
||||
prompt = self.build_prompt(item, skill)
|
||||
|
||||
# Step 2: Call the target model
|
||||
# TODO: Customize the message format for your benchmark
|
||||
messages = [
|
||||
{"role": "system", "content": skill},
|
||||
{"role": "user", "content": item.input},
|
||||
]
|
||||
response = await model.generate(messages)
|
||||
|
||||
# Step 3: Parse the model response into a prediction
|
||||
prediction = self.parse_response(response.content)
|
||||
|
||||
# Step 4: Score the prediction
|
||||
score = self.evaluate(prediction, item.ground_truth)
|
||||
|
||||
# Step 5: Return structured result
|
||||
return {
|
||||
"item_id": item.id,
|
||||
"prediction": prediction,
|
||||
"score": score,
|
||||
"trajectory": messages + [{"role": "assistant", "content": response.content}],
|
||||
}
|
||||
|
||||
def evaluate(self, prediction: str, ground_truth: str) -> float:
|
||||
"""
|
||||
Score a prediction against the ground truth.
|
||||
|
||||
Returns:
|
||||
Float between 0.0 (wrong) and 1.0 (correct)
|
||||
|
||||
TODO: Implement your scoring metric. Common options:
|
||||
- Exact match: float(pred.strip().lower() == gt.strip().lower())
|
||||
- F1 score: compute token overlap
|
||||
- ANLS: for document QA tasks
|
||||
- Custom: any float in [0, 1]
|
||||
"""
|
||||
# Placeholder — exact match
|
||||
return float(prediction.strip().lower() == ground_truth.strip().lower())
|
||||
|
||||
def build_prompt(self, item, skill: str) -> str:
|
||||
"""Combine skill document with task input."""
|
||||
return f"{skill}\n\n---\n\nQuestion: {item.input}"
|
||||
|
||||
def parse_response(self, response: str) -> str:
|
||||
"""
|
||||
Extract the answer from the model's raw response.
|
||||
|
||||
TODO: Implement extraction logic. For example:
|
||||
- Extract text after "Answer:"
|
||||
- Parse JSON output
|
||||
- Extract from code blocks
|
||||
"""
|
||||
return response.strip()
|
||||
|
||||
@@ -3,38 +3,101 @@ Benchmark Data Loader Template
|
||||
================================
|
||||
Copy this file and implement the TODO sections to load your benchmark data.
|
||||
|
||||
The SplitDataLoader is responsible for:
|
||||
1. Loading raw data from disk for ratio split mode
|
||||
2. Loading items from train/val/test directories for split_dir mode
|
||||
3. Returning list[dict] items used by the training loop
|
||||
The DataLoader is responsible for:
|
||||
1. Loading raw data from disk
|
||||
2. Splitting into train / validation / test sets
|
||||
3. Providing DataItem objects to the training loop
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from skillopt.datasets.base import SplitDataLoader
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
class TemplateBenchmarkDataLoader(SplitDataLoader):
|
||||
class TemplateBenchmarkLoader:
|
||||
"""
|
||||
Data loader for <Your Benchmark Name>.
|
||||
|
||||
|
||||
Rename this class and implement the methods below.
|
||||
"""
|
||||
|
||||
def load_raw_items(self, data_path: str) -> list[dict]:
|
||||
"""
|
||||
Parse raw benchmark data for split_mode="ratio".
|
||||
def __init__(self, data_dir: str = "data/your_benchmark", **kwargs):
|
||||
self.data_dir = Path(data_dir)
|
||||
self.items = []
|
||||
self.splits = {}
|
||||
|
||||
Return a list of normalized item dicts.
|
||||
def setup(self, cfg: dict):
|
||||
"""
|
||||
# TODO: parse your raw JSON/JSONL/CSV format and return list[dict]
|
||||
# with deterministic "id" values.
|
||||
return super().load_raw_items(data_path)
|
||||
Initialize the loader with config.
|
||||
|
||||
Called once before training starts.
|
||||
|
||||
Args:
|
||||
cfg: Dict with keys like 'split_mode', 'train_ratio', 'val_ratio', etc.
|
||||
"""
|
||||
# Step 1: Load raw data
|
||||
self.items = self._load_items()
|
||||
|
||||
def load_split_items(self, split_path: str) -> list[dict]:
|
||||
"""
|
||||
Parse one split directory for split_mode="split_dir".
|
||||
# Step 2: Create splits
|
||||
split_mode = cfg.get("split_mode", "ratio")
|
||||
if split_mode == "ratio":
|
||||
self._split_by_ratio(
|
||||
train_ratio=cfg.get("train_ratio", 0.7),
|
||||
val_ratio=cfg.get("val_ratio", 0.15),
|
||||
)
|
||||
elif split_mode == "split_dir":
|
||||
self._load_predefined_splits(cfg.get("split_dir", self.data_dir))
|
||||
|
||||
split_path points to train/, val/, or test/.
|
||||
def _load_items(self) -> list:
|
||||
"""
|
||||
# TODO: customize when split directories contain non-standard files.
|
||||
return super().load_split_items(split_path)
|
||||
Load raw data into structured items.
|
||||
|
||||
TODO: Implement data loading. Each item should have at minimum:
|
||||
- id: unique identifier
|
||||
- input: the task input (question, instruction, etc.)
|
||||
- ground_truth: the expected answer
|
||||
- metadata: optional dict with extra info
|
||||
|
||||
Example:
|
||||
items = []
|
||||
for path in self.data_dir.glob("*.json"):
|
||||
data = json.loads(path.read_text())
|
||||
for entry in data:
|
||||
items.append({
|
||||
"id": entry["id"],
|
||||
"input": entry["question"],
|
||||
"ground_truth": entry["answer"],
|
||||
"metadata": {"source": path.name},
|
||||
})
|
||||
return items
|
||||
"""
|
||||
raise NotImplementedError("Implement _load_items() for your benchmark")
|
||||
|
||||
def _split_by_ratio(self, train_ratio: float, val_ratio: float):
|
||||
"""Split items by ratio."""
|
||||
import random
|
||||
random.shuffle(self.items)
|
||||
n = len(self.items)
|
||||
n_train = int(n * train_ratio)
|
||||
n_val = int(n * val_ratio)
|
||||
self.splits = {
|
||||
"train": self.items[:n_train],
|
||||
"valid": self.items[n_train:n_train + n_val],
|
||||
"test": self.items[n_train + n_val:],
|
||||
}
|
||||
|
||||
def _load_predefined_splits(self, split_dir):
|
||||
"""Load from pre-split directories."""
|
||||
# TODO: Implement if your benchmark has pre-defined splits
|
||||
raise NotImplementedError
|
||||
|
||||
def get_split_items(self, split: str) -> list:
|
||||
"""
|
||||
Return items for a given split.
|
||||
|
||||
Args:
|
||||
split: One of "train", "valid", "test"
|
||||
|
||||
Returns:
|
||||
List of data items for the requested split
|
||||
"""
|
||||
if split not in self.splits:
|
||||
raise ValueError(f"Unknown split '{split}'. Available: {list(self.splits.keys())}")
|
||||
return self.splits[split]
|
||||
|
||||
+2
-111
@@ -6,7 +6,6 @@ from typing import Any
|
||||
|
||||
from skillopt.model import azure_openai as _openai
|
||||
from skillopt.model import claude_backend as _claude
|
||||
from skillopt.model import minimax_backend as _minimax
|
||||
from skillopt.model import qwen_backend as _qwen
|
||||
from skillopt.model.backend_config import ( # noqa: F401
|
||||
configure_claude_code_exec,
|
||||
@@ -51,10 +50,6 @@ def set_backend(name: str | None) -> str:
|
||||
set_optimizer_backend("openai_chat")
|
||||
set_target_backend("qwen_chat")
|
||||
return "qwen_chat"
|
||||
if normalized in {"minimax", "minimax_chat"}:
|
||||
set_optimizer_backend("openai_chat")
|
||||
set_target_backend("minimax_chat")
|
||||
return "minimax_chat"
|
||||
raise ValueError(f"Unsupported legacy backend: {name!r}")
|
||||
|
||||
|
||||
@@ -64,16 +59,12 @@ def get_backend_name() -> str:
|
||||
target = get_target_backend()
|
||||
if optimizer == "claude_chat" and target == "claude_chat":
|
||||
return "claude_chat"
|
||||
if optimizer == "qwen_chat" and target == "qwen_chat":
|
||||
return "qwen_chat"
|
||||
if optimizer == "openai_chat" and target == "openai_chat":
|
||||
return "azure_openai"
|
||||
if optimizer == "openai_chat" and target == "codex_exec":
|
||||
return "codex"
|
||||
if optimizer == "openai_chat" and target == "qwen_chat":
|
||||
return "qwen_chat"
|
||||
if optimizer == "openai_chat" and target == "minimax_chat":
|
||||
return "minimax_chat"
|
||||
return f"{optimizer}+{target}"
|
||||
|
||||
|
||||
@@ -95,16 +86,6 @@ def chat_optimizer(
|
||||
stage=stage,
|
||||
timeout=timeout,
|
||||
)
|
||||
if get_optimizer_backend() == "qwen_chat":
|
||||
return _qwen.chat_optimizer(
|
||||
system=system,
|
||||
user=user,
|
||||
max_completion_tokens=max_completion_tokens,
|
||||
retries=retries,
|
||||
stage=stage,
|
||||
reasoning_effort=reasoning_effort,
|
||||
timeout=timeout,
|
||||
)
|
||||
return _openai.chat_optimizer(
|
||||
system=system,
|
||||
user=user,
|
||||
@@ -143,18 +124,9 @@ def chat_target(
|
||||
stage=stage,
|
||||
reasoning_effort=reasoning_effort,
|
||||
)
|
||||
if get_target_backend() == "minimax_chat":
|
||||
return _minimax.chat_target(
|
||||
system=system,
|
||||
user=user,
|
||||
max_completion_tokens=max_completion_tokens,
|
||||
retries=retries,
|
||||
stage=stage,
|
||||
reasoning_effort=reasoning_effort,
|
||||
)
|
||||
if not is_target_chat_backend():
|
||||
raise NotImplementedError(
|
||||
"chat_target is only supported with target_backend=openai_chat, claude_chat, qwen_chat, or minimax_chat. "
|
||||
"chat_target is only supported with target_backend=openai_chat, claude_chat, or qwen_chat. "
|
||||
"Exec backends are handled in environment-specific rollout code."
|
||||
)
|
||||
return _openai.chat_target(
|
||||
@@ -191,18 +163,6 @@ def chat_optimizer_messages(
|
||||
return_message=return_message,
|
||||
timeout=timeout,
|
||||
)
|
||||
if get_optimizer_backend() == "qwen_chat":
|
||||
return _qwen.chat_optimizer_messages(
|
||||
messages=messages,
|
||||
max_completion_tokens=max_completion_tokens,
|
||||
retries=retries,
|
||||
stage=stage,
|
||||
reasoning_effort=reasoning_effort,
|
||||
tools=tools,
|
||||
tool_choice=tool_choice,
|
||||
return_message=return_message,
|
||||
timeout=timeout,
|
||||
)
|
||||
return _openai.chat_optimizer_messages(
|
||||
messages=messages,
|
||||
max_completion_tokens=max_completion_tokens,
|
||||
@@ -250,20 +210,9 @@ def chat_target_messages(
|
||||
tool_choice=tool_choice,
|
||||
return_message=return_message,
|
||||
)
|
||||
if get_target_backend() == "minimax_chat":
|
||||
return _minimax.chat_target_messages(
|
||||
messages=messages,
|
||||
max_completion_tokens=max_completion_tokens,
|
||||
retries=retries,
|
||||
stage=stage,
|
||||
reasoning_effort=reasoning_effort,
|
||||
tools=tools,
|
||||
tool_choice=tool_choice,
|
||||
return_message=return_message,
|
||||
)
|
||||
if not is_target_chat_backend():
|
||||
raise NotImplementedError(
|
||||
"chat_target_messages is only supported with target_backend=openai_chat, claude_chat, qwen_chat, or minimax_chat. "
|
||||
"chat_target_messages is only supported with target_backend=openai_chat, claude_chat, or qwen_chat. "
|
||||
"Exec backends are handled in environment-specific rollout code."
|
||||
)
|
||||
return _openai.chat_target_messages(
|
||||
@@ -352,17 +301,6 @@ def get_token_summary() -> dict:
|
||||
summary[stage]["prompt_tokens"] += values["prompt_tokens"]
|
||||
summary[stage]["completion_tokens"] += values["completion_tokens"]
|
||||
summary[stage]["total_tokens"] += values["total_tokens"]
|
||||
minimax_summary = _minimax.get_token_summary()
|
||||
for stage, values in minimax_summary.items():
|
||||
if stage == "_total":
|
||||
continue
|
||||
if stage not in summary:
|
||||
summary[stage] = values
|
||||
continue
|
||||
summary[stage]["calls"] += values["calls"]
|
||||
summary[stage]["prompt_tokens"] += values["prompt_tokens"]
|
||||
summary[stage]["completion_tokens"] += values["completion_tokens"]
|
||||
summary[stage]["total_tokens"] += values["total_tokens"]
|
||||
total = {
|
||||
"calls": 0,
|
||||
"prompt_tokens": 0,
|
||||
@@ -384,7 +322,6 @@ def reset_token_tracker() -> None:
|
||||
_openai.reset_token_tracker()
|
||||
_claude.reset_token_tracker()
|
||||
_qwen.reset_token_tracker()
|
||||
_minimax.reset_token_tracker()
|
||||
|
||||
|
||||
def configure_azure_openai(
|
||||
@@ -438,18 +375,6 @@ def configure_qwen_chat(
|
||||
timeout_seconds: float | str | None = None,
|
||||
max_tokens: int | str | None = None,
|
||||
enable_thinking: bool | str | None = None,
|
||||
optimizer_base_url: str | None = None,
|
||||
optimizer_api_key: str | None = None,
|
||||
optimizer_temperature: float | str | None = None,
|
||||
optimizer_timeout_seconds: float | str | None = None,
|
||||
optimizer_max_tokens: int | str | None = None,
|
||||
optimizer_enable_thinking: bool | str | None = None,
|
||||
target_base_url: str | None = None,
|
||||
target_api_key: str | None = None,
|
||||
target_temperature: float | str | None = None,
|
||||
target_timeout_seconds: float | str | None = None,
|
||||
target_max_tokens: int | str | None = None,
|
||||
target_enable_thinking: bool | str | None = None,
|
||||
) -> None:
|
||||
_qwen.configure_qwen_chat(
|
||||
base_url=base_url,
|
||||
@@ -458,37 +383,6 @@ def configure_qwen_chat(
|
||||
timeout_seconds=timeout_seconds,
|
||||
max_tokens=max_tokens,
|
||||
enable_thinking=enable_thinking,
|
||||
optimizer_base_url=optimizer_base_url,
|
||||
optimizer_api_key=optimizer_api_key,
|
||||
optimizer_temperature=optimizer_temperature,
|
||||
optimizer_timeout_seconds=optimizer_timeout_seconds,
|
||||
optimizer_max_tokens=optimizer_max_tokens,
|
||||
optimizer_enable_thinking=optimizer_enable_thinking,
|
||||
target_base_url=target_base_url,
|
||||
target_api_key=target_api_key,
|
||||
target_temperature=target_temperature,
|
||||
target_timeout_seconds=target_timeout_seconds,
|
||||
target_max_tokens=target_max_tokens,
|
||||
target_enable_thinking=target_enable_thinking,
|
||||
)
|
||||
|
||||
|
||||
def configure_minimax_chat(
|
||||
*,
|
||||
base_url: str | None = None,
|
||||
api_key: str | None = None,
|
||||
temperature: float | str | None = None,
|
||||
timeout_seconds: float | str | None = None,
|
||||
max_tokens: int | str | None = None,
|
||||
enable_thinking: bool | str | None = None,
|
||||
) -> None:
|
||||
_minimax.configure_minimax_chat(
|
||||
base_url=base_url,
|
||||
api_key=api_key,
|
||||
temperature=temperature,
|
||||
timeout_seconds=timeout_seconds,
|
||||
max_tokens=max_tokens,
|
||||
enable_thinking=enable_thinking,
|
||||
)
|
||||
|
||||
|
||||
@@ -496,17 +390,14 @@ def set_reasoning_effort(effort: str | None) -> None:
|
||||
_openai.set_reasoning_effort(effort)
|
||||
_claude.set_reasoning_effort(effort)
|
||||
_qwen.set_reasoning_effort(effort)
|
||||
_minimax.set_reasoning_effort(effort)
|
||||
|
||||
|
||||
def set_target_deployment(deployment: str) -> None:
|
||||
_openai.set_target_deployment(deployment)
|
||||
_claude.set_target_deployment(deployment)
|
||||
_qwen.set_target_deployment(deployment)
|
||||
_minimax.set_target_deployment(deployment)
|
||||
|
||||
|
||||
def set_optimizer_deployment(deployment: str) -> None:
|
||||
_openai.set_optimizer_deployment(deployment)
|
||||
_claude.set_optimizer_deployment(deployment)
|
||||
_qwen.set_optimizer_deployment(deployment)
|
||||
|
||||
@@ -336,10 +336,9 @@ def get_target_client() -> AzureOpenAI | OpenAI:
|
||||
from skillopt.model.backend_config import get_target_backend
|
||||
if get_target_backend() == "qwen_chat":
|
||||
from skillopt.model import qwen_backend as _qwen
|
||||
target_config = _qwen.TARGET_CONFIG
|
||||
_target_client = OpenAI(
|
||||
base_url=target_config.base_url,
|
||||
api_key=target_config.api_key or "dummy",
|
||||
base_url=_qwen.BASE_URL,
|
||||
api_key=_qwen.API_KEY or "dummy",
|
||||
)
|
||||
else:
|
||||
_target_client = _make_client("target")
|
||||
|
||||
@@ -49,10 +49,10 @@ CLAUDE_CODE_EXEC_MAX_THINKING_TOKENS = max(
|
||||
def set_optimizer_backend(backend: str) -> None:
|
||||
global OPTIMIZER_BACKEND
|
||||
OPTIMIZER_BACKEND = normalize_backend_name(backend or "openai_chat")
|
||||
if OPTIMIZER_BACKEND not in {"openai_chat", "claude_chat", "qwen_chat", "minimax_chat"}:
|
||||
if OPTIMIZER_BACKEND not in {"openai_chat", "claude_chat"}:
|
||||
raise ValueError(
|
||||
f"Unsupported optimizer backend: {OPTIMIZER_BACKEND!r}. "
|
||||
"Supported values are 'openai_chat', 'claude_chat', 'qwen_chat', and 'minimax_chat'."
|
||||
"Supported values are 'openai_chat' and 'claude_chat'."
|
||||
)
|
||||
os.environ["OPTIMIZER_BACKEND"] = OPTIMIZER_BACKEND
|
||||
|
||||
@@ -64,10 +64,10 @@ def get_optimizer_backend() -> str:
|
||||
def set_target_backend(backend: str) -> None:
|
||||
global TARGET_BACKEND
|
||||
TARGET_BACKEND = normalize_backend_name(backend or "openai_chat")
|
||||
if TARGET_BACKEND not in {"openai_chat", "claude_chat", "qwen_chat", "minimax_chat", "codex_exec", "claude_code_exec"}:
|
||||
if TARGET_BACKEND not in {"openai_chat", "claude_chat", "qwen_chat", "codex_exec", "claude_code_exec"}:
|
||||
raise ValueError(
|
||||
f"Unsupported target backend: {TARGET_BACKEND!r}. "
|
||||
"Supported values are 'openai_chat', 'claude_chat', 'qwen_chat', 'minimax_chat', 'codex_exec', and 'claude_code_exec'."
|
||||
"Supported values are 'openai_chat', 'claude_chat', 'qwen_chat', 'codex_exec', and 'claude_code_exec'."
|
||||
)
|
||||
os.environ["TARGET_BACKEND"] = TARGET_BACKEND
|
||||
|
||||
@@ -81,11 +81,11 @@ def is_target_exec_backend() -> bool:
|
||||
|
||||
|
||||
def is_optimizer_chat_backend() -> bool:
|
||||
return OPTIMIZER_BACKEND in {"openai_chat", "claude_chat", "qwen_chat", "minimax_chat"}
|
||||
return OPTIMIZER_BACKEND in {"openai_chat", "claude_chat"}
|
||||
|
||||
|
||||
def is_target_chat_backend() -> bool:
|
||||
return TARGET_BACKEND in {"openai_chat", "claude_chat", "qwen_chat", "minimax_chat"}
|
||||
return TARGET_BACKEND in {"openai_chat", "claude_chat", "qwen_chat"}
|
||||
|
||||
|
||||
def configure_codex_exec(
|
||||
|
||||
@@ -25,7 +25,6 @@ _BACKEND_DEFAULT_MODELS = {
|
||||
"claude_chat": "claude-sonnet-4-6",
|
||||
"claude_code_exec": "claude-sonnet-4-6",
|
||||
"qwen_chat": "Qwen/Qwen3.5-4B",
|
||||
"minimax_chat": "MiniMax-M2.7",
|
||||
}
|
||||
|
||||
_BACKEND_ALIASES = {
|
||||
@@ -42,8 +41,6 @@ _BACKEND_ALIASES = {
|
||||
"anthropic": "claude_chat",
|
||||
"qwen": "qwen_chat",
|
||||
"qwen_chat": "qwen_chat",
|
||||
"minimax": "minimax_chat",
|
||||
"minimax_chat": "minimax_chat",
|
||||
}
|
||||
|
||||
|
||||
|
||||
@@ -1,277 +0,0 @@
|
||||
"""OpenAI-compatible MiniMax chat backend for the target path."""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import threading
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from typing import Any
|
||||
|
||||
from skillopt.model.common import (
|
||||
CompatAssistantMessage,
|
||||
CompatToolCall,
|
||||
CompatToolFunction,
|
||||
TokenTracker,
|
||||
default_model_for_backend,
|
||||
)
|
||||
|
||||
BASE_URL = os.environ.get("MINIMAX_BASE_URL", "https://api.minimax.io/v1")
|
||||
API_KEY = os.environ.get("MINIMAX_API_KEY", "")
|
||||
TIMEOUT_SECONDS = float(os.environ.get("MINIMAX_TIMEOUT_SECONDS", "300") or 300)
|
||||
MAX_TOKENS = int(os.environ.get("MINIMAX_MAX_TOKENS", "8000") or 8000)
|
||||
TEMPERATURE: float | None = None
|
||||
_raw_temperature = os.environ.get("MINIMAX_TEMPERATURE", "0.7").strip()
|
||||
if _raw_temperature:
|
||||
TEMPERATURE = float(_raw_temperature)
|
||||
ENABLE_THINKING = os.environ.get("MINIMAX_ENABLE_THINKING", "false").strip().lower() in {
|
||||
"1",
|
||||
"true",
|
||||
"yes",
|
||||
"on",
|
||||
}
|
||||
|
||||
TARGET_DEPLOYMENT = os.environ.get(
|
||||
"TARGET_DEPLOYMENT",
|
||||
default_model_for_backend("minimax_chat"),
|
||||
)
|
||||
|
||||
_config_lock = threading.Lock()
|
||||
tracker = TokenTracker()
|
||||
|
||||
|
||||
def _chat_url() -> str:
|
||||
base = BASE_URL.rstrip("/")
|
||||
if base.endswith("/chat/completions"):
|
||||
return base
|
||||
return f"{base}/chat/completions"
|
||||
|
||||
|
||||
def _json_safe(value: Any) -> Any:
|
||||
if value is None or isinstance(value, (str, int, float, bool)):
|
||||
return value
|
||||
if isinstance(value, list):
|
||||
return [_json_safe(item) for item in value]
|
||||
if isinstance(value, dict):
|
||||
return {str(key): _json_safe(val) for key, val in value.items()}
|
||||
model_dump = getattr(value, "model_dump", None)
|
||||
if callable(model_dump):
|
||||
try:
|
||||
return model_dump(mode="json")
|
||||
except TypeError:
|
||||
return model_dump()
|
||||
return str(value)
|
||||
|
||||
|
||||
def _usage_from_payload(payload: dict[str, Any]) -> dict[str, int]:
|
||||
usage = payload.get("usage") or {}
|
||||
prompt_tokens = int(usage.get("prompt_tokens") or usage.get("input_tokens") or 0)
|
||||
completion_tokens = int(usage.get("completion_tokens") or usage.get("output_tokens") or 0)
|
||||
total_tokens = int(usage.get("total_tokens") or (prompt_tokens + completion_tokens))
|
||||
return {
|
||||
"prompt_tokens": prompt_tokens,
|
||||
"completion_tokens": completion_tokens,
|
||||
"total_tokens": total_tokens,
|
||||
}
|
||||
|
||||
|
||||
def _compat_message_from_payload(message: dict[str, Any], choice: dict[str, Any]) -> CompatAssistantMessage:
|
||||
content = message.get("content") or ""
|
||||
if not isinstance(content, str):
|
||||
content = json.dumps(content, ensure_ascii=False)
|
||||
tool_calls: list[CompatToolCall] = []
|
||||
for index, tool_call in enumerate(message.get("tool_calls") or [], start=1):
|
||||
function = tool_call.get("function") or {}
|
||||
tool_calls.append(
|
||||
CompatToolCall(
|
||||
id=str(tool_call.get("id") or f"minimax_tool_{index}"),
|
||||
type=str(tool_call.get("type") or "function"),
|
||||
function=CompatToolFunction(
|
||||
name=str(function.get("name") or ""),
|
||||
arguments=str(function.get("arguments") or "{}"),
|
||||
),
|
||||
)
|
||||
)
|
||||
return CompatAssistantMessage(
|
||||
content=content,
|
||||
tool_calls=tool_calls,
|
||||
metadata={
|
||||
"finish_reason": choice.get("finish_reason"),
|
||||
"choice0": _json_safe(choice),
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def _post_chat_completion(payload: dict[str, Any], timeout: float | None) -> dict[str, Any]:
|
||||
headers = {"Content-Type": "application/json"}
|
||||
if API_KEY:
|
||||
headers["Authorization"] = f"Bearer {API_KEY}"
|
||||
req = urllib.request.Request(
|
||||
_chat_url(),
|
||||
data=json.dumps(payload, ensure_ascii=False).encode("utf-8"),
|
||||
headers=headers,
|
||||
method="POST",
|
||||
)
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=timeout or TIMEOUT_SECONDS) as resp:
|
||||
raw = resp.read().decode("utf-8")
|
||||
except urllib.error.HTTPError as e:
|
||||
body = e.read().decode("utf-8", errors="replace")
|
||||
raise RuntimeError(f"MiniMax chat API returned HTTP {e.code}: {body}") from e
|
||||
except urllib.error.URLError as e:
|
||||
raise RuntimeError(f"MiniMax chat API request failed: {e}") from e
|
||||
try:
|
||||
return json.loads(raw)
|
||||
except json.JSONDecodeError as e:
|
||||
raise RuntimeError(f"MiniMax chat API returned non-JSON response: {raw[:1000]}") from e
|
||||
|
||||
|
||||
def _chat_messages_impl(
|
||||
messages: list[dict[str, Any]],
|
||||
max_completion_tokens: int,
|
||||
retries: int,
|
||||
stage: str,
|
||||
*,
|
||||
tools: list[dict[str, Any]] | None = None,
|
||||
tool_choice: str | dict[str, Any] | None = None,
|
||||
return_message: bool = False,
|
||||
deployment: str | None = None,
|
||||
timeout: float | None = None,
|
||||
) -> tuple[Any, dict[str, int]]:
|
||||
payload: dict[str, Any] = {
|
||||
"model": deployment or TARGET_DEPLOYMENT,
|
||||
"messages": _json_safe(messages),
|
||||
"max_tokens": min(max_completion_tokens, MAX_TOKENS),
|
||||
}
|
||||
payload["chat_template_kwargs"] = {"enable_thinking": ENABLE_THINKING}
|
||||
if TEMPERATURE is not None:
|
||||
payload["temperature"] = TEMPERATURE
|
||||
if tools:
|
||||
payload["tools"] = _json_safe(tools)
|
||||
if tool_choice is not None:
|
||||
payload["tool_choice"] = _json_safe(tool_choice)
|
||||
|
||||
last_err: Exception | None = None
|
||||
for attempt in range(retries):
|
||||
try:
|
||||
data = _post_chat_completion(payload, timeout)
|
||||
choices = data.get("choices") or []
|
||||
if not choices:
|
||||
raise RuntimeError(f"MiniMax chat API returned no choices: {data}")
|
||||
choice0 = choices[0]
|
||||
message = choice0.get("message") or {}
|
||||
text = message.get("content") or ""
|
||||
if not isinstance(text, str):
|
||||
text = json.dumps(text, ensure_ascii=False)
|
||||
usage_info = _usage_from_payload(data)
|
||||
tracker.record(stage, usage_info["prompt_tokens"], usage_info["completion_tokens"])
|
||||
if return_message:
|
||||
return _compat_message_from_payload(message, choice0), usage_info
|
||||
return text, usage_info
|
||||
except Exception as e: # noqa: BLE001
|
||||
last_err = e
|
||||
time.sleep(min(2 ** attempt, 30))
|
||||
raise RuntimeError(f"MiniMax chat call failed after {retries} retries: {last_err}")
|
||||
|
||||
|
||||
def configure_minimax_chat(
|
||||
*,
|
||||
base_url: str | None = None,
|
||||
api_key: str | None = None,
|
||||
temperature: float | str | None = None,
|
||||
timeout_seconds: float | str | None = None,
|
||||
max_tokens: int | str | None = None,
|
||||
enable_thinking: bool | str | None = None,
|
||||
) -> None:
|
||||
global BASE_URL, API_KEY, TEMPERATURE, TIMEOUT_SECONDS, MAX_TOKENS, ENABLE_THINKING
|
||||
with _config_lock:
|
||||
if base_url is not None:
|
||||
BASE_URL = str(base_url).strip() or BASE_URL
|
||||
os.environ["MINIMAX_BASE_URL"] = BASE_URL
|
||||
if api_key is not None:
|
||||
API_KEY = str(api_key).strip()
|
||||
os.environ["MINIMAX_API_KEY"] = API_KEY
|
||||
if temperature is not None:
|
||||
raw = str(temperature).strip()
|
||||
TEMPERATURE = float(raw) if raw else None
|
||||
os.environ["MINIMAX_TEMPERATURE"] = raw
|
||||
if timeout_seconds is not None:
|
||||
TIMEOUT_SECONDS = float(timeout_seconds)
|
||||
os.environ["MINIMAX_TIMEOUT_SECONDS"] = str(timeout_seconds)
|
||||
if max_tokens is not None:
|
||||
MAX_TOKENS = int(max_tokens)
|
||||
os.environ["MINIMAX_MAX_TOKENS"] = str(max_tokens)
|
||||
if enable_thinking is not None:
|
||||
if isinstance(enable_thinking, str):
|
||||
ENABLE_THINKING = enable_thinking.strip().lower() in {"1", "true", "yes", "on"}
|
||||
else:
|
||||
ENABLE_THINKING = bool(enable_thinking)
|
||||
os.environ["MINIMAX_ENABLE_THINKING"] = "true" if ENABLE_THINKING else "false"
|
||||
|
||||
|
||||
def get_max_tokens() -> int:
|
||||
return MAX_TOKENS
|
||||
|
||||
|
||||
def chat_target(
|
||||
system: str,
|
||||
user: str,
|
||||
max_completion_tokens: int = 16384,
|
||||
retries: int = 5,
|
||||
stage: str = "target",
|
||||
reasoning_effort: str | None = None,
|
||||
timeout: float | None = None,
|
||||
) -> tuple[str, dict[str, int]]:
|
||||
del reasoning_effort
|
||||
messages = [{"role": "system", "content": system}, {"role": "user", "content": user}]
|
||||
return _chat_messages_impl(
|
||||
messages,
|
||||
max_completion_tokens,
|
||||
retries,
|
||||
stage,
|
||||
timeout=timeout,
|
||||
)
|
||||
|
||||
|
||||
def chat_target_messages(
|
||||
messages: list[dict[str, Any]],
|
||||
max_completion_tokens: int = 16384,
|
||||
retries: int = 5,
|
||||
stage: str = "target",
|
||||
reasoning_effort: str | None = None,
|
||||
*,
|
||||
tools: list[dict[str, Any]] | None = None,
|
||||
tool_choice: str | dict[str, Any] | None = None,
|
||||
return_message: bool = False,
|
||||
timeout: float | None = None,
|
||||
) -> tuple[Any, dict[str, int]]:
|
||||
del reasoning_effort
|
||||
return _chat_messages_impl(
|
||||
messages,
|
||||
max_completion_tokens,
|
||||
retries,
|
||||
stage,
|
||||
tools=tools,
|
||||
tool_choice=tool_choice,
|
||||
return_message=return_message,
|
||||
timeout=timeout,
|
||||
)
|
||||
|
||||
|
||||
def get_token_summary() -> dict[str, dict[str, int]]:
|
||||
return tracker.summary()
|
||||
|
||||
|
||||
def reset_token_tracker() -> None:
|
||||
tracker.reset()
|
||||
|
||||
|
||||
def set_reasoning_effort(effort: str | None) -> None:
|
||||
del effort
|
||||
|
||||
|
||||
def set_target_deployment(deployment: str) -> None:
|
||||
global TARGET_DEPLOYMENT
|
||||
TARGET_DEPLOYMENT = deployment or default_model_for_backend("minimax_chat")
|
||||
os.environ["TARGET_DEPLOYMENT"] = TARGET_DEPLOYMENT
|
||||
+51
-229
@@ -1,7 +1,6 @@
|
||||
"""OpenAI-compatible Qwen chat backend for optimizer and target paths."""
|
||||
"""OpenAI-compatible Qwen chat backend for the target path."""
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
import json
|
||||
import os
|
||||
import threading
|
||||
@@ -18,72 +17,32 @@ from skillopt.model.common import (
|
||||
default_model_for_backend,
|
||||
)
|
||||
|
||||
BASE_URL = os.environ.get("QWEN_CHAT_BASE_URL", "http://localhost:8000/v1")
|
||||
API_KEY = os.environ.get("QWEN_CHAT_API_KEY", "")
|
||||
TIMEOUT_SECONDS = float(os.environ.get("QWEN_CHAT_TIMEOUT_SECONDS", "300") or 300)
|
||||
MAX_TOKENS = int(os.environ.get("QWEN_CHAT_MAX_TOKENS", "8000") or 8000)
|
||||
TEMPERATURE: float | None = None
|
||||
_raw_temperature = os.environ.get("QWEN_CHAT_TEMPERATURE", "0.7").strip()
|
||||
if _raw_temperature:
|
||||
TEMPERATURE = float(_raw_temperature)
|
||||
ENABLE_THINKING = os.environ.get("QWEN_CHAT_ENABLE_THINKING", "false").strip().lower() in {
|
||||
"1",
|
||||
"true",
|
||||
"yes",
|
||||
"on",
|
||||
}
|
||||
|
||||
@dataclass
|
||||
class QwenChatConfig:
|
||||
base_url: str
|
||||
api_key: str
|
||||
timeout_seconds: float
|
||||
max_tokens: int
|
||||
temperature: float | None
|
||||
enable_thinking: bool
|
||||
deployment: str
|
||||
|
||||
|
||||
def _parse_bool(value: Any, default: bool = False) -> bool:
|
||||
if value is None:
|
||||
return default
|
||||
return str(value).strip().lower() in {"1", "true", "yes", "on"}
|
||||
|
||||
|
||||
def _parse_optional_float(value: Any) -> float | None:
|
||||
if value is None:
|
||||
return None
|
||||
raw = str(value).strip()
|
||||
return float(raw) if raw else None
|
||||
|
||||
|
||||
def _parse_int(value: Any, default: int) -> int:
|
||||
if value is None:
|
||||
return default
|
||||
raw = str(value).strip()
|
||||
return int(raw) if raw else default
|
||||
|
||||
|
||||
def _role_env(role: str, key: str, default: str) -> str:
|
||||
role_key = f"{role.upper()}_QWEN_CHAT_{key}"
|
||||
generic_key = f"QWEN_CHAT_{key}"
|
||||
return os.environ.get(role_key) or os.environ.get(generic_key) or default
|
||||
|
||||
|
||||
def _initial_config(role: str) -> QwenChatConfig:
|
||||
role_upper = role.upper()
|
||||
deployment_env = "OPTIMIZER_DEPLOYMENT" if role == "optimizer" else "TARGET_DEPLOYMENT"
|
||||
return QwenChatConfig(
|
||||
base_url=_role_env(role, "BASE_URL", "http://localhost:8000/v1"),
|
||||
api_key=_role_env(role, "API_KEY", ""),
|
||||
timeout_seconds=float(_role_env(role, "TIMEOUT_SECONDS", "300") or 300),
|
||||
max_tokens=_parse_int(_role_env(role, "MAX_TOKENS", "8000"), 8000),
|
||||
temperature=_parse_optional_float(_role_env(role, "TEMPERATURE", "0.7")),
|
||||
enable_thinking=_parse_bool(_role_env(role, "ENABLE_THINKING", "false")),
|
||||
deployment=(
|
||||
os.environ.get(f"{role_upper}_QWEN_CHAT_MODEL")
|
||||
or os.environ.get("QWEN_CHAT_MODEL")
|
||||
or os.environ.get(deployment_env)
|
||||
or default_model_for_backend("qwen_chat")
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
OPTIMIZER_CONFIG = _initial_config("optimizer")
|
||||
TARGET_CONFIG = _initial_config("target")
|
||||
TARGET_DEPLOYMENT = os.environ.get(
|
||||
"TARGET_DEPLOYMENT",
|
||||
default_model_for_backend("qwen_chat"),
|
||||
)
|
||||
|
||||
_config_lock = threading.Lock()
|
||||
tracker = TokenTracker()
|
||||
|
||||
|
||||
def _chat_url(config: QwenChatConfig) -> str:
|
||||
base = config.base_url.rstrip("/")
|
||||
def _chat_url() -> str:
|
||||
base = BASE_URL.rstrip("/")
|
||||
if base.endswith("/chat/completions"):
|
||||
return base
|
||||
return f"{base}/chat/completions"
|
||||
@@ -144,22 +103,18 @@ def _compat_message_from_payload(message: dict[str, Any], choice: dict[str, Any]
|
||||
)
|
||||
|
||||
|
||||
def _post_chat_completion(
|
||||
payload: dict[str, Any],
|
||||
timeout: float | None,
|
||||
config: QwenChatConfig,
|
||||
) -> dict[str, Any]:
|
||||
def _post_chat_completion(payload: dict[str, Any], timeout: float | None) -> dict[str, Any]:
|
||||
headers = {"Content-Type": "application/json"}
|
||||
if config.api_key:
|
||||
headers["Authorization"] = f"Bearer {config.api_key}"
|
||||
if API_KEY:
|
||||
headers["Authorization"] = f"Bearer {API_KEY}"
|
||||
req = urllib.request.Request(
|
||||
_chat_url(config),
|
||||
_chat_url(),
|
||||
data=json.dumps(payload, ensure_ascii=False).encode("utf-8"),
|
||||
headers=headers,
|
||||
method="POST",
|
||||
)
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=timeout or config.timeout_seconds) as resp:
|
||||
with urllib.request.urlopen(req, timeout=timeout or TIMEOUT_SECONDS) as resp:
|
||||
raw = resp.read().decode("utf-8")
|
||||
except urllib.error.HTTPError as e:
|
||||
body = e.read().decode("utf-8", errors="replace")
|
||||
@@ -178,22 +133,20 @@ def _chat_messages_impl(
|
||||
retries: int,
|
||||
stage: str,
|
||||
*,
|
||||
role: str,
|
||||
tools: list[dict[str, Any]] | None = None,
|
||||
tool_choice: str | dict[str, Any] | None = None,
|
||||
return_message: bool = False,
|
||||
deployment: str | None = None,
|
||||
timeout: float | None = None,
|
||||
) -> tuple[Any, dict[str, int]]:
|
||||
config = OPTIMIZER_CONFIG if role == "optimizer" else TARGET_CONFIG
|
||||
payload: dict[str, Any] = {
|
||||
"model": deployment or config.deployment,
|
||||
"model": deployment or TARGET_DEPLOYMENT,
|
||||
"messages": _json_safe(messages),
|
||||
"max_tokens": min(max_completion_tokens, config.max_tokens),
|
||||
"max_tokens": min(max_completion_tokens, MAX_TOKENS),
|
||||
}
|
||||
payload["chat_template_kwargs"] = {"enable_thinking": config.enable_thinking}
|
||||
if config.temperature is not None:
|
||||
payload["temperature"] = config.temperature
|
||||
payload["chat_template_kwargs"] = {"enable_thinking": ENABLE_THINKING}
|
||||
if TEMPERATURE is not None:
|
||||
payload["temperature"] = TEMPERATURE
|
||||
if tools:
|
||||
payload["tools"] = _json_safe(tools)
|
||||
if tool_choice is not None:
|
||||
@@ -202,7 +155,7 @@ def _chat_messages_impl(
|
||||
last_err: Exception | None = None
|
||||
for attempt in range(retries):
|
||||
try:
|
||||
data = _post_chat_completion(payload, timeout, config)
|
||||
data = _post_chat_completion(payload, timeout)
|
||||
choices = data.get("choices") or []
|
||||
if not choices:
|
||||
raise RuntimeError(f"Qwen chat API returned no choices: {data}")
|
||||
@@ -230,134 +183,35 @@ def configure_qwen_chat(
|
||||
timeout_seconds: float | str | None = None,
|
||||
max_tokens: int | str | None = None,
|
||||
enable_thinking: bool | str | None = None,
|
||||
optimizer_base_url: str | None = None,
|
||||
optimizer_api_key: str | None = None,
|
||||
optimizer_temperature: float | str | None = None,
|
||||
optimizer_timeout_seconds: float | str | None = None,
|
||||
optimizer_max_tokens: int | str | None = None,
|
||||
optimizer_enable_thinking: bool | str | None = None,
|
||||
target_base_url: str | None = None,
|
||||
target_api_key: str | None = None,
|
||||
target_temperature: float | str | None = None,
|
||||
target_timeout_seconds: float | str | None = None,
|
||||
target_max_tokens: int | str | None = None,
|
||||
target_enable_thinking: bool | str | None = None,
|
||||
) -> None:
|
||||
global BASE_URL, API_KEY, TEMPERATURE, TIMEOUT_SECONDS, MAX_TOKENS, ENABLE_THINKING
|
||||
with _config_lock:
|
||||
if base_url is not None:
|
||||
os.environ["QWEN_CHAT_BASE_URL"] = str(base_url).strip()
|
||||
BASE_URL = str(base_url).strip() or BASE_URL
|
||||
os.environ["QWEN_CHAT_BASE_URL"] = BASE_URL
|
||||
if api_key is not None:
|
||||
os.environ["QWEN_CHAT_API_KEY"] = str(api_key).strip()
|
||||
API_KEY = str(api_key).strip()
|
||||
os.environ["QWEN_CHAT_API_KEY"] = API_KEY
|
||||
if temperature is not None:
|
||||
os.environ["QWEN_CHAT_TEMPERATURE"] = str(temperature).strip()
|
||||
raw = str(temperature).strip()
|
||||
TEMPERATURE = float(raw) if raw else None
|
||||
os.environ["QWEN_CHAT_TEMPERATURE"] = raw
|
||||
if timeout_seconds is not None:
|
||||
TIMEOUT_SECONDS = float(timeout_seconds)
|
||||
os.environ["QWEN_CHAT_TIMEOUT_SECONDS"] = str(timeout_seconds)
|
||||
if max_tokens is not None:
|
||||
MAX_TOKENS = int(max_tokens)
|
||||
os.environ["QWEN_CHAT_MAX_TOKENS"] = str(max_tokens)
|
||||
if enable_thinking is not None:
|
||||
os.environ["QWEN_CHAT_ENABLE_THINKING"] = (
|
||||
"true" if _parse_bool(enable_thinking) else "false"
|
||||
)
|
||||
_update_config(
|
||||
OPTIMIZER_CONFIG,
|
||||
"optimizer",
|
||||
base_url=optimizer_base_url if optimizer_base_url is not None else base_url,
|
||||
api_key=optimizer_api_key if optimizer_api_key is not None else api_key,
|
||||
temperature=(
|
||||
optimizer_temperature
|
||||
if optimizer_temperature is not None
|
||||
else temperature
|
||||
),
|
||||
timeout_seconds=(
|
||||
optimizer_timeout_seconds
|
||||
if optimizer_timeout_seconds is not None
|
||||
else timeout_seconds
|
||||
),
|
||||
max_tokens=optimizer_max_tokens if optimizer_max_tokens is not None else max_tokens,
|
||||
enable_thinking=(
|
||||
optimizer_enable_thinking
|
||||
if optimizer_enable_thinking is not None
|
||||
else enable_thinking
|
||||
),
|
||||
)
|
||||
_update_config(
|
||||
TARGET_CONFIG,
|
||||
"target",
|
||||
base_url=target_base_url if target_base_url is not None else base_url,
|
||||
api_key=target_api_key if target_api_key is not None else api_key,
|
||||
temperature=target_temperature if target_temperature is not None else temperature,
|
||||
timeout_seconds=(
|
||||
target_timeout_seconds
|
||||
if target_timeout_seconds is not None
|
||||
else timeout_seconds
|
||||
),
|
||||
max_tokens=target_max_tokens if target_max_tokens is not None else max_tokens,
|
||||
enable_thinking=(
|
||||
target_enable_thinking
|
||||
if target_enable_thinking is not None
|
||||
else enable_thinking
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def _update_config(
|
||||
config: QwenChatConfig,
|
||||
role: str,
|
||||
*,
|
||||
base_url: str | None = None,
|
||||
api_key: str | None = None,
|
||||
temperature: float | str | None = None,
|
||||
timeout_seconds: float | str | None = None,
|
||||
max_tokens: int | str | None = None,
|
||||
enable_thinking: bool | str | None = None,
|
||||
) -> None:
|
||||
env_prefix = role.upper()
|
||||
if base_url is not None:
|
||||
config.base_url = str(base_url).strip() or config.base_url
|
||||
os.environ[f"{env_prefix}_QWEN_CHAT_BASE_URL"] = config.base_url
|
||||
if api_key is not None:
|
||||
config.api_key = str(api_key).strip()
|
||||
os.environ[f"{env_prefix}_QWEN_CHAT_API_KEY"] = config.api_key
|
||||
if temperature is not None:
|
||||
raw = str(temperature).strip()
|
||||
config.temperature = float(raw) if raw else None
|
||||
os.environ[f"{env_prefix}_QWEN_CHAT_TEMPERATURE"] = raw
|
||||
if timeout_seconds is not None:
|
||||
config.timeout_seconds = float(timeout_seconds)
|
||||
os.environ[f"{env_prefix}_QWEN_CHAT_TIMEOUT_SECONDS"] = str(timeout_seconds)
|
||||
if max_tokens is not None:
|
||||
config.max_tokens = int(max_tokens)
|
||||
os.environ[f"{env_prefix}_QWEN_CHAT_MAX_TOKENS"] = str(max_tokens)
|
||||
if enable_thinking is not None:
|
||||
config.enable_thinking = _parse_bool(enable_thinking)
|
||||
os.environ[f"{env_prefix}_QWEN_CHAT_ENABLE_THINKING"] = (
|
||||
"true" if config.enable_thinking else "false"
|
||||
)
|
||||
if isinstance(enable_thinking, str):
|
||||
ENABLE_THINKING = enable_thinking.strip().lower() in {"1", "true", "yes", "on"}
|
||||
else:
|
||||
ENABLE_THINKING = bool(enable_thinking)
|
||||
os.environ["QWEN_CHAT_ENABLE_THINKING"] = "true" if ENABLE_THINKING else "false"
|
||||
|
||||
|
||||
def get_max_tokens() -> int:
|
||||
return TARGET_CONFIG.max_tokens
|
||||
|
||||
|
||||
def chat_optimizer(
|
||||
system: str,
|
||||
user: str,
|
||||
max_completion_tokens: int = 16384,
|
||||
retries: int = 5,
|
||||
stage: str = "optimizer",
|
||||
reasoning_effort: str | None = None,
|
||||
timeout: float | None = None,
|
||||
) -> tuple[str, dict[str, int]]:
|
||||
del reasoning_effort
|
||||
messages = [{"role": "system", "content": system}, {"role": "user", "content": user}]
|
||||
return _chat_messages_impl(
|
||||
messages,
|
||||
max_completion_tokens,
|
||||
retries,
|
||||
stage,
|
||||
role="optimizer",
|
||||
timeout=timeout,
|
||||
)
|
||||
return MAX_TOKENS
|
||||
|
||||
|
||||
def chat_target(
|
||||
@@ -376,33 +230,6 @@ def chat_target(
|
||||
max_completion_tokens,
|
||||
retries,
|
||||
stage,
|
||||
role="target",
|
||||
timeout=timeout,
|
||||
)
|
||||
|
||||
|
||||
def chat_optimizer_messages(
|
||||
messages: list[dict[str, Any]],
|
||||
max_completion_tokens: int = 16384,
|
||||
retries: int = 5,
|
||||
stage: str = "optimizer",
|
||||
reasoning_effort: str | None = None,
|
||||
*,
|
||||
tools: list[dict[str, Any]] | None = None,
|
||||
tool_choice: str | dict[str, Any] | None = None,
|
||||
return_message: bool = False,
|
||||
timeout: float | None = None,
|
||||
) -> tuple[Any, dict[str, int]]:
|
||||
del reasoning_effort
|
||||
return _chat_messages_impl(
|
||||
messages,
|
||||
max_completion_tokens,
|
||||
retries,
|
||||
stage,
|
||||
role="optimizer",
|
||||
tools=tools,
|
||||
tool_choice=tool_choice,
|
||||
return_message=return_message,
|
||||
timeout=timeout,
|
||||
)
|
||||
|
||||
@@ -425,7 +252,6 @@ def chat_target_messages(
|
||||
max_completion_tokens,
|
||||
retries,
|
||||
stage,
|
||||
role="target",
|
||||
tools=tools,
|
||||
tool_choice=tool_choice,
|
||||
return_message=return_message,
|
||||
@@ -446,10 +272,6 @@ def set_reasoning_effort(effort: str | None) -> None:
|
||||
|
||||
|
||||
def set_target_deployment(deployment: str) -> None:
|
||||
TARGET_CONFIG.deployment = deployment or default_model_for_backend("qwen_chat")
|
||||
os.environ["TARGET_DEPLOYMENT"] = TARGET_CONFIG.deployment
|
||||
|
||||
|
||||
def set_optimizer_deployment(deployment: str) -> None:
|
||||
OPTIMIZER_CONFIG.deployment = deployment or default_model_for_backend("qwen_chat")
|
||||
os.environ["OPTIMIZER_DEPLOYMENT"] = OPTIMIZER_CONFIG.deployment
|
||||
global TARGET_DEPLOYMENT
|
||||
TARGET_DEPLOYMENT = deployment or default_model_for_backend("qwen_chat")
|
||||
os.environ["TARGET_DEPLOYMENT"] = TARGET_DEPLOYMENT
|
||||
|
||||
@@ -1,13 +0,0 @@
|
||||
from skillopt.datasets.base import SplitDataLoader
|
||||
from skillopt.envs._template.env_template import TemplateBenchmarkAdapter
|
||||
from skillopt.envs._template.loader_template import TemplateBenchmarkDataLoader
|
||||
|
||||
|
||||
def test_template_adapter_is_concrete():
|
||||
adapter = TemplateBenchmarkAdapter()
|
||||
assert adapter.get_task_types() == ["your_benchmark"]
|
||||
|
||||
|
||||
def test_template_loader_uses_split_dataloader():
|
||||
loader = TemplateBenchmarkDataLoader()
|
||||
assert isinstance(loader, SplitDataLoader)
|
||||
@@ -1,112 +0,0 @@
|
||||
"""Tests for skillopt.utils.json_utils."""
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from skillopt.utils.json_utils import extract_json, extract_json_array
|
||||
|
||||
|
||||
class TestExtractJson:
|
||||
"""extract_json — extract a JSON object from LLM response text."""
|
||||
|
||||
def test_code_fence_json(self) -> None:
|
||||
text = 'Some text\n```json\n{"key": "value", "num": 42}\n```\nmore text'
|
||||
assert extract_json(text) == {"key": "value", "num": 42}
|
||||
|
||||
def test_bare_json_object(self) -> None:
|
||||
text = 'The result is {"answer": "yes", "score": 0.95}.'
|
||||
assert extract_json(text) == {"answer": "yes", "score": 0.95}
|
||||
|
||||
def test_code_fence_takes_precedence(self) -> None:
|
||||
"""If fence content parses successfully it should be preferred over bare."""
|
||||
text = (
|
||||
'```json\n{"source": "fence"}\n```\n'
|
||||
'Then also {"source": "bare"}'
|
||||
)
|
||||
assert extract_json(text) == {"source": "fence"}
|
||||
|
||||
def test_broken_fence_falls_back_to_bare(self) -> None:
|
||||
"""When fence content is invalid JSON, fall back to bare {...} match."""
|
||||
# Use invalid fence content that has no braces so the greedy bare
|
||||
# regex doesn't swallow the valid object.
|
||||
text = (
|
||||
'```json\nnot json at all\n```\n'
|
||||
'Answer: {"fallback": "yes"}'
|
||||
)
|
||||
assert extract_json(text) == {"fallback": "yes"}
|
||||
|
||||
def test_nested_json(self) -> None:
|
||||
text = '```json\n{"outer": {"inner": [1, 2, 3]}}\n```'
|
||||
assert extract_json(text) == {"outer": {"inner": [1, 2, 3]}}
|
||||
|
||||
def test_no_json_returns_none(self) -> None:
|
||||
assert extract_json("Just plain text without JSON.") is None
|
||||
|
||||
def test_empty_string_returns_none(self) -> None:
|
||||
assert extract_json("") is None
|
||||
|
||||
def test_malformed_json_returns_none(self) -> None:
|
||||
assert extract_json("{broken") is None
|
||||
|
||||
def test_empty_json_object(self) -> None:
|
||||
assert extract_json('{"empty": {}}') == {"empty": {}}
|
||||
|
||||
def test_json_with_escaped_chars(self) -> None:
|
||||
text = '{"message": "hello\\nworld"}'
|
||||
assert extract_json(text) == {"message": "hello\nworld"}
|
||||
|
||||
def test_only_fence_with_no_json_syntax(self) -> None:
|
||||
"""Code fences without valid JSON content should not match."""
|
||||
text = "```\nplain code block\n```"
|
||||
assert extract_json(text) is None
|
||||
|
||||
|
||||
class TestExtractJsonArray:
|
||||
"""extract_json_array — extract a JSON array from LLM response text."""
|
||||
|
||||
def test_code_fence_array(self) -> None:
|
||||
text = '```json\n["a", "b", "c"]\n```'
|
||||
assert extract_json_array(text) == ["a", "b", "c"]
|
||||
|
||||
def test_bare_array(self) -> None:
|
||||
text = "The items are [1, 2, 3]."
|
||||
assert extract_json_array(text) == [1, 2, 3]
|
||||
|
||||
def test_code_fence_takes_precedence(self) -> None:
|
||||
text = (
|
||||
'```json\n["from_fence"]\n```\n'
|
||||
'also ["from_bare"]'
|
||||
)
|
||||
assert extract_json_array(text) == ["from_fence"]
|
||||
|
||||
def test_broken_fence_falls_back_to_bare(self) -> None:
|
||||
text = (
|
||||
'```json\nnot json at all\n```\n'
|
||||
'values: [42]'
|
||||
)
|
||||
assert extract_json_array(text) == [42]
|
||||
|
||||
def test_nested_array(self) -> None:
|
||||
text = '```json\n[[1, 2], [3, 4]]\n```'
|
||||
assert extract_json_array(text) == [[1, 2], [3, 4]]
|
||||
|
||||
def test_no_array_returns_none(self) -> None:
|
||||
assert extract_json_array("no brackets here") is None
|
||||
|
||||
def test_empty_string_returns_none(self) -> None:
|
||||
assert extract_json_array("") is None
|
||||
|
||||
def test_malformed_array_returns_none(self) -> None:
|
||||
assert extract_json_array("[1, 2, ") is None
|
||||
|
||||
def test_empty_json_array(self) -> None:
|
||||
assert extract_json_array("[]") == []
|
||||
|
||||
def test_array_of_objects(self) -> None:
|
||||
text = '[{"x": 1}, {"x": 2}]'
|
||||
assert extract_json_array(text) == [{"x": 1}, {"x": 2}]
|
||||
|
||||
def test_object_not_confused_with_array(self) -> None:
|
||||
"""extract_json_array should not match a bare JSON object."""
|
||||
text = '{"this is an object": true}'
|
||||
assert extract_json_array(text) is None
|
||||
@@ -1,106 +0,0 @@
|
||||
"""Tests for skillopt.utils.scoring."""
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from skillopt.utils.scoring import compute_score, skill_hash
|
||||
|
||||
|
||||
class _ResultObject:
|
||||
"""Minimal object with hard/soft attrs (duck-typing path)."""
|
||||
|
||||
def __init__(self, hard: float, soft: float) -> None:
|
||||
self.hard = hard
|
||||
self.soft = soft
|
||||
|
||||
|
||||
class TestComputeScore:
|
||||
"""compute_score — hard/soft accuracy from a list of episode results."""
|
||||
|
||||
def test_empty_list_returns_zeros(self) -> None:
|
||||
assert compute_score([]) == (0.0, 0.0)
|
||||
|
||||
def test_dict_results_happy_path(self) -> None:
|
||||
results = [
|
||||
{"hard": 1, "soft": 0.8},
|
||||
{"hard": 0, "soft": 0.5},
|
||||
{"hard": 1, "soft": 0.9},
|
||||
]
|
||||
hard, soft = compute_score(results)
|
||||
assert hard == pytest.approx(2 / 3)
|
||||
assert soft == pytest.approx((0.8 + 0.5 + 0.9) / 3)
|
||||
|
||||
def test_object_results(self) -> None:
|
||||
results = [
|
||||
_ResultObject(1.0, 0.75),
|
||||
_ResultObject(0.0, 0.25),
|
||||
]
|
||||
hard, soft = compute_score(results)
|
||||
assert hard == 0.5
|
||||
assert soft == 0.5
|
||||
|
||||
def test_mixed_dict_and_object_results(self) -> None:
|
||||
results = [
|
||||
{"hard": 1, "soft": 1.0},
|
||||
_ResultObject(0, 0.0),
|
||||
]
|
||||
hard, soft = compute_score(results)
|
||||
assert hard == 0.5
|
||||
assert soft == 0.5
|
||||
|
||||
def test_missing_keys_default_to_zero(self) -> None:
|
||||
results = [
|
||||
{"hard": 1},
|
||||
{},
|
||||
]
|
||||
hard, soft = compute_score(results)
|
||||
assert hard == 0.5
|
||||
assert soft == 0.0
|
||||
|
||||
def test_single_result(self) -> None:
|
||||
results = [{"hard": 1, "soft": 0.95}]
|
||||
assert compute_score(results) == (1.0, 0.95)
|
||||
|
||||
def test_continuous_hard_values(self) -> None:
|
||||
"""Hard may be continuous 0.0-1.0 when using smoothed reward."""
|
||||
results = [
|
||||
{"hard": 0.75, "soft": 0.6},
|
||||
{"hard": 0.25, "soft": 0.4},
|
||||
]
|
||||
hard, soft = compute_score(results)
|
||||
assert hard == 0.5
|
||||
assert soft == 0.5
|
||||
|
||||
|
||||
class TestSkillHash:
|
||||
"""skill_hash — a short, deterministic hash of skill content."""
|
||||
|
||||
def test_deterministic(self) -> None:
|
||||
assert skill_hash("hello") == skill_hash("hello")
|
||||
|
||||
def test_different_input_produces_different_hash(self) -> None:
|
||||
assert skill_hash("hello") != skill_hash("world")
|
||||
|
||||
def test_empty_string(self) -> None:
|
||||
h = skill_hash("")
|
||||
assert isinstance(h, str)
|
||||
assert len(h) == 16
|
||||
|
||||
def test_output_length(self) -> None:
|
||||
h = skill_hash("some skill content here")
|
||||
assert len(h) == 16
|
||||
|
||||
def test_hex_characters(self) -> None:
|
||||
h = skill_hash("any content")
|
||||
assert all(c in "0123456789abcdef" for c in h)
|
||||
|
||||
def test_unicode_content(self) -> None:
|
||||
h1 = skill_hash("cafe")
|
||||
h2 = skill_hash("cafe")
|
||||
assert h1 == h2
|
||||
|
||||
def test_multiline_content(self) -> None:
|
||||
content = "line1\nline2\nline3"
|
||||
h = skill_hash(content)
|
||||
assert len(h) == 16
|
||||
assert isinstance(h, str)
|
||||
@@ -1,249 +0,0 @@
|
||||
"""Tests for skillopt.types — Edit and Patch dataclass serialization."""
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from skillopt.types import Edit, Patch
|
||||
|
||||
|
||||
# ── Edit ────────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
class TestEditCreation:
|
||||
"""Edit dataclass construction."""
|
||||
|
||||
def test_minimal_edit(self) -> None:
|
||||
e = Edit(op="append")
|
||||
assert e.op == "append"
|
||||
assert e.content == ""
|
||||
assert e.target == ""
|
||||
assert e.support_count is None
|
||||
assert e.source_type is None
|
||||
assert e.merge_level is None
|
||||
assert e.update_origin == ""
|
||||
assert e.update_target == ""
|
||||
|
||||
def test_full_edit(self) -> None:
|
||||
e = Edit(
|
||||
op="replace",
|
||||
content="new content",
|
||||
target="old content",
|
||||
support_count=5,
|
||||
source_type="failure",
|
||||
merge_level=2,
|
||||
update_origin="reflect",
|
||||
update_target="skill",
|
||||
)
|
||||
assert e.op == "replace"
|
||||
assert e.content == "new content"
|
||||
assert e.target == "old content"
|
||||
assert e.support_count == 5
|
||||
assert e.source_type == "failure"
|
||||
assert e.merge_level == 2
|
||||
assert e.update_origin == "reflect"
|
||||
assert e.update_target == "skill"
|
||||
|
||||
def test_insert_after_op(self) -> None:
|
||||
e = Edit(op="insert_after", content="insertion", target="anchor")
|
||||
assert e.op == "insert_after"
|
||||
assert e.content == "insertion"
|
||||
assert e.target == "anchor"
|
||||
|
||||
def test_delete_op(self) -> None:
|
||||
e = Edit(op="delete", target="thing_to_remove")
|
||||
assert e.op == "delete"
|
||||
assert e.target == "thing_to_remove"
|
||||
|
||||
|
||||
class TestEditRoundTrip:
|
||||
"""Edit.to_dict() / Edit.from_dict() round-trip."""
|
||||
|
||||
def test_round_trip_minimal(self) -> None:
|
||||
e = Edit(op="append")
|
||||
d = e.to_dict()
|
||||
restored = Edit.from_dict(d)
|
||||
assert restored == e
|
||||
|
||||
def test_round_trip_full(self) -> None:
|
||||
e = Edit(
|
||||
op="replace",
|
||||
content="new content",
|
||||
target="old content",
|
||||
support_count=3,
|
||||
source_type="success",
|
||||
merge_level=1,
|
||||
update_origin="meta_reflect",
|
||||
update_target="system_prompt",
|
||||
)
|
||||
d = e.to_dict()
|
||||
restored = Edit.from_dict(d)
|
||||
assert restored == e
|
||||
|
||||
def test_round_trip_delete_without_content(self) -> None:
|
||||
e = Edit(op="delete", target="obsolete_line")
|
||||
d = e.to_dict()
|
||||
restored = Edit.from_dict(d)
|
||||
assert restored == e
|
||||
|
||||
def test_optional_fields_omitted_when_default(self) -> None:
|
||||
e = Edit(op="append")
|
||||
d = e.to_dict()
|
||||
assert d == {"op": "append", "content": ""}
|
||||
# support_count, source_type, etc. should be absent
|
||||
assert "support_count" not in d
|
||||
assert "source_type" not in d
|
||||
assert "merge_level" not in d
|
||||
assert "target" not in d
|
||||
assert "update_origin" not in d
|
||||
assert "update_target" not in d
|
||||
|
||||
def test_from_dict_with_defaults(self) -> None:
|
||||
d = {"op": "replace", "content": "abc"}
|
||||
e = Edit.from_dict(d)
|
||||
assert e.op == "replace"
|
||||
assert e.content == "abc"
|
||||
assert e.target == ""
|
||||
assert e.support_count is None
|
||||
assert e.source_type is None
|
||||
|
||||
def test_from_dict_with_extra_keys(self) -> None:
|
||||
"""Extra keys in dict should be ignored."""
|
||||
d = {"op": "append", "content": "", "unknown_field": 42}
|
||||
e = Edit.from_dict(d)
|
||||
assert e.op == "append"
|
||||
assert not hasattr(e, "unknown_field")
|
||||
|
||||
|
||||
class TestEditEdgeCases:
|
||||
"""Edge cases around Edit."""
|
||||
|
||||
def test_support_count_zero(self) -> None:
|
||||
"""0 is a valid support_count and should be serialized."""
|
||||
e = Edit(op="append", support_count=0)
|
||||
d = e.to_dict()
|
||||
assert d["support_count"] == 0
|
||||
restored = Edit.from_dict(d)
|
||||
assert restored.support_count == 0
|
||||
|
||||
def test_merge_level_zero(self) -> None:
|
||||
e = Edit(op="replace", merge_level=0)
|
||||
d = e.to_dict()
|
||||
assert d["merge_level"] == 0
|
||||
restored = Edit.from_dict(d)
|
||||
assert restored.merge_level == 0
|
||||
|
||||
def test_empty_target_stays_empty(self) -> None:
|
||||
e = Edit(op="append", target="")
|
||||
d = e.to_dict()
|
||||
assert "target" not in d
|
||||
|
||||
|
||||
# ── Patch ───────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
class TestPatchCreation:
|
||||
"""Patch dataclass construction."""
|
||||
|
||||
def test_empty_patch(self) -> None:
|
||||
p = Patch()
|
||||
assert p.edits == []
|
||||
assert p.reasoning == ""
|
||||
assert p.ranking_details is None
|
||||
|
||||
def test_patch_with_edits(self) -> None:
|
||||
edits = [
|
||||
Edit(op="append", content="step 1"),
|
||||
Edit(op="append", content="step 2"),
|
||||
]
|
||||
p = Patch(edits=edits, reasoning="Added two steps")
|
||||
assert len(p.edits) == 2
|
||||
assert p.reasoning == "Added two steps"
|
||||
|
||||
def test_patch_with_ranking_details(self) -> None:
|
||||
p = Patch(ranking_details={"score": 0.95, "rank": 1})
|
||||
assert p.ranking_details == {"score": 0.95, "rank": 1}
|
||||
|
||||
|
||||
class TestPatchRoundTrip:
|
||||
"""Patch.to_dict() / Patch.from_dict() round-trip."""
|
||||
|
||||
def test_round_trip_empty(self) -> None:
|
||||
p = Patch()
|
||||
d = p.to_dict()
|
||||
restored = Patch.from_dict(d)
|
||||
assert restored.edits == []
|
||||
assert restored.reasoning == ""
|
||||
assert restored.ranking_details is None
|
||||
|
||||
def test_round_trip_with_edits(self) -> None:
|
||||
edits = [
|
||||
Edit(op="insert_after", content="new step", target="existing step"),
|
||||
Edit(op="replace", content="updated", target="old"),
|
||||
]
|
||||
p = Patch(edits=edits, reasoning="Batch update")
|
||||
d = p.to_dict()
|
||||
restored = Patch.from_dict(d)
|
||||
assert len(restored.edits) == 2
|
||||
for original, restored_edit in zip(p.edits, restored.edits):
|
||||
assert isinstance(restored_edit, Edit)
|
||||
assert original == restored_edit
|
||||
assert restored.reasoning == "Batch update"
|
||||
assert restored.ranking_details is None
|
||||
|
||||
def test_round_trip_with_ranking_details(self) -> None:
|
||||
details = {"strategy": "rouge", "scores": [0.9, 0.8, 0.7]}
|
||||
p = Patch(
|
||||
edits=[Edit(op="append", content="a")],
|
||||
reasoning="selected best",
|
||||
ranking_details=details,
|
||||
)
|
||||
d = p.to_dict()
|
||||
restored = Patch.from_dict(d)
|
||||
assert restored.ranking_details == details
|
||||
|
||||
def test_to_dict_contains_reasoning_and_edits(self) -> None:
|
||||
p = Patch(edits=[Edit(op="append", content="test")], reasoning="reason")
|
||||
d = p.to_dict()
|
||||
assert "reasoning" in d
|
||||
assert "edits" in d
|
||||
assert isinstance(d["edits"], list)
|
||||
|
||||
def test_from_dict_preserves_edit_order(self) -> None:
|
||||
edits = [
|
||||
Edit(op="append", content="first"),
|
||||
Edit(op="insert_after", content="second", target="first"),
|
||||
Edit(op="append", content="third"),
|
||||
]
|
||||
p = Patch(edits=edits, reasoning="ordered")
|
||||
d = p.to_dict()
|
||||
restored = Patch.from_dict(d)
|
||||
assert restored.edits[0].content == "first"
|
||||
assert restored.edits[1].content == "second"
|
||||
assert restored.edits[2].content == "third"
|
||||
|
||||
|
||||
class TestPatchEdgeCases:
|
||||
"""Edge cases around Patch."""
|
||||
|
||||
def test_reasoning_empty_string(self) -> None:
|
||||
p = Patch(reasoning="")
|
||||
d = p.to_dict()
|
||||
assert d["reasoning"] == ""
|
||||
|
||||
def test_zero_edits(self) -> None:
|
||||
"""Patch with explicitly empty edit list."""
|
||||
p = Patch(edits=[])
|
||||
d = p.to_dict()
|
||||
assert d["edits"] == []
|
||||
|
||||
def test_nested_edit_from_dict_handles_dicts(self) -> None:
|
||||
"""from_dict should accept dicts in the 'edits' list."""
|
||||
d = {
|
||||
"reasoning": "test",
|
||||
"edits": [{"op": "append", "content": "hello"}],
|
||||
}
|
||||
p = Patch.from_dict(d)
|
||||
assert len(p.edits) == 1
|
||||
assert isinstance(p.edits[0], Edit)
|
||||
assert p.edits[0].op == "append"
|
||||
assert p.edits[0].content == "hello"
|
||||
Reference in New Issue
Block a user