Compare commits
17 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 copilot-swe-agent[bot]
|
4f582d4f6e | ||
|
copilot-swe-agent[bot]
|
b3c7d72364 | ||
|
copilot-swe-agent[bot]
|
36284e1bb0 | ||
|
Yifan Yang
|
fb1a76371d | ||
|
Yifan Yang
|
47063e1ceb | ||
|
hwq
|
181d71b737 | ||
|
kaikai-macbook
|
41012e2d5e | ||
|
Claude Code Agent
|
dd8cd993b5 | ||
|
Yif Yang
|
8ebede0efd | ||
|
Yif Yang
|
266fca72ab | ||
|
Yif Yang
|
9265545c45 | ||
|
Yif Yang
|
b4850ce418 | ||
|
Yif Yang
|
643346c9f3 | ||
|
Declan Murphy
|
c6da31df44 | ||
|
Declan Murphy
|
e4201074aa | ||
|
Declan Murphy
|
309ea64ff4 | ||
|
Declan Murphy
|
d224d425f9 |
@@ -27,3 +27,8 @@ export AZURE_OPENAI_API_KEY=
|
||||
# ── Qwen Local Model (for qwen_chat backend) ────────────────────────
|
||||
# export QWEN_CHAT_BASE_URL=http://localhost:8000/v1
|
||||
# export QWEN_CHAT_MODEL=Qwen/Qwen3.5-4B
|
||||
|
||||
# ── MiniMax (for minimax_chat backend) ──────────────────────────────
|
||||
# export MINIMAX_BASE_URL=https://api.minimax.io/v1
|
||||
# export MINIMAX_API_KEY=...
|
||||
# export MINIMAX_MODEL=MiniMax-M2.7
|
||||
|
||||
+15
-1
@@ -5,7 +5,20 @@ build/
|
||||
dist/
|
||||
site/
|
||||
|
||||
data/
|
||||
data/*
|
||||
!data/README.md
|
||||
!data/searchqa_id_split/
|
||||
!data/searchqa_id_split/**
|
||||
!data/livemathematicianbench_id_split/
|
||||
!data/livemathematicianbench_id_split/**
|
||||
!data/docvqa_id_split/
|
||||
!data/docvqa_id_split/**
|
||||
!data/officeqa_id_split/
|
||||
!data/officeqa_id_split/**
|
||||
!data/spreadsheetbench_id_split/
|
||||
!data/spreadsheetbench_id_split/**
|
||||
!data/alfworld_path_split/
|
||||
!data/alfworld_path_split/**
|
||||
outputs/
|
||||
logs/
|
||||
external/
|
||||
@@ -40,3 +53,4 @@ docs/reflact_overview.html
|
||||
docs/render_ablation_paper_tables.py
|
||||
docs/让*
|
||||
.gradio/
|
||||
.venv
|
||||
|
||||
@@ -4,7 +4,37 @@
|
||||
|
||||
[](https://microsoft.github.io/SkillOpt/) [](https://arxiv.org/abs/2605.23904) [](https://youtu.be/JUBMDTCiM0M) [](https://www.python.org/) [](LICENSE)
|
||||
|
||||
## 🎬 SkillOpt Demo Video
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
Modern agent skills are usually hand-crafted, generated one-shot by a strong
|
||||
LLM, or evolved through loosely controlled self-revision — none of which
|
||||
behaves like a deep-learning optimizer for the skill itself, and none of
|
||||
which reliably improves over its starting point under feedback.
|
||||
|
||||
**SkillOpt treats the skill document as the trainable state of a frozen
|
||||
agent**, and trains it with the discipline that makes weight-space
|
||||
optimization reproducible. A separate optimizer model turns scored rollouts
|
||||
into bounded add / delete / replace edits on a single skill document; a
|
||||
candidate edit is accepted only when it strictly improves a held-out
|
||||
validation score. A textual learning-rate budget, a rejected-edit buffer,
|
||||
and an epoch-wise slow / meta update make skill training stable while
|
||||
adding **zero inference-time model calls** at deployment.
|
||||
|
||||
The deployed artifact is a compact `best_skill.md` (typically 300–2,000
|
||||
tokens) that runs against the unchanged target model. Across **six
|
||||
benchmarks, seven target models, and three execution harnesses** (direct
|
||||
chat, Codex CLI, Claude Code CLI), SkillOpt is best or tied-best on **all
|
||||
52 evaluated (model, benchmark, harness) cells** and on GPT-5.5 lifts the
|
||||
average no-skill accuracy by **+23.5 points in direct chat, +24.8 inside
|
||||
the Codex agentic loop, and +19.1 inside Claude Code**. Optimized skill
|
||||
artifacts transfer across model scales, between Codex and Claude Code
|
||||
harnesses, and to nearby benchmarks without further optimization.
|
||||
|
||||
For the full method, ablations, and per-cell results see the [paper](https://arxiv.org/abs/2605.23904); for a visual walkthrough of the loop see the [project page](https://microsoft.github.io/SkillOpt/); for deeper API / backend / benchmark docs see [`docs/`](docs/).
|
||||
|
||||
## 🎬 Demo Video
|
||||
|
||||
https://github.com/user-attachments/assets/eb12d3bc-371c-467f-904d-91b61f339ed7
|
||||
|
||||
@@ -16,14 +46,16 @@ https://github.com/user-attachments/assets/eb12d3bc-371c-467f-904d-91b61f339ed7
|
||||
|
||||
## Install
|
||||
|
||||
**Requirements:** Python 3.10+
|
||||
### Requirements
|
||||
|
||||
- Python 3.10+
|
||||
|
||||
```bash
|
||||
git clone https://github.com/microsoft/SkillOpt.git
|
||||
cd SkillOpt
|
||||
pip install -e .
|
||||
|
||||
# For ALFWorld benchmark (optional):
|
||||
# For the ALFWorld benchmark (optional):
|
||||
pip install -e ".[alfworld]"
|
||||
alfworld-download
|
||||
```
|
||||
@@ -36,7 +68,8 @@ cp .env.example .env
|
||||
source .env
|
||||
```
|
||||
|
||||
**Azure OpenAI** (recommended):
|
||||
#### Azure OpenAI *(recommended)*
|
||||
|
||||
```bash
|
||||
export AZURE_OPENAI_ENDPOINT="https://your-resource.openai.azure.com/"
|
||||
# Option 1: API key auth
|
||||
@@ -45,74 +78,56 @@ export AZURE_OPENAI_API_KEY="your-key"
|
||||
export AZURE_OPENAI_AUTH_MODE="azure_cli"
|
||||
```
|
||||
|
||||
> **Note:** `AZURE_OPENAI_ENDPOINT` is required for all three modes (`api_key`, `azure_cli`,
|
||||
> `openai_compatible`). Without it, all LLM calls will fail.
|
||||
> **Note:** `AZURE_OPENAI_ENDPOINT` is required for all three modes (`api_key`, `azure_cli`, `openai_compatible`). Without it, all LLM calls will fail.
|
||||
|
||||
#### OpenAI-compatible endpoints
|
||||
|
||||
**OpenAI-compatible endpoints**:
|
||||
```bash
|
||||
export AZURE_OPENAI_ENDPOINT="https://api.openai.com/v1"
|
||||
export AZURE_OPENAI_API_KEY="sk-..."
|
||||
export AZURE_OPENAI_AUTH_MODE="openai_compatible"
|
||||
```
|
||||
|
||||
This routes all calls through the plain OpenAI Python client (no Azure auth, no `api-version`
|
||||
header).
|
||||
This routes all calls through the plain OpenAI Python client (no Azure auth, no `api-version` header).
|
||||
|
||||
> **Note:** SkillOpt reuses the `AZURE_OPENAI_*` env var names even in this mode — there is no
|
||||
> separate `OPENAI_API_KEY` knob.
|
||||
> **Note:** SkillOpt reuses the `AZURE_OPENAI_*` env var names even in this mode — there is no separate `OPENAI_API_KEY` knob.
|
||||
|
||||
#### Anthropic Claude
|
||||
|
||||
**Anthropic Claude**:
|
||||
```bash
|
||||
export ANTHROPIC_API_KEY="sk-ant-..."
|
||||
```
|
||||
|
||||
**Qwen (local vLLM)**:
|
||||
#### Qwen *(local vLLM)*
|
||||
|
||||
```bash
|
||||
export QWEN_CHAT_BASE_URL="http://localhost:8000/v1"
|
||||
export QWEN_CHAT_MODEL="Qwen/Qwen3.5-4B"
|
||||
```
|
||||
|
||||
---
|
||||
`qwen_chat` can also be used as the optimizer backend. When optimizer and
|
||||
target should point to different local vLLM services, use the role-specific
|
||||
settings:
|
||||
|
||||
## Data Preparation
|
||||
|
||||
SkillOpt expects data in a **split directory** with `train/`, `val/`, `test/` subdirectories, each containing a JSON file (e.g., `items.json`).
|
||||
|
||||
```
|
||||
data/my_split/
|
||||
├── train/items.json
|
||||
├── val/items.json
|
||||
└── test/items.json
|
||||
```bash
|
||||
python scripts/train.py \
|
||||
--config configs/searchqa/default.yaml \
|
||||
--optimizer_backend qwen_chat \
|
||||
--target_backend qwen_chat \
|
||||
--optimizer_model Qwen/Qwen3.5-4B \
|
||||
--target_model Qwen/Qwen3.5-4B \
|
||||
--optimizer_qwen_chat_base_url http://localhost:8001/v1 \
|
||||
--target_qwen_chat_base_url http://localhost:8000/v1
|
||||
```
|
||||
|
||||
Each JSON file is an array of task items. The required fields depend on the benchmark. For example, SearchQA items look like:
|
||||
#### MiniMax
|
||||
|
||||
```json
|
||||
[
|
||||
{
|
||||
"id": "unique_item_id",
|
||||
"question": "Who wrote the novel ...",
|
||||
"context": "[DOC] relevant passage text ...",
|
||||
"answers": ["expected answer"]
|
||||
}
|
||||
]
|
||||
```bash
|
||||
export MINIMAX_BASE_URL="https://api.minimax.io/v1"
|
||||
export MINIMAX_API_KEY="..."
|
||||
export MINIMAX_MODEL="MiniMax-M2.7"
|
||||
```
|
||||
|
||||
See `skillopt/envs/<benchmark>/dataloader.py` for the exact format each benchmark expects.
|
||||
|
||||
> **Note:** Benchmark datasets are not included in this repository. Prepare your own data following the format above.
|
||||
|
||||
### Supported Benchmarks
|
||||
|
||||
| Benchmark | Type | Config |
|
||||
|---|---|---|
|
||||
| SearchQA | QA | `configs/searchqa/default.yaml` |
|
||||
| ALFWorld | Embodied agent | `configs/alfworld/default.yaml` |
|
||||
| DocVQA | Document QA | `configs/docvqa/default.yaml` |
|
||||
| LiveMathematicianBench | Math | `configs/livemathematicianbench/default.yaml` |
|
||||
| SpreadsheetBench | Code generation | `configs/spreadsheetbench/default.yaml` |
|
||||
| OfficeQA | Tool-augmented QA | `configs/officeqa/default.yaml` |
|
||||
|
||||
---
|
||||
|
||||
## Quick Start
|
||||
@@ -139,7 +154,7 @@ python scripts/train.py \
|
||||
# Train on ALFWorld:
|
||||
python scripts/train.py \
|
||||
--config configs/alfworld/default.yaml \
|
||||
--split_dir /path/to/your/alfworld_split \
|
||||
--split_dir data/alfworld_path_split \
|
||||
--azure_openai_endpoint https://your-resource.openai.azure.com/ \
|
||||
--optimizer_model gpt-5.5 \
|
||||
--target_model gpt-5.5
|
||||
@@ -181,8 +196,7 @@ python scripts/eval_only.py \
|
||||
--azure_openai_endpoint https://your-resource.openai.azure.com/
|
||||
```
|
||||
|
||||
To evaluate a skill produced by a training run, replace `--skill` with that
|
||||
run's best-skill path, for example `outputs/my_run/best_skill.md`.
|
||||
To evaluate a skill produced by your own training run, replace `--skill` with that run's best-skill path, for example `outputs/my_run/best_skill.md`.
|
||||
|
||||
| Split | Description |
|
||||
|---|---|
|
||||
@@ -193,7 +207,7 @@ run's best-skill path, for example `outputs/my_run/best_skill.md`.
|
||||
|
||||
### Output Structure
|
||||
|
||||
Each run writes to a structured output directory:
|
||||
Each training run writes to a structured output directory:
|
||||
|
||||
```
|
||||
outputs/<run_name>/
|
||||
@@ -209,26 +223,147 @@ outputs/<run_name>/
|
||||
|
||||
Re-running the same command auto-resumes from the last completed step.
|
||||
|
||||
### Pretrained Skill Artifacts
|
||||
|
||||
We provide a subset of the paper's main Table 1 GPT-5.5 optimized skills in
|
||||
[`ckpt/`](ckpt/) as reference artifacts. Use them with `scripts/eval_only.py`
|
||||
to evaluate the provided skills on a matching data split without re-running
|
||||
training. See [`ckpt/README.md`](ckpt/README.md) for the full per-benchmark
|
||||
command. This is the first artifact batch; we plan to continue uploading
|
||||
the remaining optimized skills and benchmark split manifests as they are
|
||||
cleaned and verified.
|
||||
|
||||
---
|
||||
|
||||
## Community-contributed configs
|
||||
## Data Preparation
|
||||
|
||||
These are **not** default SkillOpt settings — they are reference configs
|
||||
### Directory layout
|
||||
|
||||
SkillOpt expects data in a **split directory** with `train/`, `val/`, `test/` subdirectories, each containing a JSON file (e.g., `items.json`):
|
||||
|
||||
```
|
||||
data/my_split/
|
||||
├── train/items.json
|
||||
├── val/items.json
|
||||
└── test/items.json
|
||||
```
|
||||
|
||||
Each JSON file is an array of task items. The required fields depend on the benchmark. For example, SearchQA items look like:
|
||||
|
||||
```json
|
||||
[
|
||||
{
|
||||
"id": "unique_item_id",
|
||||
"question": "Who wrote the novel ...",
|
||||
"context": "[DOC] relevant passage text ...",
|
||||
"answers": ["expected answer"]
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
See `skillopt/envs/<benchmark>/dataloader.py` for the exact format each benchmark expects.
|
||||
|
||||
> **Note:** Most benchmark datasets are not included in this repository. Prepare your own data following the format above. The exact SearchQA split used in the paper is provided at [`data/searchqa_id_split/`](data/searchqa_id_split) (400 train / 200 val / 1400 test). We are preparing the remaining benchmark split manifests for upload.
|
||||
|
||||
### Supported Benchmarks
|
||||
|
||||
| Benchmark | Type | Config |
|
||||
|---|---|---|
|
||||
| SearchQA | QA | `configs/searchqa/default.yaml` |
|
||||
| ALFWorld | Embodied agent | `configs/alfworld/default.yaml` |
|
||||
| DocVQA | Document QA | `configs/docvqa/default.yaml` |
|
||||
| LiveMathematicianBench | Math | `configs/livemathematicianbench/default.yaml` |
|
||||
| SpreadsheetBench | Code generation | `configs/spreadsheetbench/default.yaml` |
|
||||
| OfficeQA | Tool-augmented QA | `configs/officeqa/default.yaml` |
|
||||
|
||||
---
|
||||
|
||||
## Configuration
|
||||
|
||||
### Default settings and paper-reproduction knobs
|
||||
|
||||
`configs/_base_/default.yaml` is the single source of truth for SkillOpt's
|
||||
runtime knobs. Out of the box, every included benchmark config inherits
|
||||
from it and keeps the paper protocol visible: 4 epochs, rollout batch 40,
|
||||
reflection minibatch 8, textual learning rate 4 with cosine decay, strict
|
||||
hard validation gating, and slow-update + meta-skill enabled. One detail to
|
||||
watch is slow-update acceptance: the current `main` default is the newer
|
||||
post-submission force-accept mode, while the paper protocol and the
|
||||
paper-aligned skills under `ckpt/` use the gated semantics described in
|
||||
paper Section 3.6.
|
||||
|
||||
### Slow-update acceptance mode
|
||||
|
||||
The epoch-boundary slow / meta update can be applied two ways, controlled
|
||||
by `optimizer.slow_update_gate_with_selection`:
|
||||
|
||||
```yaml
|
||||
optimizer:
|
||||
slow_update_gate_with_selection: false # current main default
|
||||
```
|
||||
|
||||
- **`false`** *(current `main` default)*: force-accept. The
|
||||
slow-update guidance is injected into both `current_skill` and
|
||||
`best_skill` unconditionally at the epoch boundary. This is the newer
|
||||
post-submission behavior on `main`.
|
||||
- **`true`** *(paper / ckpt-skill reproduction)*: gated, matching paper
|
||||
Section 3.6 verbatim. The slow-update candidate is evaluated on the
|
||||
selection split and accepted only if it passes the same validation gate
|
||||
as a step-level edit. Use this setting when re-running optimization to
|
||||
match the paper protocol and the provenance of the provided `ckpt/` skills.
|
||||
|
||||
The trainer prints which mode is active at startup
|
||||
(`[slow update] acceptance=...`). See issue #22 for the discussion that
|
||||
led to the flag.
|
||||
|
||||
### Gate metric (`hard` / `soft` / `mixed`)
|
||||
|
||||
The validation gate compares candidate vs. current skills on the selection
|
||||
split using `gate_metric`:
|
||||
|
||||
- **`hard`** *(default, paper)*: exact-match accuracy, strictly greater
|
||||
than the current score is required.
|
||||
- **`soft`**: per-item soft / partial-credit score. Useful when the
|
||||
selection split is small (e.g. ≤10 items) and the reward is continuous,
|
||||
where the discrete hard gate often rejects every candidate.
|
||||
- **`mixed`**: weighted average, `(1 - w) * hard + w * soft`, with `w`
|
||||
set by `gate_mixed_weight` (default `0.5`).
|
||||
|
||||
Default is `hard`. Use the optional feature config below to switch.
|
||||
|
||||
### Optional feature configs
|
||||
|
||||
These are **not** default SkillOpt settings — they are optional feature configs
|
||||
contributed by users for specific scenarios. The paper-reported numbers
|
||||
were obtained with the default settings, not these.
|
||||
|
||||
- **`configs/examples/soft_gate.yaml`** *(PR #25, contributed by
|
||||
[@lvbaocheng](https://github.com/lvbaocheng))* — switches the
|
||||
validation gate from exact-match (`hard`) to soft / partial-credit
|
||||
(`soft` or `mixed`). Useful when the held-out **selection split is
|
||||
small** (e.g. ≤ ~10 items) and the **reward is continuous**, where the
|
||||
discrete hard gate often rejects every candidate and training stalls.
|
||||
See the comment at the top of the file for details and when not to use
|
||||
it.
|
||||
- **[`configs/features/soft_gate.yaml`](configs/features/soft_gate.yaml)**
|
||||
*(PR #25, contributed by [@lvbaocheng](https://github.com/lvbaocheng))* —
|
||||
switches `gate_metric` to `soft` (or `mixed`). See the comment at the
|
||||
top of the file for when to use and when not to.
|
||||
|
||||
---
|
||||
|
||||
## WebUI
|
||||
## Extensibility & WebUI
|
||||
|
||||
### Adding a new backend
|
||||
|
||||
A backend = a chat / exec target (e.g. `openai_chat`, `claude_chat`,
|
||||
`qwen_chat`, `minimax_chat`, `codex_exec`, `claude_code_exec`). See
|
||||
[`docs/guide/new-backend.md`](docs/guide/new-backend.md) for the full
|
||||
contract; in short you add a `skillopt/model/<name>_backend.py` module,
|
||||
register it in `skillopt/model/common.py` + `backend_config.py`, and wire
|
||||
it through the router in `skillopt/model/__init__.py`. `qwen_backend.py`
|
||||
and `minimax_backend.py` are good templates.
|
||||
|
||||
### Adding a new benchmark
|
||||
|
||||
A benchmark = a `skillopt/envs/<name>/` package with a `dataloader.py`, a
|
||||
`rollout.py`, and an `initial.md` seed skill. See
|
||||
[`docs/guide/new-benchmark.md`](docs/guide/new-benchmark.md) for the full
|
||||
contract; the simplest reference is `skillopt/envs/searchqa/`.
|
||||
|
||||
### WebUI
|
||||
|
||||
Launch the monitoring dashboard (optional):
|
||||
|
||||
@@ -243,11 +378,6 @@ python -m skillopt_webui.app
|
||||
| `--host` | `0.0.0.0` | Bind address |
|
||||
| `--share` | off | Create a public Gradio share link |
|
||||
|
||||
```bash
|
||||
# With public share link (useful for remote servers)
|
||||
python -m skillopt_webui.app --share
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Citation
|
||||
|
||||
@@ -0,0 +1,79 @@
|
||||
# Paper-aligned SkillOpt reference skills (GPT-5.5)
|
||||
|
||||
This folder provides a subset of the paper's main Table 1 GPT-5.5 optimized
|
||||
skills as reference artifacts — one `gpt5.5_skill.md` per currently included
|
||||
benchmark. You can plug them into `scripts/eval_only.py` to evaluate the
|
||||
provided skills on a given split without re-running the training loop.
|
||||
|
||||
> These are checkpoints associated with the paper, not a general-purpose
|
||||
> tool. They're here so you can verify the reported numbers and use the
|
||||
> skills as portable artifacts. If you want to *train* your own skill,
|
||||
> use `scripts/train.py` per the top-level README.
|
||||
>
|
||||
> This is the first artifact batch. We plan to continue uploading the
|
||||
> remaining optimized skills and benchmark split manifests as they are
|
||||
> cleaned and verified.
|
||||
|
||||
## What's here
|
||||
|
||||
| Benchmark | Skill artifact | Matching config |
|
||||
|---|---|---|
|
||||
| SearchQA | `ckpt/searchqa/gpt5.5_skill.md` | `configs/searchqa/default.yaml` |
|
||||
| ALFWorld | `ckpt/alfworld/gpt5.5_skill.md` | `configs/alfworld/default.yaml` |
|
||||
| DocVQA | `ckpt/docvqa/gpt5.5_skill.md` | `configs/docvqa/default.yaml` |
|
||||
| LiveMathematicianBench | `ckpt/livemath/gpt5.5_skill.md` | `configs/livemathematicianbench/default.yaml` |
|
||||
| OfficeQA | `ckpt/officeqa/gpt5.5_skill.md` | `configs/officeqa/default.yaml` |
|
||||
| SpreadsheetBench | `ckpt/spreadsheetbench/gpt5.5_skill.md` | `configs/spreadsheetbench/default.yaml` |
|
||||
|
||||
Each file is a plain Markdown skill document (~2k–13k chars). It contains a
|
||||
protected `SLOW_UPDATE` section at the end that holds epoch-wise
|
||||
longitudinal guidance — that's expected, not a formatting issue.
|
||||
|
||||
## How to evaluate a provided skill
|
||||
|
||||
`scripts/eval_only.py` runs a single skill against a data split without
|
||||
invoking the optimizer. Example for SearchQA against the test split:
|
||||
|
||||
```bash
|
||||
python scripts/eval_only.py \
|
||||
--config configs/searchqa/default.yaml \
|
||||
--skill ckpt/searchqa/gpt5.5_skill.md \
|
||||
--split valid_unseen \
|
||||
--split_dir data/searchqa_id_split \
|
||||
--azure_openai_endpoint https://your-resource.openai.azure.com/ \
|
||||
--target_model gpt-5.5
|
||||
```
|
||||
|
||||
Substitute the benchmark, config, skill path, and `--split_dir` to evaluate
|
||||
any of the other five. `--split valid_unseen` is the test split, `valid_seen`
|
||||
is the selection / validation split, `train` is the training split, and
|
||||
`all` runs all three.
|
||||
|
||||
## On comparing to the paper numbers
|
||||
|
||||
To compare against the paper-reported cells, use the same dataset split and
|
||||
scorer. SearchQA's split is checked in at `data/searchqa_id_split/` (400
|
||||
train / 200 selection / 1400 test). For the other benchmarks, point
|
||||
`--split_dir` at your own materialized split; the loader is deterministic
|
||||
from `split_seed` (default `42`) + `split_ratio` (default `2:1:7`) when
|
||||
`split_mode: ratio` is used, so a given `data_path` + seed reproduces
|
||||
across machines. Explicit per-benchmark split manifests are being prepared
|
||||
for upload — see issues #14 and #21.
|
||||
|
||||
## Why force-accept vs. gated slow-update matters
|
||||
|
||||
These `ckpt/` skills were produced with the gated slow-update semantics
|
||||
described in paper Section 3.6:
|
||||
|
||||
```yaml
|
||||
optimizer:
|
||||
slow_update_gate_with_selection: true
|
||||
```
|
||||
|
||||
Current `main` defaults to `false` (force-accept mode), a newer
|
||||
post-submission behavior where the slow-update guidance is written into
|
||||
`current_skill` and `best_skill` unconditionally at the epoch boundary. If
|
||||
you re-train with the current default, you may produce a *different*
|
||||
`best_skill.md` than the one checked in here. Both modes are supported;
|
||||
see the top-level README's "Configuration -> Slow-update acceptance mode"
|
||||
section.
|
||||
@@ -44,6 +44,18 @@ model:
|
||||
target_azure_openai_ad_scope: "https://cognitiveservices.azure.com/.default"
|
||||
target_azure_openai_managed_identity_client_id: ""
|
||||
|
||||
# MiniMax backend settings (minimax_chat target)
|
||||
minimax_base_url: "" # https://api.minimax.io/v1 if blank
|
||||
minimax_api_key: ""
|
||||
minimax_model: "MiniMax-M2.7"
|
||||
minimax_temperature: "0.7"
|
||||
minimax_max_tokens: "8000"
|
||||
minimax_enable_thinking: "false"
|
||||
optimizer_minimax_base_url: "" # per-role override
|
||||
target_minimax_base_url: "" # per-role override
|
||||
optimizer_minimax_api_key: ""
|
||||
target_minimax_api_key: ""
|
||||
|
||||
train:
|
||||
num_epochs: 4
|
||||
train_size: 0 # 0 = derive from dataset split when available
|
||||
|
||||
@@ -19,7 +19,7 @@ env:
|
||||
name: alfworld
|
||||
skill_init: skillopt/envs/alfworld/skills/initial.md
|
||||
split_mode: split_dir
|
||||
split_dir: data/ablation_splits/alfworld/2-1-7_seed42
|
||||
split_dir: data/alfworld_path_split
|
||||
data_path: ""
|
||||
split_output_dir: ""
|
||||
max_steps: 50
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Example: soft / mixed validation-gate metric (community-contributed, PR #25)
|
||||
# Feature: soft / mixed validation-gate metric (community-contributed, PR #25)
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
#
|
||||
# This is NOT a default SkillOpt setting and was NOT used to produce the
|
||||
@@ -28,7 +28,7 @@
|
||||
# and matches the design described in the paper.
|
||||
#
|
||||
# To use: inherit your env config from this file, e.g.
|
||||
# _base_: ../examples/soft_gate.yaml
|
||||
# _base_: ../features/soft_gate.yaml
|
||||
# or copy the `evaluation:` block below into your config.
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
@@ -9,7 +9,7 @@ env:
|
||||
name: livemathematicianbench
|
||||
skill_init: skillopt/envs/livemathematicianbench/skills/initial.md
|
||||
split_mode: split_dir
|
||||
split_dir: data/ablation_splits/livemathematicianbench/2-1-7_seed42
|
||||
split_dir: data/livemathematicianbench_split
|
||||
data_path: ""
|
||||
split_output_dir: ""
|
||||
max_turns: 1
|
||||
|
||||
+223
@@ -0,0 +1,223 @@
|
||||
# Data Manifests
|
||||
|
||||
This directory releases lightweight split manifests for the SkillOpt paper
|
||||
splits. These manifests are not full runnable benchmark payloads. To evaluate a
|
||||
benchmark, first materialize the full examples from the raw data source when
|
||||
needed, then point `--split_dir` at the split directory listed below.
|
||||
|
||||
In this README, "coverage" describes which part of the upstream benchmark the
|
||||
manifest references. It does not mean the released manifest directory contains
|
||||
the full runnable examples.
|
||||
|
||||
## Layout
|
||||
|
||||
Every released manifest directory uses the same file layout:
|
||||
|
||||
```text
|
||||
data/<benchmark>_<manifest_type>/
|
||||
|-- split_manifest.json
|
||||
|-- train/items.json
|
||||
|-- val/items.json
|
||||
`-- test/items.json
|
||||
```
|
||||
|
||||
`split_manifest.json` records source metadata, split counts, and item fields.
|
||||
Each `items.json` contains only stable IDs or source-path hints.
|
||||
|
||||
## Released Splits
|
||||
|
||||
| Manifest directory | Benchmark | Counts | Coverage | Raw data source | `split_dir` |
|
||||
|---|---|---:|---|---|---|
|
||||
| `searchqa_id_split/` | SearchQA | 400 / 200 / 1400 | Official HF dataset IDs | [lucadiliello/searchqa](https://huggingface.co/datasets/lucadiliello/searchqa) | `data/searchqa_split` |
|
||||
| `livemathematicianbench_id_split/` | LiveMathematicianBench | 35 / 18 / 124 | Four official monthly files | [LiveMathematicianBench/LiveMathematicianBench](https://huggingface.co/datasets/LiveMathematicianBench/LiveMathematicianBench) | `data/livemathematicianbench_split` |
|
||||
| `docvqa_id_split/` | DocVQA | 107 / 53 / 374 | 10% subset of validation | [lmms-lab/DocVQA](https://huggingface.co/datasets/lmms-lab/DocVQA) | `data/docvqa/splits` |
|
||||
| `officeqa_id_split/` | OfficeQA | 50 / 24 / 172 | OfficeQA Full | [databricks/officeqa](https://huggingface.co/datasets/databricks/officeqa) | `data/officeqa_split` |
|
||||
| `spreadsheetbench_id_split/` | SpreadsheetBench | 80 / 40 / 280 | SpreadsheetBench Verified 400 | [KAKA22/SpreadsheetBench](https://huggingface.co/datasets/KAKA22/SpreadsheetBench) | `data/spreadsheetbench_split` |
|
||||
| `alfworld_path_split/` | ALFWorld | 39 / 18 / 134 | ALFWorld `json_2.1.1` paths | [alfworld/alfworld](https://github.com/alfworld/alfworld) | `data/alfworld_path_split` |
|
||||
|
||||
Counts are ordered as train / val / test.
|
||||
|
||||
## Direct Use
|
||||
|
||||
Only `alfworld_path_split/` can be used directly as `--split_dir` from this
|
||||
release, because the ALFWorld loader reads `gamefile` and `task_type` from the
|
||||
split items.
|
||||
|
||||
This does not mean the ALFWorld raw data is included. You still need to
|
||||
download ALFWorld separately with `alfworld-download` and set `$ALFWORLD_DATA`
|
||||
to the data root containing `json_2.1.1`.
|
||||
|
||||
The other manifest directories are lookup manifests. They intentionally omit
|
||||
full example fields such as questions, answers, contexts, images, or task
|
||||
instructions. Materialize those benchmarks into the `split_dir` paths listed
|
||||
above before running SkillOpt.
|
||||
|
||||
## Lookup Keys
|
||||
|
||||
The manifests are sufficient to locate the corresponding raw examples after
|
||||
the raw data has been downloaded or otherwise made available:
|
||||
|
||||
| Benchmark | Manifest lookup key |
|
||||
|---|---|
|
||||
| SearchQA | Match `items.json[].id` to the `key` field in `lucadiliello/searchqa`. |
|
||||
| LiveMathematicianBench | Open `source_file`, then match `no`; the manifest `id` is `<month>:<no>`. |
|
||||
| DocVQA | Match `questionId` within the official DocVQA `validation` split; `image_path` records the expected local image path. |
|
||||
| OfficeQA | Match `uid` in `officeqa_full.csv`; `source_files` and `source_docs` identify the supporting document. |
|
||||
| SpreadsheetBench | Match `id`; `spreadsheet_path` identifies the referenced spreadsheet directory. |
|
||||
| ALFWorld | Resolve `gamefile` relative to `$ALFWORLD_DATA`. |
|
||||
|
||||
## Manifest Item Examples
|
||||
|
||||
SearchQA:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "221c83e6630f4e7983da48fa28da1882"
|
||||
}
|
||||
```
|
||||
|
||||
LiveMathematicianBench:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "202602:22",
|
||||
"month": "202602",
|
||||
"no": 22,
|
||||
"paper_link": "http://arxiv.org/abs/2602.10700v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
}
|
||||
```
|
||||
|
||||
DocVQA:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "50877",
|
||||
"questionId": "50877",
|
||||
"docId": "14724",
|
||||
"image_path": "data/docvqa_images/q50877_d14724.png",
|
||||
"source_split": "validation"
|
||||
}
|
||||
```
|
||||
|
||||
OfficeQA:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "UID0002",
|
||||
"uid": "UID0002",
|
||||
"category": "easy",
|
||||
"source_files": "treasury_bulletin_1944_01.txt"
|
||||
}
|
||||
```
|
||||
|
||||
SpreadsheetBench:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "32438",
|
||||
"spreadsheet_path": "spreadsheet/32438",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
}
|
||||
```
|
||||
|
||||
ALFWorld:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "train:0000",
|
||||
"gamefile": "json_2.1.1/train/.../game.tw-pddl",
|
||||
"task_type": "look_at_obj_in_light"
|
||||
}
|
||||
```
|
||||
|
||||
## Benchmark Notes
|
||||
|
||||
### SearchQA
|
||||
|
||||
`searchqa_id_split/` is an ID-only manifest. Each released `id` exactly matches
|
||||
the `key` field in `lucadiliello/searchqa`.
|
||||
|
||||
Materialized examples must include the fields consumed by the SearchQA
|
||||
environment, including:
|
||||
|
||||
```text
|
||||
question
|
||||
context
|
||||
answers
|
||||
```
|
||||
|
||||
### LiveMathematicianBench
|
||||
|
||||
`livemathematicianbench_id_split/` was generated from these raw files:
|
||||
|
||||
```text
|
||||
data/202511/qa_202511_final.json
|
||||
data/202512/qa_202512_final.json
|
||||
data/202601/qa_202601_final.json
|
||||
data/202602/qa_202602_final.json
|
||||
```
|
||||
|
||||
The manifest stores IDs in the loader format:
|
||||
|
||||
```text
|
||||
<month>:<no>
|
||||
```
|
||||
|
||||
Materialized examples must include:
|
||||
|
||||
```text
|
||||
question
|
||||
choices
|
||||
correct_choice
|
||||
theorem_type
|
||||
theorem
|
||||
sketch
|
||||
paper_link
|
||||
```
|
||||
|
||||
### DocVQA
|
||||
|
||||
`docvqa_id_split/` records `docvqa_validation_10pct`: a 10% subset sampled from
|
||||
the official DocVQA `validation` split.
|
||||
|
||||
```text
|
||||
source_split: validation
|
||||
docvqa_validation_10pct: train=107, val=53, test=374
|
||||
```
|
||||
|
||||
Each manifest item contains question/document IDs plus image location metadata.
|
||||
Materialized examples must provide `question`, `answer` or `ground_truth`, and
|
||||
an `image_path` that resolves locally.
|
||||
|
||||
### OfficeQA
|
||||
|
||||
`officeqa_id_split/` records the split over OfficeQA Full
|
||||
(`officeqa_full.csv`). The official OfficeQA CSVs are gated on Hugging Face, so
|
||||
materialization requires authorized access.
|
||||
|
||||
Each manifest item contains `uid`, `category`, `source_files`, and
|
||||
`source_docs` hints. Materialized examples must include `question` and
|
||||
`ground_truth` or `answer`.
|
||||
|
||||
### SpreadsheetBench
|
||||
|
||||
`spreadsheetbench_id_split/` records the split over SpreadsheetBench Verified
|
||||
400, from `spreadsheetbench_verified_400.tar.gz`.
|
||||
|
||||
Each manifest item contains task identity metadata such as `id`,
|
||||
`spreadsheet_path`, and `instruction_type`. Materialization must also place the
|
||||
referenced spreadsheet directories at:
|
||||
|
||||
```text
|
||||
data/spreadsheetbench_verified_400
|
||||
```
|
||||
|
||||
### ALFWorld
|
||||
|
||||
`alfworld_path_split/` records `gamefile` paths relative to `$ALFWORLD_DATA`.
|
||||
The source payload is `json_2.1.1`, which must be downloaded separately with
|
||||
`alfworld-download`.
|
||||
|
||||
This manifest can be used directly as `--split_dir` after `$ALFWORLD_DATA`
|
||||
points to the local ALFWorld data root containing `json_2.1.1`.
|
||||
@@ -0,0 +1,29 @@
|
||||
{
|
||||
"benchmark": "ALFWorld",
|
||||
"manifest_type": "path_split",
|
||||
"source_repo": "alfworld/alfworld",
|
||||
"source_repo_type": "repository",
|
||||
"source_url": "https://github.com/alfworld/alfworld",
|
||||
"source_file": "json_2.1.1",
|
||||
"source_method": "generated by alfworld-download",
|
||||
"source_split_files": [
|
||||
"split_train.json",
|
||||
"split_val.json",
|
||||
"split_test.json"
|
||||
],
|
||||
"counts": {
|
||||
"train": 39,
|
||||
"val": 18,
|
||||
"test": 134
|
||||
},
|
||||
"item_fields": [
|
||||
"id",
|
||||
"gamefile",
|
||||
"task_type"
|
||||
],
|
||||
"path_root": "$ALFWORLD_DATA",
|
||||
"notes": [
|
||||
"This is a path manifest, not the ALFWorld game payload.",
|
||||
"The gamefile field is relative to ALFWORLD_DATA and must be expanded before direct use as split_dir data."
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,672 @@
|
||||
[
|
||||
{
|
||||
"id": "test:0000",
|
||||
"gamefile": "json_2.1.1/valid_unseen/look_at_obj_in_light-AlarmClock-None-DeskLamp-308/trial_T20190908_222917_366542/game.tw-pddl",
|
||||
"task_type": "look_at_obj_in_light"
|
||||
},
|
||||
{
|
||||
"id": "test:0001",
|
||||
"gamefile": "json_2.1.1/valid_unseen/look_at_obj_in_light-AlarmClock-None-DeskLamp-308/trial_T20190908_222933_607649/game.tw-pddl",
|
||||
"task_type": "look_at_obj_in_light"
|
||||
},
|
||||
{
|
||||
"id": "test:0002",
|
||||
"gamefile": "json_2.1.1/valid_unseen/look_at_obj_in_light-AlarmClock-None-DeskLamp-308/trial_T20190908_222951_616606/game.tw-pddl",
|
||||
"task_type": "look_at_obj_in_light"
|
||||
},
|
||||
{
|
||||
"id": "test:0003",
|
||||
"gamefile": "json_2.1.1/valid_unseen/look_at_obj_in_light-Book-None-DeskLamp-308/trial_T20190908_020029_636862/game.tw-pddl",
|
||||
"task_type": "look_at_obj_in_light"
|
||||
},
|
||||
{
|
||||
"id": "test:0004",
|
||||
"gamefile": "json_2.1.1/valid_unseen/look_at_obj_in_light-Book-None-DeskLamp-308/trial_T20190908_020048_814402/game.tw-pddl",
|
||||
"task_type": "look_at_obj_in_light"
|
||||
},
|
||||
{
|
||||
"id": "test:0005",
|
||||
"gamefile": "json_2.1.1/valid_unseen/look_at_obj_in_light-Book-None-DeskLamp-308/trial_T20190908_144951_587345/game.tw-pddl",
|
||||
"task_type": "look_at_obj_in_light"
|
||||
},
|
||||
{
|
||||
"id": "test:0006",
|
||||
"gamefile": "json_2.1.1/valid_unseen/look_at_obj_in_light-Bowl-None-DeskLamp-308/trial_T20190907_133919_856963/game.tw-pddl",
|
||||
"task_type": "look_at_obj_in_light"
|
||||
},
|
||||
{
|
||||
"id": "test:0007",
|
||||
"gamefile": "json_2.1.1/valid_unseen/look_at_obj_in_light-Bowl-None-DeskLamp-308/trial_T20190907_133935_066606/game.tw-pddl",
|
||||
"task_type": "look_at_obj_in_light"
|
||||
},
|
||||
{
|
||||
"id": "test:0008",
|
||||
"gamefile": "json_2.1.1/valid_unseen/look_at_obj_in_light-Bowl-None-DeskLamp-308/trial_T20190907_133953_562557/game.tw-pddl",
|
||||
"task_type": "look_at_obj_in_light"
|
||||
},
|
||||
{
|
||||
"id": "test:0009",
|
||||
"gamefile": "json_2.1.1/valid_unseen/look_at_obj_in_light-CD-None-DeskLamp-308/trial_T20190908_141942_810052/game.tw-pddl",
|
||||
"task_type": "look_at_obj_in_light"
|
||||
},
|
||||
{
|
||||
"id": "test:0010",
|
||||
"gamefile": "json_2.1.1/valid_unseen/look_at_obj_in_light-CD-None-DeskLamp-308/trial_T20190908_141958_463362/game.tw-pddl",
|
||||
"task_type": "look_at_obj_in_light"
|
||||
},
|
||||
{
|
||||
"id": "test:0011",
|
||||
"gamefile": "json_2.1.1/valid_unseen/look_at_obj_in_light-CD-None-DeskLamp-308/trial_T20190908_142046_281296/game.tw-pddl",
|
||||
"task_type": "look_at_obj_in_light"
|
||||
},
|
||||
{
|
||||
"id": "test:0012",
|
||||
"gamefile": "json_2.1.1/valid_unseen/look_at_obj_in_light-Mug-None-DeskLamp-308/trial_T20190908_161733_213242/game.tw-pddl",
|
||||
"task_type": "look_at_obj_in_light"
|
||||
},
|
||||
{
|
||||
"id": "test:0013",
|
||||
"gamefile": "json_2.1.1/valid_unseen/look_at_obj_in_light-Mug-None-DeskLamp-308/trial_T20190908_201421_021646/game.tw-pddl",
|
||||
"task_type": "look_at_obj_in_light"
|
||||
},
|
||||
{
|
||||
"id": "test:0014",
|
||||
"gamefile": "json_2.1.1/valid_unseen/look_at_obj_in_light-Mug-None-DeskLamp-308/trial_T20190908_201444_037645/game.tw-pddl",
|
||||
"task_type": "look_at_obj_in_light"
|
||||
},
|
||||
{
|
||||
"id": "test:0015",
|
||||
"gamefile": "json_2.1.1/valid_unseen/look_at_obj_in_light-Pencil-None-DeskLamp-308/trial_T20190908_220545_153480/game.tw-pddl",
|
||||
"task_type": "look_at_obj_in_light"
|
||||
},
|
||||
{
|
||||
"id": "test:0016",
|
||||
"gamefile": "json_2.1.1/valid_unseen/look_at_obj_in_light-Pencil-None-DeskLamp-308/trial_T20190908_220604_010430/game.tw-pddl",
|
||||
"task_type": "look_at_obj_in_light"
|
||||
},
|
||||
{
|
||||
"id": "test:0017",
|
||||
"gamefile": "json_2.1.1/valid_unseen/look_at_obj_in_light-Pencil-None-DeskLamp-308/trial_T20190908_220656_510400/game.tw-pddl",
|
||||
"task_type": "look_at_obj_in_light"
|
||||
},
|
||||
{
|
||||
"id": "test:0018",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_and_place_simple-Mug-None-Desk-308/trial_T20190908_125200_737896/game.tw-pddl",
|
||||
"task_type": "pick_and_place_simple"
|
||||
},
|
||||
{
|
||||
"id": "test:0019",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_and_place_simple-Mug-None-Desk-308/trial_T20190909_203041_433487/game.tw-pddl",
|
||||
"task_type": "pick_and_place_simple"
|
||||
},
|
||||
{
|
||||
"id": "test:0020",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_and_place_simple-Mug-None-Desk-308/trial_T20190909_210238_431966/game.tw-pddl",
|
||||
"task_type": "pick_and_place_simple"
|
||||
},
|
||||
{
|
||||
"id": "test:0021",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_and_place_simple-Pencil-None-Shelf-308/trial_T20190908_121952_610012/game.tw-pddl",
|
||||
"task_type": "pick_and_place_simple"
|
||||
},
|
||||
{
|
||||
"id": "test:0022",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_and_place_simple-Pencil-None-Shelf-308/trial_T20190908_122024_052056/game.tw-pddl",
|
||||
"task_type": "pick_and_place_simple"
|
||||
},
|
||||
{
|
||||
"id": "test:0023",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_and_place_simple-Pencil-None-Shelf-308/trial_T20190908_122154_042763/game.tw-pddl",
|
||||
"task_type": "pick_and_place_simple"
|
||||
},
|
||||
{
|
||||
"id": "test:0024",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_and_place_simple-PepperShaker-None-Drawer-10/trial_T20190906_184021_215264/game.tw-pddl",
|
||||
"task_type": "pick_and_place_simple"
|
||||
},
|
||||
{
|
||||
"id": "test:0025",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_and_place_simple-PepperShaker-None-Drawer-10/trial_T20190918_154326_823501/game.tw-pddl",
|
||||
"task_type": "pick_and_place_simple"
|
||||
},
|
||||
{
|
||||
"id": "test:0026",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_and_place_simple-PepperShaker-None-Drawer-10/trial_T20190918_154424_844749/game.tw-pddl",
|
||||
"task_type": "pick_and_place_simple"
|
||||
},
|
||||
{
|
||||
"id": "test:0027",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_and_place_simple-SaltShaker-None-Cabinet-10/trial_T20190906_191429_743650/game.tw-pddl",
|
||||
"task_type": "pick_and_place_simple"
|
||||
},
|
||||
{
|
||||
"id": "test:0028",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_and_place_simple-SaltShaker-None-Cabinet-10/trial_T20190906_191445_723170/game.tw-pddl",
|
||||
"task_type": "pick_and_place_simple"
|
||||
},
|
||||
{
|
||||
"id": "test:0029",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_and_place_simple-SaltShaker-None-Cabinet-10/trial_T20190906_191501_563086/game.tw-pddl",
|
||||
"task_type": "pick_and_place_simple"
|
||||
},
|
||||
{
|
||||
"id": "test:0030",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_and_place_simple-SaltShaker-None-Drawer-10/trial_T20190909_021613_077537/game.tw-pddl",
|
||||
"task_type": "pick_and_place_simple"
|
||||
},
|
||||
{
|
||||
"id": "test:0031",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_and_place_simple-SaltShaker-None-Drawer-10/trial_T20190909_021650_880235/game.tw-pddl",
|
||||
"task_type": "pick_and_place_simple"
|
||||
},
|
||||
{
|
||||
"id": "test:0032",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_and_place_simple-SaltShaker-None-Drawer-10/trial_T20190909_021728_339782/game.tw-pddl",
|
||||
"task_type": "pick_and_place_simple"
|
||||
},
|
||||
{
|
||||
"id": "test:0033",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_and_place_simple-SoapBottle-None-Toilet-424/trial_T20190907_004321_405868/game.tw-pddl",
|
||||
"task_type": "pick_and_place_simple"
|
||||
},
|
||||
{
|
||||
"id": "test:0034",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_and_place_simple-SoapBottle-None-Toilet-424/trial_T20190907_004351_281384/game.tw-pddl",
|
||||
"task_type": "pick_and_place_simple"
|
||||
},
|
||||
{
|
||||
"id": "test:0035",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_and_place_simple-SoapBottle-None-Toilet-424/trial_T20190907_004404_604165/game.tw-pddl",
|
||||
"task_type": "pick_and_place_simple"
|
||||
},
|
||||
{
|
||||
"id": "test:0036",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_and_place_simple-Vase-None-Safe-219/trial_T20190908_205204_244321/game.tw-pddl",
|
||||
"task_type": "pick_and_place_simple"
|
||||
},
|
||||
{
|
||||
"id": "test:0037",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_and_place_simple-Vase-None-Safe-219/trial_T20190908_205221_748352/game.tw-pddl",
|
||||
"task_type": "pick_and_place_simple"
|
||||
},
|
||||
{
|
||||
"id": "test:0038",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_and_place_simple-Vase-None-Safe-219/trial_T20190908_205246_776817/game.tw-pddl",
|
||||
"task_type": "pick_and_place_simple"
|
||||
},
|
||||
{
|
||||
"id": "test:0039",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_and_place_simple-Watch-None-Safe-219/trial_T20190907_074524_006355/game.tw-pddl",
|
||||
"task_type": "pick_and_place_simple"
|
||||
},
|
||||
{
|
||||
"id": "test:0040",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_and_place_simple-Watch-None-Safe-219/trial_T20190907_074556_124850/game.tw-pddl",
|
||||
"task_type": "pick_and_place_simple"
|
||||
},
|
||||
{
|
||||
"id": "test:0041",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_and_place_simple-Watch-None-Safe-219/trial_T20190907_074643_810052/game.tw-pddl",
|
||||
"task_type": "pick_and_place_simple"
|
||||
},
|
||||
{
|
||||
"id": "test:0042",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-Bowl-None-Cabinet-10/trial_T20190909_061130_844814/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0043",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-Bowl-None-Cabinet-10/trial_T20190909_061158_110530/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0044",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-Bowl-None-Cabinet-10/trial_T20190909_061232_368489/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0045",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-Cloth-None-Cabinet-424/trial_T20190908_022321_380927/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0046",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-Cloth-None-Cabinet-424/trial_T20190908_022436_073995/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0047",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-Cloth-None-CounterTop-424/trial_T20190908_100632_546757/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0048",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-Cloth-None-CounterTop-424/trial_T20190908_114340_674467/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0049",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-Egg-None-Microwave-10/trial_T20190909_120554_888709/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0050",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-Egg-None-Microwave-10/trial_T20190909_120632_691361/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0051",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-Egg-None-Microwave-10/trial_T20190909_120712_273910/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0052",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-Knife-None-CounterTop-10/trial_T20190909_110347_624008/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0053",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-Knife-None-CounterTop-10/trial_T20190909_110445_675754/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0054",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-Knife-None-CounterTop-10/trial_T20190909_110531_148235/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0055",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-Mug-None-CoffeeMachine-10/trial_T20190907_221208_560499/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0056",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-Mug-None-CoffeeMachine-10/trial_T20190907_221300_362511/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0057",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-Mug-None-CoffeeMachine-10/trial_T20190907_221355_558505/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0058",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-Pan-None-CounterTop-10/trial_T20190908_032434_013084/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0059",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-Pan-None-CounterTop-10/trial_T20190908_032518_891433/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0060",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-Pan-None-CounterTop-10/trial_T20190908_032543_712058/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0061",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-Plate-None-CounterTop-10/trial_T20190908_213356_017769/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0062",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-Plate-None-CounterTop-10/trial_T20190908_213420_728917/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0063",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-Plate-None-CounterTop-10/trial_T20190908_213533_897289/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0064",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-SoapBar-None-Cabinet-424/trial_T20190908_214926_337906/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0065",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-SoapBar-None-Cabinet-424/trial_T20190908_214946_567644/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0066",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-SoapBar-None-Cabinet-424/trial_T20190908_215019_162873/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0067",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-SoapBar-None-CounterTop-424/trial_T20190907_074045_109439/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0068",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-SoapBar-None-CounterTop-424/trial_T20190907_074106_050405/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0069",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-SoapBar-None-CounterTop-424/trial_T20190907_074124_966890/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0070",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-Spatula-None-Drawer-10/trial_T20190907_080730_211959/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0071",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-Spatula-None-Drawer-10/trial_T20190907_080800_275989/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0072",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_clean_then_place_in_recep-Spatula-None-Drawer-10/trial_T20190907_080825_222432/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0073",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_cool_then_place_in_recep-Bread-None-CounterTop-10/trial_T20190908_091747_866951/game.tw-pddl",
|
||||
"task_type": "pick_cool_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0074",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_cool_then_place_in_recep-Bread-None-CounterTop-10/trial_T20190908_091811_414150/game.tw-pddl",
|
||||
"task_type": "pick_cool_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0075",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_cool_then_place_in_recep-Bread-None-CounterTop-10/trial_T20190908_091835_825830/game.tw-pddl",
|
||||
"task_type": "pick_cool_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0076",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_cool_then_place_in_recep-Lettuce-None-CounterTop-10/trial_T20190909_123133_763972/game.tw-pddl",
|
||||
"task_type": "pick_cool_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0077",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_cool_then_place_in_recep-Lettuce-None-CounterTop-10/trial_T20190909_174807_646433/game.tw-pddl",
|
||||
"task_type": "pick_cool_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0078",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_cool_then_place_in_recep-Lettuce-None-CounterTop-10/trial_T20190909_174840_771703/game.tw-pddl",
|
||||
"task_type": "pick_cool_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0079",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_cool_then_place_in_recep-Mug-None-Cabinet-10/trial_T20190909_121559_082363/game.tw-pddl",
|
||||
"task_type": "pick_cool_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0080",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_cool_then_place_in_recep-Mug-None-Cabinet-10/trial_T20190909_121635_622676/game.tw-pddl",
|
||||
"task_type": "pick_cool_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0081",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_cool_then_place_in_recep-Mug-None-Cabinet-10/trial_T20190909_121710_650938/game.tw-pddl",
|
||||
"task_type": "pick_cool_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0082",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_cool_then_place_in_recep-Mug-None-CoffeeMachine-10/trial_T20190907_183715_299073/game.tw-pddl",
|
||||
"task_type": "pick_cool_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0083",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_cool_then_place_in_recep-Mug-None-CoffeeMachine-10/trial_T20190907_183807_477267/game.tw-pddl",
|
||||
"task_type": "pick_cool_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0084",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_cool_then_place_in_recep-Mug-None-CoffeeMachine-10/trial_T20190907_183853_958104/game.tw-pddl",
|
||||
"task_type": "pick_cool_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0085",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_cool_then_place_in_recep-Pan-None-CounterTop-10/trial_T20190908_114545_244903/game.tw-pddl",
|
||||
"task_type": "pick_cool_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0086",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_cool_then_place_in_recep-Pan-None-CounterTop-10/trial_T20190908_114622_738670/game.tw-pddl",
|
||||
"task_type": "pick_cool_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0087",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_cool_then_place_in_recep-Pan-None-CounterTop-10/trial_T20190908_114656_768805/game.tw-pddl",
|
||||
"task_type": "pick_cool_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0088",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_cool_then_place_in_recep-Potato-None-Microwave-10/trial_T20190907_033157_424297/game.tw-pddl",
|
||||
"task_type": "pick_cool_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0089",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_cool_then_place_in_recep-Potato-None-Microwave-10/trial_T20190907_033228_194678/game.tw-pddl",
|
||||
"task_type": "pick_cool_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0090",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_cool_then_place_in_recep-Potato-None-Microwave-10/trial_T20190907_033306_962974/game.tw-pddl",
|
||||
"task_type": "pick_cool_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0091",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_cool_then_place_in_recep-Tomato-None-Microwave-10/trial_T20190909_102608_318800/game.tw-pddl",
|
||||
"task_type": "pick_cool_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0092",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_cool_then_place_in_recep-Tomato-None-Microwave-10/trial_T20190909_102644_926781/game.tw-pddl",
|
||||
"task_type": "pick_cool_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0093",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_cool_then_place_in_recep-Tomato-None-Microwave-10/trial_T20190909_102710_795182/game.tw-pddl",
|
||||
"task_type": "pick_cool_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0094",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_heat_then_place_in_recep-Apple-None-Fridge-10/trial_T20190906_182259_116320/game.tw-pddl",
|
||||
"task_type": "pick_heat_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0095",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_heat_then_place_in_recep-Apple-None-Fridge-10/trial_T20190906_182353_418140/game.tw-pddl",
|
||||
"task_type": "pick_heat_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0096",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_heat_then_place_in_recep-Apple-None-Fridge-10/trial_T20190906_182435_622538/game.tw-pddl",
|
||||
"task_type": "pick_heat_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0097",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_heat_then_place_in_recep-Apple-None-GarbageCan-10/trial_T20190908_145050_918567/game.tw-pddl",
|
||||
"task_type": "pick_heat_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0098",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_heat_then_place_in_recep-Apple-None-GarbageCan-10/trial_T20190908_145143_820541/game.tw-pddl",
|
||||
"task_type": "pick_heat_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0099",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_heat_then_place_in_recep-Apple-None-GarbageCan-10/trial_T20190908_145356_918528/game.tw-pddl",
|
||||
"task_type": "pick_heat_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0100",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_heat_then_place_in_recep-Cup-None-Cabinet-10/trial_T20190907_083346_800823/game.tw-pddl",
|
||||
"task_type": "pick_heat_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0101",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_heat_then_place_in_recep-Cup-None-Cabinet-10/trial_T20190907_083429_887065/game.tw-pddl",
|
||||
"task_type": "pick_heat_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0102",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_heat_then_place_in_recep-Cup-None-Cabinet-10/trial_T20190907_083507_594820/game.tw-pddl",
|
||||
"task_type": "pick_heat_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0103",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_heat_then_place_in_recep-Egg-None-GarbageCan-10/trial_T20190908_113432_673307/game.tw-pddl",
|
||||
"task_type": "pick_heat_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0104",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_heat_then_place_in_recep-Egg-None-GarbageCan-10/trial_T20190908_113523_123938/game.tw-pddl",
|
||||
"task_type": "pick_heat_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0105",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_heat_then_place_in_recep-Egg-None-GarbageCan-10/trial_T20190908_113610_425142/game.tw-pddl",
|
||||
"task_type": "pick_heat_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0106",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_heat_then_place_in_recep-Mug-None-Cabinet-10/trial_T20190909_021100_341887/game.tw-pddl",
|
||||
"task_type": "pick_heat_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0107",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_heat_then_place_in_recep-Mug-None-Cabinet-10/trial_T20190909_021200_669381/game.tw-pddl",
|
||||
"task_type": "pick_heat_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0108",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_heat_then_place_in_recep-Mug-None-Cabinet-10/trial_T20190909_021247_306737/game.tw-pddl",
|
||||
"task_type": "pick_heat_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0109",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_heat_then_place_in_recep-Mug-None-CoffeeMachine-10/trial_T20190907_171806_406231/game.tw-pddl",
|
||||
"task_type": "pick_heat_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0110",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_heat_then_place_in_recep-Mug-None-CoffeeMachine-10/trial_T20190907_171850_960211/game.tw-pddl",
|
||||
"task_type": "pick_heat_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0111",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_heat_then_place_in_recep-Mug-None-CoffeeMachine-10/trial_T20190907_171933_349922/game.tw-pddl",
|
||||
"task_type": "pick_heat_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0112",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_heat_then_place_in_recep-Potato-None-GarbageCan-10/trial_T20190907_161745_664033/game.tw-pddl",
|
||||
"task_type": "pick_heat_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0113",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_heat_then_place_in_recep-Potato-None-GarbageCan-10/trial_T20190907_161853_945788/game.tw-pddl",
|
||||
"task_type": "pick_heat_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0114",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_heat_then_place_in_recep-Tomato-None-GarbageCan-10/trial_T20190908_225046_020282/game.tw-pddl",
|
||||
"task_type": "pick_heat_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0115",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_heat_then_place_in_recep-Tomato-None-GarbageCan-10/trial_T20190908_225359_617900/game.tw-pddl",
|
||||
"task_type": "pick_heat_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0116",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_heat_then_place_in_recep-Tomato-None-GarbageCan-10/trial_T20190908_225453_272533/game.tw-pddl",
|
||||
"task_type": "pick_heat_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "test:0117",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_two_obj_and_place-CD-None-Safe-308/trial_T20190907_050942_897916/game.tw-pddl",
|
||||
"task_type": "pick_two_obj_and_place"
|
||||
},
|
||||
{
|
||||
"id": "test:0118",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_two_obj_and_place-CD-None-Safe-308/trial_T20190907_051013_060265/game.tw-pddl",
|
||||
"task_type": "pick_two_obj_and_place"
|
||||
},
|
||||
{
|
||||
"id": "test:0119",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_two_obj_and_place-CD-None-Safe-308/trial_T20190907_051056_585414/game.tw-pddl",
|
||||
"task_type": "pick_two_obj_and_place"
|
||||
},
|
||||
{
|
||||
"id": "test:0120",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_two_obj_and_place-KeyChain-None-Safe-219/trial_T20190909_011803_423115/game.tw-pddl",
|
||||
"task_type": "pick_two_obj_and_place"
|
||||
},
|
||||
{
|
||||
"id": "test:0121",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_two_obj_and_place-KeyChain-None-Safe-219/trial_T20190909_012027_782483/game.tw-pddl",
|
||||
"task_type": "pick_two_obj_and_place"
|
||||
},
|
||||
{
|
||||
"id": "test:0122",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_two_obj_and_place-PepperShaker-None-Drawer-10/trial_T20190908_010306_215435/game.tw-pddl",
|
||||
"task_type": "pick_two_obj_and_place"
|
||||
},
|
||||
{
|
||||
"id": "test:0123",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_two_obj_and_place-PepperShaker-None-Drawer-10/trial_T20190912_221016_460197/game.tw-pddl",
|
||||
"task_type": "pick_two_obj_and_place"
|
||||
},
|
||||
{
|
||||
"id": "test:0124",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_two_obj_and_place-PepperShaker-None-Drawer-10/trial_T20190912_221141_608117/game.tw-pddl",
|
||||
"task_type": "pick_two_obj_and_place"
|
||||
},
|
||||
{
|
||||
"id": "test:0125",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_two_obj_and_place-Pillow-None-Sofa-219/trial_T20190907_163240_345855/game.tw-pddl",
|
||||
"task_type": "pick_two_obj_and_place"
|
||||
},
|
||||
{
|
||||
"id": "test:0126",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_two_obj_and_place-Pillow-None-Sofa-219/trial_T20190907_163327_486300/game.tw-pddl",
|
||||
"task_type": "pick_two_obj_and_place"
|
||||
},
|
||||
{
|
||||
"id": "test:0127",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_two_obj_and_place-Pillow-None-Sofa-219/trial_T20190907_163408_914117/game.tw-pddl",
|
||||
"task_type": "pick_two_obj_and_place"
|
||||
},
|
||||
{
|
||||
"id": "test:0128",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_two_obj_and_place-SoapBar-None-Cabinet-424/trial_T20190909_081720_491733/game.tw-pddl",
|
||||
"task_type": "pick_two_obj_and_place"
|
||||
},
|
||||
{
|
||||
"id": "test:0129",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_two_obj_and_place-SoapBar-None-Cabinet-424/trial_T20190909_081746_857594/game.tw-pddl",
|
||||
"task_type": "pick_two_obj_and_place"
|
||||
},
|
||||
{
|
||||
"id": "test:0130",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_two_obj_and_place-SoapBar-None-GarbageCan-424/trial_T20190909_064053_839817/game.tw-pddl",
|
||||
"task_type": "pick_two_obj_and_place"
|
||||
},
|
||||
{
|
||||
"id": "test:0131",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_two_obj_and_place-SoapBar-None-GarbageCan-424/trial_T20190909_064221_368939/game.tw-pddl",
|
||||
"task_type": "pick_two_obj_and_place"
|
||||
},
|
||||
{
|
||||
"id": "test:0132",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_two_obj_and_place-SoapBar-None-GarbageCan-424/trial_T20190909_064309_357168/game.tw-pddl",
|
||||
"task_type": "pick_two_obj_and_place"
|
||||
},
|
||||
{
|
||||
"id": "test:0133",
|
||||
"gamefile": "json_2.1.1/valid_unseen/pick_two_obj_and_place-ToiletPaper-None-Cabinet-424/trial_T20190906_202926_527010/game.tw-pddl",
|
||||
"task_type": "pick_two_obj_and_place"
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,197 @@
|
||||
[
|
||||
{
|
||||
"id": "train:0000",
|
||||
"gamefile": "json_2.1.1/train/look_at_obj_in_light-AlarmClock-None-DeskLamp-305/trial_T20190908_082736_108723/game.tw-pddl",
|
||||
"task_type": "look_at_obj_in_light"
|
||||
},
|
||||
{
|
||||
"id": "train:0001",
|
||||
"gamefile": "json_2.1.1/train/look_at_obj_in_light-CD-None-DeskLamp-304/trial_T20190907_185649_782438/game.tw-pddl",
|
||||
"task_type": "look_at_obj_in_light"
|
||||
},
|
||||
{
|
||||
"id": "train:0002",
|
||||
"gamefile": "json_2.1.1/train/look_at_obj_in_light-CD-None-DeskLamp-320/trial_T20190907_224439_174735/game.tw-pddl",
|
||||
"task_type": "look_at_obj_in_light"
|
||||
},
|
||||
{
|
||||
"id": "train:0003",
|
||||
"gamefile": "json_2.1.1/train/look_at_obj_in_light-Pillow-None-DeskLamp-316/trial_T20190908_232421_645610/game.tw-pddl",
|
||||
"task_type": "look_at_obj_in_light"
|
||||
},
|
||||
{
|
||||
"id": "train:0004",
|
||||
"gamefile": "json_2.1.1/train/look_at_obj_in_light-Statue-None-DeskLamp-319/trial_T20190907_035546_167548/game.tw-pddl",
|
||||
"task_type": "look_at_obj_in_light"
|
||||
},
|
||||
{
|
||||
"id": "train:0005",
|
||||
"gamefile": "json_2.1.1/train/pick_and_place_simple-CellPhone-None-Shelf-313/trial_T20190908_123725_452958/game.tw-pddl",
|
||||
"task_type": "pick_and_place_simple"
|
||||
},
|
||||
{
|
||||
"id": "train:0006",
|
||||
"gamefile": "json_2.1.1/train/pick_and_place_simple-Newspaper-None-Sofa-211/trial_T20190906_175004_203092/game.tw-pddl",
|
||||
"task_type": "pick_and_place_simple"
|
||||
},
|
||||
{
|
||||
"id": "train:0007",
|
||||
"gamefile": "json_2.1.1/train/pick_and_place_simple-Pencil-None-Desk-302/trial_T20190908_032836_462632/game.tw-pddl",
|
||||
"task_type": "pick_and_place_simple"
|
||||
},
|
||||
{
|
||||
"id": "train:0008",
|
||||
"gamefile": "json_2.1.1/train/pick_and_place_simple-SoapBar-None-GarbageCan-416/trial_T20190908_020839_714699/game.tw-pddl",
|
||||
"task_type": "pick_and_place_simple"
|
||||
},
|
||||
{
|
||||
"id": "train:0009",
|
||||
"gamefile": "json_2.1.1/train/pick_and_place_simple-Statue-None-CoffeeTable-222/trial_T20190907_131249_788749/game.tw-pddl",
|
||||
"task_type": "pick_and_place_simple"
|
||||
},
|
||||
{
|
||||
"id": "train:0010",
|
||||
"gamefile": "json_2.1.1/train/pick_and_place_simple-ToiletPaper-None-ToiletPaperHanger-406/trial_T20190908_122807_136741/game.tw-pddl",
|
||||
"task_type": "pick_and_place_simple"
|
||||
},
|
||||
{
|
||||
"id": "train:0011",
|
||||
"gamefile": "json_2.1.1/train/pick_and_place_simple-ToiletPaper-None-ToiletPaperHanger-415/trial_T20190908_050443_333939/game.tw-pddl",
|
||||
"task_type": "pick_and_place_simple"
|
||||
},
|
||||
{
|
||||
"id": "train:0012",
|
||||
"gamefile": "json_2.1.1/train/pick_clean_then_place_in_recep-Apple-None-DiningTable-4/trial_T20190908_104413_450768/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "train:0013",
|
||||
"gamefile": "json_2.1.1/train/pick_clean_then_place_in_recep-DishSponge-None-Shelf-20/trial_T20190907_222429_992578/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "train:0014",
|
||||
"gamefile": "json_2.1.1/train/pick_clean_then_place_in_recep-DishSponge-None-Shelf-401/trial_T20190908_072225_397518/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "train:0015",
|
||||
"gamefile": "json_2.1.1/train/pick_clean_then_place_in_recep-Kettle-None-Cabinet-2/trial_T20190909_043103_418752/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "train:0016",
|
||||
"gamefile": "json_2.1.1/train/pick_clean_then_place_in_recep-Knife-None-Drawer-22/trial_T20190907_224827_746945/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "train:0017",
|
||||
"gamefile": "json_2.1.1/train/pick_clean_then_place_in_recep-Lettuce-None-DiningTable-20/trial_T20190906_191148_519826/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "train:0018",
|
||||
"gamefile": "json_2.1.1/train/pick_clean_then_place_in_recep-Lettuce-None-Fridge-13/trial_T20190908_203022_601787/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "train:0019",
|
||||
"gamefile": "json_2.1.1/train/pick_clean_then_place_in_recep-Plate-None-Fridge-5/trial_T20190909_112954_869911/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "train:0020",
|
||||
"gamefile": "json_2.1.1/train/pick_clean_then_place_in_recep-Spoon-None-DiningTable-18/trial_T20190909_102159_277894/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "train:0021",
|
||||
"gamefile": "json_2.1.1/train/pick_cool_then_place_in_recep-Bread-None-CounterTop-1/trial_T20190908_212439_711334/game.tw-pddl",
|
||||
"task_type": "pick_cool_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "train:0022",
|
||||
"gamefile": "json_2.1.1/train/pick_cool_then_place_in_recep-Bread-None-CounterTop-15/trial_T20190909_085448_256298/game.tw-pddl",
|
||||
"task_type": "pick_cool_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "train:0023",
|
||||
"gamefile": "json_2.1.1/train/pick_cool_then_place_in_recep-Bread-None-CounterTop-16/trial_T20190908_143948_082471/game.tw-pddl",
|
||||
"task_type": "pick_cool_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "train:0024",
|
||||
"gamefile": "json_2.1.1/train/pick_cool_then_place_in_recep-Pan-None-StoveBurner-27/trial_T20190906_212619_469871/game.tw-pddl",
|
||||
"task_type": "pick_cool_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "train:0025",
|
||||
"gamefile": "json_2.1.1/train/pick_cool_then_place_in_recep-Plate-None-DiningTable-17/trial_T20190909_122939_032098/game.tw-pddl",
|
||||
"task_type": "pick_cool_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "train:0026",
|
||||
"gamefile": "json_2.1.1/train/pick_cool_then_place_in_recep-Pot-None-CounterTop-1/trial_T20190909_124252_504581/game.tw-pddl",
|
||||
"task_type": "pick_cool_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "train:0027",
|
||||
"gamefile": "json_2.1.1/train/pick_heat_then_place_in_recep-Apple-None-Fridge-20/trial_T20190908_013911_274341/game.tw-pddl",
|
||||
"task_type": "pick_heat_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "train:0028",
|
||||
"gamefile": "json_2.1.1/train/pick_heat_then_place_in_recep-Egg-None-CounterTop-12/trial_T20190908_215527_416490/game.tw-pddl",
|
||||
"task_type": "pick_heat_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "train:0029",
|
||||
"gamefile": "json_2.1.1/train/pick_heat_then_place_in_recep-Mug-None-CoffeeMachine-1/trial_T20190907_222924_821086/game.tw-pddl",
|
||||
"task_type": "pick_heat_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "train:0030",
|
||||
"gamefile": "json_2.1.1/train/pick_heat_then_place_in_recep-Mug-None-CoffeeMachine-28/trial_T20190908_062730_537428/game.tw-pddl",
|
||||
"task_type": "pick_heat_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "train:0031",
|
||||
"gamefile": "json_2.1.1/train/pick_heat_then_place_in_recep-Plate-None-Cabinet-13/trial_T20190907_062749_759882/game.tw-pddl",
|
||||
"task_type": "pick_heat_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "train:0032",
|
||||
"gamefile": "json_2.1.1/train/pick_heat_then_place_in_recep-Potato-None-Fridge-2/trial_T20190909_030845_198194/game.tw-pddl",
|
||||
"task_type": "pick_heat_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "train:0033",
|
||||
"gamefile": "json_2.1.1/train/pick_heat_then_place_in_recep-Tomato-None-CounterTop-26/trial_T20190907_005525_499114/game.tw-pddl",
|
||||
"task_type": "pick_heat_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "train:0034",
|
||||
"gamefile": "json_2.1.1/train/pick_two_obj_and_place-CD-None-Drawer-319/trial_T20190907_145515_348252/game.tw-pddl",
|
||||
"task_type": "pick_two_obj_and_place"
|
||||
},
|
||||
{
|
||||
"id": "train:0035",
|
||||
"gamefile": "json_2.1.1/train/pick_two_obj_and_place-Candle-None-Drawer-427/trial_T20190909_043917_251333/game.tw-pddl",
|
||||
"task_type": "pick_two_obj_and_place"
|
||||
},
|
||||
{
|
||||
"id": "train:0036",
|
||||
"gamefile": "json_2.1.1/train/pick_two_obj_and_place-KeyChain-None-ArmChair-222/trial_T20190909_100312_677332/game.tw-pddl",
|
||||
"task_type": "pick_two_obj_and_place"
|
||||
},
|
||||
{
|
||||
"id": "train:0037",
|
||||
"gamefile": "json_2.1.1/train/pick_two_obj_and_place-Newspaper-None-Sofa-212/trial_T20190908_112632_208041/game.tw-pddl",
|
||||
"task_type": "pick_two_obj_and_place"
|
||||
},
|
||||
{
|
||||
"id": "train:0038",
|
||||
"gamefile": "json_2.1.1/train/pick_two_obj_and_place-SaltShaker-None-SideTable-21/trial_T20190909_041626_844806/game.tw-pddl",
|
||||
"task_type": "pick_two_obj_and_place"
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,92 @@
|
||||
[
|
||||
{
|
||||
"id": "val:0000",
|
||||
"gamefile": "json_2.1.1/valid_seen/look_at_obj_in_light-AlarmClock-None-DeskLamp-323/trial_T20190909_044715_250790/game.tw-pddl",
|
||||
"task_type": "look_at_obj_in_light"
|
||||
},
|
||||
{
|
||||
"id": "val:0001",
|
||||
"gamefile": "json_2.1.1/valid_seen/look_at_obj_in_light-Bowl-None-DeskLamp-301/trial_T20190909_150719_492274/game.tw-pddl",
|
||||
"task_type": "look_at_obj_in_light"
|
||||
},
|
||||
{
|
||||
"id": "val:0002",
|
||||
"gamefile": "json_2.1.1/valid_seen/look_at_obj_in_light-Pillow-None-DeskLamp-323/trial_T20190908_053153_077977/game.tw-pddl",
|
||||
"task_type": "look_at_obj_in_light"
|
||||
},
|
||||
{
|
||||
"id": "val:0003",
|
||||
"gamefile": "json_2.1.1/valid_seen/pick_and_place_simple-Mug-None-SideTable-329/trial_T20190909_032318_169393/game.tw-pddl",
|
||||
"task_type": "pick_and_place_simple"
|
||||
},
|
||||
{
|
||||
"id": "val:0004",
|
||||
"gamefile": "json_2.1.1/valid_seen/pick_and_place_simple-Mug-None-SideTable-329/trial_T20190909_032340_274147/game.tw-pddl",
|
||||
"task_type": "pick_and_place_simple"
|
||||
},
|
||||
{
|
||||
"id": "val:0005",
|
||||
"gamefile": "json_2.1.1/valid_seen/pick_and_place_simple-Pencil-None-Desk-310/trial_T20190909_113054_894334/game.tw-pddl",
|
||||
"task_type": "pick_and_place_simple"
|
||||
},
|
||||
{
|
||||
"id": "val:0006",
|
||||
"gamefile": "json_2.1.1/valid_seen/pick_clean_then_place_in_recep-ButterKnife-None-Drawer-30/trial_T20190908_052007_212776/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "val:0007",
|
||||
"gamefile": "json_2.1.1/valid_seen/pick_clean_then_place_in_recep-ButterKnife-None-Drawer-8/trial_T20190909_124425_112757/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "val:0008",
|
||||
"gamefile": "json_2.1.1/valid_seen/pick_clean_then_place_in_recep-SoapBar-None-Cabinet-402/trial_T20190908_055221_984342/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "val:0009",
|
||||
"gamefile": "json_2.1.1/valid_seen/pick_clean_then_place_in_recep-SoapBar-None-Toilet-410/trial_T20190906_201106_979461/game.tw-pddl",
|
||||
"task_type": "pick_clean_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "val:0010",
|
||||
"gamefile": "json_2.1.1/valid_seen/pick_cool_then_place_in_recep-Apple-None-Microwave-19/trial_T20190906_210937_878489/game.tw-pddl",
|
||||
"task_type": "pick_cool_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "val:0011",
|
||||
"gamefile": "json_2.1.1/valid_seen/pick_cool_then_place_in_recep-Plate-None-CounterTop-1/trial_T20190906_205324_559361/game.tw-pddl",
|
||||
"task_type": "pick_cool_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "val:0012",
|
||||
"gamefile": "json_2.1.1/valid_seen/pick_cool_then_place_in_recep-Tomato-None-Microwave-18/trial_T20190909_012524_159092/game.tw-pddl",
|
||||
"task_type": "pick_cool_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "val:0013",
|
||||
"gamefile": "json_2.1.1/valid_seen/pick_heat_then_place_in_recep-Apple-None-DiningTable-26/trial_T20190907_060234_011675/game.tw-pddl",
|
||||
"task_type": "pick_heat_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "val:0014",
|
||||
"gamefile": "json_2.1.1/valid_seen/pick_heat_then_place_in_recep-Tomato-None-Fridge-15/trial_T20190909_020200_054379/game.tw-pddl",
|
||||
"task_type": "pick_heat_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "val:0015",
|
||||
"gamefile": "json_2.1.1/valid_seen/pick_heat_then_place_in_recep-Tomato-None-Fridge-23/trial_T20190909_082320_103350/game.tw-pddl",
|
||||
"task_type": "pick_heat_then_place_in_recep"
|
||||
},
|
||||
{
|
||||
"id": "val:0016",
|
||||
"gamefile": "json_2.1.1/valid_seen/pick_two_obj_and_place-Book-None-Desk-313/trial_T20190908_125930_920681/game.tw-pddl",
|
||||
"task_type": "pick_two_obj_and_place"
|
||||
},
|
||||
{
|
||||
"id": "val:0017",
|
||||
"gamefile": "json_2.1.1/valid_seen/pick_two_obj_and_place-CreditCard-None-Safe-323/trial_T20190907_001129_214240/game.tw-pddl",
|
||||
"task_type": "pick_two_obj_and_place"
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,36 @@
|
||||
{
|
||||
"benchmark": "DocVQA",
|
||||
"manifest_type": "id_split",
|
||||
"source_repo": "lmms-lab/DocVQA",
|
||||
"source_repo_type": "dataset",
|
||||
"source_url": "https://huggingface.co/datasets/lmms-lab/DocVQA",
|
||||
"source_revision": "539088ef8a8ada01ac8e2e6d4e372586748a265e",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"source_split_name": "docvqa_validation_10pct",
|
||||
"split_method": "10% subset sampled from the DocVQA validation split",
|
||||
"counts": {
|
||||
"train": 107,
|
||||
"val": 53,
|
||||
"test": 374
|
||||
},
|
||||
"item_fields": [
|
||||
"id",
|
||||
"questionId",
|
||||
"docId",
|
||||
"image_path",
|
||||
"ucsf_document_id",
|
||||
"ucsf_document_page_no",
|
||||
"topic",
|
||||
"source_dataset",
|
||||
"source_config",
|
||||
"source_split",
|
||||
"sample_seed"
|
||||
],
|
||||
"notes": [
|
||||
"This is a split manifest, not the full DocVQA payload.",
|
||||
"Materialize full CSV rows and image files before evaluation.",
|
||||
"This manifest corresponds to docvqa_validation_10pct.",
|
||||
"All released train/val/test items originate from a 10% subset of the official DocVQA validation split."
|
||||
]
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,691 @@
|
||||
[
|
||||
{
|
||||
"id": "62409",
|
||||
"questionId": "62409",
|
||||
"docId": "8554",
|
||||
"image_path": "data/docvqa_images/q62409_d8554.png",
|
||||
"ucsf_document_id": "pgjw0227",
|
||||
"ucsf_document_page_no": "5",
|
||||
"topic": "table/list",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "50961",
|
||||
"questionId": "50961",
|
||||
"docId": "549",
|
||||
"image_path": "data/docvqa_images/q50961_d549.png",
|
||||
"ucsf_document_id": "qtjf0226",
|
||||
"ucsf_document_page_no": "2",
|
||||
"topic": "free_text",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "46461",
|
||||
"questionId": "46461",
|
||||
"docId": "13361",
|
||||
"image_path": "data/docvqa_images/q46461_d13361.png",
|
||||
"ucsf_document_id": "ysbw0217",
|
||||
"ucsf_document_page_no": "5",
|
||||
"topic": "layout",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "3041",
|
||||
"questionId": "3041",
|
||||
"docId": "1204",
|
||||
"image_path": "data/docvqa_images/q3041_d1204.png",
|
||||
"ucsf_document_id": "xfjv0228",
|
||||
"ucsf_document_page_no": "3",
|
||||
"topic": "form",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "41716",
|
||||
"questionId": "41716",
|
||||
"docId": "11835",
|
||||
"image_path": "data/docvqa_images/q41716_d11835.png",
|
||||
"ucsf_document_id": "qjgn0226",
|
||||
"ucsf_document_page_no": "131",
|
||||
"topic": "form",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "61123",
|
||||
"questionId": "61123",
|
||||
"docId": "7374",
|
||||
"image_path": "data/docvqa_images/q61123_d7374.png",
|
||||
"ucsf_document_id": "mldg0227",
|
||||
"ucsf_document_page_no": "5",
|
||||
"topic": "layout",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "43068",
|
||||
"questionId": "43068",
|
||||
"docId": "12393",
|
||||
"image_path": "data/docvqa_images/q43068_d12393.png",
|
||||
"ucsf_document_id": "rmwn0226",
|
||||
"ucsf_document_page_no": "52",
|
||||
"topic": "figure/diagram",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "51221",
|
||||
"questionId": "51221",
|
||||
"docId": "764",
|
||||
"image_path": "data/docvqa_images/q51221_d764.png",
|
||||
"ucsf_document_id": "kzbn0226",
|
||||
"ucsf_document_page_no": "14",
|
||||
"topic": "layout",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "6397",
|
||||
"questionId": "6397",
|
||||
"docId": "2242",
|
||||
"image_path": "data/docvqa_images/q6397_d2242.png",
|
||||
"ucsf_document_id": "jkcn0000",
|
||||
"ucsf_document_page_no": "2",
|
||||
"topic": "form",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "57428",
|
||||
"questionId": "57428",
|
||||
"docId": "4779",
|
||||
"image_path": "data/docvqa_images/q57428_d4779.png",
|
||||
"ucsf_document_id": "rnbx0223",
|
||||
"ucsf_document_page_no": "208",
|
||||
"topic": "Image/Photo",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "3135",
|
||||
"questionId": "3135",
|
||||
"docId": "1221",
|
||||
"image_path": "data/docvqa_images/q3135_d1221.png",
|
||||
"ucsf_document_id": "ngph0227",
|
||||
"ucsf_document_page_no": "5",
|
||||
"topic": "layout",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "18819",
|
||||
"questionId": "18819",
|
||||
"docId": "5749",
|
||||
"image_path": "data/docvqa_images/q18819_d5749.png",
|
||||
"ucsf_document_id": "jhfd0079",
|
||||
"ucsf_document_page_no": "9",
|
||||
"topic": "form",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "15382",
|
||||
"questionId": "15382",
|
||||
"docId": "4890",
|
||||
"image_path": "data/docvqa_images/q15382_d4890.png",
|
||||
"ucsf_document_id": "kjvw0217",
|
||||
"ucsf_document_page_no": "3",
|
||||
"topic": "table/list",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "5772",
|
||||
"questionId": "5772",
|
||||
"docId": "1940",
|
||||
"image_path": "data/docvqa_images/q5772_d1940.png",
|
||||
"ucsf_document_id": "pzyw0224",
|
||||
"ucsf_document_page_no": "10",
|
||||
"topic": "form",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "49077",
|
||||
"questionId": "49077",
|
||||
"docId": "14179",
|
||||
"image_path": "data/docvqa_images/q49077_d14179.png",
|
||||
"ucsf_document_id": "nrxb0228",
|
||||
"ucsf_document_page_no": "3",
|
||||
"topic": "layout",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "58519",
|
||||
"questionId": "58519",
|
||||
"docId": "5347",
|
||||
"image_path": "data/docvqa_images/q58519_d5347.png",
|
||||
"ucsf_document_id": "sjbw0217",
|
||||
"ucsf_document_page_no": "11",
|
||||
"topic": "table/list",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "50720",
|
||||
"questionId": "50720",
|
||||
"docId": "281",
|
||||
"image_path": "data/docvqa_images/q50720_d281.png",
|
||||
"ucsf_document_id": "nrcj0037",
|
||||
"ucsf_document_page_no": "7",
|
||||
"topic": "table/list",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "56785",
|
||||
"questionId": "56785",
|
||||
"docId": "14289",
|
||||
"image_path": "data/docvqa_images/q56785_d14289.png",
|
||||
"ucsf_document_id": "xkbv0228",
|
||||
"ucsf_document_page_no": "1",
|
||||
"topic": "layout",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "59653",
|
||||
"questionId": "59653",
|
||||
"docId": "6579",
|
||||
"image_path": "data/docvqa_images/q59653_d6579.png",
|
||||
"ucsf_document_id": "mzbx0227",
|
||||
"ucsf_document_page_no": "2",
|
||||
"topic": "layout",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "61791",
|
||||
"questionId": "61791",
|
||||
"docId": "8072",
|
||||
"image_path": "data/docvqa_images/q61791_d8072.png",
|
||||
"ucsf_document_id": "hfmf0227",
|
||||
"ucsf_document_page_no": "1",
|
||||
"topic": "layout",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "37229",
|
||||
"questionId": "37229",
|
||||
"docId": "10742",
|
||||
"image_path": "data/docvqa_images/q37229_d10742.png",
|
||||
"ucsf_document_id": "nkcd0227",
|
||||
"ucsf_document_page_no": "2",
|
||||
"topic": "free_text",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "60407",
|
||||
"questionId": "60407",
|
||||
"docId": "7135",
|
||||
"image_path": "data/docvqa_images/q60407_d7135.png",
|
||||
"ucsf_document_id": "gkpk0226",
|
||||
"ucsf_document_page_no": "1",
|
||||
"topic": "free_text",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "64420",
|
||||
"questionId": "64420",
|
||||
"docId": "10230",
|
||||
"image_path": "data/docvqa_images/q64420_d10230.png",
|
||||
"ucsf_document_id": "jnjm0223",
|
||||
"ucsf_document_page_no": "107",
|
||||
"topic": "table/list",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "47365",
|
||||
"questionId": "47365",
|
||||
"docId": "13813",
|
||||
"image_path": "data/docvqa_images/q47365_d13813.png",
|
||||
"ucsf_document_id": "nxym0227",
|
||||
"ucsf_document_page_no": "28",
|
||||
"topic": "table/list",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "47458",
|
||||
"questionId": "47458",
|
||||
"docId": "13639",
|
||||
"image_path": "data/docvqa_images/q47458_d13639.png",
|
||||
"ucsf_document_id": "skdv0228",
|
||||
"ucsf_document_page_no": "5",
|
||||
"topic": "layout",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "7621",
|
||||
"questionId": "7621",
|
||||
"docId": "2668",
|
||||
"image_path": "data/docvqa_images/q7621_d2668.png",
|
||||
"ucsf_document_id": "flxn0020",
|
||||
"ucsf_document_page_no": "1",
|
||||
"topic": "form",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "53575",
|
||||
"questionId": "53575",
|
||||
"docId": "2766",
|
||||
"image_path": "data/docvqa_images/q53575_d2766.png",
|
||||
"ucsf_document_id": "hsfn0020",
|
||||
"ucsf_document_page_no": "2",
|
||||
"topic": "free_text|table/list",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "60913",
|
||||
"questionId": "60913",
|
||||
"docId": "7349",
|
||||
"image_path": "data/docvqa_images/q60913_d7349.png",
|
||||
"ucsf_document_id": "jzhd0227",
|
||||
"ucsf_document_page_no": "61",
|
||||
"topic": "free_text",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "60454",
|
||||
"questionId": "60454",
|
||||
"docId": "7163",
|
||||
"image_path": "data/docvqa_images/q60454_d7163.png",
|
||||
"ucsf_document_id": "jgyk0226",
|
||||
"ucsf_document_page_no": "1",
|
||||
"topic": "free_text",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "57978",
|
||||
"questionId": "57978",
|
||||
"docId": "4920",
|
||||
"image_path": "data/docvqa_images/q57978_d4920.png",
|
||||
"ucsf_document_id": "lkvw0217",
|
||||
"ucsf_document_page_no": "2",
|
||||
"topic": "table/list",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "64547",
|
||||
"questionId": "64547",
|
||||
"docId": "10361",
|
||||
"image_path": "data/docvqa_images/q64547_d10361.png",
|
||||
"ucsf_document_id": "lpdl0226",
|
||||
"ucsf_document_page_no": "32",
|
||||
"topic": "table/list",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "59481",
|
||||
"questionId": "59481",
|
||||
"docId": "6243",
|
||||
"image_path": "data/docvqa_images/q59481_d6243.png",
|
||||
"ucsf_document_id": "psgv0228",
|
||||
"ucsf_document_page_no": "5",
|
||||
"topic": "free_text",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "61472",
|
||||
"questionId": "61472",
|
||||
"docId": "7757",
|
||||
"image_path": "data/docvqa_images/q61472_d7757.png",
|
||||
"ucsf_document_id": "ymkp0227",
|
||||
"ucsf_document_page_no": "13",
|
||||
"topic": "handwritten|table/list",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "5673",
|
||||
"questionId": "5673",
|
||||
"docId": "1908",
|
||||
"image_path": "data/docvqa_images/q5673_d1908.png",
|
||||
"ucsf_document_id": "lldj0224",
|
||||
"ucsf_document_page_no": "2",
|
||||
"topic": "form",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "49109",
|
||||
"questionId": "49109",
|
||||
"docId": "13644",
|
||||
"image_path": "data/docvqa_images/q49109_d13644.png",
|
||||
"ucsf_document_id": "mzdv0228",
|
||||
"ucsf_document_page_no": "1",
|
||||
"topic": "table/list",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "46123",
|
||||
"questionId": "46123",
|
||||
"docId": "13503",
|
||||
"image_path": "data/docvqa_images/q46123_d13503.png",
|
||||
"ucsf_document_id": "xmww0217",
|
||||
"ucsf_document_page_no": "17",
|
||||
"topic": "table/list",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "48158",
|
||||
"questionId": "48158",
|
||||
"docId": "13976",
|
||||
"image_path": "data/docvqa_images/q48158_d13976.png",
|
||||
"ucsf_document_id": "zqhm0227",
|
||||
"ucsf_document_page_no": "1",
|
||||
"topic": "free_text",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "1955",
|
||||
"questionId": "1955",
|
||||
"docId": "892",
|
||||
"image_path": "data/docvqa_images/q1955_d892.png",
|
||||
"ucsf_document_id": "jsbn0226",
|
||||
"ucsf_document_page_no": "2",
|
||||
"topic": "table/list",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "8127",
|
||||
"questionId": "8127",
|
||||
"docId": "2754",
|
||||
"image_path": "data/docvqa_images/q8127_d2754.png",
|
||||
"ucsf_document_id": "xtvn0020",
|
||||
"ucsf_document_page_no": "2",
|
||||
"topic": "form",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "57431",
|
||||
"questionId": "57431",
|
||||
"docId": "4779",
|
||||
"image_path": "data/docvqa_images/q57431_d4779.png",
|
||||
"ucsf_document_id": "rnbx0223",
|
||||
"ucsf_document_page_no": "208",
|
||||
"topic": "Image/Photo",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "64306",
|
||||
"questionId": "64306",
|
||||
"docId": "10149",
|
||||
"image_path": "data/docvqa_images/q64306_d10149.png",
|
||||
"ucsf_document_id": "lpjm0223",
|
||||
"ucsf_document_page_no": "23",
|
||||
"topic": "table/list",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "64887",
|
||||
"questionId": "64887",
|
||||
"docId": "9754",
|
||||
"image_path": "data/docvqa_images/q64887_d9754.png",
|
||||
"ucsf_document_id": "szpg0227",
|
||||
"ucsf_document_page_no": "9",
|
||||
"topic": "table/list",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "58680",
|
||||
"questionId": "58680",
|
||||
"docId": "5545",
|
||||
"image_path": "data/docvqa_images/q58680_d5545.png",
|
||||
"ucsf_document_id": "hhwh0078",
|
||||
"ucsf_document_page_no": "1",
|
||||
"topic": "table/list",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "5287",
|
||||
"questionId": "5287",
|
||||
"docId": "1785",
|
||||
"image_path": "data/docvqa_images/q5287_d1785.png",
|
||||
"ucsf_document_id": "mtnh0227",
|
||||
"ucsf_document_page_no": "10",
|
||||
"topic": "form",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "55471",
|
||||
"questionId": "55471",
|
||||
"docId": "4340",
|
||||
"image_path": "data/docvqa_images/q55471_d4340.png",
|
||||
"ucsf_document_id": "fsgj0223",
|
||||
"ucsf_document_page_no": "96",
|
||||
"topic": "free_text",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "53095",
|
||||
"questionId": "53095",
|
||||
"docId": "296",
|
||||
"image_path": "data/docvqa_images/q53095_d296.png",
|
||||
"ucsf_document_id": "qhxj0037",
|
||||
"ucsf_document_page_no": "3",
|
||||
"topic": "layout",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "53726",
|
||||
"questionId": "53726",
|
||||
"docId": "2008",
|
||||
"image_path": "data/docvqa_images/q53726_d2008.png",
|
||||
"ucsf_document_id": "hhnf0094",
|
||||
"ucsf_document_page_no": "5",
|
||||
"topic": "layout",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "57321",
|
||||
"questionId": "57321",
|
||||
"docId": "4722",
|
||||
"image_path": "data/docvqa_images/q57321_d4722.png",
|
||||
"ucsf_document_id": "xybx0223",
|
||||
"ucsf_document_page_no": "32",
|
||||
"topic": "table/list",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "26659",
|
||||
"questionId": "26659",
|
||||
"docId": "7470",
|
||||
"image_path": "data/docvqa_images/q26659_d7470.png",
|
||||
"ucsf_document_id": "lhmg0227",
|
||||
"ucsf_document_page_no": "1",
|
||||
"topic": "layout",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "38920",
|
||||
"questionId": "38920",
|
||||
"docId": "11157",
|
||||
"image_path": "data/docvqa_images/q38920_d11157.png",
|
||||
"ucsf_document_id": "klnf0227",
|
||||
"ucsf_document_page_no": "1",
|
||||
"topic": "table/list|layout",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "50837",
|
||||
"questionId": "50837",
|
||||
"docId": "14742",
|
||||
"image_path": "data/docvqa_images/q50837_d14742.png",
|
||||
"ucsf_document_id": "ysmc0228",
|
||||
"ucsf_document_page_no": "4",
|
||||
"topic": "table/list",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "59615",
|
||||
"questionId": "59615",
|
||||
"docId": "6569",
|
||||
"image_path": "data/docvqa_images/q59615_d6569.png",
|
||||
"ucsf_document_id": "hnnp0227",
|
||||
"ucsf_document_page_no": "45",
|
||||
"topic": "handwritten|table/list|layout",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
},
|
||||
{
|
||||
"id": "58687",
|
||||
"questionId": "58687",
|
||||
"docId": "5545",
|
||||
"image_path": "data/docvqa_images/q58687_d5545.png",
|
||||
"ucsf_document_id": "hhwh0078",
|
||||
"ucsf_document_page_no": "1",
|
||||
"topic": "table/list",
|
||||
"source_dataset": "lmms-lab/DocVQA",
|
||||
"source_config": "DocVQA",
|
||||
"source_split": "validation",
|
||||
"sample_seed": "full_validation_5349"
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,34 @@
|
||||
{
|
||||
"benchmark": "LiveMathematicianBench",
|
||||
"manifest_type": "id_split",
|
||||
"source_repo": "LiveMathematicianBench/LiveMathematicianBench",
|
||||
"source_repo_type": "dataset",
|
||||
"source_url": "https://huggingface.co/datasets/LiveMathematicianBench/LiveMathematicianBench",
|
||||
"source_revision": "b72450f6ce96c26158d64d945a5d31ef7727be41",
|
||||
"source_files": [
|
||||
"data/202511/qa_202511_final.json",
|
||||
"data/202512/qa_202512_final.json",
|
||||
"data/202601/qa_202601_final.json",
|
||||
"data/202602/qa_202602_final.json"
|
||||
],
|
||||
"split_mode": "ratio",
|
||||
"split_ratio": "2:1:7",
|
||||
"split_seed": 42,
|
||||
"counts": {
|
||||
"train": 35,
|
||||
"val": 18,
|
||||
"test": 124
|
||||
},
|
||||
"item_fields": [
|
||||
"id",
|
||||
"month",
|
||||
"no",
|
||||
"paper_link",
|
||||
"source_file"
|
||||
],
|
||||
"id_format": "<month>:<no>",
|
||||
"notes": [
|
||||
"This is an ID split manifest, not the full LiveMathematicianBench payload.",
|
||||
"Materialize full split items from the official LiveMathematicianBench raw qa_*_final.json files before evaluation."
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,870 @@
|
||||
[
|
||||
{
|
||||
"id": "202602:12",
|
||||
"month": "202602",
|
||||
"no": 12,
|
||||
"paper_link": "http://arxiv.org/abs/2602.07171v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:3",
|
||||
"month": "202601",
|
||||
"no": 3,
|
||||
"paper_link": "http://arxiv.org/abs/2601.01447v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202511:4",
|
||||
"month": "202511",
|
||||
"no": 4,
|
||||
"paper_link": "http://arxiv.org/abs/2511.23123v1",
|
||||
"source_file": "data/202511/qa_202511_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:20",
|
||||
"month": "202601",
|
||||
"no": 20,
|
||||
"paper_link": "http://arxiv.org/abs/2601.13212v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:42",
|
||||
"month": "202601",
|
||||
"no": 42,
|
||||
"paper_link": "http://arxiv.org/abs/2601.09348v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:38",
|
||||
"month": "202512",
|
||||
"no": 38,
|
||||
"paper_link": "http://arxiv.org/abs/2512.19831v2",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:4",
|
||||
"month": "202512",
|
||||
"no": 4,
|
||||
"paper_link": "http://arxiv.org/abs/2512.03141v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:4",
|
||||
"month": "202602",
|
||||
"no": 4,
|
||||
"paper_link": "http://arxiv.org/abs/2602.14368v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202511:15",
|
||||
"month": "202511",
|
||||
"no": 15,
|
||||
"paper_link": "http://arxiv.org/abs/2511.17325v1",
|
||||
"source_file": "data/202511/qa_202511_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:32",
|
||||
"month": "202602",
|
||||
"no": 32,
|
||||
"paper_link": "http://arxiv.org/abs/2602.14817v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:51",
|
||||
"month": "202512",
|
||||
"no": 51,
|
||||
"paper_link": "http://arxiv.org/abs/2512.14581v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:26",
|
||||
"month": "202512",
|
||||
"no": 26,
|
||||
"paper_link": "http://arxiv.org/abs/2512.19586v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:13",
|
||||
"month": "202601",
|
||||
"no": 13,
|
||||
"paper_link": "http://arxiv.org/abs/2601.10017v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:1",
|
||||
"month": "202602",
|
||||
"no": 1,
|
||||
"paper_link": "http://arxiv.org/abs/2602.23137v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202511:18",
|
||||
"month": "202511",
|
||||
"no": 18,
|
||||
"paper_link": "http://arxiv.org/abs/2511.10795v1",
|
||||
"source_file": "data/202511/qa_202511_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:5",
|
||||
"month": "202512",
|
||||
"no": 5,
|
||||
"paper_link": "http://arxiv.org/abs/2512.00348v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202511:19",
|
||||
"month": "202511",
|
||||
"no": 19,
|
||||
"paper_link": "http://arxiv.org/abs/2511.06951v1",
|
||||
"source_file": "data/202511/qa_202511_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:40",
|
||||
"month": "202602",
|
||||
"no": 40,
|
||||
"paper_link": "http://arxiv.org/abs/2602.20462v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:29",
|
||||
"month": "202602",
|
||||
"no": 29,
|
||||
"paper_link": "http://arxiv.org/abs/2602.10676v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:35",
|
||||
"month": "202512",
|
||||
"no": 35,
|
||||
"paper_link": "http://arxiv.org/abs/2512.08840v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:48",
|
||||
"month": "202512",
|
||||
"no": 48,
|
||||
"paper_link": "http://arxiv.org/abs/2512.03482v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:52",
|
||||
"month": "202512",
|
||||
"no": 52,
|
||||
"paper_link": "http://arxiv.org/abs/2512.11246v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:44",
|
||||
"month": "202512",
|
||||
"no": 44,
|
||||
"paper_link": "http://arxiv.org/abs/2512.10385v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202511:28",
|
||||
"month": "202511",
|
||||
"no": 28,
|
||||
"paper_link": "http://arxiv.org/abs/2511.03812v1",
|
||||
"source_file": "data/202511/qa_202511_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:43",
|
||||
"month": "202601",
|
||||
"no": 43,
|
||||
"paper_link": "http://arxiv.org/abs/2601.22555v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:9",
|
||||
"month": "202602",
|
||||
"no": 9,
|
||||
"paper_link": "http://arxiv.org/abs/2602.19882v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:23",
|
||||
"month": "202512",
|
||||
"no": 23,
|
||||
"paper_link": "http://arxiv.org/abs/2512.09180v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:21",
|
||||
"month": "202602",
|
||||
"no": 21,
|
||||
"paper_link": "http://arxiv.org/abs/2602.10509v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202511:5",
|
||||
"month": "202511",
|
||||
"no": 5,
|
||||
"paper_link": "http://arxiv.org/abs/2511.20164v1",
|
||||
"source_file": "data/202511/qa_202511_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:35",
|
||||
"month": "202601",
|
||||
"no": 35,
|
||||
"paper_link": "http://arxiv.org/abs/2601.15606v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:50",
|
||||
"month": "202602",
|
||||
"no": 50,
|
||||
"paper_link": "http://arxiv.org/abs/2602.05652v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:13",
|
||||
"month": "202512",
|
||||
"no": 13,
|
||||
"paper_link": "http://arxiv.org/abs/2512.22861v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:49",
|
||||
"month": "202602",
|
||||
"no": 49,
|
||||
"paper_link": "http://arxiv.org/abs/2602.07167v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:18",
|
||||
"month": "202602",
|
||||
"no": 18,
|
||||
"paper_link": "http://arxiv.org/abs/2602.20124v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:15",
|
||||
"month": "202601",
|
||||
"no": 15,
|
||||
"paper_link": "http://arxiv.org/abs/2601.05327v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:21",
|
||||
"month": "202601",
|
||||
"no": 21,
|
||||
"paper_link": "http://arxiv.org/abs/2601.04994v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:32",
|
||||
"month": "202601",
|
||||
"no": 32,
|
||||
"paper_link": "http://arxiv.org/abs/2601.09183v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:34",
|
||||
"month": "202602",
|
||||
"no": 34,
|
||||
"paper_link": "http://arxiv.org/abs/2602.21118v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:20",
|
||||
"month": "202602",
|
||||
"no": 20,
|
||||
"paper_link": "http://arxiv.org/abs/2602.16506v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:5",
|
||||
"month": "202602",
|
||||
"no": 5,
|
||||
"paper_link": "http://arxiv.org/abs/2602.09806v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:40",
|
||||
"month": "202512",
|
||||
"no": 40,
|
||||
"paper_link": "http://arxiv.org/abs/2512.16535v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202511:22",
|
||||
"month": "202511",
|
||||
"no": 22,
|
||||
"paper_link": "http://arxiv.org/abs/2511.07607v2",
|
||||
"source_file": "data/202511/qa_202511_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:36",
|
||||
"month": "202601",
|
||||
"no": 36,
|
||||
"paper_link": "http://arxiv.org/abs/2601.12457v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:49",
|
||||
"month": "202512",
|
||||
"no": 49,
|
||||
"paper_link": "http://arxiv.org/abs/2512.21565v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202511:10",
|
||||
"month": "202511",
|
||||
"no": 10,
|
||||
"paper_link": "http://arxiv.org/abs/2511.06484v1",
|
||||
"source_file": "data/202511/qa_202511_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:2",
|
||||
"month": "202601",
|
||||
"no": 2,
|
||||
"paper_link": "http://arxiv.org/abs/2601.07068v4",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:19",
|
||||
"month": "202602",
|
||||
"no": 19,
|
||||
"paper_link": "http://arxiv.org/abs/2602.18179v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:9",
|
||||
"month": "202601",
|
||||
"no": 9,
|
||||
"paper_link": "http://arxiv.org/abs/2601.17765v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:6",
|
||||
"month": "202512",
|
||||
"no": 6,
|
||||
"paper_link": "http://arxiv.org/abs/2512.23079v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:5",
|
||||
"month": "202601",
|
||||
"no": 5,
|
||||
"paper_link": "http://arxiv.org/abs/2601.20344v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:14",
|
||||
"month": "202602",
|
||||
"no": 14,
|
||||
"paper_link": "http://arxiv.org/abs/2602.09177v2",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:17",
|
||||
"month": "202512",
|
||||
"no": 17,
|
||||
"paper_link": "http://arxiv.org/abs/2512.11657v2",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:19",
|
||||
"month": "202512",
|
||||
"no": 19,
|
||||
"paper_link": "http://arxiv.org/abs/2512.16655v2",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:33",
|
||||
"month": "202602",
|
||||
"no": 33,
|
||||
"paper_link": "http://arxiv.org/abs/2602.13734v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:18",
|
||||
"month": "202512",
|
||||
"no": 18,
|
||||
"paper_link": "http://arxiv.org/abs/2512.22960v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:26",
|
||||
"month": "202601",
|
||||
"no": 26,
|
||||
"paper_link": "http://arxiv.org/abs/2601.06814v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:1",
|
||||
"month": "202601",
|
||||
"no": 1,
|
||||
"paper_link": "http://arxiv.org/abs/2601.18276v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:30",
|
||||
"month": "202512",
|
||||
"no": 30,
|
||||
"paper_link": "http://arxiv.org/abs/2512.07260v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:44",
|
||||
"month": "202602",
|
||||
"no": 44,
|
||||
"paper_link": "http://arxiv.org/abs/2602.01138v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:20",
|
||||
"month": "202512",
|
||||
"no": 20,
|
||||
"paper_link": "http://arxiv.org/abs/2512.14575v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202511:13",
|
||||
"month": "202511",
|
||||
"no": 13,
|
||||
"paper_link": "http://arxiv.org/abs/2511.16910v1",
|
||||
"source_file": "data/202511/qa_202511_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:30",
|
||||
"month": "202601",
|
||||
"no": 30,
|
||||
"paper_link": "http://arxiv.org/abs/2601.12140v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:40",
|
||||
"month": "202601",
|
||||
"no": 40,
|
||||
"paper_link": "http://arxiv.org/abs/2601.05146v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:29",
|
||||
"month": "202601",
|
||||
"no": 29,
|
||||
"paper_link": "http://arxiv.org/abs/2601.12846v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202511:11",
|
||||
"month": "202511",
|
||||
"no": 11,
|
||||
"paper_link": "http://arxiv.org/abs/2511.17548v1",
|
||||
"source_file": "data/202511/qa_202511_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:9",
|
||||
"month": "202512",
|
||||
"no": 9,
|
||||
"paper_link": "http://arxiv.org/abs/2512.08817v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:18",
|
||||
"month": "202601",
|
||||
"no": 18,
|
||||
"paper_link": "http://arxiv.org/abs/2601.01797v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:1",
|
||||
"month": "202512",
|
||||
"no": 1,
|
||||
"paper_link": "http://arxiv.org/abs/2512.20055v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:4",
|
||||
"month": "202601",
|
||||
"no": 4,
|
||||
"paper_link": "http://arxiv.org/abs/2601.21223v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202511:6",
|
||||
"month": "202511",
|
||||
"no": 6,
|
||||
"paper_link": "http://arxiv.org/abs/2511.14959v1",
|
||||
"source_file": "data/202511/qa_202511_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:38",
|
||||
"month": "202602",
|
||||
"no": 38,
|
||||
"paper_link": "http://arxiv.org/abs/2602.08398v2",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:10",
|
||||
"month": "202601",
|
||||
"no": 10,
|
||||
"paper_link": "http://arxiv.org/abs/2601.15524v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:11",
|
||||
"month": "202602",
|
||||
"no": 11,
|
||||
"paper_link": "http://arxiv.org/abs/2602.11045v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:45",
|
||||
"month": "202512",
|
||||
"no": 45,
|
||||
"paper_link": "http://arxiv.org/abs/2512.08395v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:12",
|
||||
"month": "202601",
|
||||
"no": 12,
|
||||
"paper_link": "http://arxiv.org/abs/2601.11877v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:47",
|
||||
"month": "202512",
|
||||
"no": 47,
|
||||
"paper_link": "http://arxiv.org/abs/2512.09683v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202511:21",
|
||||
"month": "202511",
|
||||
"no": 21,
|
||||
"paper_link": "http://arxiv.org/abs/2511.21288v1",
|
||||
"source_file": "data/202511/qa_202511_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:16",
|
||||
"month": "202601",
|
||||
"no": 16,
|
||||
"paper_link": "http://arxiv.org/abs/2601.05008v2",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:3",
|
||||
"month": "202512",
|
||||
"no": 3,
|
||||
"paper_link": "http://arxiv.org/abs/2512.13450v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:37",
|
||||
"month": "202601",
|
||||
"no": 37,
|
||||
"paper_link": "http://arxiv.org/abs/2601.09443v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202511:12",
|
||||
"month": "202511",
|
||||
"no": 12,
|
||||
"paper_link": "http://arxiv.org/abs/2511.04978v1",
|
||||
"source_file": "data/202511/qa_202511_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:39",
|
||||
"month": "202512",
|
||||
"no": 39,
|
||||
"paper_link": "http://arxiv.org/abs/2512.19003v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:8",
|
||||
"month": "202601",
|
||||
"no": 8,
|
||||
"paper_link": "http://arxiv.org/abs/2601.19754v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:11",
|
||||
"month": "202601",
|
||||
"no": 11,
|
||||
"paper_link": "http://arxiv.org/abs/2601.13552v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202511:25",
|
||||
"month": "202511",
|
||||
"no": 25,
|
||||
"paper_link": "http://arxiv.org/abs/2511.10548v1",
|
||||
"source_file": "data/202511/qa_202511_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:17",
|
||||
"month": "202601",
|
||||
"no": 17,
|
||||
"paper_link": "http://arxiv.org/abs/2601.02655v2",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:36",
|
||||
"month": "202602",
|
||||
"no": 36,
|
||||
"paper_link": "http://arxiv.org/abs/2602.13001v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:43",
|
||||
"month": "202602",
|
||||
"no": 43,
|
||||
"paper_link": "http://arxiv.org/abs/2602.06897v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:6",
|
||||
"month": "202601",
|
||||
"no": 6,
|
||||
"paper_link": "http://arxiv.org/abs/2601.04747v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:35",
|
||||
"month": "202602",
|
||||
"no": 35,
|
||||
"paper_link": "http://arxiv.org/abs/2602.20938v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:11",
|
||||
"month": "202512",
|
||||
"no": 11,
|
||||
"paper_link": "http://arxiv.org/abs/2512.03294v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:23",
|
||||
"month": "202602",
|
||||
"no": 23,
|
||||
"paper_link": "http://arxiv.org/abs/2602.09201v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:7",
|
||||
"month": "202601",
|
||||
"no": 7,
|
||||
"paper_link": "http://arxiv.org/abs/2601.02859v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:39",
|
||||
"month": "202602",
|
||||
"no": 39,
|
||||
"paper_link": "http://arxiv.org/abs/2602.21659v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:12",
|
||||
"month": "202512",
|
||||
"no": 12,
|
||||
"paper_link": "http://arxiv.org/abs/2512.00690v3",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202511:2",
|
||||
"month": "202511",
|
||||
"no": 2,
|
||||
"paper_link": "http://arxiv.org/abs/2511.19681v1",
|
||||
"source_file": "data/202511/qa_202511_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:43",
|
||||
"month": "202512",
|
||||
"no": 43,
|
||||
"paper_link": "http://arxiv.org/abs/2512.10820v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:24",
|
||||
"month": "202602",
|
||||
"no": 24,
|
||||
"paper_link": "http://arxiv.org/abs/2602.08680v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:34",
|
||||
"month": "202601",
|
||||
"no": 34,
|
||||
"paper_link": "http://arxiv.org/abs/2601.07318v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:28",
|
||||
"month": "202512",
|
||||
"no": 28,
|
||||
"paper_link": "http://arxiv.org/abs/2512.11294v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:27",
|
||||
"month": "202601",
|
||||
"no": 27,
|
||||
"paper_link": "http://arxiv.org/abs/2601.05692v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:42",
|
||||
"month": "202602",
|
||||
"no": 42,
|
||||
"paper_link": "http://arxiv.org/abs/2602.09749v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:22",
|
||||
"month": "202512",
|
||||
"no": 22,
|
||||
"paper_link": "http://arxiv.org/abs/2512.11658v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:17",
|
||||
"month": "202602",
|
||||
"no": 17,
|
||||
"paper_link": "http://arxiv.org/abs/2602.22504v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:48",
|
||||
"month": "202602",
|
||||
"no": 48,
|
||||
"paper_link": "http://arxiv.org/abs/2602.08760v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:28",
|
||||
"month": "202602",
|
||||
"no": 28,
|
||||
"paper_link": "http://arxiv.org/abs/2602.11595v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:3",
|
||||
"month": "202602",
|
||||
"no": 3,
|
||||
"paper_link": "http://arxiv.org/abs/2602.17369v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:31",
|
||||
"month": "202512",
|
||||
"no": 31,
|
||||
"paper_link": "http://arxiv.org/abs/2512.23668v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:27",
|
||||
"month": "202512",
|
||||
"no": 27,
|
||||
"paper_link": "http://arxiv.org/abs/2512.16505v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202511:24",
|
||||
"month": "202511",
|
||||
"no": 24,
|
||||
"paper_link": "http://arxiv.org/abs/2511.12549v2",
|
||||
"source_file": "data/202511/qa_202511_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202511:8",
|
||||
"month": "202511",
|
||||
"no": 8,
|
||||
"paper_link": "http://arxiv.org/abs/2511.12657v1",
|
||||
"source_file": "data/202511/qa_202511_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202511:9",
|
||||
"month": "202511",
|
||||
"no": 9,
|
||||
"paper_link": "http://arxiv.org/abs/2511.09015v1",
|
||||
"source_file": "data/202511/qa_202511_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:28",
|
||||
"month": "202601",
|
||||
"no": 28,
|
||||
"paper_link": "http://arxiv.org/abs/2601.14825v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:25",
|
||||
"month": "202602",
|
||||
"no": 25,
|
||||
"paper_link": "http://arxiv.org/abs/2602.16048v3",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202511:23",
|
||||
"month": "202511",
|
||||
"no": 23,
|
||||
"paper_link": "http://arxiv.org/abs/2511.06595v1",
|
||||
"source_file": "data/202511/qa_202511_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:13",
|
||||
"month": "202602",
|
||||
"no": 13,
|
||||
"paper_link": "http://arxiv.org/abs/2602.12261v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202511:27",
|
||||
"month": "202511",
|
||||
"no": 27,
|
||||
"paper_link": "http://arxiv.org/abs/2511.04407v1",
|
||||
"source_file": "data/202511/qa_202511_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:7",
|
||||
"month": "202512",
|
||||
"no": 7,
|
||||
"paper_link": "http://arxiv.org/abs/2512.09490v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:29",
|
||||
"month": "202512",
|
||||
"no": 29,
|
||||
"paper_link": "http://arxiv.org/abs/2512.08562v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:34",
|
||||
"month": "202512",
|
||||
"no": 34,
|
||||
"paper_link": "http://arxiv.org/abs/2512.09598v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:42",
|
||||
"month": "202512",
|
||||
"no": 42,
|
||||
"paper_link": "http://arxiv.org/abs/2512.10845v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202511:7",
|
||||
"month": "202511",
|
||||
"no": 7,
|
||||
"paper_link": "http://arxiv.org/abs/2511.13976v1",
|
||||
"source_file": "data/202511/qa_202511_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202511:29",
|
||||
"month": "202511",
|
||||
"no": 29,
|
||||
"paper_link": "http://arxiv.org/abs/2511.03722v1",
|
||||
"source_file": "data/202511/qa_202511_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:37",
|
||||
"month": "202602",
|
||||
"no": 37,
|
||||
"paper_link": "http://arxiv.org/abs/2602.08644v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,247 @@
|
||||
[
|
||||
{
|
||||
"id": "202602:22",
|
||||
"month": "202602",
|
||||
"no": 22,
|
||||
"paper_link": "http://arxiv.org/abs/2602.10700v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:8",
|
||||
"month": "202512",
|
||||
"no": 8,
|
||||
"paper_link": "http://arxiv.org/abs/2512.08863v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202511:16",
|
||||
"month": "202511",
|
||||
"no": 16,
|
||||
"paper_link": "http://arxiv.org/abs/2511.15668v1",
|
||||
"source_file": "data/202511/qa_202511_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:44",
|
||||
"month": "202601",
|
||||
"no": 44,
|
||||
"paper_link": "http://arxiv.org/abs/2601.21267v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202511:14",
|
||||
"month": "202511",
|
||||
"no": 14,
|
||||
"paper_link": "http://arxiv.org/abs/2511.13447v1",
|
||||
"source_file": "data/202511/qa_202511_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:30",
|
||||
"month": "202602",
|
||||
"no": 30,
|
||||
"paper_link": "http://arxiv.org/abs/2602.16692v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:2",
|
||||
"month": "202602",
|
||||
"no": 2,
|
||||
"paper_link": "http://arxiv.org/abs/2602.22933v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:41",
|
||||
"month": "202601",
|
||||
"no": 41,
|
||||
"paper_link": "http://arxiv.org/abs/2601.01164v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:23",
|
||||
"month": "202601",
|
||||
"no": 23,
|
||||
"paper_link": "http://arxiv.org/abs/2601.02528v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202511:20",
|
||||
"month": "202511",
|
||||
"no": 20,
|
||||
"paper_link": "http://arxiv.org/abs/2511.02963v1",
|
||||
"source_file": "data/202511/qa_202511_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:22",
|
||||
"month": "202601",
|
||||
"no": 22,
|
||||
"paper_link": "http://arxiv.org/abs/2601.03984v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:14",
|
||||
"month": "202512",
|
||||
"no": 14,
|
||||
"paper_link": "http://arxiv.org/abs/2512.22459v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202511:26",
|
||||
"month": "202511",
|
||||
"no": 26,
|
||||
"paper_link": "http://arxiv.org/abs/2511.07817v1",
|
||||
"source_file": "data/202511/qa_202511_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202511:3",
|
||||
"month": "202511",
|
||||
"no": 3,
|
||||
"paper_link": "http://arxiv.org/abs/2511.11409v1",
|
||||
"source_file": "data/202511/qa_202511_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:33",
|
||||
"month": "202601",
|
||||
"no": 33,
|
||||
"paper_link": "http://arxiv.org/abs/2601.07747v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:7",
|
||||
"month": "202602",
|
||||
"no": 7,
|
||||
"paper_link": "http://arxiv.org/abs/2602.22912v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:27",
|
||||
"month": "202602",
|
||||
"no": 27,
|
||||
"paper_link": "http://arxiv.org/abs/2602.13968v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:31",
|
||||
"month": "202602",
|
||||
"no": 31,
|
||||
"paper_link": "http://arxiv.org/abs/2602.15528v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:41",
|
||||
"month": "202602",
|
||||
"no": 41,
|
||||
"paper_link": "http://arxiv.org/abs/2602.10707v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:25",
|
||||
"month": "202512",
|
||||
"no": 25,
|
||||
"paper_link": "http://arxiv.org/abs/2512.04531v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:10",
|
||||
"month": "202602",
|
||||
"no": 10,
|
||||
"paper_link": "http://arxiv.org/abs/2602.17863v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:16",
|
||||
"month": "202602",
|
||||
"no": 16,
|
||||
"paper_link": "http://arxiv.org/abs/2602.02723v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:16",
|
||||
"month": "202512",
|
||||
"no": 16,
|
||||
"paper_link": "http://arxiv.org/abs/2512.11601v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:2",
|
||||
"month": "202512",
|
||||
"no": 2,
|
||||
"paper_link": "http://arxiv.org/abs/2512.16120v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:24",
|
||||
"month": "202512",
|
||||
"no": 24,
|
||||
"paper_link": "http://arxiv.org/abs/2512.08391v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:32",
|
||||
"month": "202512",
|
||||
"no": 32,
|
||||
"paper_link": "http://arxiv.org/abs/2512.23224v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:47",
|
||||
"month": "202602",
|
||||
"no": 47,
|
||||
"paper_link": "http://arxiv.org/abs/2602.10391v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:46",
|
||||
"month": "202602",
|
||||
"no": 46,
|
||||
"paper_link": "http://arxiv.org/abs/2602.13727v2",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:21",
|
||||
"month": "202512",
|
||||
"no": 21,
|
||||
"paper_link": "http://arxiv.org/abs/2512.12835v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:33",
|
||||
"month": "202512",
|
||||
"no": 33,
|
||||
"paper_link": "http://arxiv.org/abs/2512.19500v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:45",
|
||||
"month": "202602",
|
||||
"no": 45,
|
||||
"paper_link": "http://arxiv.org/abs/2602.23912v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:26",
|
||||
"month": "202602",
|
||||
"no": 26,
|
||||
"paper_link": "http://arxiv.org/abs/2602.14658v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:41",
|
||||
"month": "202512",
|
||||
"no": 41,
|
||||
"paper_link": "http://arxiv.org/abs/2512.15177v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:38",
|
||||
"month": "202601",
|
||||
"no": 38,
|
||||
"paper_link": "http://arxiv.org/abs/2601.07817v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:14",
|
||||
"month": "202601",
|
||||
"no": 14,
|
||||
"paper_link": "http://arxiv.org/abs/2601.08704v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,128 @@
|
||||
[
|
||||
{
|
||||
"id": "202602:8",
|
||||
"month": "202602",
|
||||
"no": 8,
|
||||
"paper_link": "http://arxiv.org/abs/2602.19529v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:50",
|
||||
"month": "202512",
|
||||
"no": 50,
|
||||
"paper_link": "http://arxiv.org/abs/2512.15277v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:36",
|
||||
"month": "202512",
|
||||
"no": 36,
|
||||
"paper_link": "http://arxiv.org/abs/2512.06696v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202511:1",
|
||||
"month": "202511",
|
||||
"no": 1,
|
||||
"paper_link": "http://arxiv.org/abs/2511.04651v1",
|
||||
"source_file": "data/202511/qa_202511_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:31",
|
||||
"month": "202601",
|
||||
"no": 31,
|
||||
"paper_link": "http://arxiv.org/abs/2601.10298v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202511:17",
|
||||
"month": "202511",
|
||||
"no": 17,
|
||||
"paper_link": "http://arxiv.org/abs/2511.13215v1",
|
||||
"source_file": "data/202511/qa_202511_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:37",
|
||||
"month": "202512",
|
||||
"no": 37,
|
||||
"paper_link": "http://arxiv.org/abs/2512.20498v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:39",
|
||||
"month": "202601",
|
||||
"no": 39,
|
||||
"paper_link": "http://arxiv.org/abs/2601.06601v2",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:25",
|
||||
"month": "202601",
|
||||
"no": 25,
|
||||
"paper_link": "http://arxiv.org/abs/2601.10996v3",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:24",
|
||||
"month": "202601",
|
||||
"no": 24,
|
||||
"paper_link": "http://arxiv.org/abs/2601.12250v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:45",
|
||||
"month": "202601",
|
||||
"no": 45,
|
||||
"paper_link": "http://arxiv.org/abs/2601.12113v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:19",
|
||||
"month": "202601",
|
||||
"no": 19,
|
||||
"paper_link": "http://arxiv.org/abs/2601.00779v1",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:10",
|
||||
"month": "202512",
|
||||
"no": 10,
|
||||
"paper_link": "http://arxiv.org/abs/2512.07073v2",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202601:46",
|
||||
"month": "202601",
|
||||
"no": 46,
|
||||
"paper_link": "http://arxiv.org/abs/2601.07793v2",
|
||||
"source_file": "data/202601/qa_202601_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:15",
|
||||
"month": "202512",
|
||||
"no": 15,
|
||||
"paper_link": "http://arxiv.org/abs/2512.16165v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:15",
|
||||
"month": "202602",
|
||||
"no": 15,
|
||||
"paper_link": "http://arxiv.org/abs/2602.05303v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202602:6",
|
||||
"month": "202602",
|
||||
"no": 6,
|
||||
"paper_link": "http://arxiv.org/abs/2602.01571v1",
|
||||
"source_file": "data/202602/qa_202602_final.json"
|
||||
},
|
||||
{
|
||||
"id": "202512:46",
|
||||
"month": "202512",
|
||||
"no": 46,
|
||||
"paper_link": "http://arxiv.org/abs/2512.05945v1",
|
||||
"source_file": "data/202512/qa_202512_final.json"
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,27 @@
|
||||
{
|
||||
"benchmark": "OfficeQA",
|
||||
"manifest_type": "id_split",
|
||||
"source_repo": "databricks/officeqa",
|
||||
"source_repo_type": "dataset",
|
||||
"source_url": "https://huggingface.co/datasets/databricks/officeqa",
|
||||
"source_revision": "8ecbf18d3833daf4750a903d14963e4c4c1d4cd8",
|
||||
"source_file": "officeqa_full.csv",
|
||||
"source_split_name": "officeqa_split",
|
||||
"counts": {
|
||||
"train": 50,
|
||||
"val": 24,
|
||||
"test": 172
|
||||
},
|
||||
"item_fields": [
|
||||
"id",
|
||||
"uid",
|
||||
"category",
|
||||
"source_files",
|
||||
"source_docs",
|
||||
"source_split"
|
||||
],
|
||||
"notes": [
|
||||
"This is a split manifest, not the full OfficeQA payload.",
|
||||
"The official OfficeQA CSV is gated on Hugging Face; materialization requires authorized access."
|
||||
]
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,402 @@
|
||||
[
|
||||
{
|
||||
"id": "UID0002",
|
||||
"uid": "UID0002",
|
||||
"category": "easy",
|
||||
"source_files": "treasury_bulletin_1944_01.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/january-1944-6565?page=18",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0007",
|
||||
"uid": "UID0007",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_1950_02.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/february-1950-6637?page=15",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0014",
|
||||
"uid": "UID0014",
|
||||
"category": "easy",
|
||||
"source_files": "treasury_bulletin_1942_07.txt\r\ntreasury_bulletin_2001_12.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/july-1942-6547?page=76\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/december-2001-7108?page=17&deep=true",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0017",
|
||||
"uid": "UID0017",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_1982_08.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/august-1982-7028?page=13",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0018",
|
||||
"uid": "UID0018",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_1985_03.txt\r\ntreasury_bulletin_1986_03.txt\r\ntreasury_bulletin_1987_03.txt\r\ntreasury_bulletin_1988_03.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/march-1985-7040?page=22\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/march-1986-7045?page=26\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/march-1987-7049?page=24\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/march-1988-7052?page=36",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0019",
|
||||
"uid": "UID0019",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_2016_09.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/september-2016-533966?page=54\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/september-2016-533966?page=58",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0026",
|
||||
"uid": "UID0026",
|
||||
"category": "easy",
|
||||
"source_files": "treasury_bulletin_1963_01.txt\r\ntreasury_bulletin_1962_01.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/january-1963-6793?page=88\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/january-1962-6781?page=82&deep=true",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0028",
|
||||
"uid": "UID0028",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_1970_06.txt\r\ntreasury_bulletin_1964_12.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/june-1970-6882?page=89&deep=true\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/december-1964-6816?page=25&deep=true",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0030",
|
||||
"uid": "UID0030",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_1990_09.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/september-1990-7062?page=19&deep=true",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0031",
|
||||
"uid": "UID0031",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_1992_03.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/march-1992-7068?page=158&deep=true",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0033",
|
||||
"uid": "UID0033",
|
||||
"category": "easy",
|
||||
"source_files": "treasury_bulletin_1977_04.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/april-1977-6964?page=9",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0034",
|
||||
"uid": "UID0034",
|
||||
"category": "easy",
|
||||
"source_files": "treasury_bulletin_1992_06.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/june-1992-7069?page=32",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0044",
|
||||
"uid": "UID0044",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_1939_04.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/april-1939-6506?page=61",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0046",
|
||||
"uid": "UID0046",
|
||||
"category": "easy",
|
||||
"source_files": "treasury_bulletin_1988_09.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/september-1988-7054?page=37",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0049",
|
||||
"uid": "UID0049",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_1942_02.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/february-1942-6542?page=19&deep=true",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0056",
|
||||
"uid": "UID0056",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_1991_09.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/september-1991-7066?page=30&deep=true",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0063",
|
||||
"uid": "UID0063",
|
||||
"category": "easy",
|
||||
"source_files": "treasury_bulletin_1990_06.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/june-1990-7061?page=127",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0065",
|
||||
"uid": "UID0065",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_1998_06.txt\r\ntreasury_bulletin_1995_12.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/june-1998-7094?page=7\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/december-1995-7084?page=16",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0073",
|
||||
"uid": "UID0073",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_1982_03.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/march-1982-7023?page=24",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0079",
|
||||
"uid": "UID0079",
|
||||
"category": "easy",
|
||||
"source_files": "treasury_bulletin_2011_12.txt\r\ntreasury_bulletin_2013_12.txt\r\ntreasury_bulletin_2015_12.txt\r\ntreasury_bulletin_2017_12.txt\r\ntreasury_bulletin_2019_12.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/december-2011-7149?page=25\r\n \r\n https://fraser.stlouisfed.org/title/treasury-bulletin-407/december-2013-7155?page=24\r\n \r\n https://fraser.stlouisfed.org/title/treasury-bulletin-407/december-2015-519209?page=23\r\n \r\n https://fraser.stlouisfed.org/title/treasury-bulletin-407/december-2017-575188?page=23\r\n \r\n https://fraser.stlouisfed.org/title/treasury-bulletin-407/december-2019-584842?page=22",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0083",
|
||||
"uid": "UID0083",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_1981_12.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/december-1981-7020?page=24",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0085",
|
||||
"uid": "UID0085",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_2019_12.txt\r\ntreasury_bulletin_2018_12.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/december-2019-584842?page=23\r\n\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/december-2018-581283?page=22",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0087",
|
||||
"uid": "UID0087",
|
||||
"category": "easy",
|
||||
"source_files": "treasury_bulletin_2013_12.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/december-2013-7155?page=17",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0092",
|
||||
"uid": "UID0092",
|
||||
"category": "easy",
|
||||
"source_files": "treasury_bulletin_1987_12.txt\r\ntreasury_bulletin_1992_12.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/december-1987-7051?page=69\r\n\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/december-1992-7071?page=84",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0098",
|
||||
"uid": "UID0098",
|
||||
"category": "easy",
|
||||
"source_files": "treasury_bulletin_2020_12.txt\r\ntreasury_bulletin_2024_12.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/december-2020-598551?page=21\r\n \r\n https://fraser.stlouisfed.org/title/treasury-bulletin-407/december-2024-679984?page=22",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0101",
|
||||
"uid": "UID0101",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_2011_12.txt\r\ntreasury_bulletin_2019_12.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/december-2011-7149?page=25\r\n \r\n https://fraser.stlouisfed.org/title/treasury-bulletin-407/december-2019-584842?page=22",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0110",
|
||||
"uid": "UID0110",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_2020_03.txt\r\ntreasury_bulletin_2016_03.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/march-2020-587316?page=10\r\n \r\n https://fraser.stlouisfed.org/title/treasury-bulletin-407/march-2016-527290?page=9",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0115",
|
||||
"uid": "UID0115",
|
||||
"category": "easy",
|
||||
"source_files": "treasury_bulletin_1980_02.txt\r\ntreasury_bulletin_1981_02.txt\r\ntreasury_bulletin_1982_02.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/february-1980-6998?page=27\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/february-1981-7010?page=38\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/february-1982-7022?page=31",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0122",
|
||||
"uid": "UID0122",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_2001_03.txt\r\ntreasury_bulletin_2002_03.txt\r\ntreasury_bulletin_2003_03.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/march-2001-7105?page=112\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/march-2002-7110?page=115\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/march-2003-7109?page=113",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0123",
|
||||
"uid": "UID0123",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_1941_12.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/december-1941-6540?page=91",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0133",
|
||||
"uid": "UID0133",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_2004_09.txt\r\ntreasury_bulletin_2013_12.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/september-2004-7119?page=63\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/december-2013-7155?page=68",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0141",
|
||||
"uid": "UID0141",
|
||||
"category": "easy",
|
||||
"source_files": "treasury_bulletin_1962_04.txt\r\ntreasury_bulletin_1963_04.txt\r\ntreasury_bulletin_1964_04.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/april-1962-6784?page=75\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/april-1963-6796?page=79\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/april-1964-6808?page=82",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0144",
|
||||
"uid": "UID0144",
|
||||
"category": "easy",
|
||||
"source_files": "treasury_bulletin_1980_11.txt\r\ntreasury_bulletin_1981_11.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/november-1980-7007?page=76\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/november-1981-7019?page=67",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0145",
|
||||
"uid": "UID0145",
|
||||
"category": "easy",
|
||||
"source_files": "treasury_bulletin_1943_01.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/january-1943-6553?page=41",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0150",
|
||||
"uid": "UID0150",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_1972_04.txt\r\ntreasury_bulletin_1973_04.txt\r\ntreasury_bulletin_1974_04.txt\r\ntreasury_bulletin_1975_04.txt\r\ntreasury_bulletin_1976_04.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/april-1972-6905?page=89\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/april-1973-6916?page=91\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/april-1974-6927?page=88\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/april-1975-6940?page=88\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/april-1976-6952?page=104",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0151",
|
||||
"uid": "UID0151",
|
||||
"category": "easy",
|
||||
"source_files": "treasury_bulletin_1953_03.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/march-1953-6674?page=54",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0162",
|
||||
"uid": "UID0162",
|
||||
"category": "easy",
|
||||
"source_files": "treasury_bulletin_2011_06.txt\r\ntreasury_bulletin_2012_06.txt\r\ntreasury_bulletin_2013_06.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/june-2011-7129?page=105\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/june-2012-7151?page=105\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/june-2013-7153?page=104",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0163",
|
||||
"uid": "UID0163",
|
||||
"category": "easy",
|
||||
"source_files": "treasury_bulletin_1981_12.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/december-1981-7020?page=28",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0165",
|
||||
"uid": "UID0165",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_2010_03.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/march-2010-7143?page=49",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0166",
|
||||
"uid": "UID0166",
|
||||
"category": "easy",
|
||||
"source_files": "treasury_bulletin_1943_03.txt\r\ntreasury_bulletin_1944_03.txt\r\ntreasury_bulletin_1945_03.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/march-1943-6555?page=68\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/march-1944-6567?page=83\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/march-1945-6577?page=71",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0169",
|
||||
"uid": "UID0169",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_1982_03.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/march-1982-7023?page=73",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0189",
|
||||
"uid": "UID0189",
|
||||
"category": "easy",
|
||||
"source_files": "treasury_bulletin_1970_08.txt\r\ntreasury_bulletin_1970_09.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/august-1970-6884?page=70\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/september-1970-6885?page=70",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0195",
|
||||
"uid": "UID0195",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_1956_08.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/august-1956-6715?page=59",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0202",
|
||||
"uid": "UID0202",
|
||||
"category": "easy",
|
||||
"source_files": "treasury_bulletin_1939_07.txt\r\ntreasury_bulletin_1939_08.txt\r\ntreasury_bulletin_1939_09.txt\r\ntreasury_bulletin_1939_10.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/july-1939-6509?page=99\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/august-1939-6510?page=107\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/september-1939-6511?page=60\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/october-1939-6520?page=62",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0212",
|
||||
"uid": "UID0212",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_1964_01.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/january-1964-6805?page=99",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0222",
|
||||
"uid": "UID0222",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_2001_06.txt\r\ntreasury_bulletin_2006_06.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/june-2001-7106?page=50\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/june-2006-7126?page=50",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0228",
|
||||
"uid": "UID0228",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_1956_03.txt\r\ntreasury_bulletin_1956_04.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/march-1956-6710?page=22\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/april-1956-6711?page=22",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0229",
|
||||
"uid": "UID0229",
|
||||
"category": "easy",
|
||||
"source_files": "treasury_bulletin_2005_03.txt\r\ntreasury_bulletin_2006_03.txt\r\ntreasury_bulletin_2007_03.txt\r\ntreasury_bulletin_2008_03.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/march-2005-7121?page=109\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/march-2006-7125?page=106\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/march-2007-7130?page=109\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/march-2008-7134?page=107",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0238",
|
||||
"uid": "UID0238",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_1982_03.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/march-1982-7023?page=44",
|
||||
"source_split": "train"
|
||||
},
|
||||
{
|
||||
"id": "UID0241",
|
||||
"uid": "UID0241",
|
||||
"category": "easy",
|
||||
"source_files": "treasury_bulletin_1963_06.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/june-1963-6798?page=13",
|
||||
"source_split": "train"
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,194 @@
|
||||
[
|
||||
{
|
||||
"id": "UID0001",
|
||||
"uid": "UID0001",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_1941_01.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/january-1941-6529?page=15",
|
||||
"source_split": "val"
|
||||
},
|
||||
{
|
||||
"id": "UID0027",
|
||||
"uid": "UID0027",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_1970_06.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/june-1970-6882?page=89&deep=true",
|
||||
"source_split": "val"
|
||||
},
|
||||
{
|
||||
"id": "UID0039",
|
||||
"uid": "UID0039",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_2004_03.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/march-2004-7117?page=20\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/march-2004-7117?page=21&deep=true",
|
||||
"source_split": "val"
|
||||
},
|
||||
{
|
||||
"id": "UID0041",
|
||||
"uid": "UID0041",
|
||||
"category": "easy",
|
||||
"source_files": "treasury_bulletin_1970_10.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/october-1970-6886?page=35",
|
||||
"source_split": "val"
|
||||
},
|
||||
{
|
||||
"id": "UID0052",
|
||||
"uid": "UID0052",
|
||||
"category": "easy",
|
||||
"source_files": "treasury_bulletin_2000_06.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/june-2000-7102?page=56",
|
||||
"source_split": "val"
|
||||
},
|
||||
{
|
||||
"id": "UID0070",
|
||||
"uid": "UID0070",
|
||||
"category": "easy",
|
||||
"source_files": "treasury_bulletin_1939_01.txt\r\ntreasury_bulletin_1939_02.txt\r\ntreasury_bulletin_1939_03.txt\r\ntreasury_bulletin_1939_04.txt\r\ntreasury_bulletin_1939_05.txt\r\ntreasury_bulletin_1939_06.txt\r\ntreasury_bulletin_1939_07.txt\r\ntreasury_bulletin_1939_08.txt\r\ntreasury_bulletin_1939_09.txt\r\ntreasury_bulletin_1939_10.txt\r\ntreasury_bulletin_1939_11.txt\r\ntreasury_bulletin_1939_12.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/january-1939-6518?page=81\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/february-1939-6505?page=111\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/march-1939-6519?page=117\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/april-1939-6506?page=95\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/may-1939-6507?page=109\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/june-1939-6508?page=117\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/july-1939-6509?page=109\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/august-1939-6510?page=117\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/september-1939-6511?page=66&deep=true \r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/october-1939-6520?page=68\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/november-1939-6512?page=70\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/december-1939-6513?page=72",
|
||||
"source_split": "val"
|
||||
},
|
||||
{
|
||||
"id": "UID0072",
|
||||
"uid": "UID0072",
|
||||
"category": "easy",
|
||||
"source_files": "treasury_bulletin_2011_12.txt\r\ntreasury_bulletin_2016_12.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/december-2011-7149?page=58\r\n\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/december-2016-535293?page=57",
|
||||
"source_split": "val"
|
||||
},
|
||||
{
|
||||
"id": "UID0086",
|
||||
"uid": "UID0086",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_2022_12.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/december-2022-627778?page=86",
|
||||
"source_split": "val"
|
||||
},
|
||||
{
|
||||
"id": "UID0091",
|
||||
"uid": "UID0091",
|
||||
"category": "easy",
|
||||
"source_files": "treasury_bulletin_1940_12.txt\r\ntreasury_bulletin_1941_12.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/december-1940-6528?page=21\r\n\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/december-1941-6540?page=64",
|
||||
"source_split": "val"
|
||||
},
|
||||
{
|
||||
"id": "UID0109",
|
||||
"uid": "UID0109",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_2015_12.txt\r\ntreasury_bulletin_2020_12.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/december-2015-519209?page=21\r\n \r\n https://fraser.stlouisfed.org/title/treasury-bulletin-407/december-2020-598551?page=24",
|
||||
"source_split": "val"
|
||||
},
|
||||
{
|
||||
"id": "UID0142",
|
||||
"uid": "UID0142",
|
||||
"category": "easy",
|
||||
"source_files": "treasury_bulletin_1944_03.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/march-1944-6567?page=93",
|
||||
"source_split": "val"
|
||||
},
|
||||
{
|
||||
"id": "UID0154",
|
||||
"uid": "UID0154",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_1977_03.txt\r\ntreasury_bulletin_1978_03.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/march-1977-6963?page=83\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/march-1978-6975?page=84",
|
||||
"source_split": "val"
|
||||
},
|
||||
{
|
||||
"id": "UID0159",
|
||||
"uid": "UID0159",
|
||||
"category": "easy",
|
||||
"source_files": "treasury_bulletin_2000_09.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/september-2000-7103?page=109",
|
||||
"source_split": "val"
|
||||
},
|
||||
{
|
||||
"id": "UID0161",
|
||||
"uid": "UID0161",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_1980_03.txt\r\ntreasury_bulletin_1985_03.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/march-1980-6999?page=88\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/march-1985-7040?page=48",
|
||||
"source_split": "val"
|
||||
},
|
||||
{
|
||||
"id": "UID0170",
|
||||
"uid": "UID0170",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_1960_03.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/march-1960-6759?page=64",
|
||||
"source_split": "val"
|
||||
},
|
||||
{
|
||||
"id": "UID0190",
|
||||
"uid": "UID0190",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_1939_10.txt\r\ntreasury_bulletin_1939_11.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/october-1939-6520?page=14\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/november-1939-6512?page=14",
|
||||
"source_split": "val"
|
||||
},
|
||||
{
|
||||
"id": "UID0213",
|
||||
"uid": "UID0213",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_1947_04.txt\r\ntreasury_bulletin_1948_04.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/april-1947-6603?page=28\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/april-1948-6615?page=18",
|
||||
"source_split": "val"
|
||||
},
|
||||
{
|
||||
"id": "UID0217",
|
||||
"uid": "UID0217",
|
||||
"category": "easy",
|
||||
"source_files": "treasury_bulletin_1963_10.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/october-1963-6802?page=15",
|
||||
"source_split": "val"
|
||||
},
|
||||
{
|
||||
"id": "UID0220",
|
||||
"uid": "UID0220",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_1939_02.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/february-1939-6505?page=25",
|
||||
"source_split": "val"
|
||||
},
|
||||
{
|
||||
"id": "UID0233",
|
||||
"uid": "UID0233",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_1948_04.txt\r\ntreasury_bulletin_1958_04.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/april-1948-6615?page=42\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/april-1958-6735?page=54",
|
||||
"source_split": "val"
|
||||
},
|
||||
{
|
||||
"id": "UID0234",
|
||||
"uid": "UID0234",
|
||||
"category": "easy",
|
||||
"source_files": "treasury_bulletin_1958_01.txt\r\ntreasury_bulletin_1958_02.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/january-1958-6732?page=28\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/february-1958-6733?page=32",
|
||||
"source_split": "val"
|
||||
},
|
||||
{
|
||||
"id": "UID0235",
|
||||
"uid": "UID0235",
|
||||
"category": "easy",
|
||||
"source_files": "treasury_bulletin_1948_04.txt\r\ntreasury_bulletin_1948_05.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/april-1948-6615?page=27\r\nhttps://fraser.stlouisfed.org/title/treasury-bulletin-407/may-1948-6616?page=27",
|
||||
"source_split": "val"
|
||||
},
|
||||
{
|
||||
"id": "UID0239",
|
||||
"uid": "UID0239",
|
||||
"category": "easy",
|
||||
"source_files": "treasury_bulletin_1953_01.txt\r\ntreasury_bulletin_1954_01.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/january-1953-6672?page=62\r\nhttp://fraser.stlouisfed.org/title/treasury-bulletin-407/january-1954-6684?page=51",
|
||||
"source_split": "val"
|
||||
},
|
||||
{
|
||||
"id": "UID0240",
|
||||
"uid": "UID0240",
|
||||
"category": "hard",
|
||||
"source_files": "treasury_bulletin_1957_12.txt",
|
||||
"source_docs": "https://fraser.stlouisfed.org/title/treasury-bulletin-407/december-1957-6731?page=26",
|
||||
"source_split": "val"
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,21 @@
|
||||
{
|
||||
"benchmark": "SearchQA",
|
||||
"manifest_type": "id_split",
|
||||
"source_repo": "lucadiliello/searchqa",
|
||||
"source_repo_type": "dataset",
|
||||
"source_url": "https://huggingface.co/datasets/lucadiliello/searchqa",
|
||||
"source_id_field": "key",
|
||||
"counts": {
|
||||
"train": 400,
|
||||
"val": 200,
|
||||
"test": 1400
|
||||
},
|
||||
"item_fields": [
|
||||
"id"
|
||||
],
|
||||
"notes": [
|
||||
"This is a split manifest, not the full SearchQA payload.",
|
||||
"Materialize full split items from lucadiliello/searchqa before evaluation.",
|
||||
"The IDs in items.json exactly match the key field in lucadiliello/searchqa."
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,24 @@
|
||||
{
|
||||
"benchmark": "SpreadsheetBench",
|
||||
"manifest_type": "id_split",
|
||||
"source_repo": "KAKA22/SpreadsheetBench",
|
||||
"source_repo_type": "dataset",
|
||||
"source_url": "https://huggingface.co/datasets/KAKA22/SpreadsheetBench",
|
||||
"source_revision": "ab0b742b0fc95b946f212d80ac7771b5531272e4",
|
||||
"source_file": "spreadsheetbench_verified_400.tar.gz",
|
||||
"source_split_name": "spreadsheetbench_split",
|
||||
"counts": {
|
||||
"train": 80,
|
||||
"val": 40,
|
||||
"test": 280
|
||||
},
|
||||
"item_fields": [
|
||||
"id",
|
||||
"spreadsheet_path",
|
||||
"instruction_type"
|
||||
],
|
||||
"notes": [
|
||||
"This is a split manifest, not the full SpreadsheetBench payload.",
|
||||
"Materialize full task JSON rows plus spreadsheet files from SpreadsheetBench Verified 400 before evaluation."
|
||||
]
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,402 @@
|
||||
[
|
||||
{
|
||||
"id": "32438",
|
||||
"spreadsheet_path": "spreadsheet/32438",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "398-14",
|
||||
"spreadsheet_path": "spreadsheet/398-14",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "47766",
|
||||
"spreadsheet_path": "spreadsheet/47766",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "48365",
|
||||
"spreadsheet_path": "spreadsheet/48365",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "32255",
|
||||
"spreadsheet_path": "spreadsheet/32255",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "10747",
|
||||
"spreadsheet_path": "spreadsheet/10747",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "50916",
|
||||
"spreadsheet_path": "spreadsheet/50916",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "577-40",
|
||||
"spreadsheet_path": "spreadsheet/577-40",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "35742",
|
||||
"spreadsheet_path": "spreadsheet/35742",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "46121",
|
||||
"spreadsheet_path": "spreadsheet/46121",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "51090",
|
||||
"spreadsheet_path": "spreadsheet/51090",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "51249",
|
||||
"spreadsheet_path": "spreadsheet/51249",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "82-30",
|
||||
"spreadsheet_path": "spreadsheet/82-30",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "56274",
|
||||
"spreadsheet_path": "spreadsheet/56274",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "57445",
|
||||
"spreadsheet_path": "spreadsheet/57445",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "46646",
|
||||
"spreadsheet_path": "spreadsheet/46646",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "105-24",
|
||||
"spreadsheet_path": "spreadsheet/105-24",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "6239",
|
||||
"spreadsheet_path": "spreadsheet/6239",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "414-20",
|
||||
"spreadsheet_path": "spreadsheet/414-20",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "165-23",
|
||||
"spreadsheet_path": "spreadsheet/165-23",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "40892",
|
||||
"spreadsheet_path": "spreadsheet/40892",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "48745",
|
||||
"spreadsheet_path": "spreadsheet/48745",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "32612",
|
||||
"spreadsheet_path": "spreadsheet/32612",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "325-44",
|
||||
"spreadsheet_path": "spreadsheet/325-44",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "262-17",
|
||||
"spreadsheet_path": "spreadsheet/262-17",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "141-20",
|
||||
"spreadsheet_path": "spreadsheet/141-20",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "52216",
|
||||
"spreadsheet_path": "spreadsheet/52216",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "22-47",
|
||||
"spreadsheet_path": "spreadsheet/22-47",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "55421",
|
||||
"spreadsheet_path": "spreadsheet/55421",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "56427",
|
||||
"spreadsheet_path": "spreadsheet/56427",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "36097",
|
||||
"spreadsheet_path": "spreadsheet/36097",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "32902",
|
||||
"spreadsheet_path": "spreadsheet/32902",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "32023",
|
||||
"spreadsheet_path": "spreadsheet/32023",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "1818",
|
||||
"spreadsheet_path": "spreadsheet/1818",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "170-13",
|
||||
"spreadsheet_path": "spreadsheet/170-13",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "66-24",
|
||||
"spreadsheet_path": "spreadsheet/66-24",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "58949",
|
||||
"spreadsheet_path": "spreadsheet/58949",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "42354",
|
||||
"spreadsheet_path": "spreadsheet/42354",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "194-19",
|
||||
"spreadsheet_path": "spreadsheet/194-19",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "31915",
|
||||
"spreadsheet_path": "spreadsheet/31915",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "58499",
|
||||
"spreadsheet_path": "spreadsheet/58499",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "45372",
|
||||
"spreadsheet_path": "spreadsheet/45372",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "11842",
|
||||
"spreadsheet_path": "spreadsheet/11842",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "57558",
|
||||
"spreadsheet_path": "spreadsheet/57558",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "472-15",
|
||||
"spreadsheet_path": "spreadsheet/472-15",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "55060",
|
||||
"spreadsheet_path": "spreadsheet/55060",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "31011",
|
||||
"spreadsheet_path": "spreadsheet/31011",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "408-39",
|
||||
"spreadsheet_path": "spreadsheet/408-39",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "54085",
|
||||
"spreadsheet_path": "spreadsheet/54085",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "39903",
|
||||
"spreadsheet_path": "spreadsheet/39903",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "48983",
|
||||
"spreadsheet_path": "spreadsheet/48983",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "108-24",
|
||||
"spreadsheet_path": "spreadsheet/108-24",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "58484",
|
||||
"spreadsheet_path": "spreadsheet/58484",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "118-50",
|
||||
"spreadsheet_path": "spreadsheet/118-50",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "10452",
|
||||
"spreadsheet_path": "spreadsheet/10452",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "39931",
|
||||
"spreadsheet_path": "spreadsheet/39931",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "3413",
|
||||
"spreadsheet_path": "spreadsheet/3413",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "247-24",
|
||||
"spreadsheet_path": "spreadsheet/247-24",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "56786",
|
||||
"spreadsheet_path": "spreadsheet/56786",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "55965",
|
||||
"spreadsheet_path": "spreadsheet/55965",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "379-36",
|
||||
"spreadsheet_path": "spreadsheet/379-36",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "58109",
|
||||
"spreadsheet_path": "spreadsheet/58109",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "433-47",
|
||||
"spreadsheet_path": "spreadsheet/433-47",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "192-22",
|
||||
"spreadsheet_path": "spreadsheet/192-22",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "49333",
|
||||
"spreadsheet_path": "spreadsheet/49333",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "493-18",
|
||||
"spreadsheet_path": "spreadsheet/493-18",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "54638",
|
||||
"spreadsheet_path": "spreadsheet/54638",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "34033",
|
||||
"spreadsheet_path": "spreadsheet/34033",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "30930",
|
||||
"spreadsheet_path": "spreadsheet/30930",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "585-41",
|
||||
"spreadsheet_path": "spreadsheet/585-41",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "32337",
|
||||
"spreadsheet_path": "spreadsheet/32337",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "55427",
|
||||
"spreadsheet_path": "spreadsheet/55427",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "263-1",
|
||||
"spreadsheet_path": "spreadsheet/263-1",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "254-34",
|
||||
"spreadsheet_path": "spreadsheet/254-34",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "57113",
|
||||
"spreadsheet_path": "spreadsheet/57113",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "57743",
|
||||
"spreadsheet_path": "spreadsheet/57743",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "43589",
|
||||
"spreadsheet_path": "spreadsheet/43589",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "250-20",
|
||||
"spreadsheet_path": "spreadsheet/250-20",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "48080",
|
||||
"spreadsheet_path": "spreadsheet/48080",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "370-43",
|
||||
"spreadsheet_path": "spreadsheet/370-43",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,202 @@
|
||||
[
|
||||
{
|
||||
"id": "45635",
|
||||
"spreadsheet_path": "spreadsheet/45635",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "560-12",
|
||||
"spreadsheet_path": "spreadsheet/560-12",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "55049",
|
||||
"spreadsheet_path": "spreadsheet/55049",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "9569",
|
||||
"spreadsheet_path": "spreadsheet/9569",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "7902",
|
||||
"spreadsheet_path": "spreadsheet/7902",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "227-40",
|
||||
"spreadsheet_path": "spreadsheet/227-40",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "463-17",
|
||||
"spreadsheet_path": "spreadsheet/463-17",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "54144",
|
||||
"spreadsheet_path": "spreadsheet/54144",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "80-42",
|
||||
"spreadsheet_path": "spreadsheet/80-42",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "2768",
|
||||
"spreadsheet_path": "spreadsheet/2768",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "37456",
|
||||
"spreadsheet_path": "spreadsheet/37456",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "12864",
|
||||
"spreadsheet_path": "spreadsheet/12864",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "55979",
|
||||
"spreadsheet_path": "spreadsheet/55979",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "48620",
|
||||
"spreadsheet_path": "spreadsheet/48620",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "48588",
|
||||
"spreadsheet_path": "spreadsheet/48588",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "395-36",
|
||||
"spreadsheet_path": "spreadsheet/395-36",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "382-10",
|
||||
"spreadsheet_path": "spreadsheet/382-10",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "59595",
|
||||
"spreadsheet_path": "spreadsheet/59595",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "53383",
|
||||
"spreadsheet_path": "spreadsheet/53383",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "48921",
|
||||
"spreadsheet_path": "spreadsheet/48921",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "416-15",
|
||||
"spreadsheet_path": "spreadsheet/416-15",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "47798",
|
||||
"spreadsheet_path": "spreadsheet/47798",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "56563",
|
||||
"spreadsheet_path": "spreadsheet/56563",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "46897",
|
||||
"spreadsheet_path": "spreadsheet/46897",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "9726",
|
||||
"spreadsheet_path": "spreadsheet/9726",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "50768",
|
||||
"spreadsheet_path": "spreadsheet/50768",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "51-12",
|
||||
"spreadsheet_path": "spreadsheet/51-12",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "31628",
|
||||
"spreadsheet_path": "spreadsheet/31628",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "39046",
|
||||
"spreadsheet_path": "spreadsheet/39046",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "8942",
|
||||
"spreadsheet_path": "spreadsheet/8942",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "48527",
|
||||
"spreadsheet_path": "spreadsheet/48527",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "59196",
|
||||
"spreadsheet_path": "spreadsheet/59196",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "6698",
|
||||
"spreadsheet_path": "spreadsheet/6698",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "43436",
|
||||
"spreadsheet_path": "spreadsheet/43436",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "38462",
|
||||
"spreadsheet_path": "spreadsheet/38462",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "402-43",
|
||||
"spreadsheet_path": "spreadsheet/402-43",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "267-18",
|
||||
"spreadsheet_path": "spreadsheet/267-18",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "37378",
|
||||
"spreadsheet_path": "spreadsheet/37378",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "53647",
|
||||
"spreadsheet_path": "spreadsheet/53647",
|
||||
"instruction_type": "Cell-Level Manipulation"
|
||||
},
|
||||
{
|
||||
"id": "142-12",
|
||||
"spreadsheet_path": "spreadsheet/142-12",
|
||||
"instruction_type": "Sheet-Level Manipulation"
|
||||
}
|
||||
]
|
||||
@@ -25,10 +25,10 @@ Open an issue with:
|
||||
See [Add a New Benchmark](guide/new-benchmark.md) for the implementation guide.
|
||||
|
||||
**Checklist:**
|
||||
- [ ] Data loader in `skillopt/envs/<benchmark>/loader.py`
|
||||
- [ ] Environment adapter in `skillopt/envs/<benchmark>/env.py`
|
||||
- [ ] Data loader in `skillopt/envs/<benchmark>/dataloader.py`
|
||||
- [ ] Environment adapter in `skillopt/envs/<benchmark>/adapter.py`
|
||||
- [ ] Config file in `configs/<benchmark>/default.yaml`
|
||||
- [ ] Registration in `skillopt/envs/__init__.py`
|
||||
- [ ] Registration in `scripts/train.py` (`_ENV_REGISTRY`)
|
||||
- [ ] Documentation page in `docs/`
|
||||
|
||||
### 🤖 New Model Backend
|
||||
|
||||
+56
-107
@@ -6,9 +6,10 @@ Extend SkillOpt with your own benchmark in ~100 lines of code.
|
||||
|
||||
To add a benchmark, you need:
|
||||
|
||||
1. **Data Loader** — Loads and splits your dataset
|
||||
2. **Environment Adapter** — Executes tasks and returns scores
|
||||
1. **Data Loader** — Subclass `SplitDataLoader` to load your split data
|
||||
2. **Environment Adapter** — Subclass `EnvAdapter` and implement rollout/reflect hooks
|
||||
3. **Config** — YAML configuration file
|
||||
4. **Registration** — Add your adapter to the train script registry
|
||||
|
||||
## Step 1: Create the Benchmark Package
|
||||
|
||||
@@ -19,126 +20,73 @@ touch skillopt/envs/my_benchmark/__init__.py
|
||||
|
||||
## Step 2: Implement the Data Loader
|
||||
|
||||
Create `skillopt/envs/my_benchmark/loader.py`:
|
||||
Create `skillopt/envs/my_benchmark/dataloader.py`:
|
||||
|
||||
```python
|
||||
from skillopt.data.base import DataLoader, DataItem
|
||||
from skillopt.datasets.base import SplitDataLoader
|
||||
|
||||
class MyBenchmarkDataLoader(DataLoader):
|
||||
"""Load and split your benchmark data."""
|
||||
|
||||
def __init__(self, data_dir: str, **kwargs):
|
||||
super().__init__(**kwargs)
|
||||
self.data_dir = data_dir
|
||||
|
||||
def setup(self, cfg: dict):
|
||||
"""Initialize splits based on config."""
|
||||
self.split_mode = cfg.get('split_mode', 'ratio')
|
||||
# Load your data here
|
||||
self.items = self._load_items()
|
||||
self._create_splits(cfg)
|
||||
|
||||
def _load_items(self) -> list[DataItem]:
|
||||
"""Load raw data into DataItem objects."""
|
||||
items = []
|
||||
# TODO: Load your data
|
||||
for entry in your_data:
|
||||
items.append(DataItem(
|
||||
id=entry['id'],
|
||||
input=entry['question'],
|
||||
ground_truth=entry['answer'],
|
||||
metadata=entry.get('metadata', {})
|
||||
))
|
||||
return items
|
||||
|
||||
def get_split_items(self, split: str) -> list[DataItem]:
|
||||
"""Return items for a given split (train/valid/test)."""
|
||||
return self.splits[split]
|
||||
|
||||
class MyBenchmarkDataLoader(SplitDataLoader):
|
||||
"""Load benchmark items from raw data and/or split directories."""
|
||||
|
||||
def load_raw_items(self, data_path: str) -> list[dict]:
|
||||
# For ratio mode, parse your source dataset from data_path.
|
||||
# Return list[dict] where each item has at least a unique, deterministic "id".
|
||||
return super().load_raw_items(data_path)
|
||||
|
||||
def load_split_items(self, split_path: str) -> list[dict]:
|
||||
# For split_dir mode, parse one split directory.
|
||||
return super().load_split_items(split_path)
|
||||
```
|
||||
|
||||
## Step 3: Implement the Environment Adapter
|
||||
|
||||
Create `skillopt/envs/my_benchmark/env.py`:
|
||||
Create `skillopt/envs/my_benchmark/adapter.py`:
|
||||
|
||||
```python
|
||||
from skillopt.envs.base import EnvAdapter, TaskResult
|
||||
from skillopt.envs.base import EnvAdapter
|
||||
from skillopt.envs.my_benchmark.dataloader import MyBenchmarkDataLoader
|
||||
|
||||
class MyBenchmarkEnv(EnvAdapter):
|
||||
"""Execute tasks and evaluate results."""
|
||||
|
||||
def __init__(self, cfg: dict):
|
||||
super().__init__(cfg)
|
||||
|
||||
async def execute(self, item: DataItem, skill: str, model) -> TaskResult:
|
||||
"""
|
||||
Execute a single task.
|
||||
|
||||
Args:
|
||||
item: The data item to process
|
||||
skill: Current skill document content
|
||||
model: The target model instance
|
||||
|
||||
Returns:
|
||||
TaskResult with prediction, score, and trajectory
|
||||
"""
|
||||
# Build prompt with skill document
|
||||
prompt = self.build_prompt(item, skill)
|
||||
|
||||
# Get model response
|
||||
response = await model.generate(prompt)
|
||||
|
||||
# Extract prediction
|
||||
prediction = self.parse_response(response)
|
||||
|
||||
# Score against ground truth
|
||||
score = self.evaluate(prediction, item.ground_truth)
|
||||
|
||||
return TaskResult(
|
||||
item_id=item.id,
|
||||
prediction=prediction,
|
||||
score=score,
|
||||
trajectory=[
|
||||
{"role": "system", "content": skill},
|
||||
{"role": "user", "content": item.input},
|
||||
{"role": "assistant", "content": response}
|
||||
]
|
||||
)
|
||||
|
||||
def evaluate(self, prediction: str, ground_truth: str) -> float:
|
||||
"""
|
||||
Score a prediction against ground truth.
|
||||
|
||||
Returns:
|
||||
Float between 0.0 and 1.0
|
||||
"""
|
||||
# TODO: Implement your scoring logic
|
||||
# Examples: exact match, F1, ANLS, etc.
|
||||
return float(prediction.strip() == ground_truth.strip())
|
||||
|
||||
def build_prompt(self, item, skill: str) -> str:
|
||||
"""Combine skill document with task input."""
|
||||
return f"{skill}\n\n---\n\nQuestion: {item.input}"
|
||||
|
||||
def parse_response(self, response: str) -> str:
|
||||
"""Extract the answer from model response."""
|
||||
return response.strip()
|
||||
class MyBenchmarkAdapter(EnvAdapter):
|
||||
def __init__(self, split_dir: str = "", data_path: str = "", **kwargs):
|
||||
self.dataloader = MyBenchmarkDataLoader(split_dir=split_dir, data_path=data_path, **kwargs)
|
||||
|
||||
def setup(self, cfg: dict) -> None:
|
||||
super().setup(cfg)
|
||||
self.dataloader.setup(cfg)
|
||||
|
||||
def get_dataloader(self):
|
||||
return self.dataloader
|
||||
|
||||
def build_train_env(self, batch_size: int, seed: int, **kwargs):
|
||||
return self.dataloader.build_train_batch(batch_size=batch_size, seed=seed, **kwargs).payload
|
||||
|
||||
def build_eval_env(self, env_num: int, split: str, seed: int, **kwargs):
|
||||
return self.dataloader.build_eval_batch(env_num=env_num, split=split, seed=seed, **kwargs).payload
|
||||
|
||||
def rollout(self, env_manager, skill_content: str, out_dir: str, **kwargs) -> list[dict]:
|
||||
# env_manager is the payload returned by build_train_env/build_eval_env
|
||||
# (commonly list[dict] task items).
|
||||
# Run target model on each item and return list[dict].
|
||||
# Required keys per row: "id", "hard" (0/1), "soft" (0.0-1.0)
|
||||
raise NotImplementedError
|
||||
|
||||
def reflect(self, results: list[dict], skill_content: str, out_dir: str, **kwargs) -> list[dict | None]:
|
||||
# Convert failure/success analysis into RawPatch-like dicts.
|
||||
raise NotImplementedError
|
||||
|
||||
def get_task_types(self) -> list[str]:
|
||||
return ["my_benchmark"]
|
||||
```
|
||||
|
||||
## Step 4: Register the Benchmark
|
||||
|
||||
Add to `skillopt/envs/__init__.py`:
|
||||
Add your adapter to `_register_builtins()` in `scripts/train.py`:
|
||||
|
||||
```python
|
||||
from .my_benchmark.env import MyBenchmarkEnv
|
||||
from .my_benchmark.loader import MyBenchmarkDataLoader
|
||||
from skillopt.envs.my_benchmark.adapter import MyBenchmarkAdapter
|
||||
|
||||
BENCHMARK_REGISTRY = {
|
||||
# ... existing benchmarks ...
|
||||
'my_benchmark': {
|
||||
'env': MyBenchmarkEnv,
|
||||
'loader': MyBenchmarkDataLoader,
|
||||
},
|
||||
}
|
||||
_ENV_REGISTRY["my_benchmark"] = MyBenchmarkAdapter
|
||||
```
|
||||
|
||||
## Step 5: Create Config
|
||||
@@ -146,7 +94,7 @@ BENCHMARK_REGISTRY = {
|
||||
Create `configs/my_benchmark/default.yaml`:
|
||||
|
||||
```yaml
|
||||
_base_: ['../_base_/default.yaml']
|
||||
_base_: ../_base_/default.yaml
|
||||
|
||||
env:
|
||||
name: my_benchmark
|
||||
@@ -178,4 +126,5 @@ python scripts/train.py --config configs/my_benchmark/default.yaml
|
||||
|
||||
!!! tip
|
||||
- Use a small `batch_size` (10-20) for initial testing
|
||||
- The `evaluate()` method is critical — a noisy metric will confuse the optimizer
|
||||
- Start from `skillopt/envs/_template/` and adapt from there
|
||||
- Use an existing adapter (for example `skillopt/envs/officeqa/adapter.py`) as a concrete reference
|
||||
|
||||
+44
-50
@@ -4,78 +4,72 @@
|
||||
|
||||
### `EnvAdapter`
|
||||
|
||||
Abstract base class for benchmark environments.
|
||||
Abstract base class for benchmark environments (`skillopt/envs/base.py`).
|
||||
|
||||
```python
|
||||
class EnvAdapter(ABC):
|
||||
async def execute(self, item, skill, model) -> TaskResult
|
||||
def evaluate(self, prediction, ground_truth) -> float
|
||||
def build_prompt(self, item, skill) -> str
|
||||
```
|
||||
|
||||
### `DataLoader`
|
||||
|
||||
Abstract base class for data loading and splitting.
|
||||
|
||||
```python
|
||||
class DataLoader(ABC):
|
||||
def setup(self, cfg: dict) -> None
|
||||
def get_split_items(self, split: str) -> list[DataItem]
|
||||
def get_dataloader(self) -> BaseDataLoader | None
|
||||
def build_train_env(self, batch_size: int, seed: int, **kwargs)
|
||||
def build_eval_env(self, env_num: int, split: str, seed: int, **kwargs)
|
||||
def rollout(self, env_manager, skill_content: str, out_dir: str, **kwargs) -> list[dict]
|
||||
def reflect(self, results: list[dict], skill_content: str, out_dir: str, **kwargs) -> list[dict | None]
|
||||
def get_task_types(self) -> list[str]
|
||||
```
|
||||
|
||||
### `ModelBackend`
|
||||
|
||||
Abstract base class for LLM backends.
|
||||
The rollout contract expects result rows with at least:
|
||||
|
||||
```python
|
||||
class ModelBackend(ABC):
|
||||
async def generate(self, messages, **kwargs) -> ModelResponse
|
||||
async def generate_with_tools(self, messages, tools, **kwargs) -> ModelResponse
|
||||
{"id": str, "hard": int, "soft": float}
|
||||
```
|
||||
|
||||
### `Trainer`
|
||||
### `BaseDataLoader` / `SplitDataLoader`
|
||||
|
||||
Main training loop orchestrator.
|
||||
Data loader abstractions (`skillopt/datasets/base.py`).
|
||||
|
||||
```python
|
||||
class Trainer:
|
||||
def __init__(self, cfg: dict)
|
||||
async def train(self) -> TrainResult
|
||||
async def evaluate(self, skill: str, split: str) -> EvalResult
|
||||
class BaseDataLoader(ABC):
|
||||
def setup(self, cfg: dict) -> None
|
||||
def build_train_batch(self, batch_size: int, seed: int, **kwargs) -> BatchSpec
|
||||
def build_eval_batch(self, env_num: int, split: str, seed: int, **kwargs) -> BatchSpec
|
||||
|
||||
class SplitDataLoader(BaseDataLoader):
|
||||
def load_raw_items(self, data_path: str) -> list[dict]
|
||||
def load_split_items(self, split_path: str) -> list[dict]
|
||||
def get_split_items(self, split: str) -> list[dict]
|
||||
```
|
||||
|
||||
## Data Classes
|
||||
### `BatchSpec`
|
||||
|
||||
### `DataItem`
|
||||
Represents one concrete batch request.
|
||||
|
||||
```python
|
||||
@dataclass(slots=True)
|
||||
class BatchSpec:
|
||||
phase: str
|
||||
split: str
|
||||
seed: int
|
||||
batch_size: int
|
||||
payload: object | None = None
|
||||
metadata: dict[str, Any] = field(default_factory=dict)
|
||||
```
|
||||
|
||||
### `RolloutResult` / `RawPatch`
|
||||
|
||||
Typed helpers for stage I/O in `skillopt/types.py`.
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class DataItem:
|
||||
class RolloutResult:
|
||||
id: str
|
||||
input: str
|
||||
ground_truth: str
|
||||
metadata: dict = field(default_factory=dict)
|
||||
```
|
||||
hard: int
|
||||
soft: float
|
||||
# optional benchmark-specific fields
|
||||
|
||||
### `TaskResult`
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class TaskResult:
|
||||
item_id: str
|
||||
prediction: str
|
||||
score: float
|
||||
trajectory: list[dict]
|
||||
```
|
||||
|
||||
### `ModelResponse`
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class ModelResponse:
|
||||
content: str
|
||||
usage: dict
|
||||
model: str
|
||||
class RawPatch:
|
||||
patch: Patch
|
||||
source_type: Literal["failure", "success"] = "failure"
|
||||
```
|
||||
|
||||
For detailed source code, see the [`skillopt/`](https://github.com/microsoft/SkillOpt/tree/main/skillopt) directory.
|
||||
|
||||
@@ -10,6 +10,12 @@ Complete reference for all SkillOpt configuration parameters.
|
||||
| `model.optimizer` | str | `gpt-5.5` | Optimizer model (for reflection & slow update) |
|
||||
| `model.target` | str | `gpt-5.5` | Target model (for rollout execution) |
|
||||
| `model.reasoning_effort` | str | `medium` | Reasoning effort level |
|
||||
| `model.optimizer_backend` | str | `openai_chat` | Optimizer backend: `openai_chat` / `claude_chat` / `qwen_chat` / `minimax_chat` |
|
||||
| `model.target_backend` | str | `openai_chat` | Target backend: chat backends plus execution harnesses |
|
||||
| `model.qwen_chat_base_url` | str | `http://localhost:8000/v1` | Shared Qwen/vLLM OpenAI-compatible endpoint |
|
||||
| `model.qwen_chat_enable_thinking` | bool | `false` | Shared Qwen thinking flag |
|
||||
| `model.optimizer_qwen_chat_base_url` | str | — | Optimizer-specific Qwen/vLLM endpoint; overrides shared `qwen_chat_base_url` |
|
||||
| `model.target_qwen_chat_base_url` | str | — | Target-specific Qwen/vLLM endpoint; overrides shared `qwen_chat_base_url` |
|
||||
|
||||
## Training (`train`)
|
||||
|
||||
@@ -70,3 +76,10 @@ Complete reference for all SkillOpt configuration parameters.
|
||||
| `AZURE_OPENAI_API_KEY` / `model.azure_openai_api_key` | Azure API key |
|
||||
| `OPENAI_API_KEY` | OpenAI API key (for `openai_chat` backend) |
|
||||
| `ANTHROPIC_API_KEY` | Anthropic API key (for `claude_code_exec` backend) |
|
||||
| `QWEN_CHAT_BASE_URL` | Shared local vLLM endpoint for `qwen_chat` |
|
||||
| `QWEN_CHAT_MODEL` | Shared served model name for `qwen_chat` |
|
||||
| `QWEN_CHAT_API_KEY` | Optional API key for the shared Qwen endpoint |
|
||||
| `OPTIMIZER_QWEN_CHAT_BASE_URL` | Optimizer-specific local vLLM endpoint |
|
||||
| `OPTIMIZER_QWEN_CHAT_MODEL` | Optimizer-specific served model name |
|
||||
| `TARGET_QWEN_CHAT_BASE_URL` | Target-specific local vLLM endpoint |
|
||||
| `TARGET_QWEN_CHAT_MODEL` | Target-specific served model name |
|
||||
|
||||
+55
-1
@@ -137,7 +137,7 @@ def parse_args() -> argparse.Namespace:
|
||||
# Legacy flat CLI overrides (still work, prefer --cfg-options for new usage)
|
||||
p.add_argument("--env", type=str)
|
||||
p.add_argument("--backend", type=str,
|
||||
choices=["azure_openai", "codex", "codex_exec", "claude", "claude_chat", "claude_code_exec", "qwen", "qwen_chat"])
|
||||
choices=["azure_openai", "codex", "codex_exec", "claude", "claude_chat", "claude_code_exec", "qwen", "qwen_chat", "minimax", "minimax_chat"])
|
||||
p.add_argument("--optimizer_model", type=str)
|
||||
p.add_argument("--target_model", type=str)
|
||||
p.add_argument("--optimizer_backend", type=str)
|
||||
@@ -173,6 +173,24 @@ def parse_args() -> argparse.Namespace:
|
||||
p.add_argument("--qwen_chat_timeout_seconds", type=float)
|
||||
p.add_argument("--qwen_chat_max_tokens", type=int)
|
||||
p.add_argument("--qwen_chat_enable_thinking", type=_BOOL)
|
||||
p.add_argument("--optimizer_qwen_chat_base_url", type=str)
|
||||
p.add_argument("--optimizer_qwen_chat_api_key", type=str)
|
||||
p.add_argument("--optimizer_qwen_chat_temperature", type=float)
|
||||
p.add_argument("--optimizer_qwen_chat_timeout_seconds", type=float)
|
||||
p.add_argument("--optimizer_qwen_chat_max_tokens", type=int)
|
||||
p.add_argument("--optimizer_qwen_chat_enable_thinking", type=_BOOL)
|
||||
p.add_argument("--target_qwen_chat_base_url", type=str)
|
||||
p.add_argument("--target_qwen_chat_api_key", type=str)
|
||||
p.add_argument("--target_qwen_chat_temperature", type=float)
|
||||
p.add_argument("--target_qwen_chat_timeout_seconds", type=float)
|
||||
p.add_argument("--target_qwen_chat_max_tokens", type=int)
|
||||
p.add_argument("--target_qwen_chat_enable_thinking", type=_BOOL)
|
||||
p.add_argument("--minimax_base_url", type=str)
|
||||
p.add_argument("--minimax_api_key", type=str)
|
||||
p.add_argument("--minimax_model", type=str)
|
||||
p.add_argument("--minimax_temperature", type=float)
|
||||
p.add_argument("--minimax_max_tokens", type=int)
|
||||
p.add_argument("--minimax_enable_thinking", type=_BOOL)
|
||||
p.add_argument("--codex_exec_path", type=str)
|
||||
p.add_argument("--codex_exec_sandbox", type=str)
|
||||
p.add_argument("--codex_exec_profile", type=str)
|
||||
@@ -289,6 +307,24 @@ _LEGACY_TO_STRUCTURED: dict[str, str] = {
|
||||
"qwen_chat_timeout_seconds": "model.qwen_chat_timeout_seconds",
|
||||
"qwen_chat_max_tokens": "model.qwen_chat_max_tokens",
|
||||
"qwen_chat_enable_thinking": "model.qwen_chat_enable_thinking",
|
||||
"optimizer_qwen_chat_base_url": "model.optimizer_qwen_chat_base_url",
|
||||
"optimizer_qwen_chat_api_key": "model.optimizer_qwen_chat_api_key",
|
||||
"optimizer_qwen_chat_temperature": "model.optimizer_qwen_chat_temperature",
|
||||
"optimizer_qwen_chat_timeout_seconds": "model.optimizer_qwen_chat_timeout_seconds",
|
||||
"optimizer_qwen_chat_max_tokens": "model.optimizer_qwen_chat_max_tokens",
|
||||
"optimizer_qwen_chat_enable_thinking": "model.optimizer_qwen_chat_enable_thinking",
|
||||
"target_qwen_chat_base_url": "model.target_qwen_chat_base_url",
|
||||
"target_qwen_chat_api_key": "model.target_qwen_chat_api_key",
|
||||
"target_qwen_chat_temperature": "model.target_qwen_chat_temperature",
|
||||
"target_qwen_chat_timeout_seconds": "model.target_qwen_chat_timeout_seconds",
|
||||
"target_qwen_chat_max_tokens": "model.target_qwen_chat_max_tokens",
|
||||
"target_qwen_chat_enable_thinking": "model.target_qwen_chat_enable_thinking",
|
||||
"minimax_base_url": "model.minimax_base_url",
|
||||
"minimax_api_key": "model.minimax_api_key",
|
||||
"minimax_model": "model.minimax_model",
|
||||
"minimax_temperature": "model.minimax_temperature",
|
||||
"minimax_max_tokens": "model.minimax_max_tokens",
|
||||
"minimax_enable_thinking": "model.minimax_enable_thinking",
|
||||
"codex_exec_path": "model.codex_exec_path",
|
||||
"codex_exec_sandbox": "model.codex_exec_sandbox",
|
||||
"codex_exec_profile": "model.codex_exec_profile",
|
||||
@@ -403,6 +439,9 @@ def load_config(args: argparse.Namespace) -> dict:
|
||||
elif backend in {"qwen", "qwen_chat"}:
|
||||
flat.setdefault("optimizer_backend", "openai_chat")
|
||||
flat.setdefault("target_backend", "qwen_chat")
|
||||
elif backend in {"minimax", "minimax_chat"}:
|
||||
flat.setdefault("optimizer_backend", "openai_chat")
|
||||
flat.setdefault("target_backend", "minimax_chat")
|
||||
else:
|
||||
flat.setdefault("optimizer_backend", "openai_chat")
|
||||
flat.setdefault("target_backend", "openai_chat")
|
||||
@@ -416,6 +455,12 @@ def load_config(args: argparse.Namespace) -> dict:
|
||||
and not _has_model_override("model.optimizer", "optimizer_model")
|
||||
):
|
||||
flat["optimizer_model"] = default_model_for_backend("claude_chat")
|
||||
if flat.get("optimizer_backend") == "qwen_chat":
|
||||
if (
|
||||
str(flat.get("optimizer_model", "") or "").strip() in _OPENAI_DEFAULT_MODEL_SENTINELS
|
||||
and not _has_model_override("model.optimizer", "optimizer_model")
|
||||
):
|
||||
flat["optimizer_model"] = default_model_for_backend("qwen_chat")
|
||||
if flat.get("target_backend") == "claude_chat":
|
||||
if (
|
||||
str(flat.get("target_model", "") or "").strip() in _OPENAI_DEFAULT_MODEL_SENTINELS
|
||||
@@ -434,6 +479,15 @@ def load_config(args: argparse.Namespace) -> dict:
|
||||
and not _has_model_override("model.target", "target_model")
|
||||
):
|
||||
flat["target_model"] = default_model_for_backend("qwen_chat")
|
||||
if flat.get("target_backend") == "minimax_chat":
|
||||
if (
|
||||
str(flat.get("target_model", "") or "").strip() in _OPENAI_DEFAULT_MODEL_SENTINELS
|
||||
and not _has_model_override("model.target", "target_model")
|
||||
):
|
||||
flat["target_model"] = (
|
||||
flat.get("minimax_model")
|
||||
or default_model_for_backend("minimax_chat")
|
||||
)
|
||||
|
||||
# Auto-generate output root
|
||||
if not flat.get("out_root"):
|
||||
|
||||
@@ -79,6 +79,24 @@ _FLATTEN_MAP: dict[str, str] = {
|
||||
"model.qwen_chat_timeout_seconds": "qwen_chat_timeout_seconds",
|
||||
"model.qwen_chat_max_tokens": "qwen_chat_max_tokens",
|
||||
"model.qwen_chat_enable_thinking": "qwen_chat_enable_thinking",
|
||||
"model.optimizer_qwen_chat_base_url": "optimizer_qwen_chat_base_url",
|
||||
"model.optimizer_qwen_chat_api_key": "optimizer_qwen_chat_api_key",
|
||||
"model.optimizer_qwen_chat_temperature": "optimizer_qwen_chat_temperature",
|
||||
"model.optimizer_qwen_chat_timeout_seconds": "optimizer_qwen_chat_timeout_seconds",
|
||||
"model.optimizer_qwen_chat_max_tokens": "optimizer_qwen_chat_max_tokens",
|
||||
"model.optimizer_qwen_chat_enable_thinking": "optimizer_qwen_chat_enable_thinking",
|
||||
"model.target_qwen_chat_base_url": "target_qwen_chat_base_url",
|
||||
"model.target_qwen_chat_api_key": "target_qwen_chat_api_key",
|
||||
"model.target_qwen_chat_temperature": "target_qwen_chat_temperature",
|
||||
"model.target_qwen_chat_timeout_seconds": "target_qwen_chat_timeout_seconds",
|
||||
"model.target_qwen_chat_max_tokens": "target_qwen_chat_max_tokens",
|
||||
"model.target_qwen_chat_enable_thinking": "target_qwen_chat_enable_thinking",
|
||||
"model.minimax_base_url": "minimax_base_url",
|
||||
"model.minimax_api_key": "minimax_api_key",
|
||||
"model.minimax_model": "minimax_model",
|
||||
"model.minimax_temperature": "minimax_temperature",
|
||||
"model.minimax_max_tokens": "minimax_max_tokens",
|
||||
"model.minimax_enable_thinking": "minimax_enable_thinking",
|
||||
"train.num_epochs": "num_epochs",
|
||||
"train.train_size": "train_size",
|
||||
"train.steps_per_epoch": "steps_per_epoch",
|
||||
|
||||
@@ -51,6 +51,7 @@ from skillopt.model import (
|
||||
configure_azure_openai,
|
||||
configure_claude_code_exec,
|
||||
configure_codex_exec,
|
||||
configure_minimax_chat,
|
||||
configure_qwen_chat,
|
||||
get_token_summary,
|
||||
reset_token_tracker,
|
||||
@@ -628,14 +629,36 @@ class ReflACTTrainer:
|
||||
effort=cfg.get("claude_code_exec_effort", cfg.get("reasoning_effort", "medium")),
|
||||
max_thinking_tokens=cfg.get("claude_code_exec_max_thinking_tokens", 16384),
|
||||
)
|
||||
configure_qwen_chat(
|
||||
base_url=cfg.get("qwen_chat_base_url") or None,
|
||||
api_key=cfg.get("qwen_chat_api_key") or None,
|
||||
temperature=cfg.get("qwen_chat_temperature"),
|
||||
timeout_seconds=cfg.get("qwen_chat_timeout_seconds"),
|
||||
max_tokens=cfg.get("qwen_chat_max_tokens"),
|
||||
enable_thinking=cfg.get("qwen_chat_enable_thinking"),
|
||||
configure_qwen_chat(
|
||||
base_url=cfg.get("qwen_chat_base_url") or None,
|
||||
api_key=cfg.get("qwen_chat_api_key") or None,
|
||||
temperature=cfg.get("qwen_chat_temperature"),
|
||||
timeout_seconds=cfg.get("qwen_chat_timeout_seconds"),
|
||||
max_tokens=cfg.get("qwen_chat_max_tokens"),
|
||||
enable_thinking=cfg.get("qwen_chat_enable_thinking"),
|
||||
optimizer_base_url=cfg.get("optimizer_qwen_chat_base_url") or None,
|
||||
optimizer_api_key=cfg.get("optimizer_qwen_chat_api_key") or None,
|
||||
optimizer_temperature=cfg.get("optimizer_qwen_chat_temperature"),
|
||||
optimizer_timeout_seconds=cfg.get("optimizer_qwen_chat_timeout_seconds"),
|
||||
optimizer_max_tokens=cfg.get("optimizer_qwen_chat_max_tokens"),
|
||||
optimizer_enable_thinking=cfg.get("optimizer_qwen_chat_enable_thinking"),
|
||||
target_base_url=cfg.get("target_qwen_chat_base_url") or None,
|
||||
target_api_key=cfg.get("target_qwen_chat_api_key") or None,
|
||||
target_temperature=cfg.get("target_qwen_chat_temperature"),
|
||||
target_timeout_seconds=cfg.get("target_qwen_chat_timeout_seconds"),
|
||||
target_max_tokens=cfg.get("target_qwen_chat_max_tokens"),
|
||||
target_enable_thinking=cfg.get("target_qwen_chat_enable_thinking"),
|
||||
)
|
||||
configure_minimax_chat(
|
||||
base_url=cfg.get("minimax_base_url") or None,
|
||||
api_key=cfg.get("minimax_api_key") or None,
|
||||
temperature=cfg.get("minimax_temperature"),
|
||||
max_tokens=cfg.get("minimax_max_tokens"),
|
||||
enable_thinking=cfg.get("minimax_enable_thinking"),
|
||||
)
|
||||
minimax_model_cfg = cfg.get("minimax_model")
|
||||
if minimax_model_cfg and cfg.get("target_backend") == "minimax_chat":
|
||||
set_target_deployment(str(minimax_model_cfg))
|
||||
os.environ["REFLACT_CODEX_TRACE_TO_OPTIMIZER"] = (
|
||||
"1"
|
||||
if target_backend == "codex_exec" and cfg.get("codex_trace_to_optimizer", False)
|
||||
|
||||
@@ -13,7 +13,7 @@ This directory provides scaffold files for adding a new benchmark to SkillOpt.
|
||||
1. Copy this directory: `cp -r skillopt/envs/_template skillopt/envs/your_benchmark`
|
||||
2. Rename files: remove `_template` suffix
|
||||
3. Implement the `TODO` sections
|
||||
4. Register in `skillopt/envs/__init__.py`
|
||||
4. Register your adapter in `_ENV_REGISTRY` inside `scripts/train.py`
|
||||
5. Create config at `configs/your_benchmark/default.yaml`
|
||||
|
||||
See the [documentation](../../docs/guide/new-benchmark.md) for the full guide.
|
||||
|
||||
@@ -5,11 +5,11 @@
|
||||
# and customize the values below.
|
||||
|
||||
# Inherit global defaults
|
||||
_base_: ['../_base_/default.yaml']
|
||||
_base_: ../_base_/default.yaml
|
||||
|
||||
# ── Environment ──────────────────────────────────
|
||||
env:
|
||||
name: your_benchmark # Must match registry key
|
||||
name: your_benchmark # Must match _ENV_REGISTRY key in scripts/train.py
|
||||
data_path: data/your_benchmark # Path to your data
|
||||
split_mode: ratio # "ratio" or "split_dir"
|
||||
split_ratio: "2:1:7" # train:val:test
|
||||
|
||||
@@ -4,89 +4,80 @@ Benchmark Environment Template
|
||||
Copy this file and implement the TODO sections to add a new benchmark.
|
||||
|
||||
The EnvAdapter is responsible for:
|
||||
1. Executing tasks using the target model + current skill document
|
||||
2. Evaluating predictions against ground truth
|
||||
3. Returning structured results for the training loop
|
||||
1. Building train/eval environment payloads
|
||||
2. Running rollout and returning scored result rows
|
||||
3. Reflecting on results and returning patch candidates
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from skillopt.datasets.base import BatchSpec
|
||||
from skillopt.envs._template.loader_template import TemplateBenchmarkDataLoader
|
||||
from skillopt.envs.base import EnvAdapter
|
||||
|
||||
|
||||
class TemplateBenchmarkEnv(EnvAdapter):
|
||||
class TemplateBenchmarkAdapter(EnvAdapter):
|
||||
"""
|
||||
Environment adapter for <Your Benchmark Name>.
|
||||
|
||||
|
||||
Rename this class and implement the abstract methods below.
|
||||
"""
|
||||
|
||||
def __init__(self, cfg: dict):
|
||||
super().__init__(cfg)
|
||||
# TODO: Initialize benchmark-specific state
|
||||
# Example: self.tools = load_tools(cfg)
|
||||
def __init__(
|
||||
self,
|
||||
split_dir: str = "",
|
||||
data_path: str = "",
|
||||
split_mode: str = "ratio",
|
||||
split_ratio: str = "2:1:7",
|
||||
split_seed: int = 42,
|
||||
split_output_dir: str = "",
|
||||
seed: int = 42,
|
||||
limit: int = 0,
|
||||
**kwargs,
|
||||
) -> None:
|
||||
self.dataloader = TemplateBenchmarkDataLoader(
|
||||
split_dir=split_dir,
|
||||
data_path=data_path,
|
||||
split_mode=split_mode,
|
||||
split_ratio=split_ratio,
|
||||
split_seed=split_seed,
|
||||
split_output_dir=split_output_dir,
|
||||
seed=seed,
|
||||
limit=limit,
|
||||
)
|
||||
# TODO: initialize runtime options, e.g.
|
||||
# self.max_retries = int(kwargs.get("max_retries", 3))
|
||||
# self.timeout_s = int(kwargs.get("timeout_s", 120))
|
||||
|
||||
async def execute(self, item, skill: str, model):
|
||||
def setup(self, cfg: dict) -> None:
|
||||
super().setup(cfg)
|
||||
self.dataloader.setup(cfg)
|
||||
|
||||
def get_dataloader(self):
|
||||
return self.dataloader
|
||||
|
||||
def build_env_from_batch(self, batch: BatchSpec, **kwargs):
|
||||
return list(batch.payload or [])
|
||||
|
||||
def build_train_env(self, batch_size: int, seed: int, **kwargs):
|
||||
batch = self.dataloader.build_train_batch(batch_size=batch_size, seed=seed, **kwargs)
|
||||
return self.build_env_from_batch(batch, **kwargs)
|
||||
|
||||
def build_eval_env(self, env_num: int, split: str, seed: int, **kwargs):
|
||||
batch = self.dataloader.build_eval_batch(env_num=env_num, split=split, seed=seed, **kwargs)
|
||||
return self.build_env_from_batch(batch, **kwargs)
|
||||
|
||||
def rollout(self, env_manager, skill_content: str, out_dir: str, **kwargs) -> list[dict]:
|
||||
"""
|
||||
Execute a single task with the target model.
|
||||
|
||||
Args:
|
||||
item: DataItem with .id, .input, .ground_truth, .metadata
|
||||
skill: Current skill document content (Markdown string)
|
||||
model: Target model backend instance
|
||||
|
||||
Returns:
|
||||
TaskResult with prediction, score, and trajectory
|
||||
Run one batch and return list[dict] with at least:
|
||||
{"id": str, "hard": int, "soft": float}
|
||||
"""
|
||||
# Step 1: Build the prompt combining skill + task input
|
||||
prompt = self.build_prompt(item, skill)
|
||||
raise NotImplementedError("Implement rollout() for your benchmark")
|
||||
|
||||
# Step 2: Call the target model
|
||||
# TODO: Customize the message format for your benchmark
|
||||
messages = [
|
||||
{"role": "system", "content": skill},
|
||||
{"role": "user", "content": item.input},
|
||||
]
|
||||
response = await model.generate(messages)
|
||||
|
||||
# Step 3: Parse the model response into a prediction
|
||||
prediction = self.parse_response(response.content)
|
||||
|
||||
# Step 4: Score the prediction
|
||||
score = self.evaluate(prediction, item.ground_truth)
|
||||
|
||||
# Step 5: Return structured result
|
||||
return {
|
||||
"item_id": item.id,
|
||||
"prediction": prediction,
|
||||
"score": score,
|
||||
"trajectory": messages + [{"role": "assistant", "content": response.content}],
|
||||
}
|
||||
|
||||
def evaluate(self, prediction: str, ground_truth: str) -> float:
|
||||
def reflect(self, results: list[dict], skill_content: str, out_dir: str, **kwargs) -> list[dict | None]:
|
||||
"""
|
||||
Score a prediction against the ground truth.
|
||||
|
||||
Returns:
|
||||
Float between 0.0 (wrong) and 1.0 (correct)
|
||||
|
||||
TODO: Implement your scoring metric. Common options:
|
||||
- Exact match: float(pred.strip().lower() == gt.strip().lower())
|
||||
- F1 score: compute token overlap
|
||||
- ANLS: for document QA tasks
|
||||
- Custom: any float in [0, 1]
|
||||
Reflect on rollout results and return patch dicts (or None entries).
|
||||
"""
|
||||
# Placeholder — exact match
|
||||
return float(prediction.strip().lower() == ground_truth.strip().lower())
|
||||
raise NotImplementedError("Implement reflect() for your benchmark")
|
||||
|
||||
def build_prompt(self, item, skill: str) -> str:
|
||||
"""Combine skill document with task input."""
|
||||
return f"{skill}\n\n---\n\nQuestion: {item.input}"
|
||||
|
||||
def parse_response(self, response: str) -> str:
|
||||
"""
|
||||
Extract the answer from the model's raw response.
|
||||
|
||||
TODO: Implement extraction logic. For example:
|
||||
- Extract text after "Answer:"
|
||||
- Parse JSON output
|
||||
- Extract from code blocks
|
||||
"""
|
||||
return response.strip()
|
||||
def get_task_types(self) -> list[str]:
|
||||
return ["your_benchmark"]
|
||||
|
||||
@@ -3,101 +3,38 @@ Benchmark Data Loader Template
|
||||
================================
|
||||
Copy this file and implement the TODO sections to load your benchmark data.
|
||||
|
||||
The DataLoader is responsible for:
|
||||
1. Loading raw data from disk
|
||||
2. Splitting into train / validation / test sets
|
||||
3. Providing DataItem objects to the training loop
|
||||
The SplitDataLoader is responsible for:
|
||||
1. Loading raw data from disk for ratio split mode
|
||||
2. Loading items from train/val/test directories for split_dir mode
|
||||
3. Returning list[dict] items used by the training loop
|
||||
"""
|
||||
from pathlib import Path
|
||||
from __future__ import annotations
|
||||
|
||||
from skillopt.datasets.base import SplitDataLoader
|
||||
|
||||
|
||||
class TemplateBenchmarkLoader:
|
||||
class TemplateBenchmarkDataLoader(SplitDataLoader):
|
||||
"""
|
||||
Data loader for <Your Benchmark Name>.
|
||||
|
||||
|
||||
Rename this class and implement the methods below.
|
||||
"""
|
||||
|
||||
def __init__(self, data_dir: str = "data/your_benchmark", **kwargs):
|
||||
self.data_dir = Path(data_dir)
|
||||
self.items = []
|
||||
self.splits = {}
|
||||
|
||||
def setup(self, cfg: dict):
|
||||
def load_raw_items(self, data_path: str) -> list[dict]:
|
||||
"""
|
||||
Initialize the loader with config.
|
||||
|
||||
Called once before training starts.
|
||||
|
||||
Args:
|
||||
cfg: Dict with keys like 'split_mode', 'train_ratio', 'val_ratio', etc.
|
||||
"""
|
||||
# Step 1: Load raw data
|
||||
self.items = self._load_items()
|
||||
Parse raw benchmark data for split_mode="ratio".
|
||||
|
||||
# Step 2: Create splits
|
||||
split_mode = cfg.get("split_mode", "ratio")
|
||||
if split_mode == "ratio":
|
||||
self._split_by_ratio(
|
||||
train_ratio=cfg.get("train_ratio", 0.7),
|
||||
val_ratio=cfg.get("val_ratio", 0.15),
|
||||
)
|
||||
elif split_mode == "split_dir":
|
||||
self._load_predefined_splits(cfg.get("split_dir", self.data_dir))
|
||||
|
||||
def _load_items(self) -> list:
|
||||
Return a list of normalized item dicts.
|
||||
"""
|
||||
Load raw data into structured items.
|
||||
|
||||
TODO: Implement data loading. Each item should have at minimum:
|
||||
- id: unique identifier
|
||||
- input: the task input (question, instruction, etc.)
|
||||
- ground_truth: the expected answer
|
||||
- metadata: optional dict with extra info
|
||||
|
||||
Example:
|
||||
items = []
|
||||
for path in self.data_dir.glob("*.json"):
|
||||
data = json.loads(path.read_text())
|
||||
for entry in data:
|
||||
items.append({
|
||||
"id": entry["id"],
|
||||
"input": entry["question"],
|
||||
"ground_truth": entry["answer"],
|
||||
"metadata": {"source": path.name},
|
||||
})
|
||||
return items
|
||||
"""
|
||||
raise NotImplementedError("Implement _load_items() for your benchmark")
|
||||
# TODO: parse your raw JSON/JSONL/CSV format and return list[dict]
|
||||
# with deterministic "id" values.
|
||||
return super().load_raw_items(data_path)
|
||||
|
||||
def _split_by_ratio(self, train_ratio: float, val_ratio: float):
|
||||
"""Split items by ratio."""
|
||||
import random
|
||||
random.shuffle(self.items)
|
||||
n = len(self.items)
|
||||
n_train = int(n * train_ratio)
|
||||
n_val = int(n * val_ratio)
|
||||
self.splits = {
|
||||
"train": self.items[:n_train],
|
||||
"valid": self.items[n_train:n_train + n_val],
|
||||
"test": self.items[n_train + n_val:],
|
||||
}
|
||||
|
||||
def _load_predefined_splits(self, split_dir):
|
||||
"""Load from pre-split directories."""
|
||||
# TODO: Implement if your benchmark has pre-defined splits
|
||||
raise NotImplementedError
|
||||
|
||||
def get_split_items(self, split: str) -> list:
|
||||
def load_split_items(self, split_path: str) -> list[dict]:
|
||||
"""
|
||||
Return items for a given split.
|
||||
|
||||
Args:
|
||||
split: One of "train", "valid", "test"
|
||||
|
||||
Returns:
|
||||
List of data items for the requested split
|
||||
Parse one split directory for split_mode="split_dir".
|
||||
|
||||
split_path points to train/, val/, or test/.
|
||||
"""
|
||||
if split not in self.splits:
|
||||
raise ValueError(f"Unknown split '{split}'. Available: {list(self.splits.keys())}")
|
||||
return self.splits[split]
|
||||
# TODO: customize when split directories contain non-standard files.
|
||||
return super().load_split_items(split_path)
|
||||
|
||||
+111
-2
@@ -6,6 +6,7 @@ from typing import Any
|
||||
|
||||
from skillopt.model import azure_openai as _openai
|
||||
from skillopt.model import claude_backend as _claude
|
||||
from skillopt.model import minimax_backend as _minimax
|
||||
from skillopt.model import qwen_backend as _qwen
|
||||
from skillopt.model.backend_config import ( # noqa: F401
|
||||
configure_claude_code_exec,
|
||||
@@ -50,6 +51,10 @@ def set_backend(name: str | None) -> str:
|
||||
set_optimizer_backend("openai_chat")
|
||||
set_target_backend("qwen_chat")
|
||||
return "qwen_chat"
|
||||
if normalized in {"minimax", "minimax_chat"}:
|
||||
set_optimizer_backend("openai_chat")
|
||||
set_target_backend("minimax_chat")
|
||||
return "minimax_chat"
|
||||
raise ValueError(f"Unsupported legacy backend: {name!r}")
|
||||
|
||||
|
||||
@@ -59,12 +64,16 @@ def get_backend_name() -> str:
|
||||
target = get_target_backend()
|
||||
if optimizer == "claude_chat" and target == "claude_chat":
|
||||
return "claude_chat"
|
||||
if optimizer == "qwen_chat" and target == "qwen_chat":
|
||||
return "qwen_chat"
|
||||
if optimizer == "openai_chat" and target == "openai_chat":
|
||||
return "azure_openai"
|
||||
if optimizer == "openai_chat" and target == "codex_exec":
|
||||
return "codex"
|
||||
if optimizer == "openai_chat" and target == "qwen_chat":
|
||||
return "qwen_chat"
|
||||
if optimizer == "openai_chat" and target == "minimax_chat":
|
||||
return "minimax_chat"
|
||||
return f"{optimizer}+{target}"
|
||||
|
||||
|
||||
@@ -86,6 +95,16 @@ def chat_optimizer(
|
||||
stage=stage,
|
||||
timeout=timeout,
|
||||
)
|
||||
if get_optimizer_backend() == "qwen_chat":
|
||||
return _qwen.chat_optimizer(
|
||||
system=system,
|
||||
user=user,
|
||||
max_completion_tokens=max_completion_tokens,
|
||||
retries=retries,
|
||||
stage=stage,
|
||||
reasoning_effort=reasoning_effort,
|
||||
timeout=timeout,
|
||||
)
|
||||
return _openai.chat_optimizer(
|
||||
system=system,
|
||||
user=user,
|
||||
@@ -124,9 +143,18 @@ def chat_target(
|
||||
stage=stage,
|
||||
reasoning_effort=reasoning_effort,
|
||||
)
|
||||
if get_target_backend() == "minimax_chat":
|
||||
return _minimax.chat_target(
|
||||
system=system,
|
||||
user=user,
|
||||
max_completion_tokens=max_completion_tokens,
|
||||
retries=retries,
|
||||
stage=stage,
|
||||
reasoning_effort=reasoning_effort,
|
||||
)
|
||||
if not is_target_chat_backend():
|
||||
raise NotImplementedError(
|
||||
"chat_target is only supported with target_backend=openai_chat, claude_chat, or qwen_chat. "
|
||||
"chat_target is only supported with target_backend=openai_chat, claude_chat, qwen_chat, or minimax_chat. "
|
||||
"Exec backends are handled in environment-specific rollout code."
|
||||
)
|
||||
return _openai.chat_target(
|
||||
@@ -163,6 +191,18 @@ def chat_optimizer_messages(
|
||||
return_message=return_message,
|
||||
timeout=timeout,
|
||||
)
|
||||
if get_optimizer_backend() == "qwen_chat":
|
||||
return _qwen.chat_optimizer_messages(
|
||||
messages=messages,
|
||||
max_completion_tokens=max_completion_tokens,
|
||||
retries=retries,
|
||||
stage=stage,
|
||||
reasoning_effort=reasoning_effort,
|
||||
tools=tools,
|
||||
tool_choice=tool_choice,
|
||||
return_message=return_message,
|
||||
timeout=timeout,
|
||||
)
|
||||
return _openai.chat_optimizer_messages(
|
||||
messages=messages,
|
||||
max_completion_tokens=max_completion_tokens,
|
||||
@@ -210,9 +250,20 @@ def chat_target_messages(
|
||||
tool_choice=tool_choice,
|
||||
return_message=return_message,
|
||||
)
|
||||
if get_target_backend() == "minimax_chat":
|
||||
return _minimax.chat_target_messages(
|
||||
messages=messages,
|
||||
max_completion_tokens=max_completion_tokens,
|
||||
retries=retries,
|
||||
stage=stage,
|
||||
reasoning_effort=reasoning_effort,
|
||||
tools=tools,
|
||||
tool_choice=tool_choice,
|
||||
return_message=return_message,
|
||||
)
|
||||
if not is_target_chat_backend():
|
||||
raise NotImplementedError(
|
||||
"chat_target_messages is only supported with target_backend=openai_chat, claude_chat, or qwen_chat. "
|
||||
"chat_target_messages is only supported with target_backend=openai_chat, claude_chat, qwen_chat, or minimax_chat. "
|
||||
"Exec backends are handled in environment-specific rollout code."
|
||||
)
|
||||
return _openai.chat_target_messages(
|
||||
@@ -301,6 +352,17 @@ def get_token_summary() -> dict:
|
||||
summary[stage]["prompt_tokens"] += values["prompt_tokens"]
|
||||
summary[stage]["completion_tokens"] += values["completion_tokens"]
|
||||
summary[stage]["total_tokens"] += values["total_tokens"]
|
||||
minimax_summary = _minimax.get_token_summary()
|
||||
for stage, values in minimax_summary.items():
|
||||
if stage == "_total":
|
||||
continue
|
||||
if stage not in summary:
|
||||
summary[stage] = values
|
||||
continue
|
||||
summary[stage]["calls"] += values["calls"]
|
||||
summary[stage]["prompt_tokens"] += values["prompt_tokens"]
|
||||
summary[stage]["completion_tokens"] += values["completion_tokens"]
|
||||
summary[stage]["total_tokens"] += values["total_tokens"]
|
||||
total = {
|
||||
"calls": 0,
|
||||
"prompt_tokens": 0,
|
||||
@@ -322,6 +384,7 @@ def reset_token_tracker() -> None:
|
||||
_openai.reset_token_tracker()
|
||||
_claude.reset_token_tracker()
|
||||
_qwen.reset_token_tracker()
|
||||
_minimax.reset_token_tracker()
|
||||
|
||||
|
||||
def configure_azure_openai(
|
||||
@@ -375,6 +438,18 @@ def configure_qwen_chat(
|
||||
timeout_seconds: float | str | None = None,
|
||||
max_tokens: int | str | None = None,
|
||||
enable_thinking: bool | str | None = None,
|
||||
optimizer_base_url: str | None = None,
|
||||
optimizer_api_key: str | None = None,
|
||||
optimizer_temperature: float | str | None = None,
|
||||
optimizer_timeout_seconds: float | str | None = None,
|
||||
optimizer_max_tokens: int | str | None = None,
|
||||
optimizer_enable_thinking: bool | str | None = None,
|
||||
target_base_url: str | None = None,
|
||||
target_api_key: str | None = None,
|
||||
target_temperature: float | str | None = None,
|
||||
target_timeout_seconds: float | str | None = None,
|
||||
target_max_tokens: int | str | None = None,
|
||||
target_enable_thinking: bool | str | None = None,
|
||||
) -> None:
|
||||
_qwen.configure_qwen_chat(
|
||||
base_url=base_url,
|
||||
@@ -383,6 +458,37 @@ def configure_qwen_chat(
|
||||
timeout_seconds=timeout_seconds,
|
||||
max_tokens=max_tokens,
|
||||
enable_thinking=enable_thinking,
|
||||
optimizer_base_url=optimizer_base_url,
|
||||
optimizer_api_key=optimizer_api_key,
|
||||
optimizer_temperature=optimizer_temperature,
|
||||
optimizer_timeout_seconds=optimizer_timeout_seconds,
|
||||
optimizer_max_tokens=optimizer_max_tokens,
|
||||
optimizer_enable_thinking=optimizer_enable_thinking,
|
||||
target_base_url=target_base_url,
|
||||
target_api_key=target_api_key,
|
||||
target_temperature=target_temperature,
|
||||
target_timeout_seconds=target_timeout_seconds,
|
||||
target_max_tokens=target_max_tokens,
|
||||
target_enable_thinking=target_enable_thinking,
|
||||
)
|
||||
|
||||
|
||||
def configure_minimax_chat(
|
||||
*,
|
||||
base_url: str | None = None,
|
||||
api_key: str | None = None,
|
||||
temperature: float | str | None = None,
|
||||
timeout_seconds: float | str | None = None,
|
||||
max_tokens: int | str | None = None,
|
||||
enable_thinking: bool | str | None = None,
|
||||
) -> None:
|
||||
_minimax.configure_minimax_chat(
|
||||
base_url=base_url,
|
||||
api_key=api_key,
|
||||
temperature=temperature,
|
||||
timeout_seconds=timeout_seconds,
|
||||
max_tokens=max_tokens,
|
||||
enable_thinking=enable_thinking,
|
||||
)
|
||||
|
||||
|
||||
@@ -390,14 +496,17 @@ def set_reasoning_effort(effort: str | None) -> None:
|
||||
_openai.set_reasoning_effort(effort)
|
||||
_claude.set_reasoning_effort(effort)
|
||||
_qwen.set_reasoning_effort(effort)
|
||||
_minimax.set_reasoning_effort(effort)
|
||||
|
||||
|
||||
def set_target_deployment(deployment: str) -> None:
|
||||
_openai.set_target_deployment(deployment)
|
||||
_claude.set_target_deployment(deployment)
|
||||
_qwen.set_target_deployment(deployment)
|
||||
_minimax.set_target_deployment(deployment)
|
||||
|
||||
|
||||
def set_optimizer_deployment(deployment: str) -> None:
|
||||
_openai.set_optimizer_deployment(deployment)
|
||||
_claude.set_optimizer_deployment(deployment)
|
||||
_qwen.set_optimizer_deployment(deployment)
|
||||
|
||||
@@ -336,9 +336,10 @@ def get_target_client() -> AzureOpenAI | OpenAI:
|
||||
from skillopt.model.backend_config import get_target_backend
|
||||
if get_target_backend() == "qwen_chat":
|
||||
from skillopt.model import qwen_backend as _qwen
|
||||
target_config = _qwen.TARGET_CONFIG
|
||||
_target_client = OpenAI(
|
||||
base_url=_qwen.BASE_URL,
|
||||
api_key=_qwen.API_KEY or "dummy",
|
||||
base_url=target_config.base_url,
|
||||
api_key=target_config.api_key or "dummy",
|
||||
)
|
||||
else:
|
||||
_target_client = _make_client("target")
|
||||
|
||||
@@ -49,10 +49,10 @@ CLAUDE_CODE_EXEC_MAX_THINKING_TOKENS = max(
|
||||
def set_optimizer_backend(backend: str) -> None:
|
||||
global OPTIMIZER_BACKEND
|
||||
OPTIMIZER_BACKEND = normalize_backend_name(backend or "openai_chat")
|
||||
if OPTIMIZER_BACKEND not in {"openai_chat", "claude_chat"}:
|
||||
if OPTIMIZER_BACKEND not in {"openai_chat", "claude_chat", "qwen_chat", "minimax_chat"}:
|
||||
raise ValueError(
|
||||
f"Unsupported optimizer backend: {OPTIMIZER_BACKEND!r}. "
|
||||
"Supported values are 'openai_chat' and 'claude_chat'."
|
||||
"Supported values are 'openai_chat', 'claude_chat', 'qwen_chat', and 'minimax_chat'."
|
||||
)
|
||||
os.environ["OPTIMIZER_BACKEND"] = OPTIMIZER_BACKEND
|
||||
|
||||
@@ -64,10 +64,10 @@ def get_optimizer_backend() -> str:
|
||||
def set_target_backend(backend: str) -> None:
|
||||
global TARGET_BACKEND
|
||||
TARGET_BACKEND = normalize_backend_name(backend or "openai_chat")
|
||||
if TARGET_BACKEND not in {"openai_chat", "claude_chat", "qwen_chat", "codex_exec", "claude_code_exec"}:
|
||||
if TARGET_BACKEND not in {"openai_chat", "claude_chat", "qwen_chat", "minimax_chat", "codex_exec", "claude_code_exec"}:
|
||||
raise ValueError(
|
||||
f"Unsupported target backend: {TARGET_BACKEND!r}. "
|
||||
"Supported values are 'openai_chat', 'claude_chat', 'qwen_chat', 'codex_exec', and 'claude_code_exec'."
|
||||
"Supported values are 'openai_chat', 'claude_chat', 'qwen_chat', 'minimax_chat', 'codex_exec', and 'claude_code_exec'."
|
||||
)
|
||||
os.environ["TARGET_BACKEND"] = TARGET_BACKEND
|
||||
|
||||
@@ -81,11 +81,11 @@ def is_target_exec_backend() -> bool:
|
||||
|
||||
|
||||
def is_optimizer_chat_backend() -> bool:
|
||||
return OPTIMIZER_BACKEND in {"openai_chat", "claude_chat"}
|
||||
return OPTIMIZER_BACKEND in {"openai_chat", "claude_chat", "qwen_chat", "minimax_chat"}
|
||||
|
||||
|
||||
def is_target_chat_backend() -> bool:
|
||||
return TARGET_BACKEND in {"openai_chat", "claude_chat", "qwen_chat"}
|
||||
return TARGET_BACKEND in {"openai_chat", "claude_chat", "qwen_chat", "minimax_chat"}
|
||||
|
||||
|
||||
def configure_codex_exec(
|
||||
|
||||
@@ -25,6 +25,7 @@ _BACKEND_DEFAULT_MODELS = {
|
||||
"claude_chat": "claude-sonnet-4-6",
|
||||
"claude_code_exec": "claude-sonnet-4-6",
|
||||
"qwen_chat": "Qwen/Qwen3.5-4B",
|
||||
"minimax_chat": "MiniMax-M2.7",
|
||||
}
|
||||
|
||||
_BACKEND_ALIASES = {
|
||||
@@ -41,6 +42,8 @@ _BACKEND_ALIASES = {
|
||||
"anthropic": "claude_chat",
|
||||
"qwen": "qwen_chat",
|
||||
"qwen_chat": "qwen_chat",
|
||||
"minimax": "minimax_chat",
|
||||
"minimax_chat": "minimax_chat",
|
||||
}
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,277 @@
|
||||
"""OpenAI-compatible MiniMax chat backend for the target path."""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import threading
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from typing import Any
|
||||
|
||||
from skillopt.model.common import (
|
||||
CompatAssistantMessage,
|
||||
CompatToolCall,
|
||||
CompatToolFunction,
|
||||
TokenTracker,
|
||||
default_model_for_backend,
|
||||
)
|
||||
|
||||
BASE_URL = os.environ.get("MINIMAX_BASE_URL", "https://api.minimax.io/v1")
|
||||
API_KEY = os.environ.get("MINIMAX_API_KEY", "")
|
||||
TIMEOUT_SECONDS = float(os.environ.get("MINIMAX_TIMEOUT_SECONDS", "300") or 300)
|
||||
MAX_TOKENS = int(os.environ.get("MINIMAX_MAX_TOKENS", "8000") or 8000)
|
||||
TEMPERATURE: float | None = None
|
||||
_raw_temperature = os.environ.get("MINIMAX_TEMPERATURE", "0.7").strip()
|
||||
if _raw_temperature:
|
||||
TEMPERATURE = float(_raw_temperature)
|
||||
ENABLE_THINKING = os.environ.get("MINIMAX_ENABLE_THINKING", "false").strip().lower() in {
|
||||
"1",
|
||||
"true",
|
||||
"yes",
|
||||
"on",
|
||||
}
|
||||
|
||||
TARGET_DEPLOYMENT = os.environ.get(
|
||||
"TARGET_DEPLOYMENT",
|
||||
default_model_for_backend("minimax_chat"),
|
||||
)
|
||||
|
||||
_config_lock = threading.Lock()
|
||||
tracker = TokenTracker()
|
||||
|
||||
|
||||
def _chat_url() -> str:
|
||||
base = BASE_URL.rstrip("/")
|
||||
if base.endswith("/chat/completions"):
|
||||
return base
|
||||
return f"{base}/chat/completions"
|
||||
|
||||
|
||||
def _json_safe(value: Any) -> Any:
|
||||
if value is None or isinstance(value, (str, int, float, bool)):
|
||||
return value
|
||||
if isinstance(value, list):
|
||||
return [_json_safe(item) for item in value]
|
||||
if isinstance(value, dict):
|
||||
return {str(key): _json_safe(val) for key, val in value.items()}
|
||||
model_dump = getattr(value, "model_dump", None)
|
||||
if callable(model_dump):
|
||||
try:
|
||||
return model_dump(mode="json")
|
||||
except TypeError:
|
||||
return model_dump()
|
||||
return str(value)
|
||||
|
||||
|
||||
def _usage_from_payload(payload: dict[str, Any]) -> dict[str, int]:
|
||||
usage = payload.get("usage") or {}
|
||||
prompt_tokens = int(usage.get("prompt_tokens") or usage.get("input_tokens") or 0)
|
||||
completion_tokens = int(usage.get("completion_tokens") or usage.get("output_tokens") or 0)
|
||||
total_tokens = int(usage.get("total_tokens") or (prompt_tokens + completion_tokens))
|
||||
return {
|
||||
"prompt_tokens": prompt_tokens,
|
||||
"completion_tokens": completion_tokens,
|
||||
"total_tokens": total_tokens,
|
||||
}
|
||||
|
||||
|
||||
def _compat_message_from_payload(message: dict[str, Any], choice: dict[str, Any]) -> CompatAssistantMessage:
|
||||
content = message.get("content") or ""
|
||||
if not isinstance(content, str):
|
||||
content = json.dumps(content, ensure_ascii=False)
|
||||
tool_calls: list[CompatToolCall] = []
|
||||
for index, tool_call in enumerate(message.get("tool_calls") or [], start=1):
|
||||
function = tool_call.get("function") or {}
|
||||
tool_calls.append(
|
||||
CompatToolCall(
|
||||
id=str(tool_call.get("id") or f"minimax_tool_{index}"),
|
||||
type=str(tool_call.get("type") or "function"),
|
||||
function=CompatToolFunction(
|
||||
name=str(function.get("name") or ""),
|
||||
arguments=str(function.get("arguments") or "{}"),
|
||||
),
|
||||
)
|
||||
)
|
||||
return CompatAssistantMessage(
|
||||
content=content,
|
||||
tool_calls=tool_calls,
|
||||
metadata={
|
||||
"finish_reason": choice.get("finish_reason"),
|
||||
"choice0": _json_safe(choice),
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def _post_chat_completion(payload: dict[str, Any], timeout: float | None) -> dict[str, Any]:
|
||||
headers = {"Content-Type": "application/json"}
|
||||
if API_KEY:
|
||||
headers["Authorization"] = f"Bearer {API_KEY}"
|
||||
req = urllib.request.Request(
|
||||
_chat_url(),
|
||||
data=json.dumps(payload, ensure_ascii=False).encode("utf-8"),
|
||||
headers=headers,
|
||||
method="POST",
|
||||
)
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=timeout or TIMEOUT_SECONDS) as resp:
|
||||
raw = resp.read().decode("utf-8")
|
||||
except urllib.error.HTTPError as e:
|
||||
body = e.read().decode("utf-8", errors="replace")
|
||||
raise RuntimeError(f"MiniMax chat API returned HTTP {e.code}: {body}") from e
|
||||
except urllib.error.URLError as e:
|
||||
raise RuntimeError(f"MiniMax chat API request failed: {e}") from e
|
||||
try:
|
||||
return json.loads(raw)
|
||||
except json.JSONDecodeError as e:
|
||||
raise RuntimeError(f"MiniMax chat API returned non-JSON response: {raw[:1000]}") from e
|
||||
|
||||
|
||||
def _chat_messages_impl(
|
||||
messages: list[dict[str, Any]],
|
||||
max_completion_tokens: int,
|
||||
retries: int,
|
||||
stage: str,
|
||||
*,
|
||||
tools: list[dict[str, Any]] | None = None,
|
||||
tool_choice: str | dict[str, Any] | None = None,
|
||||
return_message: bool = False,
|
||||
deployment: str | None = None,
|
||||
timeout: float | None = None,
|
||||
) -> tuple[Any, dict[str, int]]:
|
||||
payload: dict[str, Any] = {
|
||||
"model": deployment or TARGET_DEPLOYMENT,
|
||||
"messages": _json_safe(messages),
|
||||
"max_tokens": min(max_completion_tokens, MAX_TOKENS),
|
||||
}
|
||||
payload["chat_template_kwargs"] = {"enable_thinking": ENABLE_THINKING}
|
||||
if TEMPERATURE is not None:
|
||||
payload["temperature"] = TEMPERATURE
|
||||
if tools:
|
||||
payload["tools"] = _json_safe(tools)
|
||||
if tool_choice is not None:
|
||||
payload["tool_choice"] = _json_safe(tool_choice)
|
||||
|
||||
last_err: Exception | None = None
|
||||
for attempt in range(retries):
|
||||
try:
|
||||
data = _post_chat_completion(payload, timeout)
|
||||
choices = data.get("choices") or []
|
||||
if not choices:
|
||||
raise RuntimeError(f"MiniMax chat API returned no choices: {data}")
|
||||
choice0 = choices[0]
|
||||
message = choice0.get("message") or {}
|
||||
text = message.get("content") or ""
|
||||
if not isinstance(text, str):
|
||||
text = json.dumps(text, ensure_ascii=False)
|
||||
usage_info = _usage_from_payload(data)
|
||||
tracker.record(stage, usage_info["prompt_tokens"], usage_info["completion_tokens"])
|
||||
if return_message:
|
||||
return _compat_message_from_payload(message, choice0), usage_info
|
||||
return text, usage_info
|
||||
except Exception as e: # noqa: BLE001
|
||||
last_err = e
|
||||
time.sleep(min(2 ** attempt, 30))
|
||||
raise RuntimeError(f"MiniMax chat call failed after {retries} retries: {last_err}")
|
||||
|
||||
|
||||
def configure_minimax_chat(
|
||||
*,
|
||||
base_url: str | None = None,
|
||||
api_key: str | None = None,
|
||||
temperature: float | str | None = None,
|
||||
timeout_seconds: float | str | None = None,
|
||||
max_tokens: int | str | None = None,
|
||||
enable_thinking: bool | str | None = None,
|
||||
) -> None:
|
||||
global BASE_URL, API_KEY, TEMPERATURE, TIMEOUT_SECONDS, MAX_TOKENS, ENABLE_THINKING
|
||||
with _config_lock:
|
||||
if base_url is not None:
|
||||
BASE_URL = str(base_url).strip() or BASE_URL
|
||||
os.environ["MINIMAX_BASE_URL"] = BASE_URL
|
||||
if api_key is not None:
|
||||
API_KEY = str(api_key).strip()
|
||||
os.environ["MINIMAX_API_KEY"] = API_KEY
|
||||
if temperature is not None:
|
||||
raw = str(temperature).strip()
|
||||
TEMPERATURE = float(raw) if raw else None
|
||||
os.environ["MINIMAX_TEMPERATURE"] = raw
|
||||
if timeout_seconds is not None:
|
||||
TIMEOUT_SECONDS = float(timeout_seconds)
|
||||
os.environ["MINIMAX_TIMEOUT_SECONDS"] = str(timeout_seconds)
|
||||
if max_tokens is not None:
|
||||
MAX_TOKENS = int(max_tokens)
|
||||
os.environ["MINIMAX_MAX_TOKENS"] = str(max_tokens)
|
||||
if enable_thinking is not None:
|
||||
if isinstance(enable_thinking, str):
|
||||
ENABLE_THINKING = enable_thinking.strip().lower() in {"1", "true", "yes", "on"}
|
||||
else:
|
||||
ENABLE_THINKING = bool(enable_thinking)
|
||||
os.environ["MINIMAX_ENABLE_THINKING"] = "true" if ENABLE_THINKING else "false"
|
||||
|
||||
|
||||
def get_max_tokens() -> int:
|
||||
return MAX_TOKENS
|
||||
|
||||
|
||||
def chat_target(
|
||||
system: str,
|
||||
user: str,
|
||||
max_completion_tokens: int = 16384,
|
||||
retries: int = 5,
|
||||
stage: str = "target",
|
||||
reasoning_effort: str | None = None,
|
||||
timeout: float | None = None,
|
||||
) -> tuple[str, dict[str, int]]:
|
||||
del reasoning_effort
|
||||
messages = [{"role": "system", "content": system}, {"role": "user", "content": user}]
|
||||
return _chat_messages_impl(
|
||||
messages,
|
||||
max_completion_tokens,
|
||||
retries,
|
||||
stage,
|
||||
timeout=timeout,
|
||||
)
|
||||
|
||||
|
||||
def chat_target_messages(
|
||||
messages: list[dict[str, Any]],
|
||||
max_completion_tokens: int = 16384,
|
||||
retries: int = 5,
|
||||
stage: str = "target",
|
||||
reasoning_effort: str | None = None,
|
||||
*,
|
||||
tools: list[dict[str, Any]] | None = None,
|
||||
tool_choice: str | dict[str, Any] | None = None,
|
||||
return_message: bool = False,
|
||||
timeout: float | None = None,
|
||||
) -> tuple[Any, dict[str, int]]:
|
||||
del reasoning_effort
|
||||
return _chat_messages_impl(
|
||||
messages,
|
||||
max_completion_tokens,
|
||||
retries,
|
||||
stage,
|
||||
tools=tools,
|
||||
tool_choice=tool_choice,
|
||||
return_message=return_message,
|
||||
timeout=timeout,
|
||||
)
|
||||
|
||||
|
||||
def get_token_summary() -> dict[str, dict[str, int]]:
|
||||
return tracker.summary()
|
||||
|
||||
|
||||
def reset_token_tracker() -> None:
|
||||
tracker.reset()
|
||||
|
||||
|
||||
def set_reasoning_effort(effort: str | None) -> None:
|
||||
del effort
|
||||
|
||||
|
||||
def set_target_deployment(deployment: str) -> None:
|
||||
global TARGET_DEPLOYMENT
|
||||
TARGET_DEPLOYMENT = deployment or default_model_for_backend("minimax_chat")
|
||||
os.environ["TARGET_DEPLOYMENT"] = TARGET_DEPLOYMENT
|
||||
+229
-51
@@ -1,6 +1,7 @@
|
||||
"""OpenAI-compatible Qwen chat backend for the target path."""
|
||||
"""OpenAI-compatible Qwen chat backend for optimizer and target paths."""
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
import json
|
||||
import os
|
||||
import threading
|
||||
@@ -17,32 +18,72 @@ from skillopt.model.common import (
|
||||
default_model_for_backend,
|
||||
)
|
||||
|
||||
BASE_URL = os.environ.get("QWEN_CHAT_BASE_URL", "http://localhost:8000/v1")
|
||||
API_KEY = os.environ.get("QWEN_CHAT_API_KEY", "")
|
||||
TIMEOUT_SECONDS = float(os.environ.get("QWEN_CHAT_TIMEOUT_SECONDS", "300") or 300)
|
||||
MAX_TOKENS = int(os.environ.get("QWEN_CHAT_MAX_TOKENS", "8000") or 8000)
|
||||
TEMPERATURE: float | None = None
|
||||
_raw_temperature = os.environ.get("QWEN_CHAT_TEMPERATURE", "0.7").strip()
|
||||
if _raw_temperature:
|
||||
TEMPERATURE = float(_raw_temperature)
|
||||
ENABLE_THINKING = os.environ.get("QWEN_CHAT_ENABLE_THINKING", "false").strip().lower() in {
|
||||
"1",
|
||||
"true",
|
||||
"yes",
|
||||
"on",
|
||||
}
|
||||
|
||||
TARGET_DEPLOYMENT = os.environ.get(
|
||||
"TARGET_DEPLOYMENT",
|
||||
default_model_for_backend("qwen_chat"),
|
||||
)
|
||||
@dataclass
|
||||
class QwenChatConfig:
|
||||
base_url: str
|
||||
api_key: str
|
||||
timeout_seconds: float
|
||||
max_tokens: int
|
||||
temperature: float | None
|
||||
enable_thinking: bool
|
||||
deployment: str
|
||||
|
||||
|
||||
def _parse_bool(value: Any, default: bool = False) -> bool:
|
||||
if value is None:
|
||||
return default
|
||||
return str(value).strip().lower() in {"1", "true", "yes", "on"}
|
||||
|
||||
|
||||
def _parse_optional_float(value: Any) -> float | None:
|
||||
if value is None:
|
||||
return None
|
||||
raw = str(value).strip()
|
||||
return float(raw) if raw else None
|
||||
|
||||
|
||||
def _parse_int(value: Any, default: int) -> int:
|
||||
if value is None:
|
||||
return default
|
||||
raw = str(value).strip()
|
||||
return int(raw) if raw else default
|
||||
|
||||
|
||||
def _role_env(role: str, key: str, default: str) -> str:
|
||||
role_key = f"{role.upper()}_QWEN_CHAT_{key}"
|
||||
generic_key = f"QWEN_CHAT_{key}"
|
||||
return os.environ.get(role_key) or os.environ.get(generic_key) or default
|
||||
|
||||
|
||||
def _initial_config(role: str) -> QwenChatConfig:
|
||||
role_upper = role.upper()
|
||||
deployment_env = "OPTIMIZER_DEPLOYMENT" if role == "optimizer" else "TARGET_DEPLOYMENT"
|
||||
return QwenChatConfig(
|
||||
base_url=_role_env(role, "BASE_URL", "http://localhost:8000/v1"),
|
||||
api_key=_role_env(role, "API_KEY", ""),
|
||||
timeout_seconds=float(_role_env(role, "TIMEOUT_SECONDS", "300") or 300),
|
||||
max_tokens=_parse_int(_role_env(role, "MAX_TOKENS", "8000"), 8000),
|
||||
temperature=_parse_optional_float(_role_env(role, "TEMPERATURE", "0.7")),
|
||||
enable_thinking=_parse_bool(_role_env(role, "ENABLE_THINKING", "false")),
|
||||
deployment=(
|
||||
os.environ.get(f"{role_upper}_QWEN_CHAT_MODEL")
|
||||
or os.environ.get("QWEN_CHAT_MODEL")
|
||||
or os.environ.get(deployment_env)
|
||||
or default_model_for_backend("qwen_chat")
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
OPTIMIZER_CONFIG = _initial_config("optimizer")
|
||||
TARGET_CONFIG = _initial_config("target")
|
||||
|
||||
_config_lock = threading.Lock()
|
||||
tracker = TokenTracker()
|
||||
|
||||
|
||||
def _chat_url() -> str:
|
||||
base = BASE_URL.rstrip("/")
|
||||
def _chat_url(config: QwenChatConfig) -> str:
|
||||
base = config.base_url.rstrip("/")
|
||||
if base.endswith("/chat/completions"):
|
||||
return base
|
||||
return f"{base}/chat/completions"
|
||||
@@ -103,18 +144,22 @@ def _compat_message_from_payload(message: dict[str, Any], choice: dict[str, Any]
|
||||
)
|
||||
|
||||
|
||||
def _post_chat_completion(payload: dict[str, Any], timeout: float | None) -> dict[str, Any]:
|
||||
def _post_chat_completion(
|
||||
payload: dict[str, Any],
|
||||
timeout: float | None,
|
||||
config: QwenChatConfig,
|
||||
) -> dict[str, Any]:
|
||||
headers = {"Content-Type": "application/json"}
|
||||
if API_KEY:
|
||||
headers["Authorization"] = f"Bearer {API_KEY}"
|
||||
if config.api_key:
|
||||
headers["Authorization"] = f"Bearer {config.api_key}"
|
||||
req = urllib.request.Request(
|
||||
_chat_url(),
|
||||
_chat_url(config),
|
||||
data=json.dumps(payload, ensure_ascii=False).encode("utf-8"),
|
||||
headers=headers,
|
||||
method="POST",
|
||||
)
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=timeout or TIMEOUT_SECONDS) as resp:
|
||||
with urllib.request.urlopen(req, timeout=timeout or config.timeout_seconds) as resp:
|
||||
raw = resp.read().decode("utf-8")
|
||||
except urllib.error.HTTPError as e:
|
||||
body = e.read().decode("utf-8", errors="replace")
|
||||
@@ -133,20 +178,22 @@ def _chat_messages_impl(
|
||||
retries: int,
|
||||
stage: str,
|
||||
*,
|
||||
role: str,
|
||||
tools: list[dict[str, Any]] | None = None,
|
||||
tool_choice: str | dict[str, Any] | None = None,
|
||||
return_message: bool = False,
|
||||
deployment: str | None = None,
|
||||
timeout: float | None = None,
|
||||
) -> tuple[Any, dict[str, int]]:
|
||||
config = OPTIMIZER_CONFIG if role == "optimizer" else TARGET_CONFIG
|
||||
payload: dict[str, Any] = {
|
||||
"model": deployment or TARGET_DEPLOYMENT,
|
||||
"model": deployment or config.deployment,
|
||||
"messages": _json_safe(messages),
|
||||
"max_tokens": min(max_completion_tokens, MAX_TOKENS),
|
||||
"max_tokens": min(max_completion_tokens, config.max_tokens),
|
||||
}
|
||||
payload["chat_template_kwargs"] = {"enable_thinking": ENABLE_THINKING}
|
||||
if TEMPERATURE is not None:
|
||||
payload["temperature"] = TEMPERATURE
|
||||
payload["chat_template_kwargs"] = {"enable_thinking": config.enable_thinking}
|
||||
if config.temperature is not None:
|
||||
payload["temperature"] = config.temperature
|
||||
if tools:
|
||||
payload["tools"] = _json_safe(tools)
|
||||
if tool_choice is not None:
|
||||
@@ -155,7 +202,7 @@ def _chat_messages_impl(
|
||||
last_err: Exception | None = None
|
||||
for attempt in range(retries):
|
||||
try:
|
||||
data = _post_chat_completion(payload, timeout)
|
||||
data = _post_chat_completion(payload, timeout, config)
|
||||
choices = data.get("choices") or []
|
||||
if not choices:
|
||||
raise RuntimeError(f"Qwen chat API returned no choices: {data}")
|
||||
@@ -183,35 +230,134 @@ def configure_qwen_chat(
|
||||
timeout_seconds: float | str | None = None,
|
||||
max_tokens: int | str | None = None,
|
||||
enable_thinking: bool | str | None = None,
|
||||
optimizer_base_url: str | None = None,
|
||||
optimizer_api_key: str | None = None,
|
||||
optimizer_temperature: float | str | None = None,
|
||||
optimizer_timeout_seconds: float | str | None = None,
|
||||
optimizer_max_tokens: int | str | None = None,
|
||||
optimizer_enable_thinking: bool | str | None = None,
|
||||
target_base_url: str | None = None,
|
||||
target_api_key: str | None = None,
|
||||
target_temperature: float | str | None = None,
|
||||
target_timeout_seconds: float | str | None = None,
|
||||
target_max_tokens: int | str | None = None,
|
||||
target_enable_thinking: bool | str | None = None,
|
||||
) -> None:
|
||||
global BASE_URL, API_KEY, TEMPERATURE, TIMEOUT_SECONDS, MAX_TOKENS, ENABLE_THINKING
|
||||
with _config_lock:
|
||||
if base_url is not None:
|
||||
BASE_URL = str(base_url).strip() or BASE_URL
|
||||
os.environ["QWEN_CHAT_BASE_URL"] = BASE_URL
|
||||
os.environ["QWEN_CHAT_BASE_URL"] = str(base_url).strip()
|
||||
if api_key is not None:
|
||||
API_KEY = str(api_key).strip()
|
||||
os.environ["QWEN_CHAT_API_KEY"] = API_KEY
|
||||
os.environ["QWEN_CHAT_API_KEY"] = str(api_key).strip()
|
||||
if temperature is not None:
|
||||
raw = str(temperature).strip()
|
||||
TEMPERATURE = float(raw) if raw else None
|
||||
os.environ["QWEN_CHAT_TEMPERATURE"] = raw
|
||||
os.environ["QWEN_CHAT_TEMPERATURE"] = str(temperature).strip()
|
||||
if timeout_seconds is not None:
|
||||
TIMEOUT_SECONDS = float(timeout_seconds)
|
||||
os.environ["QWEN_CHAT_TIMEOUT_SECONDS"] = str(timeout_seconds)
|
||||
if max_tokens is not None:
|
||||
MAX_TOKENS = int(max_tokens)
|
||||
os.environ["QWEN_CHAT_MAX_TOKENS"] = str(max_tokens)
|
||||
if enable_thinking is not None:
|
||||
if isinstance(enable_thinking, str):
|
||||
ENABLE_THINKING = enable_thinking.strip().lower() in {"1", "true", "yes", "on"}
|
||||
else:
|
||||
ENABLE_THINKING = bool(enable_thinking)
|
||||
os.environ["QWEN_CHAT_ENABLE_THINKING"] = "true" if ENABLE_THINKING else "false"
|
||||
os.environ["QWEN_CHAT_ENABLE_THINKING"] = (
|
||||
"true" if _parse_bool(enable_thinking) else "false"
|
||||
)
|
||||
_update_config(
|
||||
OPTIMIZER_CONFIG,
|
||||
"optimizer",
|
||||
base_url=optimizer_base_url if optimizer_base_url is not None else base_url,
|
||||
api_key=optimizer_api_key if optimizer_api_key is not None else api_key,
|
||||
temperature=(
|
||||
optimizer_temperature
|
||||
if optimizer_temperature is not None
|
||||
else temperature
|
||||
),
|
||||
timeout_seconds=(
|
||||
optimizer_timeout_seconds
|
||||
if optimizer_timeout_seconds is not None
|
||||
else timeout_seconds
|
||||
),
|
||||
max_tokens=optimizer_max_tokens if optimizer_max_tokens is not None else max_tokens,
|
||||
enable_thinking=(
|
||||
optimizer_enable_thinking
|
||||
if optimizer_enable_thinking is not None
|
||||
else enable_thinking
|
||||
),
|
||||
)
|
||||
_update_config(
|
||||
TARGET_CONFIG,
|
||||
"target",
|
||||
base_url=target_base_url if target_base_url is not None else base_url,
|
||||
api_key=target_api_key if target_api_key is not None else api_key,
|
||||
temperature=target_temperature if target_temperature is not None else temperature,
|
||||
timeout_seconds=(
|
||||
target_timeout_seconds
|
||||
if target_timeout_seconds is not None
|
||||
else timeout_seconds
|
||||
),
|
||||
max_tokens=target_max_tokens if target_max_tokens is not None else max_tokens,
|
||||
enable_thinking=(
|
||||
target_enable_thinking
|
||||
if target_enable_thinking is not None
|
||||
else enable_thinking
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def _update_config(
|
||||
config: QwenChatConfig,
|
||||
role: str,
|
||||
*,
|
||||
base_url: str | None = None,
|
||||
api_key: str | None = None,
|
||||
temperature: float | str | None = None,
|
||||
timeout_seconds: float | str | None = None,
|
||||
max_tokens: int | str | None = None,
|
||||
enable_thinking: bool | str | None = None,
|
||||
) -> None:
|
||||
env_prefix = role.upper()
|
||||
if base_url is not None:
|
||||
config.base_url = str(base_url).strip() or config.base_url
|
||||
os.environ[f"{env_prefix}_QWEN_CHAT_BASE_URL"] = config.base_url
|
||||
if api_key is not None:
|
||||
config.api_key = str(api_key).strip()
|
||||
os.environ[f"{env_prefix}_QWEN_CHAT_API_KEY"] = config.api_key
|
||||
if temperature is not None:
|
||||
raw = str(temperature).strip()
|
||||
config.temperature = float(raw) if raw else None
|
||||
os.environ[f"{env_prefix}_QWEN_CHAT_TEMPERATURE"] = raw
|
||||
if timeout_seconds is not None:
|
||||
config.timeout_seconds = float(timeout_seconds)
|
||||
os.environ[f"{env_prefix}_QWEN_CHAT_TIMEOUT_SECONDS"] = str(timeout_seconds)
|
||||
if max_tokens is not None:
|
||||
config.max_tokens = int(max_tokens)
|
||||
os.environ[f"{env_prefix}_QWEN_CHAT_MAX_TOKENS"] = str(max_tokens)
|
||||
if enable_thinking is not None:
|
||||
config.enable_thinking = _parse_bool(enable_thinking)
|
||||
os.environ[f"{env_prefix}_QWEN_CHAT_ENABLE_THINKING"] = (
|
||||
"true" if config.enable_thinking else "false"
|
||||
)
|
||||
|
||||
|
||||
def get_max_tokens() -> int:
|
||||
return MAX_TOKENS
|
||||
return TARGET_CONFIG.max_tokens
|
||||
|
||||
|
||||
def chat_optimizer(
|
||||
system: str,
|
||||
user: str,
|
||||
max_completion_tokens: int = 16384,
|
||||
retries: int = 5,
|
||||
stage: str = "optimizer",
|
||||
reasoning_effort: str | None = None,
|
||||
timeout: float | None = None,
|
||||
) -> tuple[str, dict[str, int]]:
|
||||
del reasoning_effort
|
||||
messages = [{"role": "system", "content": system}, {"role": "user", "content": user}]
|
||||
return _chat_messages_impl(
|
||||
messages,
|
||||
max_completion_tokens,
|
||||
retries,
|
||||
stage,
|
||||
role="optimizer",
|
||||
timeout=timeout,
|
||||
)
|
||||
|
||||
|
||||
def chat_target(
|
||||
@@ -230,6 +376,33 @@ def chat_target(
|
||||
max_completion_tokens,
|
||||
retries,
|
||||
stage,
|
||||
role="target",
|
||||
timeout=timeout,
|
||||
)
|
||||
|
||||
|
||||
def chat_optimizer_messages(
|
||||
messages: list[dict[str, Any]],
|
||||
max_completion_tokens: int = 16384,
|
||||
retries: int = 5,
|
||||
stage: str = "optimizer",
|
||||
reasoning_effort: str | None = None,
|
||||
*,
|
||||
tools: list[dict[str, Any]] | None = None,
|
||||
tool_choice: str | dict[str, Any] | None = None,
|
||||
return_message: bool = False,
|
||||
timeout: float | None = None,
|
||||
) -> tuple[Any, dict[str, int]]:
|
||||
del reasoning_effort
|
||||
return _chat_messages_impl(
|
||||
messages,
|
||||
max_completion_tokens,
|
||||
retries,
|
||||
stage,
|
||||
role="optimizer",
|
||||
tools=tools,
|
||||
tool_choice=tool_choice,
|
||||
return_message=return_message,
|
||||
timeout=timeout,
|
||||
)
|
||||
|
||||
@@ -252,6 +425,7 @@ def chat_target_messages(
|
||||
max_completion_tokens,
|
||||
retries,
|
||||
stage,
|
||||
role="target",
|
||||
tools=tools,
|
||||
tool_choice=tool_choice,
|
||||
return_message=return_message,
|
||||
@@ -272,6 +446,10 @@ def set_reasoning_effort(effort: str | None) -> None:
|
||||
|
||||
|
||||
def set_target_deployment(deployment: str) -> None:
|
||||
global TARGET_DEPLOYMENT
|
||||
TARGET_DEPLOYMENT = deployment or default_model_for_backend("qwen_chat")
|
||||
os.environ["TARGET_DEPLOYMENT"] = TARGET_DEPLOYMENT
|
||||
TARGET_CONFIG.deployment = deployment or default_model_for_backend("qwen_chat")
|
||||
os.environ["TARGET_DEPLOYMENT"] = TARGET_CONFIG.deployment
|
||||
|
||||
|
||||
def set_optimizer_deployment(deployment: str) -> None:
|
||||
OPTIMIZER_CONFIG.deployment = deployment or default_model_for_backend("qwen_chat")
|
||||
os.environ["OPTIMIZER_DEPLOYMENT"] = OPTIMIZER_CONFIG.deployment
|
||||
|
||||
@@ -0,0 +1,13 @@
|
||||
from skillopt.datasets.base import SplitDataLoader
|
||||
from skillopt.envs._template.env_template import TemplateBenchmarkAdapter
|
||||
from skillopt.envs._template.loader_template import TemplateBenchmarkDataLoader
|
||||
|
||||
|
||||
def test_template_adapter_is_concrete():
|
||||
adapter = TemplateBenchmarkAdapter()
|
||||
assert adapter.get_task_types() == ["your_benchmark"]
|
||||
|
||||
|
||||
def test_template_loader_uses_split_dataloader():
|
||||
loader = TemplateBenchmarkDataLoader()
|
||||
assert isinstance(loader, SplitDataLoader)
|
||||
@@ -0,0 +1,112 @@
|
||||
"""Tests for skillopt.utils.json_utils."""
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from skillopt.utils.json_utils import extract_json, extract_json_array
|
||||
|
||||
|
||||
class TestExtractJson:
|
||||
"""extract_json — extract a JSON object from LLM response text."""
|
||||
|
||||
def test_code_fence_json(self) -> None:
|
||||
text = 'Some text\n```json\n{"key": "value", "num": 42}\n```\nmore text'
|
||||
assert extract_json(text) == {"key": "value", "num": 42}
|
||||
|
||||
def test_bare_json_object(self) -> None:
|
||||
text = 'The result is {"answer": "yes", "score": 0.95}.'
|
||||
assert extract_json(text) == {"answer": "yes", "score": 0.95}
|
||||
|
||||
def test_code_fence_takes_precedence(self) -> None:
|
||||
"""If fence content parses successfully it should be preferred over bare."""
|
||||
text = (
|
||||
'```json\n{"source": "fence"}\n```\n'
|
||||
'Then also {"source": "bare"}'
|
||||
)
|
||||
assert extract_json(text) == {"source": "fence"}
|
||||
|
||||
def test_broken_fence_falls_back_to_bare(self) -> None:
|
||||
"""When fence content is invalid JSON, fall back to bare {...} match."""
|
||||
# Use invalid fence content that has no braces so the greedy bare
|
||||
# regex doesn't swallow the valid object.
|
||||
text = (
|
||||
'```json\nnot json at all\n```\n'
|
||||
'Answer: {"fallback": "yes"}'
|
||||
)
|
||||
assert extract_json(text) == {"fallback": "yes"}
|
||||
|
||||
def test_nested_json(self) -> None:
|
||||
text = '```json\n{"outer": {"inner": [1, 2, 3]}}\n```'
|
||||
assert extract_json(text) == {"outer": {"inner": [1, 2, 3]}}
|
||||
|
||||
def test_no_json_returns_none(self) -> None:
|
||||
assert extract_json("Just plain text without JSON.") is None
|
||||
|
||||
def test_empty_string_returns_none(self) -> None:
|
||||
assert extract_json("") is None
|
||||
|
||||
def test_malformed_json_returns_none(self) -> None:
|
||||
assert extract_json("{broken") is None
|
||||
|
||||
def test_empty_json_object(self) -> None:
|
||||
assert extract_json('{"empty": {}}') == {"empty": {}}
|
||||
|
||||
def test_json_with_escaped_chars(self) -> None:
|
||||
text = '{"message": "hello\\nworld"}'
|
||||
assert extract_json(text) == {"message": "hello\nworld"}
|
||||
|
||||
def test_only_fence_with_no_json_syntax(self) -> None:
|
||||
"""Code fences without valid JSON content should not match."""
|
||||
text = "```\nplain code block\n```"
|
||||
assert extract_json(text) is None
|
||||
|
||||
|
||||
class TestExtractJsonArray:
|
||||
"""extract_json_array — extract a JSON array from LLM response text."""
|
||||
|
||||
def test_code_fence_array(self) -> None:
|
||||
text = '```json\n["a", "b", "c"]\n```'
|
||||
assert extract_json_array(text) == ["a", "b", "c"]
|
||||
|
||||
def test_bare_array(self) -> None:
|
||||
text = "The items are [1, 2, 3]."
|
||||
assert extract_json_array(text) == [1, 2, 3]
|
||||
|
||||
def test_code_fence_takes_precedence(self) -> None:
|
||||
text = (
|
||||
'```json\n["from_fence"]\n```\n'
|
||||
'also ["from_bare"]'
|
||||
)
|
||||
assert extract_json_array(text) == ["from_fence"]
|
||||
|
||||
def test_broken_fence_falls_back_to_bare(self) -> None:
|
||||
text = (
|
||||
'```json\nnot json at all\n```\n'
|
||||
'values: [42]'
|
||||
)
|
||||
assert extract_json_array(text) == [42]
|
||||
|
||||
def test_nested_array(self) -> None:
|
||||
text = '```json\n[[1, 2], [3, 4]]\n```'
|
||||
assert extract_json_array(text) == [[1, 2], [3, 4]]
|
||||
|
||||
def test_no_array_returns_none(self) -> None:
|
||||
assert extract_json_array("no brackets here") is None
|
||||
|
||||
def test_empty_string_returns_none(self) -> None:
|
||||
assert extract_json_array("") is None
|
||||
|
||||
def test_malformed_array_returns_none(self) -> None:
|
||||
assert extract_json_array("[1, 2, ") is None
|
||||
|
||||
def test_empty_json_array(self) -> None:
|
||||
assert extract_json_array("[]") == []
|
||||
|
||||
def test_array_of_objects(self) -> None:
|
||||
text = '[{"x": 1}, {"x": 2}]'
|
||||
assert extract_json_array(text) == [{"x": 1}, {"x": 2}]
|
||||
|
||||
def test_object_not_confused_with_array(self) -> None:
|
||||
"""extract_json_array should not match a bare JSON object."""
|
||||
text = '{"this is an object": true}'
|
||||
assert extract_json_array(text) is None
|
||||
@@ -0,0 +1,106 @@
|
||||
"""Tests for skillopt.utils.scoring."""
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from skillopt.utils.scoring import compute_score, skill_hash
|
||||
|
||||
|
||||
class _ResultObject:
|
||||
"""Minimal object with hard/soft attrs (duck-typing path)."""
|
||||
|
||||
def __init__(self, hard: float, soft: float) -> None:
|
||||
self.hard = hard
|
||||
self.soft = soft
|
||||
|
||||
|
||||
class TestComputeScore:
|
||||
"""compute_score — hard/soft accuracy from a list of episode results."""
|
||||
|
||||
def test_empty_list_returns_zeros(self) -> None:
|
||||
assert compute_score([]) == (0.0, 0.0)
|
||||
|
||||
def test_dict_results_happy_path(self) -> None:
|
||||
results = [
|
||||
{"hard": 1, "soft": 0.8},
|
||||
{"hard": 0, "soft": 0.5},
|
||||
{"hard": 1, "soft": 0.9},
|
||||
]
|
||||
hard, soft = compute_score(results)
|
||||
assert hard == pytest.approx(2 / 3)
|
||||
assert soft == pytest.approx((0.8 + 0.5 + 0.9) / 3)
|
||||
|
||||
def test_object_results(self) -> None:
|
||||
results = [
|
||||
_ResultObject(1.0, 0.75),
|
||||
_ResultObject(0.0, 0.25),
|
||||
]
|
||||
hard, soft = compute_score(results)
|
||||
assert hard == 0.5
|
||||
assert soft == 0.5
|
||||
|
||||
def test_mixed_dict_and_object_results(self) -> None:
|
||||
results = [
|
||||
{"hard": 1, "soft": 1.0},
|
||||
_ResultObject(0, 0.0),
|
||||
]
|
||||
hard, soft = compute_score(results)
|
||||
assert hard == 0.5
|
||||
assert soft == 0.5
|
||||
|
||||
def test_missing_keys_default_to_zero(self) -> None:
|
||||
results = [
|
||||
{"hard": 1},
|
||||
{},
|
||||
]
|
||||
hard, soft = compute_score(results)
|
||||
assert hard == 0.5
|
||||
assert soft == 0.0
|
||||
|
||||
def test_single_result(self) -> None:
|
||||
results = [{"hard": 1, "soft": 0.95}]
|
||||
assert compute_score(results) == (1.0, 0.95)
|
||||
|
||||
def test_continuous_hard_values(self) -> None:
|
||||
"""Hard may be continuous 0.0-1.0 when using smoothed reward."""
|
||||
results = [
|
||||
{"hard": 0.75, "soft": 0.6},
|
||||
{"hard": 0.25, "soft": 0.4},
|
||||
]
|
||||
hard, soft = compute_score(results)
|
||||
assert hard == 0.5
|
||||
assert soft == 0.5
|
||||
|
||||
|
||||
class TestSkillHash:
|
||||
"""skill_hash — a short, deterministic hash of skill content."""
|
||||
|
||||
def test_deterministic(self) -> None:
|
||||
assert skill_hash("hello") == skill_hash("hello")
|
||||
|
||||
def test_different_input_produces_different_hash(self) -> None:
|
||||
assert skill_hash("hello") != skill_hash("world")
|
||||
|
||||
def test_empty_string(self) -> None:
|
||||
h = skill_hash("")
|
||||
assert isinstance(h, str)
|
||||
assert len(h) == 16
|
||||
|
||||
def test_output_length(self) -> None:
|
||||
h = skill_hash("some skill content here")
|
||||
assert len(h) == 16
|
||||
|
||||
def test_hex_characters(self) -> None:
|
||||
h = skill_hash("any content")
|
||||
assert all(c in "0123456789abcdef" for c in h)
|
||||
|
||||
def test_unicode_content(self) -> None:
|
||||
h1 = skill_hash("cafe")
|
||||
h2 = skill_hash("cafe")
|
||||
assert h1 == h2
|
||||
|
||||
def test_multiline_content(self) -> None:
|
||||
content = "line1\nline2\nline3"
|
||||
h = skill_hash(content)
|
||||
assert len(h) == 16
|
||||
assert isinstance(h, str)
|
||||
@@ -0,0 +1,249 @@
|
||||
"""Tests for skillopt.types — Edit and Patch dataclass serialization."""
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from skillopt.types import Edit, Patch
|
||||
|
||||
|
||||
# ── Edit ────────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
class TestEditCreation:
|
||||
"""Edit dataclass construction."""
|
||||
|
||||
def test_minimal_edit(self) -> None:
|
||||
e = Edit(op="append")
|
||||
assert e.op == "append"
|
||||
assert e.content == ""
|
||||
assert e.target == ""
|
||||
assert e.support_count is None
|
||||
assert e.source_type is None
|
||||
assert e.merge_level is None
|
||||
assert e.update_origin == ""
|
||||
assert e.update_target == ""
|
||||
|
||||
def test_full_edit(self) -> None:
|
||||
e = Edit(
|
||||
op="replace",
|
||||
content="new content",
|
||||
target="old content",
|
||||
support_count=5,
|
||||
source_type="failure",
|
||||
merge_level=2,
|
||||
update_origin="reflect",
|
||||
update_target="skill",
|
||||
)
|
||||
assert e.op == "replace"
|
||||
assert e.content == "new content"
|
||||
assert e.target == "old content"
|
||||
assert e.support_count == 5
|
||||
assert e.source_type == "failure"
|
||||
assert e.merge_level == 2
|
||||
assert e.update_origin == "reflect"
|
||||
assert e.update_target == "skill"
|
||||
|
||||
def test_insert_after_op(self) -> None:
|
||||
e = Edit(op="insert_after", content="insertion", target="anchor")
|
||||
assert e.op == "insert_after"
|
||||
assert e.content == "insertion"
|
||||
assert e.target == "anchor"
|
||||
|
||||
def test_delete_op(self) -> None:
|
||||
e = Edit(op="delete", target="thing_to_remove")
|
||||
assert e.op == "delete"
|
||||
assert e.target == "thing_to_remove"
|
||||
|
||||
|
||||
class TestEditRoundTrip:
|
||||
"""Edit.to_dict() / Edit.from_dict() round-trip."""
|
||||
|
||||
def test_round_trip_minimal(self) -> None:
|
||||
e = Edit(op="append")
|
||||
d = e.to_dict()
|
||||
restored = Edit.from_dict(d)
|
||||
assert restored == e
|
||||
|
||||
def test_round_trip_full(self) -> None:
|
||||
e = Edit(
|
||||
op="replace",
|
||||
content="new content",
|
||||
target="old content",
|
||||
support_count=3,
|
||||
source_type="success",
|
||||
merge_level=1,
|
||||
update_origin="meta_reflect",
|
||||
update_target="system_prompt",
|
||||
)
|
||||
d = e.to_dict()
|
||||
restored = Edit.from_dict(d)
|
||||
assert restored == e
|
||||
|
||||
def test_round_trip_delete_without_content(self) -> None:
|
||||
e = Edit(op="delete", target="obsolete_line")
|
||||
d = e.to_dict()
|
||||
restored = Edit.from_dict(d)
|
||||
assert restored == e
|
||||
|
||||
def test_optional_fields_omitted_when_default(self) -> None:
|
||||
e = Edit(op="append")
|
||||
d = e.to_dict()
|
||||
assert d == {"op": "append", "content": ""}
|
||||
# support_count, source_type, etc. should be absent
|
||||
assert "support_count" not in d
|
||||
assert "source_type" not in d
|
||||
assert "merge_level" not in d
|
||||
assert "target" not in d
|
||||
assert "update_origin" not in d
|
||||
assert "update_target" not in d
|
||||
|
||||
def test_from_dict_with_defaults(self) -> None:
|
||||
d = {"op": "replace", "content": "abc"}
|
||||
e = Edit.from_dict(d)
|
||||
assert e.op == "replace"
|
||||
assert e.content == "abc"
|
||||
assert e.target == ""
|
||||
assert e.support_count is None
|
||||
assert e.source_type is None
|
||||
|
||||
def test_from_dict_with_extra_keys(self) -> None:
|
||||
"""Extra keys in dict should be ignored."""
|
||||
d = {"op": "append", "content": "", "unknown_field": 42}
|
||||
e = Edit.from_dict(d)
|
||||
assert e.op == "append"
|
||||
assert not hasattr(e, "unknown_field")
|
||||
|
||||
|
||||
class TestEditEdgeCases:
|
||||
"""Edge cases around Edit."""
|
||||
|
||||
def test_support_count_zero(self) -> None:
|
||||
"""0 is a valid support_count and should be serialized."""
|
||||
e = Edit(op="append", support_count=0)
|
||||
d = e.to_dict()
|
||||
assert d["support_count"] == 0
|
||||
restored = Edit.from_dict(d)
|
||||
assert restored.support_count == 0
|
||||
|
||||
def test_merge_level_zero(self) -> None:
|
||||
e = Edit(op="replace", merge_level=0)
|
||||
d = e.to_dict()
|
||||
assert d["merge_level"] == 0
|
||||
restored = Edit.from_dict(d)
|
||||
assert restored.merge_level == 0
|
||||
|
||||
def test_empty_target_stays_empty(self) -> None:
|
||||
e = Edit(op="append", target="")
|
||||
d = e.to_dict()
|
||||
assert "target" not in d
|
||||
|
||||
|
||||
# ── Patch ───────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
class TestPatchCreation:
|
||||
"""Patch dataclass construction."""
|
||||
|
||||
def test_empty_patch(self) -> None:
|
||||
p = Patch()
|
||||
assert p.edits == []
|
||||
assert p.reasoning == ""
|
||||
assert p.ranking_details is None
|
||||
|
||||
def test_patch_with_edits(self) -> None:
|
||||
edits = [
|
||||
Edit(op="append", content="step 1"),
|
||||
Edit(op="append", content="step 2"),
|
||||
]
|
||||
p = Patch(edits=edits, reasoning="Added two steps")
|
||||
assert len(p.edits) == 2
|
||||
assert p.reasoning == "Added two steps"
|
||||
|
||||
def test_patch_with_ranking_details(self) -> None:
|
||||
p = Patch(ranking_details={"score": 0.95, "rank": 1})
|
||||
assert p.ranking_details == {"score": 0.95, "rank": 1}
|
||||
|
||||
|
||||
class TestPatchRoundTrip:
|
||||
"""Patch.to_dict() / Patch.from_dict() round-trip."""
|
||||
|
||||
def test_round_trip_empty(self) -> None:
|
||||
p = Patch()
|
||||
d = p.to_dict()
|
||||
restored = Patch.from_dict(d)
|
||||
assert restored.edits == []
|
||||
assert restored.reasoning == ""
|
||||
assert restored.ranking_details is None
|
||||
|
||||
def test_round_trip_with_edits(self) -> None:
|
||||
edits = [
|
||||
Edit(op="insert_after", content="new step", target="existing step"),
|
||||
Edit(op="replace", content="updated", target="old"),
|
||||
]
|
||||
p = Patch(edits=edits, reasoning="Batch update")
|
||||
d = p.to_dict()
|
||||
restored = Patch.from_dict(d)
|
||||
assert len(restored.edits) == 2
|
||||
for original, restored_edit in zip(p.edits, restored.edits):
|
||||
assert isinstance(restored_edit, Edit)
|
||||
assert original == restored_edit
|
||||
assert restored.reasoning == "Batch update"
|
||||
assert restored.ranking_details is None
|
||||
|
||||
def test_round_trip_with_ranking_details(self) -> None:
|
||||
details = {"strategy": "rouge", "scores": [0.9, 0.8, 0.7]}
|
||||
p = Patch(
|
||||
edits=[Edit(op="append", content="a")],
|
||||
reasoning="selected best",
|
||||
ranking_details=details,
|
||||
)
|
||||
d = p.to_dict()
|
||||
restored = Patch.from_dict(d)
|
||||
assert restored.ranking_details == details
|
||||
|
||||
def test_to_dict_contains_reasoning_and_edits(self) -> None:
|
||||
p = Patch(edits=[Edit(op="append", content="test")], reasoning="reason")
|
||||
d = p.to_dict()
|
||||
assert "reasoning" in d
|
||||
assert "edits" in d
|
||||
assert isinstance(d["edits"], list)
|
||||
|
||||
def test_from_dict_preserves_edit_order(self) -> None:
|
||||
edits = [
|
||||
Edit(op="append", content="first"),
|
||||
Edit(op="insert_after", content="second", target="first"),
|
||||
Edit(op="append", content="third"),
|
||||
]
|
||||
p = Patch(edits=edits, reasoning="ordered")
|
||||
d = p.to_dict()
|
||||
restored = Patch.from_dict(d)
|
||||
assert restored.edits[0].content == "first"
|
||||
assert restored.edits[1].content == "second"
|
||||
assert restored.edits[2].content == "third"
|
||||
|
||||
|
||||
class TestPatchEdgeCases:
|
||||
"""Edge cases around Patch."""
|
||||
|
||||
def test_reasoning_empty_string(self) -> None:
|
||||
p = Patch(reasoning="")
|
||||
d = p.to_dict()
|
||||
assert d["reasoning"] == ""
|
||||
|
||||
def test_zero_edits(self) -> None:
|
||||
"""Patch with explicitly empty edit list."""
|
||||
p = Patch(edits=[])
|
||||
d = p.to_dict()
|
||||
assert d["edits"] == []
|
||||
|
||||
def test_nested_edit_from_dict_handles_dicts(self) -> None:
|
||||
"""from_dict should accept dicts in the 'edits' list."""
|
||||
d = {
|
||||
"reasoning": "test",
|
||||
"edits": [{"op": "append", "content": "hello"}],
|
||||
}
|
||||
p = Patch.from_dict(d)
|
||||
assert len(p.edits) == 1
|
||||
assert isinstance(p.edits[0], Edit)
|
||||
assert p.edits[0].op == "append"
|
||||
assert p.edits[0].content == "hello"
|
||||
Reference in New Issue
Block a user