url/SkillOpt
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

SkillOpt v0.1.0: initial release

Browse Source 
- Skill optimization framework with training loop analogy
- 11 benchmarks, 4 model backends (Azure OpenAI, Claude, Codex, Qwen)
- WebUI for browser-based training control
- Pluggable architecture for extending benchmarks and backends
This commit is contained in:
CharlesYang030
2026-05-21 17:22:04 +00:00
commit 244e346b83
237 changed files with 30248 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.env.example
+24
View File
@@ -0,0 +1,24 @@
# SkillOpt Environment Variables
# Copy this file to .env and fill in your values.
# ── Azure OpenAI (required for openai_chat backend) ──────────────────
AZURE_OPENAI_ENDPOINT=https://your-resource.openai.azure.com/
AZURE_OPENAI_API_VERSION=2024-12-01-preview
# Authentication: choose one method
# Option 1: API Key
AZURE_OPENAI_API_KEY=
# Option 2: Azure CLI (set auth_mode=azure_cli in config)
# Option 3: Managed Identity (set auth_mode=managed_identity + client_id in config)
# ── OpenAI (alternative to Azure) ────────────────────────────────────
# OPENAI_API_KEY=sk-...
# ── Anthropic / Claude (for claude_chat backend) ─────────────────────
# ANTHROPIC_API_KEY=sk-ant-...
# ── Qwen Local Model (for qwen_chat backend) ────────────────────────
# QWEN_CHAT_BASE_URL=http://localhost:8000/v1
# QWEN_CHAT_MODEL=Qwen/Qwen3.5-4B
# ── Ray (optional, for distributed rollout) ──────────────────────────
# RAY_ADDRESS=auto
.gitignore
+41
View File
@@ -0,0 +1,41 @@
__pycache__/
*.pyc
*.egg-info/
build/
dist/
site/
data/
outputs/
logs/
external/
/BabyVision/
/MMRB/
/SpreadsheetBench/
/dl4ir-searchQA/
configs/local/
configs/**/*.local.yaml
*.local.md
*.secret.md
*.bak
.env
.secrets/
.codex_azure*/
# Internal docs (not for open-source release)
docs/ablation_plan.md
docs/ablation_paper_tables.md
docs/ablation_paper_tables.html
docs/experiment_commands.md
docs/slow_update_flowchart.md
docs/session_memory.md
docs/harness_fresh_machine_handoff.md
docs/harness_monitoring_memory.md
docs/harness_reproduction_secrets.secret.md
docs/reflact_conda_env_export.yml
docs/reflact_overview.html
docs/render_ablation_paper_tables.py
docs/让*
CONTRIBUTING.md
+43
View File
@@ -0,0 +1,43 @@
# Contributing to SkillOpt
Thank you for your interest in contributing! SkillOpt welcomes contributions of all kinds.
## Getting Started
```bash
git clone https://github.com/microsoft/SkillOpt.git
cd SkillOpt
pip install -e ".[dev]"
```
## How to Contribute
### 🐛 Bug Reports
Open a GitHub issue with reproduction steps, expected/actual behavior, and your config file (remove API keys).
### 🔧 Add a Benchmark
See the [guide](docs/guide/new-benchmark.md) and use the scaffold at `skillopt/envs/_template/`.
### 🤖 Add a Model Backend
See the [guide](docs/guide/new-backend.md).
### 📝 Improve Documentation
```bash
pip install -e ".[docs]"
mkdocs serve # Preview at http://localhost:8000
```
## Pull Request Process
1. Fork the repo and create a feature branch
2. Make changes and test with an existing benchmark
3. Submit a PR with a clear description
4. Ensure CI passes
## Code Style
- Follow existing patterns in the codebase
- Use type hints for function signatures
- Keep docstrings concise
## License
By contributing, you agree your contributions are licensed under the [MIT License](LICENSE).
LICENSE
+21
View File
@@ -0,0 +1,21 @@
MIT License
Copyright (c) 2026 Microsoft Corporation
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
README.md
+421
View File
@@ -0,0 +1,421 @@
# SkillOpt
**Executive Strategy for Self-Evolving Agent Skills**
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
[![Python 3.10+](https://img.shields.io/badge/Python-3.10%2B-blue.svg)](https://www.python.org/)
*Train agent skills like you train neural networks — with epochs, learning rates, and validation gates — but without touching model weights.*
---
## What is SkillOpt?
SkillOpt is a framework for optimizing a natural-language **skill document** through iterative rollout, reflection, editing, and gated validation.
It does **not** fine-tune model parameters. Instead, it treats the skill document as the optimization target:
- The **student** model executes tasks with the current skill
- The **teacher** model analyzes trajectories and proposes edits
- The framework merges, ranks, applies, and validates those edits
- Only validated skill updates are kept
| Deep Learning | SkillOpt |
|---|---|
| Model weights | Skill document (Markdown) |
| Forward pass | Rollout (student executes tasks) |
| Loss computation | Reflect (teacher analyzes trajectories) |
| Gradient | Edit patches (proposed skill improvements) |
| Gradient clipping | Edit ranking & selection (`learning_rate`) |
| Weight update | Patch application to skill document |
| Validation | Gated evaluation on held-out split |
| Learning rate schedule | `lr_scheduler`: cosine, linear decay |
| Epochs | Multi-epoch training with slow update & meta skill |
---
## Method Overview
### Optimization Target
Each run maintains a mutable markdown skill document. The framework repeatedly improves that document instead of changing model parameters.
This gives a training-style loop for prompt / policy optimization:
1. Roll out the current skill on a batch of tasks.
2. Reflect on failures and successes.
3. Merge patch proposals into a coherent candidate update.
4. Rank and select a bounded number of edits.
5. Apply those edits to produce a candidate skill.
6. Validate the candidate skill on a held-out selection split.
7. Keep the update only if the gate accepts it.
### Per-Step Pipeline
Every training step executes the following pipeline in `skillopt/engine/trainer.py`:
1. **Rollout**
The student model runs a batch of tasks using the current skill.
2. **Reflect**
The teacher analyzes minibatches of trajectories and emits raw patches.
Failure-driven and success-driven patches are tracked separately.
3. **Aggregate**
Raw patches are merged hierarchically. Metadata such as `support_count` and `source_type` is carried into the merged patch so later ranking can use it.
4. **Select**
The teacher ranks the merged edit pool and keeps up to `edit_budget` edits.
5. **Update**
The selected edits are applied to the skill document. The framework records an `edit_apply_report.json` so you can see which edits actually landed, which were skipped, and why.
6. **Evaluate / Gate**
The candidate skill is evaluated on the selection split. A candidate update is accepted only if it improves over the current selection score; a new global best is tracked separately.
### Within-Epoch Memory
Inside an epoch, the trainer maintains a step buffer containing:
- Compact failure-pattern summaries from previous steps
- Rejected edits and their score deltas
That context is fed back into later reflection calls so the teacher can avoid repeating ineffective edits and can focus on unsolved error patterns.
### Epoch-Level Mechanisms
#### Slow Update
At the end of each epoch, `slow_update` compares the previous epoch's terminal skill and current epoch's terminal skill on a sampled train subset. It then writes longitudinal guidance into a protected slow-update region inside the skill document.
This guidance is **not** blindly written through — it is converted into a candidate skill and sent through the same selection gate as step-level updates.
#### Meta Skill
`meta_skill` is teacher-side cross-epoch memory. It does not directly edit the current skill. Instead, it writes a compact memory artifact describing longer-term patterns across adjacent epochs. That memory is loaded into later reflection / merge / ranking calls as extra context.
#### Meta Reflect
`meta_reflect` runs at epoch end over the step history of the current epoch. It looks at accepted and rejected directions from the whole epoch, proposes higher-level patch edits, applies them to a meta candidate, and then sends that candidate through the same selection gate.
---
## Quick Start
### Install
```bash
git clone https://github.com/AgenticOpt/SkillOpt.git
cd SkillOpt
pip install -e .
```
### Configure API Credentials
```bash
cp .env.example .env
# Edit .env with your API credentials, then:
source .env
```
**Azure OpenAI** (API key or managed identity):
```bash
export AZURE_OPENAI_ENDPOINT="https://your-resource.openai.azure.com/"
export AZURE_OPENAI_API_KEY="your-key"
# Or use managed identity: set azure_openai_auth_mode=managed_identity in config
```
**OpenAI** directly:
```bash
export OPENAI_API_KEY="sk-..."
```
**Anthropic Claude**:
```bash
export ANTHROPIC_API_KEY="sk-ant-..."
```
**Qwen (local vLLM)**:
```bash
export QWEN_CHAT_BASE_URL="http://localhost:8000/v1"
export QWEN_CHAT_MODEL="Qwen/Qwen3.5-4B"
```
### Run Training
```bash
python scripts/train.py --config configs/searchqa/default.yaml
```
---
## Configuration
SkillOpt uses a hierarchical YAML configuration system. Each benchmark config inherits from `configs/_base_/default.yaml`.
### Configuration Structure
```yaml
model:
teacher_backend: openai_chat # openai_chat | claude_chat | qwen_chat
student_backend: openai_chat # openai_chat | claude_chat | codex_exec | qwen_chat
teacher: gpt-5.5 # teacher model deployment name
student: gpt-5.5 # student model deployment name
reasoning_effort: medium # low | medium | high
train:
num_epochs: 4
batch_size: 40
seed: 42
gradient:
minibatch_size: 8 # trajectories per reflection call
analyst_workers: 16 # parallel reflection workers
use_deep_reflect: false # deep multi-turn probing
deep_reflect_failures: 4
deep_reflect_successes: 2
optimizer:
learning_rate: 4 # max edits per step (edit_budget)
min_learning_rate: 2 # min edits for decay schedulers
lr_scheduler: cosine # constant | linear | cosine | autonomous
skill_update_mode: patch # patch | rewrite_from_suggestions | full_rewrite_minibatch
use_slow_update: true
use_meta_skill: true
use_meta_reflect: false
evaluation:
use_gate: true # gated validation (always recommended)
env:
name: "" # benchmark name
skill_init: "" # path to initial skill document
split_mode: ratio # ratio | split_dir
split_ratio: "2:1:7" # train:val:test
```
### CLI Overrides
Override any config key from the command line:
```bash
python scripts/train.py \
--config configs/searchqa/default.yaml \
--cfg-options model.teacher_backend=openai_chat \
model.student_backend=codex_exec \
train.batch_size=40 \
optimizer.learning_rate=4
# Legacy flat overrides also work for common keys:
python scripts/train.py \
--config configs/searchqa/default.yaml \
--backend azure_openai \
--teacher_model gpt-5.5 \
--student_model gpt-5.5 \
--reasoning_effort medium
```
---
## Model Backends
All model access goes through the unified backend router in `skillopt/model/`.
| Backend | Use case | Config key |
|---|---|---|
| `openai_chat` | Azure OpenAI / OpenAI API | teacher / student |
| `claude_chat` | Anthropic Claude | teacher / student |
| `codex_exec` | Codex execution harness | student only |
| `qwen_chat` | Local Qwen via vLLM | teacher / student |
Separate teacher/student endpoints are supported:
```yaml
model:
teacher_backend: openai_chat
student_backend: codex_exec
teacher: gpt-5.5
student: gpt-5.5-codex
```
---
## Data Splits
SkillOpt supports two split modes:
**Ratio split** — auto-generate from raw data:
```bash
python scripts/train.py \
--config configs/searchqa/default.yaml \
--split_mode ratio \
--data_path /path/to/searchqa_data.json
```
**Pre-split directory** — consume prepared splits:
```bash
python scripts/train.py \
--config configs/searchqa/default.yaml \
--split_mode split_dir \
--split_dir /path/to/searchqa_split
```
---
## Supported Benchmarks
| Benchmark | Type | Config |
|---|---|---|
| SearchQA | QA | `configs/searchqa/default.yaml` |
| SpreadsheetBench | Code generation | `configs/spreadsheetbench/default.yaml` |
| ALFWorld | Embodied agent | `configs/alfworld/default.yaml` |
| DocVQA | Document QA | `configs/docvqa/default.yaml` |
| OfficeQA | Tool-augmented QA | `configs/officeqa/default.yaml` |
| SealQA | Tool-augmented QA | `configs/sealqa/default.yaml` |
| BabyVision | Vision QA | `configs/babyvision/default.yaml` |
| LiveMathematicianBench | Math | `configs/livemathematicianbench/default.yaml` |
| MathVerse | Multimodal math | `configs/mathverse/default.yaml` |
| MMRB | Multimodal reasoning | `configs/mmrb/default.yaml` |
| SWEBench | Software engineering | `configs/swebench/default.yaml` |
---
## Running Training
Basic training:
```bash
python scripts/train.py --config configs/searchqa/default.yaml
```
Exec harness (Codex student):
```bash
python scripts/train.py \
--config configs/searchqa/default.yaml \
--teacher_backend openai_chat \
--student_backend codex_exec \
--teacher_model gpt-5.5 \
--student_model gpt-5.5-codex \
--use_deep_reflect true \
--skill_update_mode rewrite_from_suggestions
```
SWEBench:
```bash
python scripts/train.py \
--config configs/swebench/default.yaml \
--cfg-options env.dataset_name=lite env.split_ratio=2:1:7
```
### Eval Only
Evaluate a specific skill without training:
```bash
python scripts/eval_only.py \
--config configs/searchqa/default.yaml \
--skill skillopt/envs/searchqa/skills/initial.md
```
---
## Output Structure
Each run writes a structured output directory:
```
outputs/<run_name>/
├── config.json # Flattened runtime config
├── history.json # Per-step history records
├── runtime_state.json # Resume state (for auto-resume)
├── best_skill.md # Current best validated skill
├── skills/skill_vXXXX.md # Skill snapshot per step
├── steps/step_XXXX/ # Per-step artifacts
│ ├── merged_patch.json
│ ├── ranked_edits.json
│ ├── candidate_skill.md
│ ├── edit_apply_report.json
│ ├── rewrite_result.json # when rewrite mode is enabled
│ └── selection_eval/
├── slow_update/epoch_XX/
├── meta_skill/epoch_XX/
└── meta_reflect/epoch_XX/
```
### Resume Behavior
The trainer resumes from `runtime_state.json` when present. That state tracks:
- Last completed step
- Current skill path and score
- Best skill path and score
- Origin tags for current and best skill
---
## Extending SkillOpt
### Add a New Benchmark
1. Create `skillopt/envs/<your_env>/` with:
- `adapter.py` — implements `EnvAdapter`
- `dataloader.py` — data loading logic
- `rollout.py` — student execution logic
- `skills/initial.md` — initial skill document
2. Add a config at `configs/<your_env>/default.yaml`
3. Register in `skillopt/envs/__init__.py`
See `skillopt/envs/_template/` for a scaffold.
### Add a New Model Backend
Implement a backend in `skillopt/model/` following the interface in `skillopt/model/common.py`, then register it in `skillopt/model/router.py`.
---
## WebUI
Launch the monitoring dashboard (optional):
```bash
pip install -e ".[webui]"
python -m skillopt_webui.app
```
Provides browser-based config selection, training launch, and real-time log monitoring.
---
## Minimal Setup
```bash
conda create -n skillopt python=3.11
conda activate skillopt
pip install -e .
```
Depending on the benchmark, you may also need:
```bash
pip install datasets gymnasium numpy
```
For SWEBench, you also need a working Docker environment plus the SWE-bench harness dependencies.
---
## Citation
```bibtex
@article{skillopt2026,
title={SkillOpt: Executive Strategy for Self-Evolving Agent Skills},
author={SkillOpt Team},
year={2026}
}
```
## License
MIT — see [LICENSE](LICENSE).
configs/_base_/default.yaml
+93
View File
@@ -0,0 +1,93 @@
# SkillOpt default configuration — base for all environments.
# Environment configs should inherit via: _base_: default.yaml
model:
backend: azure_openai
teacher: gpt-5.5
student: gpt-5.5
teacher_backend: openai_chat
student_backend: openai_chat
reasoning_effort: medium
rewrite_reasoning_effort: ""
rewrite_max_completion_tokens: 64000
codex_exec_path: codex
codex_exec_sandbox: workspace-write
codex_exec_profile: ""
codex_exec_full_auto: false
codex_exec_reasoning_effort: none
codex_exec_use_sdk: auto
codex_exec_network_access: false
codex_exec_web_search: false
codex_exec_approval_policy: never
claude_code_exec_path: claude
claude_code_exec_profile: ""
claude_code_exec_use_sdk: auto
claude_code_exec_effort: medium
claude_code_exec_max_thinking_tokens: 16384
codex_trace_to_teacher: true
azure_openai_endpoint: "" # e.g. "https://your-resource.openai.azure.com/"
azure_openai_api_version: "2024-12-01-preview"
azure_openai_api_key: "" # Fill locally if you do not export AZURE_OPENAI_API_KEY
azure_openai_auth_mode: azure_cli
azure_openai_ad_scope: "https://cognitiveservices.azure.com/.default"
azure_openai_managed_identity_client_id: ""
teacher_azure_openai_endpoint: "" # e.g. "https://your-resource.openai.azure.com/"
teacher_azure_openai_api_version: "2024-12-01-preview"
teacher_azure_openai_api_key: ""
teacher_azure_openai_auth_mode: azure_cli
teacher_azure_openai_ad_scope: "https://cognitiveservices.azure.com/.default"
teacher_azure_openai_managed_identity_client_id: ""
student_azure_openai_endpoint: "" # e.g. "https://your-resource.openai.azure.com/"
student_azure_openai_api_version: "2024-12-01-preview"
student_azure_openai_api_key: ""
student_azure_openai_auth_mode: azure_cli
student_azure_openai_ad_scope: "https://cognitiveservices.azure.com/.default"
student_azure_openai_managed_identity_client_id: ""
train:
num_epochs: 4
train_size: 0 # 0 = derive from dataset split when available
batch_size: 40
accumulation: 1
seed: 42
gradient:
minibatch_size: 8
merge_batch_size: 8
analyst_workers: 16
max_analyst_rounds: 3
failure_only: false
use_deep_reflect: false
deep_reflect_failures: 4
deep_reflect_successes: 2
optimizer:
learning_rate: 4 # max edits per step (edit_budget)
min_learning_rate: 2 # min edits for decay schedulers
lr_scheduler: cosine # constant / linear / cosine / autonomous
lr_control_mode: fixed # fixed / autonomous / none
skill_update_mode: patch # patch / rewrite_from_suggestions / full_rewrite_minibatch
use_meta_reflect: false
meta_learning_rate: 4 # max edits per epoch-level meta-reflect
use_slow_update: true
slow_update_samples: 20
longitudinal_pair_policy: mixed # mixed / changed / unchanged
use_meta_skill: true
evaluation:
use_gate: true
sel_env_num: 0
test_env_num: 0
eval_test: true
env:
name: ""
skill_init: ""
split_mode: ratio # ratio = build deterministic split from data_path; split_dir = use pre-split train/val/test
split_ratio: "2:1:7" # explicit default for dataset-backed benchmarks: train:val:test
split_seed: 42
split_dir: ""
data_path: ""
split_output_dir: ""
exec_timeout: 120 # per student model/code-agent call timeout in seconds
out_root: ""
configs/alfworld/default.yaml
+30
View File
@@ -0,0 +1,30 @@
_base_: ../_base_/default.yaml
train:
train_size: 0
accumulation: 1
gradient:
minibatch_size: 8
merge_batch_size: 8
optimizer:
learning_rate: 4
use_meta_reflect: false
evaluation:
sel_env_num: 0
test_env_num: 0
env:
name: alfworld
skill_init: skillopt/envs/alfworld/skills/initial.md
split_mode: split_dir
split_ratio: "2:1:7"
split_dir: data/ablation_splits/alfworld/2-1-7_seed42
data_path: ""
split_output_dir: ""
max_steps: 50
workers: 8
max_api_workers: 8
limit: 0
configs/babyvision/default.yaml
+21
View File
@@ -0,0 +1,21 @@
_base_: ../_base_/default.yaml
train:
batch_size: 64
accumulation: 1
env:
name: babyvision
skill_init: skillopt/envs/babyvision/skills/initial.md
split_mode: ratio
split_ratio: "2:1:7"
split_dir: ""
data_path: ""
split_output_dir: ""
max_turns: 1
workers: 16
limit: 0
image_detail: auto
judge_model: gpt-5.4
judge_max_completion_tokens: 256
judge_retries: 5
configs/docvqa/default.yaml
+28
View File
@@ -0,0 +1,28 @@
_base_: ../_base_/default.yaml
model:
reasoning_effort: medium
train:
batch_size: 40
accumulation: 1
gradient:
minibatch_size: 8
merge_batch_size: 8
optimizer:
learning_rate: 4
env:
name: docvqa
skill_init: skillopt/envs/docvqa/skills/initial.md
split_mode: split_dir
split_ratio: "2:1:7"
split_dir: data/docvqa/splits
data_path: ""
split_output_dir: ""
max_turns: 1
workers: 16
image_detail: auto
limit: 0
configs/livemathematicianbench/default.yaml
+22
View File
@@ -0,0 +1,22 @@
_base_: ../_base_/default.yaml
train:
train_size: 0
batch_size: 40
accumulation: 1
env:
name: livemathematicianbench
skill_init: skillopt/envs/livemathematicianbench/skills/initial.md
split_mode: split_dir
split_ratio: "2:1:7"
split_dir: data/ablation_splits/livemathematicianbench/2-1-7_seed42
data_path: ""
split_output_dir: ""
max_turns: 1
exec_timeout: 300
workers: 64
limit: 0
shuffle_choices: true
use_theorem: false
use_sketch: false
configs/mathverse/default.yaml
+23
View File
@@ -0,0 +1,23 @@
_base_: ../_base_/default.yaml
model:
codex_exec_sandbox: danger-full-access
train:
batch_size: 64
accumulation: 1
env:
name: mathverse
skill_init: skillopt/envs/mathverse/skills/initial.md
split_dir: ""
data_root: data/MathVerse
problem_version: Text Lite
use_text_dominant_reference: false
max_turns: 1
workers: 16
limit: 0
image_detail: auto
judge_model: gpt-5.4
judge_max_completion_tokens: 256
judge_retries: 5
configs/mmrb/default.yaml
+18
View File
@@ -0,0 +1,18 @@
_base_: ../_base_/default.yaml
train:
batch_size: 128
accumulation: 1
env:
name: mmrb
skill_init: skillopt/envs/mmrb/skills/initial.md
split_mode: ratio
split_ratio: "2:1:7"
split_dir: ""
data_path: ""
split_output_dir: ""
max_turns: 1
workers: 16
limit: 0
image_detail: auto
configs/officeqa/default.yaml
+34
View File
@@ -0,0 +1,34 @@
_base_: ../_base_/default.yaml
model:
reasoning_effort: medium
train:
batch_size: 40
accumulation: 1
gradient:
minibatch_size: 8
merge_batch_size: 8
optimizer:
learning_rate: 4
env:
name: officeqa
skill_init: skillopt/envs/officeqa/skills/initial.md
split_mode: split_dir
split_dir: data/officeqa_split
data_dirs:
- data/officeqa_docs_official
workers: 4
max_tool_turns: 24
max_completion_tokens: 10000
search_mode: offline
max_queries_per_turn: 4
search_api_url: http://apisix.westus2.cloudapp.azure.com/search_tool/search
search_auth_env: OFFICEQA_CUSTOM_SEARCH_AUTH
search_provider: duckduckgo
search_max_num_results: 4
search_timeout_seconds: 20
limit: 0
configs/sealqa/default.yaml
+23
View File
@@ -0,0 +1,23 @@
_base_: ../_base_/default.yaml
model:
reasoning_effort: medium
train:
batch_size: 10
accumulation: 1
gradient:
minibatch_size: 8
merge_batch_size: 8
optimizer:
learning_rate: 4
env:
name: sealqa
skill_init: skillopt/envs/sealqa/skills/initial.md
split_dir: data/sealqa_split
workers: 4
max_tool_turns: 12
limit: 0
configs/searchqa/default.yaml
+32
View File
@@ -0,0 +1,32 @@
_base_: ../_base_/default.yaml
model:
reasoning_effort: medium
train:
train_size: 400
batch_size: 40
accumulation: 1
gradient:
minibatch_size: 8
merge_batch_size: 8
optimizer:
learning_rate: 4
evaluation:
sel_env_num: 0
test_env_num: 0
env:
name: searchqa
skill_init: skillopt/envs/searchqa/skills/initial.md
split_mode: split_dir
split_ratio: "2:1:7"
split_dir: data/searchqa_split
data_path: ""
split_output_dir: ""
max_turns: 1
workers: 24
limit: 0
configs/spreadsheetbench/default.yaml
+34
View File
@@ -0,0 +1,34 @@
_base_: ../_base_/default.yaml
model:
reasoning_effort: medium
train:
train_size: 80
batch_size: 40
accumulation: 1
gradient:
minibatch_size: 8
merge_batch_size: 8
optimizer:
learning_rate: 4
evaluation:
sel_env_num: 0
test_env_num: 0
env:
name: spreadsheetbench
skill_init: skillopt/envs/spreadsheetbench/skills/initial.md
split_mode: split_dir
split_ratio: "2:1:7"
split_dir: data/spreadsheetbench_split
data_path: ""
split_output_dir: ""
data_root: data/spreadsheetbench_verified_400
mode: multi
max_turns: 30
exec_timeout: 600
workers: 24
configs/swebench/default.yaml
+36
View File
@@ -0,0 +1,36 @@
_base_: ../_base_/default.yaml
model:
reasoning_effort: medium
train:
batch_size: 20
accumulation: 1
gradient:
minibatch_size: 4
merge_batch_size: 8
optimizer:
learning_rate: 4
evaluation:
sel_env_num: 0
test_env_num: 0
env:
name: swebench
skill_init: skillopt/envs/swebench/skills/initial.md
split_mode: ratio
split_ratio: "2:1:7"
split_dir: ""
data_path: ""
split_output_dir: ""
dataset_name: lite
hf_split: test
workers: 8
eval_workers: 8
step_limit: 50
cost_limit: 3.0
timeout_per_instance: 600
limit: 0
docs/contributing.md
+69
View File
@@ -0,0 +1,69 @@
# Contributing to SkillOpt
Thank you for your interest in contributing to SkillOpt! This guide covers how to get started.
## Development Setup
```bash
git clone https://github.com/microsoft/SkillOpt.git
cd SkillOpt
pip install -e ".[dev]"
```
## Ways to Contribute
### 🐛 Bug Reports
Open an issue with:
- Steps to reproduce
- Expected vs actual behavior
- Config file used (sanitize API keys)
- Python version and OS
### 🔧 New Benchmark
See [Add a New Benchmark](guide/new-benchmark.md) for the implementation guide.
**Checklist:**
- [ ] Data loader in `skillopt/envs/<benchmark>/loader.py`
- [ ] Environment adapter in `skillopt/envs/<benchmark>/env.py`
- [ ] Config file in `configs/<benchmark>/default.yaml`
- [ ] Registration in `skillopt/envs/__init__.py`
- [ ] Documentation page in `docs/`
### 🤖 New Model Backend
See [Add a New Model Backend](guide/new-backend.md) for the implementation guide.
**Checklist:**
- [ ] Backend in `skillopt/model/<backend>.py`
- [ ] Registration in `skillopt/model/__init__.py`
- [ ] API key entry in `.env.example`
- [ ] Documentation update
### 📝 Documentation
Documentation is built with MkDocs Material:
```bash
pip install -e ".[docs]"
mkdocs serve # Preview at http://localhost:8000
```
## Code Style
- Follow existing patterns in the codebase
- Use type hints for function signatures
- Keep docstrings concise
## Pull Request Process
1. Fork the repository
2. Create a feature branch: `git checkout -b feature/my-benchmark`
3. Make your changes
4. Test with an existing benchmark config
5. Submit a PR with a clear description
## License
By contributing, you agree that your contributions will be licensed under the MIT License.
docs/guide/configuration.md
+109
View File
@@ -0,0 +1,109 @@
# Configuration Guide
SkillOpt uses YAML configuration files with a hierarchical override system.
## Config Structure
```
configs/
├── _base_/
│ └── default.yaml # Global defaults
├── searchqa/
│ └── default.yaml # SearchQA overrides
├── docvqa/
│ └── default.yaml # DocVQA overrides
└── alfworld/
└── default.yaml # ALFWorld overrides
```
Benchmark configs inherit from `_base_/default.yaml` and override specific values.
## Key Parameters
### Model
```yaml
model:
backend: azure_openai # azure_openai | openai_chat | claude_code_exec | qwen
teacher: gpt-5.5 # Teacher model (for reflection)
student: gpt-5.5 # Student model (for rollout)
```
### Training
```yaml
train:
num_epochs: 4 # Number of training epochs
batch_size: 40 # Tasks per step (batch size)
accumulation: 1 # Gradient accumulation
seed: 42
```
### Gradient (Reflection)
```yaml
gradient:
minibatch_size: 8 # Reflect minibatch size
analyst_workers: 16 # Parallel reflection workers
max_analyst_rounds: 3 # Max rounds of analyst reflection
failure_only: false # Only reflect on failures
```
### Optimizer
```yaml
optimizer:
learning_rate: 4 # Max edits per step (edit budget)
min_learning_rate: 2 # Min edits for decay schedulers
lr_scheduler: cosine # constant | linear | cosine | autonomous
use_slow_update: true # Momentum-like blending at epoch boundary
slow_update_samples: 20 # Samples for slow update evaluation
use_meta_skill: true # Cross-epoch strategy memory
```
### Evaluation
```yaml
evaluation:
use_gate: true # Validation gating (accept/reject updates)
eval_test: true # Run test evaluation after training
```
### Environment (Data)
```yaml
env:
name: searchqa # Benchmark name
split_mode: ratio # ratio | split_dir
split_ratio: "2:1:7" # train:val:test ratio
data_path: "" # Path to dataset
exec_timeout: 120 # Per-task timeout (seconds)
```
## CLI Overrides
Override any config value from the command line:
```bash
python scripts/train.py \
--config configs/searchqa/default.yaml \
optimizer.learning_rate=16 \
optimizer.lr_scheduler=linear \
gradient.analyst_workers=8
```
## Environment Variables
Model credentials are loaded from environment variables:
| Variable | Backend | Description |
|---|---|---|
| `AZURE_OPENAI_ENDPOINT` | azure_openai | Azure resource endpoint |
| `AZURE_OPENAI_API_KEY` | azure_openai | Azure API key |
| `OPENAI_API_KEY` | openai | OpenAI API key |
| `ANTHROPIC_API_KEY` | claude | Anthropic API key |
| `QWEN_API_BASE` | qwen | Local Qwen vLLM endpoint |
## Full Reference
See [Configuration Reference](../reference/config.md) for the complete parameter list.
docs/guide/dl-analogy.md
+51
View File
@@ -0,0 +1,51 @@
# Deep Learning ↔ SkillOpt Analogy
SkillOpt is designed around a core insight: **optimizing natural-language prompts follows the same structure as training neural networks**. This page maps every DL concept to its SkillOpt counterpart.
## Complete Mapping
| Deep Learning | SkillOpt | Description |
|---|---|---|
| **Model weights** | Skill document (Markdown) | The thing being optimized |
| **Forward pass** | Rollout | Student executes tasks using current skill |
| **Loss function** | Task evaluator | Scores task execution quality |
| **Backpropagation** | Reflect | Teacher analyzes failures → edit patches |
| **Gradients** | Edit patches | Proposed changes to the skill |
| **Gradient aggregation** | Patch aggregation | Merge similar edits |
| **Gradient clipping** | Edit selection | Cap max edits per step |
| **Learning rate** | `learning_rate` | Max number of edits applied per step |
| **LR scheduler** | `lr_scheduler` | Decay schedule: cosine, linear, constant |
| **SGD step** | Skill update | Apply selected patches to document |
| **Validation set** | Selection split | Gate checks improvement before accepting |
| **Early stopping** | Gate patience | Reject updates that don't improve |
| **Training step** | Step | One rollout → reflect → update cycle |
| **Epoch** | Epoch | Full pass with slow update + meta memory |
| **Momentum** | Slow update | Longitudinal comparison at epoch boundary |
| **Meta-learning** | Meta skill | Cross-epoch teacher strategy memory |
| **Batch size** | `batch_size` | Tasks sampled per rollout |
| **Data parallelism** | `analyst_workers` | Parallel reflection workers |
| **Training set** | Train split | Items used for rollout |
| **Test set** | Test split | Held-out final evaluation |
| **Warm-up** | (implicit) | High LR early steps explore broadly |
| **Checkpointing** | Skill snapshots | Saved after each accepted step |
| **Transfer learning** | Seed skill / cross-benchmark init | Start from pre-trained skill |
## Why This Analogy Matters
1. **Familiar mental model**: ML practitioners immediately understand how to tune SkillOpt
2. **Principled hyperparameter search**: Grid search over `learning_rate` × `lr_scheduler` works just like in DL
3. **Proven mechanisms**: Gating ≈ validation-based selection, patience ≈ early stopping, slow update ≈ momentum — all with strong theoretical motivation
## Hyperparameter Transfer Rules
From our experiments, these DL intuitions transfer well:
!!! success "What transfers"
- **Cosine schedule > constant** — same as in DL, cosine annealing helps convergence
- **Moderate LR (4-16) > very high/low** — too few edits = slow learning, too many = noisy
- **Slow update helps** — longitudinal comparison prevents catastrophic forgetting across epochs
- **Meta skill memory improves reflection** — teacher benefits from cross-epoch strategy notes
!!! warning "What doesn't transfer"
- **Batch size ≠ better** — larger rollout batches have diminishing returns due to API costs
- **More epochs ≠ better** — skills converge faster than neural networks (2-4 epochs usually enough)
docs/guide/first-experiment.md
+110
View File
@@ -0,0 +1,110 @@
# Your First Experiment
This guide walks through running a complete SkillOpt training on SearchQA.
## 1. Choose a Benchmark
SkillOpt includes ready-to-use configs for several benchmarks:
| Benchmark | Difficulty | Typical Runtime |
|---|---|---|
| SearchQA | ⭐ Easy | ~30 min |
| DocVQA | ⭐⭐ Medium | ~2 hours |
| ALFWorld | ⭐⭐⭐ Hard | ~3 hours |
We'll use **SearchQA** as it's the fastest to complete.
## 2. Configure
Review the config file:
```bash
cat configs/searchqa/default.yaml
```
Key parameters (deep learning analogy in parentheses):
```yaml
train:
num_epochs: 4 # (epochs)
batch_size: 40 # (batch size)
optimizer:
learning_rate: 4 # (max edits per step)
lr_scheduler: cosine # (learning rate schedule)
use_slow_update: true # (momentum at epoch boundary)
use_meta_skill: true # (cross-epoch teacher memory)
gradient:
analyst_workers: 16 # (parallel reflection workers)
evaluation:
use_gate: true # (validation gating)
```
## 3. Train
```bash
python scripts/train.py --config configs/searchqa/default.yaml
```
You'll see output like:
```
[Step 1/8] Rollout: 20 items, 4 workers...
[Step 1/8] Score: 0.65 → Reflect...
[Step 1/8] 6 edit patches generated
[Step 1/8] Selected 4 edits (lr=8, cosine → 7.7)
[Step 1/8] Gate: val score 0.68 > 0.65 ✓ ACCEPT
[Step 2/8] ...
```
## 4. Monitor
Training outputs are saved to `outputs/<benchmark>/<run_id>/`:
```
outputs/searchqa/2024-01-15_10-30-00/
├── steps/
│ ├── step_0001/
│ │ ├── candidate_skill.md
│ │ ├── step_record.json
│ │ └── trajectory_digest.json
│ └── step_0002/
├── slow_update/
│ └── epoch_02/
├── meta_skill/
│ └── epoch_02/
├── skills/
│ └── step_0001.md
├── best_skill.md
├── history.json
└── config.yaml
```
## 5. Evaluate
Evaluate the best skill on the test split:
```bash
python scripts/eval_only.py \
--config configs/searchqa/default.yaml \
--skill outputs/searchqa/<run_id>/skills/best_skill.md
```
## WebUI
Prefer a graphical interface? Launch the WebUI:
```bash
pip install -e ".[webui]"
python -m skillopt_webui.app
```
Then open `http://localhost:7860` in your browser to configure parameters and launch training.
## Next Steps
- [Understand the training loop](training-loop.md)
- [Configuration reference](../reference/config.md)
- [Add a new benchmark](new-benchmark.md)
docs/guide/installation.md
+89
View File
@@ -0,0 +1,89 @@
# Installation
## Requirements
- Python ≥ 3.10
- At least one model API key (Azure OpenAI, OpenAI, Anthropic, or local Qwen)
## Quick Install
```bash
git clone https://github.com/microsoft/SkillOpt.git
cd SkillOpt
pip install -e .
```
## Optional Dependencies
Install extras for specific benchmarks or backends:
=== "ALFWorld"
```bash
pip install -e ".[alfworld]"
```
=== "Claude Backend"
```bash
pip install -e ".[claude]"
```
=== "Qwen (Local)"
```bash
pip install -e ".[qwen]"
```
=== "WebUI"
```bash
pip install -e ".[webui]"
```
=== "Development"
```bash
pip install -e ".[dev]"
```
=== "All"
```bash
pip install -e ".[alfworld,claude,qwen,webui,dev]"
```
## Environment Variables
Copy the example `.env` file and fill in your credentials:
```bash
cp .env.example .env
```
Edit `.env` with your API keys:
```ini
# Azure OpenAI (default backend)
AZURE_OPENAI_ENDPOINT=https://your-resource.openai.azure.com/
AZURE_OPENAI_API_KEY=your-key
# Or use OpenAI directly
OPENAI_API_KEY=sk-...
# Or Anthropic Claude
ANTHROPIC_API_KEY=sk-ant-...
```
!!! tip
You only need credentials for the backend you plan to use. Azure OpenAI is the default.
## Verify Installation
```bash
python -c "import skillopt; print('SkillOpt ready!')"
```
## Next Steps
→ [Run your first experiment](first-experiment.md)
docs/guide/new-backend.md
+130
View File
@@ -0,0 +1,130 @@
# Add a New Model Backend
SkillOpt supports multiple LLM backends. This guide shows how to add your own.
## Backend Architecture
```
skillopt/model/
├── base.py # Abstract base class
├── azure_openai.py # Azure OpenAI backend
├── openai_model.py # Direct OpenAI backend
├── claude.py # Anthropic Claude backend
├── qwen.py # Local Qwen (vLLM) backend
└── your_backend.py # Your new backend
```
## Step 1: Create the Backend
Create `skillopt/model/your_backend.py`:
```python
from skillopt.model.base import ModelBackend, ModelResponse
class YourBackend(ModelBackend):
"""Your custom model backend."""
def __init__(self, cfg: dict):
super().__init__(cfg)
self.model_name = cfg.get('model_name', 'your-default-model')
self.api_key = os.environ.get('YOUR_API_KEY', '')
self.client = self._init_client()
def _init_client(self):
"""Initialize API client."""
# TODO: Set up your API client
pass
async def generate(
self,
messages: list[dict],
temperature: float = 0.7,
max_tokens: int = 4096,
**kwargs
) -> ModelResponse:
"""
Generate a completion.
Args:
messages: Chat messages [{"role": "...", "content": "..."}]
temperature: Sampling temperature
max_tokens: Maximum tokens in response
Returns:
ModelResponse with content, usage, and metadata
"""
response = await self.client.chat(
model=self.model_name,
messages=messages,
temperature=temperature,
max_tokens=max_tokens,
)
return ModelResponse(
content=response.text,
usage={
'prompt_tokens': response.usage.input,
'completion_tokens': response.usage.output,
},
model=self.model_name,
)
async def generate_with_tools(
self,
messages: list[dict],
tools: list[dict],
**kwargs
) -> ModelResponse:
"""Generate with tool/function calling support."""
# Optional: implement if your model supports tool use
raise NotImplementedError("Tool use not supported")
```
## Step 2: Register the Backend
Add to `skillopt/model/__init__.py`:
```python
from .your_backend import YourBackend
BACKEND_REGISTRY = {
# ... existing backends ...
'your_backend': YourBackend,
}
```
## Step 3: Configure
Use your backend in any config:
```yaml
model:
backend: your_backend
model_name: your-model-id
temperature: 0.7
max_tokens: 4096
```
Set credentials via environment variable:
```bash
export YOUR_API_KEY="your-key"
```
## Required Interface
Your backend must implement these methods:
| Method | Required | Description |
|---|---|---|
| `generate()` | ✅ | Basic text generation |
| `generate_with_tools()` | Optional | Tool/function calling |
| `count_tokens()` | Optional | Token counting for context management |
## Tips
!!! tip
- Test your backend with `python -c "from skillopt.model.your_backend import YourBackend"` first
- Use `async` methods for all API calls — SkillOpt uses asyncio throughout
- Implement retry logic with exponential backoff for production use
- Add your API key to `.env.example` when submitting a PR
docs/guide/new-benchmark.md
+181
View File
@@ -0,0 +1,181 @@
# Add a New Benchmark
Extend SkillOpt with your own benchmark in ~100 lines of code.
## Overview
To add a benchmark, you need:
1. **Data Loader** — Loads and splits your dataset
2. **Environment Adapter** — Executes tasks and returns scores
3. **Config** — YAML configuration file
## Step 1: Create the Benchmark Package
```bash
mkdir -p skillopt/envs/my_benchmark
touch skillopt/envs/my_benchmark/__init__.py
```
## Step 2: Implement the Data Loader
Create `skillopt/envs/my_benchmark/loader.py`:
```python
from skillopt.data.base import DataLoader, DataItem
class MyBenchmarkDataLoader(DataLoader):
"""Load and split your benchmark data."""
def __init__(self, data_dir: str, **kwargs):
super().__init__(**kwargs)
self.data_dir = data_dir
def setup(self, cfg: dict):
"""Initialize splits based on config."""
self.split_mode = cfg.get('split_mode', 'ratio')
# Load your data here
self.items = self._load_items()
self._create_splits(cfg)
def _load_items(self) -> list[DataItem]:
"""Load raw data into DataItem objects."""
items = []
# TODO: Load your data
for entry in your_data:
items.append(DataItem(
id=entry['id'],
input=entry['question'],
ground_truth=entry['answer'],
metadata=entry.get('metadata', {})
))
return items
def get_split_items(self, split: str) -> list[DataItem]:
"""Return items for a given split (train/valid/test)."""
return self.splits[split]
```
## Step 3: Implement the Environment Adapter
Create `skillopt/envs/my_benchmark/env.py`:
```python
from skillopt.envs.base import EnvAdapter, TaskResult
class MyBenchmarkEnv(EnvAdapter):
"""Execute tasks and evaluate results."""
def __init__(self, cfg: dict):
super().__init__(cfg)
async def execute(self, item: DataItem, skill: str, model) -> TaskResult:
"""
Execute a single task.
Args:
item: The data item to process
skill: Current skill document content
model: The student model instance
Returns:
TaskResult with prediction, score, and trajectory
"""
# Build prompt with skill document
prompt = self.build_prompt(item, skill)
# Get model response
response = await model.generate(prompt)
# Extract prediction
prediction = self.parse_response(response)
# Score against ground truth
score = self.evaluate(prediction, item.ground_truth)
return TaskResult(
item_id=item.id,
prediction=prediction,
score=score,
trajectory=[
{"role": "system", "content": skill},
{"role": "user", "content": item.input},
{"role": "assistant", "content": response}
]
)
def evaluate(self, prediction: str, ground_truth: str) -> float:
"""
Score a prediction against ground truth.
Returns:
Float between 0.0 and 1.0
"""
# TODO: Implement your scoring logic
# Examples: exact match, F1, ANLS, etc.
return float(prediction.strip() == ground_truth.strip())
def build_prompt(self, item, skill: str) -> str:
"""Combine skill document with task input."""
return f"{skill}\n\n---\n\nQuestion: {item.input}"
def parse_response(self, response: str) -> str:
"""Extract the answer from model response."""
return response.strip()
```
## Step 4: Register the Benchmark
Add to `skillopt/envs/__init__.py`:
```python
from .my_benchmark.env import MyBenchmarkEnv
from .my_benchmark.loader import MyBenchmarkDataLoader
BENCHMARK_REGISTRY = {
# ... existing benchmarks ...
'my_benchmark': {
'env': MyBenchmarkEnv,
'loader': MyBenchmarkDataLoader,
},
}
```
## Step 5: Create Config
Create `configs/my_benchmark/default.yaml`:
```yaml
_base_: ['../_base_/default.yaml']
env:
name: my_benchmark
data_path: data/my_benchmark
split_mode: ratio
split_ratio: "2:1:7"
train:
num_epochs: 4
batch_size: 40
optimizer:
learning_rate: 4
lr_scheduler: cosine
use_slow_update: true
use_meta_skill: true
gradient:
analyst_workers: 16
```
## Step 6: Run
```bash
python scripts/train.py --config configs/my_benchmark/default.yaml
```
## Tips
!!! tip
- Use a small `batch_size` (10-20) for initial testing
- The `evaluate()` method is critical — a noisy metric will confuse the optimizer
docs/guide/skill-document.md
+78
View File
@@ -0,0 +1,78 @@
# Skill Document
A **skill document** is a Markdown file that serves as the "prompt weights" of your agent. SkillOpt trains this document through iterative optimization.
## What is a Skill Document?
A skill document is a structured set of instructions that tells a language model **how** to approach a specific type of task. It's analogous to learned weights in a neural network — encoding task-specific knowledge in natural language rather than floating-point parameters.
## Structure
A typical skill document contains:
```markdown
# Task Strategy
## General Approach
- Break complex problems into sub-steps
- Always verify intermediate results
## Common Patterns
- When you see X, try approach Y
- Avoid Z because it leads to errors
## Edge Cases
- If the input contains A, handle it specially by...
- Watch out for B — it requires C
## Output Format
- Always include reasoning before the answer
- Format numbers with proper units
```
## How It Evolves
During training, the skill document is modified by **edit patches**:
1. **Additions**: New rules or strategies discovered from failed trajectories
2. **Modifications**: Refining existing rules that are partially correct
3. **Deletions**: Removing rules that consistently lead to errors
Each edit is validated through the **gate** mechanism before being permanently accepted.
## Initial Skill
You can start training with:
- **Empty skill**: The system learns everything from scratch
- **Seed skill**: Provide initial instructions to bootstrap training
- **Pre-trained skill**: Transfer a skill from a related benchmark
Configure the initial skill in your YAML:
```yaml
train:
init_skill: "path/to/initial_skill.md" # or omit for empty
```
## Skill Quality Metrics
Track your skill's evolution through:
- **Validation score**: Primary metric on the selection split
- **Test score**: Final metric on held-out test data
- **Skill length**: Total tokens in the document
- **Edit acceptance rate**: Fraction of proposed edits that pass gating
## Best Practices
!!! tip "Tips for better skills"
1. **Start with a seed skill** (`env.skill_init`) if you have domain knowledge — it converges faster
2. **Use cosine LR schedule** — aggressive early exploration + careful late refinement
3. **Enable slow update** (`use_slow_update: true`) to prevent forgetting across epochs
4. **Enable meta skill** (`use_meta_skill: true`) so the teacher accumulates strategy memory
## Next Steps
- [Deep Learning Analogy](dl-analogy.md)
- [Configuration Reference](../reference/config.md)
docs/guide/training-loop.md
+92
View File
@@ -0,0 +1,92 @@
# The Training Loop
SkillOpt's core insight: **optimizing natural-language skill documents follows the same structure as training neural networks**.
## Overview
```
┌─────────────────────────────────────────────────────────┐
│ Training Loop │
│ │
│ for epoch in epochs: │
│ for step in steps: │
│ 1. Rollout — Student executes tasks │
│ 2. Reflect — Teacher analyzes trajectories │
│ 3. Aggregate — Hierarchical merge of patches │
│ 4. Select — Rank & clip edits (learning rate) │
│ 5. Update — Apply patches to skill doc │
│ 6. Gate — Validate & accept/reject │
│ │
│ Epoch Boundary: │
│ • Slow Update (longitudinal comparison & guidance) │
│ • Meta Skill (cross-epoch strategy memory) │
└─────────────────────────────────────────────────────────┘
```
## Stage Details
### 1. Rollout (Forward Pass)
The **student** model executes tasks using the current skill document as its prompt. Each task produces a trajectory and a score.
```python
# Analogy: forward pass through the network
predictions = model(input, skill_document)
scores = evaluate(predictions, ground_truth)
```
### 2. Reflect (Backward Pass)
The **teacher** model analyzes failed trajectories and produces **edit patches** — structured suggestions for improving the skill document.
Two modes:
- **Shallow**: Analyze each trajectory independently
- **Deep**: Cross-reference multiple failures to find systemic issues
```python
# Analogy: computing gradients
gradients = loss.backward() # → edit patches
```
### 3. Aggregate
Semantically similar edit patches are merged to avoid redundant edits.
### 4. Select (Gradient Clipping)
Edits are ranked by relevance score. The `learning_rate` parameter caps how many edits are applied per step — just like gradient clipping prevents overshooting.
```python
# Analogy: gradient clipping + optimizer step size
selected = top_k(edits, k=learning_rate)
```
The `lr_scheduler` adjusts this over training:
- **cosine**: Start aggressive, taper smoothly
- **linear**: Linear decay
- **constant**: Fixed rate
### 5. Update (Parameter Update)
Selected edits are applied to the skill document, producing a new version.
### 6. Gate (Validation)
The updated skill is evaluated on a **selection split** (analogous to a validation set). The update is only accepted if performance improves.
## Epoch Boundary Mechanisms
### Slow Update
At the end of each epoch (starting from epoch 2), the system performs a **longitudinal comparison**: it rolls out both the previous epoch's skill and the current skill on the same samples, categorizes items as improved/regressed/persistent_fail/stable_success, then generates high-level **guidance** that is injected into the skill document. This prevents catastrophic forgetting of earlier improvements.
### Meta Skill
A **meta-skill memory** accumulates high-level strategy notes across the entire training run. At the end of each epoch, the teacher reflects on what changed between epochs and produces a compact memory that is provided as additional context during future reflection steps.
## Next Steps
- [Understand Skill Documents](skill-document.md)
- [DL ↔ SkillOpt analogy table](dl-analogy.md)
docs/index.md
+170
View File
@@ -0,0 +1,170 @@
---
hide:
- navigation
---
<div class="hero" markdown>
# SkillOpt
### Train Agent Skills Like Neural Networks
*Optimize natural-language skill documents through iterative rollout, reflection, and gated validation — with epochs, learning rates, and validation gates — without touching model weights.*
[Get Started :material-rocket-launch:](guide/installation.md){ .md-button .md-button--primary }
[View on GitHub :material-github:](https://github.com/microsoft/SkillOpt){ .md-button }
</div>
---
## How It Works
<div class="pipeline-container" markdown>
<div class="pipeline-wrapper">
<div class="pipeline-stage" id="stage-rollout">
<div class="stage-icon">🎯</div>
<div class="stage-label">Rollout</div>
<div class="stage-desc">Student executes tasks</div>
</div>
<div class="pipeline-arrow"><div class="flow-line"></div></div>
<div class="pipeline-stage" id="stage-reflect">
<div class="stage-icon">🔍</div>
<div class="stage-label">Reflect</div>
<div class="stage-desc">Teacher analyzes trajectories</div>
</div>
<div class="pipeline-arrow"><div class="flow-line"></div></div>
<div class="pipeline-stage" id="stage-aggregate">
<div class="stage-icon">🔗</div>
<div class="stage-label">Aggregate</div>
<div class="stage-desc">Merge edit patches</div>
</div>
<div class="pipeline-arrow"><div class="flow-line"></div></div>
<div class="pipeline-stage" id="stage-select">
<div class="stage-icon">✂️</div>
<div class="stage-label">Select</div>
<div class="stage-desc">Rank & clip edits</div>
</div>
<div class="pipeline-arrow"><div class="flow-line"></div></div>
<div class="pipeline-stage" id="stage-update">
<div class="stage-icon">📝</div>
<div class="stage-label">Update</div>
<div class="stage-desc">Apply to skill doc</div>
</div>
<div class="pipeline-arrow"><div class="flow-line"></div></div>
<div class="pipeline-stage" id="stage-gate">
<div class="stage-icon">🚦</div>
<div class="stage-label">Gate</div>
<div class="stage-desc">Validate & accept</div>
</div>
</div>
<div class="pipeline-epoch-bar">
<div class="epoch-mechanism">🔄 Slow Update</div>
<div class="epoch-mechanism">🧠 Meta Skill</div>
<div class="epoch-label">Epoch Boundary</div>
</div>
</div>
---
## Deep Learning Analogy
SkillOpt brings the familiar deep-learning training paradigm to agentic prompt optimization:
| Deep Learning | SkillOpt |
|---|---|
| Model weights | Skill document (Markdown) |
| Forward pass | Rollout (student executes tasks) |
| Loss / gradient | Reflect (teacher produces edit patches) |
| Gradient clipping | Edit selection (`learning_rate` = max edits) |
| SGD step | Patch application to skill |
| Validation set | Gated evaluation on selection split |
| LR schedule | `lr_scheduler`: cosine, linear, constant |
| Epochs | Multi-epoch with slow update & meta skill memory |
---
## Supported Benchmarks
| Benchmark | Type | Config |
|---|---|---|
| **DocVQA** | Document QA | `configs/docvqa/` |
| **ALFWorld** | Embodied AI | `configs/alfworld/` |
| **OfficeQA** | Enterprise QA | `configs/officeqa/` |
| **SearchQA** | Open-domain QA | `configs/searchqa/` |
| **LiveMathBench** | Math reasoning | `configs/livemathematicianbench/` |
| **SWEBench** | Software Engineering | `configs/swebench/` |
| + 5 more | Various | See [docs](guide/first-experiment.md) |
---
## Quick Example
```bash
# Install
pip install -e .
# Configure credentials
export AZURE_OPENAI_ENDPOINT="https://your-resource.openai.azure.com/"
export AZURE_OPENAI_API_KEY="your-key"
# Train on SearchQA
python scripts/train.py --config configs/searchqa/default.yaml
# Evaluate best skill
python scripts/eval_only.py \
--config configs/searchqa/default.yaml \
--skill outputs/best_skill.md
```
---
<div class="grid cards" markdown>
- :material-book-open-variant:{ .lg .middle } **Getting Started**
---
Install SkillOpt, configure your API keys, and run your first experiment in 5 minutes.
[:octicons-arrow-right-24: Installation](guide/installation.md)
- :material-puzzle:{ .lg .middle } **Add a Benchmark**
---
Extend SkillOpt with your own benchmark in ~100 lines of code.
[:octicons-arrow-right-24: Extension Guide](guide/new-benchmark.md)
- :material-cog:{ .lg .middle } **Configuration**
---
Full reference for all hyperparameters with deep learning analogies.
[:octicons-arrow-right-24: Config Reference](reference/config.md)
- :material-monitor-dashboard:{ .lg .middle } **WebUI**
---
Configure, launch, and monitor training from your browser.
[:octicons-arrow-right-24: WebUI Guide](guide/first-experiment.md#webui)
</div>
docs/reference/api.md
+81
View File
@@ -0,0 +1,81 @@
# API Reference
## Core Classes
### `EnvAdapter`
Abstract base class for benchmark environments.
```python
class EnvAdapter(ABC):
async def execute(self, item, skill, model) -> TaskResult
def evaluate(self, prediction, ground_truth) -> float
def build_prompt(self, item, skill) -> str
```
### `DataLoader`
Abstract base class for data loading and splitting.
```python
class DataLoader(ABC):
def setup(self, cfg: dict) -> None
def get_split_items(self, split: str) -> list[DataItem]
```
### `ModelBackend`
Abstract base class for LLM backends.
```python
class ModelBackend(ABC):
async def generate(self, messages, **kwargs) -> ModelResponse
async def generate_with_tools(self, messages, tools, **kwargs) -> ModelResponse
```
### `Trainer`
Main training loop orchestrator.
```python
class Trainer:
def __init__(self, cfg: dict)
async def train(self) -> TrainResult
async def evaluate(self, skill: str, split: str) -> EvalResult
```
## Data Classes
### `DataItem`
```python
@dataclass
class DataItem:
id: str
input: str
ground_truth: str
metadata: dict = field(default_factory=dict)
```
### `TaskResult`
```python
@dataclass
class TaskResult:
item_id: str
prediction: str
score: float
trajectory: list[dict]
```
### `ModelResponse`
```python
@dataclass
class ModelResponse:
content: str
usage: dict
model: str
```
For detailed source code, see the [`skillopt/`](https://github.com/microsoft/SkillOpt/tree/main/skillopt) directory.
docs/reference/cli.md
+71
View File
@@ -0,0 +1,71 @@
# CLI Reference
## Training
```bash
python scripts/train.py --config <config.yaml> [overrides...]
```
### Arguments
| Argument | Description |
|---|---|
| `--config` | Path to YAML config file (required) |
| `key=value` | Override any config parameter |
### Examples
```bash
# Basic training
python scripts/train.py --config configs/searchqa/default.yaml
# With overrides
python scripts/train.py \
--config configs/searchqa/default.yaml \
--cfg-options optimizer.learning_rate=16 optimizer.lr_scheduler=linear
# With custom initial skill
python scripts/train.py \
--config configs/searchqa/default.yaml \
--cfg-options env.skill_init=skills/my_seed.md
```
## Evaluation
```bash
python scripts/eval_only.py --config <config.yaml> --skill <skill.md>
```
### Arguments
| Argument | Description |
|---|---|
| `--config` | Path to YAML config file (required) |
| `--skill` | Path to skill document to evaluate (required) |
| `--split` | Evaluation split: `test` (default), `valid`, `train` |
### Examples
```bash
# Evaluate best skill on test set
python scripts/eval_only.py \
--config configs/searchqa/default.yaml \
--skill outputs/searchqa/run_001/skills/best_skill.md
# Evaluate on validation set
python scripts/eval_only.py \
--config configs/searchqa/default.yaml \
--skill outputs/searchqa/run_001/skills/best_skill.md \
--split valid
```
## WebUI
```bash
python -m skillopt_webui.app [--port PORT] [--share]
```
| Argument | Default | Description |
|---|---|---|
| `--port` | 7860 | Port number |
| `--share` | false | Create public Gradio link |
docs/reference/config.md
+72
View File
@@ -0,0 +1,72 @@
# Configuration Reference
Complete reference for all SkillOpt configuration parameters.
## Model
| Parameter | Type | Default | Description |
|---|---|---|---|
| `model.backend` | str | `azure_openai` | Backend: `azure_openai` / `openai_chat` / `claude_code_exec` / `qwen` |
| `model.teacher` | str | `gpt-5.5` | Teacher model (for reflection & slow update) |
| `model.student` | str | `gpt-5.5` | Student model (for rollout execution) |
| `model.reasoning_effort` | str | `medium` | Reasoning effort level |
## Training (`train`)
| Parameter | Type | Default | DL Analogy | Description |
|---|---|---|---|---|
| `train.num_epochs` | int | 4 | Epochs | Number of training epochs |
| `train.batch_size` | int | 40 | Batch size | Tasks sampled per step |
| `train.accumulation` | int | 1 | Gradient accumulation | Accumulation rounds per step |
| `train.seed` | int | 42 | Random seed | Reproducibility seed |
## Gradient / Reflection (`gradient`)
| Parameter | Type | Default | Description |
|---|---|---|---|
| `gradient.minibatch_size` | int | 8 | Reflect minibatch size |
| `gradient.merge_batch_size` | int | 8 | Patch merge batch size |
| `gradient.analyst_workers` | int | 16 | Parallel reflection workers |
| `gradient.max_analyst_rounds` | int | 3 | Max rounds of analyst reflection |
| `gradient.failure_only` | bool | `false` | Only reflect on failures |
## Optimizer (`optimizer`)
| Parameter | Type | Default | DL Analogy | Description |
|---|---|---|---|---|
| `optimizer.learning_rate` | int | 4 | Learning rate | Max edit patches per step (edit budget) |
| `optimizer.min_learning_rate` | int | 2 | Min LR | Min edits for decay schedulers |
| `optimizer.lr_scheduler` | str | `cosine` | LR schedule | `constant` / `linear` / `cosine` / `autonomous` |
| `optimizer.skill_update_mode` | str | `patch` | — | `patch` / `rewrite_from_suggestions` / `full_rewrite_minibatch` |
| `optimizer.use_slow_update` | bool | `true` | Momentum | Epoch-boundary longitudinal comparison & guidance |
| `optimizer.slow_update_samples` | int | 20 | — | Samples for slow update evaluation |
| `optimizer.use_meta_skill` | bool | `true` | Meta-learning | Cross-epoch teacher-side strategy memory |
| `optimizer.longitudinal_pair_policy` | str | `mixed` | — | `mixed` / `changed` / `unchanged` |
## Evaluation (`evaluation`)
| Parameter | Type | Default | Description |
|---|---|---|---|
| `evaluation.use_gate` | bool | `true` | Enable validation gating (accept/reject updates) |
| `evaluation.eval_test` | bool | `true` | Run test evaluation after training |
## Environment (`env`)
| Parameter | Type | Default | Description |
|---|---|---|---|
| `env.name` | str | — | Benchmark name (e.g., `searchqa`, `docvqa`) |
| `env.data_path` | str | — | Path to dataset |
| `env.skill_init` | str | — | Path to initial seed skill (optional) |
| `env.split_mode` | str | `ratio` | `ratio` or `split_dir` |
| `env.split_ratio` | str | `2:1:7` | Train:val:test ratio |
| `env.exec_timeout` | int | 120 | Per-task timeout in seconds |
| `env.out_root` | str | — | Output directory |
## Azure OpenAI Credentials
| Variable | Description |
|---|---|
| `AZURE_OPENAI_ENDPOINT` / `model.azure_openai_endpoint` | Azure resource endpoint |
| `AZURE_OPENAI_API_KEY` / `model.azure_openai_api_key` | Azure API key |
| `OPENAI_API_KEY` | OpenAI API key (for `openai_chat` backend) |
| `ANTHROPIC_API_KEY` | Anthropic API key (for `claude_code_exec` backend) |
mkdocs.yml
+77
View File
@@ -0,0 +1,77 @@
site_name: SkillOpt Documentation
site_url: https://microsoft.github.io/SkillOpt
site_description: "SkillOpt: Agentic Skill Optimization via Reflective Training Loops"
repo_url: https://github.com/microsoft/SkillOpt
repo_name: microsoft/SkillOpt
theme:
name: material
palette:
- scheme: default
primary: indigo
accent: deep purple
toggle:
icon: material/brightness-7
name: Switch to dark mode
- scheme: slate
primary: indigo
accent: deep purple
toggle:
icon: material/brightness-4
name: Switch to light mode
features:
- navigation.instant
- navigation.tracking
- navigation.sections
- navigation.expand
- navigation.top
- content.code.copy
- content.tabs.link
- search.suggest
- search.highlight
icon:
repo: fontawesome/brands/github
font:
text: Inter
code: JetBrains Mono
nav:
- Home: index.md
- Getting Started:
- Installation: guide/installation.md
- First Experiment: guide/first-experiment.md
- Configuration: guide/configuration.md
- Core Concepts:
- Training Loop: guide/training-loop.md
- Skill Document: guide/skill-document.md
- Deep Learning Analogy: guide/dl-analogy.md
- Extension Guides:
- Add a New Benchmark: guide/new-benchmark.md
- Add a New Model Backend: guide/new-backend.md
- Reference:
- Configuration Reference: reference/config.md
- CLI Reference: reference/cli.md
- API Reference: reference/api.md
- Contributing: contributing.md
markdown_extensions:
- admonition
- pymdownx.details
- pymdownx.superfences
- pymdownx.tabbed:
alternate_style: true
- pymdownx.highlight:
anchor_linenums: true
- pymdownx.inlinehilite
- pymdownx.emoji:
emoji_index: !!python/name:material.extensions.emoji.twemoji
emoji_generator: !!python/name:material.extensions.emoji.to_svg
- attr_list
- md_in_html
- toc:
permalink: true
plugins:
- search
pyproject.toml
+75
View File
@@ -0,0 +1,75 @@
[build-system]
requires = ["setuptools>=68.0", "wheel"]
build-backend = "setuptools.build_meta"
[project]
name = "skillopt"
version = "0.1.0"
description = "SkillOpt: Agentic Skill Optimization via Reflective Training Loops"
readme = "README.md"
license = {text = "MIT"}
requires-python = ">=3.10"
authors = [
{name = "SkillOpt Team"},
]
keywords = ["agent", "prompt-optimization", "skill-learning", "LLM", "agentic"]
classifiers = [
"Development Status :: 3 - Alpha",
"Intended Audience :: Science/Research",
"License :: OSI Approved :: MIT License",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12",
"Topic :: Scientific/Engineering :: Artificial Intelligence",
]
dependencies = [
"openai>=1.30.0",
"pyyaml>=6.0",
"numpy>=1.24.0",
"openpyxl>=3.1.0",
"azure-identity>=1.15.0",
"azure-core>=1.30.0",
"httpx>=0.27.0",
]
[project.optional-dependencies]
# Benchmark-specific dependencies
alfworld = ["alfworld>=0.4.0", "gymnasium>=0.29.0"]
# Claude model backend
claude = ["claude-agent-sdk>=0.1.0"]
# Qwen local model backend (via vLLM)
qwen = ["vllm>=0.4.0"]
# Documentation site
docs = ["mkdocs-material>=9.5.0", "mkdocstrings[python]>=0.24.0"]
# WebUI dashboard
webui = ["gradio>=4.0.0"]
# Development tools
dev = ["ruff>=0.4.0", "pytest>=8.0.0"]
# All optional dependencies (except docs/dev/webui)
all = [
"alfworld>=0.4.0",
"gymnasium>=0.29.0",
"claude-agent-sdk>=0.1.0",
]
[project.scripts]
skillopt-train = "scripts.train:main"
skillopt-eval = "scripts.eval_only:main"
[project.urls]
Homepage = "https://github.com/microsoft/SkillOpt"
Documentation = "https://microsoft.github.io/SkillOpt"
Repository = "https://github.com/microsoft/SkillOpt"
Issues = "https://github.com/microsoft/SkillOpt/issues"
[tool.setuptools.packages.find]
include = ["skillopt*", "scripts*"]
[tool.ruff]
line-length = 120
target-version = "py310"
[tool.ruff.lint]
select = ["E", "F", "I", "W"]
ignore = ["E501"]
requirements.txt
+25
View File
@@ -0,0 +1,25 @@
# ── Core ──────────────────────────────────────────
openai>=1.30.0
pyyaml>=6.0
numpy>=1.24.0
openpyxl>=3.1.0
azure-identity>=1.15.0
azure-core>=1.30.0
httpx>=0.27.0
# ── Optional: ALFWorld benchmark ──────────────────
# alfworld>=0.4.0
# gymnasium>=0.29.0
# ── Optional: Claude model backend ────────────────
# claude-agent-sdk>=0.1.0
# ── Optional: Qwen local model (via vLLM) ────────
# vllm>=0.4.0
# ── Optional: WebUI dashboard ────────────────────
# gradio>=4.0.0
# ── Optional: Documentation site ─────────────────
# mkdocs-material>=9.5.0
# mkdocstrings[python]>=0.24.0
scripts/__init__.py
View File
scripts/download_babyvision.py
+53
View File
@@ -0,0 +1,53 @@
#!/usr/bin/env python3
"""Download BabyVision from Hugging Face and convert it to local meta_data.jsonl + images/ format."""
from __future__ import annotations
import argparse
import json
import os
from pathlib import Path
def parse_args() -> argparse.Namespace:
p = argparse.ArgumentParser(description=__doc__)
p.add_argument("--out_dir", type=str, required=True)
p.add_argument("--dataset", type=str, default="UnipatAI/BabyVision")
p.add_argument("--split", type=str, default="train")
return p.parse_args()
def main() -> None:
args = parse_args()
try:
from datasets import load_dataset
except ImportError as exc: # pragma: no cover
raise SystemExit("Please install `datasets` first: pip install datasets pillow") from exc
out_dir = Path(args.out_dir).resolve()
images_dir = out_dir / "images"
meta_path = out_dir / "meta_data.jsonl"
images_dir.mkdir(parents=True, exist_ok=True)
dataset = load_dataset(args.dataset, split=args.split)
with open(meta_path, "w", encoding="utf-8") as outf:
for idx, row in enumerate(dataset):
image = row.get("image")
if image is None:
continue
task_id = str(row.get("taskId") or row.get("id") or idx + 1)
image_name = f"{task_id}.png"
image_path = images_dir / image_name
image.save(image_path)
record = dict(row)
record["image"] = image_name
outf.write(json.dumps(record, ensure_ascii=False) + "\n")
print(f"Saved BabyVision to {out_dir}")
print(f"Metadata: {meta_path}")
print(f"Images: {images_dir}")
if __name__ == "__main__":
main()
scripts/eval_only.py
+451
View File
@@ -0,0 +1,451 @@
#!/usr/bin/env python3
"""ReflACT eval-only: run a single skill on a dataset without training.
Usage
-----
python scripts/eval_only.py \
--config configs/spreadsheetbench/default.yaml \
--skill skillopt/envs/spreadsheetbench/skills/initial.md \
--split_dir /path/to/split \
--out_root outputs/eval_skill0
All YAML keys can be overridden from the CLI, same as train.py.
"""
from __future__ import annotations
import argparse
import datetime
import json
import os
import sys
_SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__))
_PROJECT_ROOT = os.path.dirname(_SCRIPT_DIR)
if _PROJECT_ROOT not in sys.path:
sys.path.insert(0, _PROJECT_ROOT)
from skillopt.model import (
configure_azure_openai,
configure_claude_code_exec,
configure_codex_exec,
set_reasoning_effort,
set_student_backend,
set_student_deployment,
set_teacher_backend,
set_teacher_deployment,
)
from skillopt.model.common import default_model_for_backend, normalize_backend_name
_OPENAI_DEFAULT_MODEL_SENTINELS = {"gpt-5.4", "gpt-5.5"}
from skillopt.utils import compute_score
# ── Reuse registry from train.py ───────────────────────────────────────────
_ENV_REGISTRY: dict[str, type] = {}
def _register_builtins() -> None:
try:
from skillopt.envs.alfworld.adapter import ALFWorldAdapter
_ENV_REGISTRY["alfworld"] = ALFWorldAdapter
except ImportError:
pass
try:
from skillopt.envs.searchqa.adapter import SearchQAAdapter
_ENV_REGISTRY["searchqa"] = SearchQAAdapter
except ImportError:
pass
try:
from skillopt.envs.livemathematicianbench.adapter import LiveMathematicianBenchAdapter
_ENV_REGISTRY["livemathematicianbench"] = LiveMathematicianBenchAdapter
except ImportError:
pass
try:
from skillopt.envs.babyvision.adapter import BabyVisionAdapter
_ENV_REGISTRY["babyvision"] = BabyVisionAdapter
except ImportError:
pass
try:
from skillopt.envs.spreadsheetbench.adapter import SpreadsheetBenchAdapter
_ENV_REGISTRY["spreadsheetbench"] = SpreadsheetBenchAdapter
except ImportError:
pass
try:
from skillopt.envs.mmrb.adapter import MMRBAdapter
_ENV_REGISTRY["mmrb"] = MMRBAdapter
except ImportError:
pass
try:
from skillopt.envs.docvqa.adapter import DocVQAAdapter
_ENV_REGISTRY["docvqa"] = DocVQAAdapter
except ImportError:
pass
try:
from skillopt.envs.mathverse.adapter import MathVerseAdapter
_ENV_REGISTRY["mathverse"] = MathVerseAdapter
except ImportError:
pass
try:
from skillopt.envs.officeqa.adapter import OfficeQAAdapter
_ENV_REGISTRY["officeqa"] = OfficeQAAdapter
except ImportError:
pass
try:
from skillopt.envs.sealqa.adapter import SealQAAdapter
_ENV_REGISTRY["sealqa"] = SealQAAdapter
except ImportError:
pass
try:
from skillopt.envs.swebench.adapter import SWEBenchAdapter
_ENV_REGISTRY["swebench"] = SWEBenchAdapter
except ImportError:
pass
def get_adapter(cfg: dict):
_register_builtins()
env_name = cfg.get("env", "alfworld")
if env_name not in _ENV_REGISTRY:
raise ValueError(
f"Unknown environment '{env_name}'. "
f"Available: {list(_ENV_REGISTRY.keys())}"
)
adapter_cls = _ENV_REGISTRY[env_name]
import inspect
sig = inspect.signature(adapter_cls.__init__)
accepted = set(sig.parameters.keys()) - {"self"}
adapter_kwargs = {k: cfg[k] for k in accepted if k in cfg}
return adapter_cls(**adapter_kwargs)
# ── CLI ────────────────────────────────────────────────────────────────────
_BOOL = lambda x: str(x).lower() in ("true", "1", "yes") # noqa: E731
def parse_args() -> argparse.Namespace:
p = argparse.ArgumentParser(description="ReflACT eval-only")
p.add_argument("--config", type=str, required=True)
p.add_argument("--skill", type=str, required=True,
help="Path to skill .md file to evaluate")
p.add_argument("--split", type=str, default="all",
help="Which split to eval: train/valid_seen/valid_unseen/all (default: all)")
p.add_argument("--cfg-options", nargs="+", default=[],
help="Override config: section.key=value")
# Legacy flat overrides
p.add_argument("--env", type=str)
p.add_argument("--backend", type=str,
choices=["azure_openai", "codex", "codex_exec", "claude", "claude_chat", "claude_code_exec"])
p.add_argument("--teacher_model", type=str)
p.add_argument("--student_model", type=str)
p.add_argument("--teacher_backend", type=str)
p.add_argument("--student_backend", type=str)
p.add_argument("--reasoning_effort", type=str,
choices=["", "low", "medium", "high", "xhigh", "max"])
p.add_argument("--azure_endpoint", type=str)
p.add_argument("--azure_api_version", type=str)
p.add_argument("--azure_api_key", type=str)
p.add_argument("--azure_openai_endpoint", type=str)
p.add_argument("--azure_openai_api_version", type=str)
p.add_argument("--azure_openai_api_key", type=str)
p.add_argument("--azure_openai_auth_mode", type=str)
p.add_argument("--azure_openai_ad_scope", type=str)
p.add_argument("--azure_openai_managed_identity_client_id", type=str)
p.add_argument("--teacher_azure_openai_endpoint", type=str)
p.add_argument("--teacher_azure_openai_api_version", type=str)
p.add_argument("--teacher_azure_openai_api_key", type=str)
p.add_argument("--teacher_azure_openai_auth_mode", type=str)
p.add_argument("--teacher_azure_openai_ad_scope", type=str)
p.add_argument("--teacher_azure_openai_managed_identity_client_id", type=str)
p.add_argument("--student_azure_openai_endpoint", type=str)
p.add_argument("--student_azure_openai_api_version", type=str)
p.add_argument("--student_azure_openai_api_key", type=str)
p.add_argument("--student_azure_openai_auth_mode", type=str)
p.add_argument("--student_azure_openai_ad_scope", type=str)
p.add_argument("--student_azure_openai_managed_identity_client_id", type=str)
p.add_argument("--codex_exec_path", type=str)
p.add_argument("--codex_exec_sandbox", type=str)
p.add_argument("--codex_exec_profile", type=str)
p.add_argument("--codex_exec_full_auto", type=_BOOL)
p.add_argument("--codex_exec_reasoning_effort", type=str)
p.add_argument("--codex_exec_use_sdk", type=str)
p.add_argument("--codex_exec_network_access", type=_BOOL)
p.add_argument("--codex_exec_web_search", type=_BOOL)
p.add_argument("--codex_exec_approval_policy", type=str)
p.add_argument("--claude_code_exec_path", type=str)
p.add_argument("--claude_code_exec_profile", type=str)
p.add_argument("--claude_code_exec_use_sdk", type=str)
p.add_argument("--claude_code_exec_effort", type=str)
p.add_argument("--claude_code_exec_max_thinking_tokens", type=int)
p.add_argument("--out_root", type=str)
p.add_argument("--data_path", type=str)
p.add_argument("--split_mode", type=str,
choices=["ratio", "split_dir"])
p.add_argument("--split_ratio", type=str)
p.add_argument("--split_seed", type=int)
p.add_argument("--split_dir", type=str)
p.add_argument("--split_output_dir", type=str)
p.add_argument("--data_root", type=str)
p.add_argument("--max_turns", type=int)
p.add_argument("--workers", type=int)
p.add_argument("--max_api_workers", type=int)
p.add_argument("--seed", type=int)
p.add_argument("--test_env_num", type=int)
p.add_argument("--mode", type=str,
help="SpreadsheetBench: single/multi/react (default comes from config)")
return p.parse_args()
def main() -> None:
args = parse_args()
from skillopt.config import load_config as _load, flatten_config, is_structured
cfg = _load(args.config, overrides=args.cfg_options)
structured = is_structured(cfg)
# Apply legacy --key value overrides
cli = {k: v for k, v in vars(args).items()
if v is not None and k not in ("config", "skill", "split", "cfg_options")}
if cli:
if structured:
from skillopt.config import apply_overrides
_MAP = {
"backend": "model.backend",
"teacher_model": "model.teacher",
"student_model": "model.student",
"teacher_backend": "model.teacher_backend",
"student_backend": "model.student_backend",
"reasoning_effort": "model.reasoning_effort",
"azure_endpoint": "model.azure_endpoint",
"azure_api_version": "model.azure_api_version",
"azure_api_key": "model.azure_api_key",
"azure_openai_endpoint": "model.azure_openai_endpoint",
"azure_openai_api_version": "model.azure_openai_api_version",
"azure_openai_api_key": "model.azure_openai_api_key",
"azure_openai_auth_mode": "model.azure_openai_auth_mode",
"azure_openai_ad_scope": "model.azure_openai_ad_scope",
"azure_openai_managed_identity_client_id": "model.azure_openai_managed_identity_client_id",
"teacher_azure_openai_endpoint": "model.teacher_azure_openai_endpoint",
"teacher_azure_openai_api_version": "model.teacher_azure_openai_api_version",
"teacher_azure_openai_api_key": "model.teacher_azure_openai_api_key",
"teacher_azure_openai_auth_mode": "model.teacher_azure_openai_auth_mode",
"teacher_azure_openai_ad_scope": "model.teacher_azure_openai_ad_scope",
"teacher_azure_openai_managed_identity_client_id": "model.teacher_azure_openai_managed_identity_client_id",
"student_azure_openai_endpoint": "model.student_azure_openai_endpoint",
"student_azure_openai_api_version": "model.student_azure_openai_api_version",
"student_azure_openai_api_key": "model.student_azure_openai_api_key",
"student_azure_openai_auth_mode": "model.student_azure_openai_auth_mode",
"student_azure_openai_ad_scope": "model.student_azure_openai_ad_scope",
"student_azure_openai_managed_identity_client_id": "model.student_azure_openai_managed_identity_client_id",
"codex_exec_path": "model.codex_exec_path",
"codex_exec_sandbox": "model.codex_exec_sandbox",
"codex_exec_profile": "model.codex_exec_profile",
"codex_exec_full_auto": "model.codex_exec_full_auto",
"codex_exec_reasoning_effort": "model.codex_exec_reasoning_effort",
"codex_exec_use_sdk": "model.codex_exec_use_sdk",
"codex_exec_network_access": "model.codex_exec_network_access",
"codex_exec_web_search": "model.codex_exec_web_search",
"codex_exec_approval_policy": "model.codex_exec_approval_policy",
"claude_code_exec_path": "model.claude_code_exec_path",
"claude_code_exec_profile": "model.claude_code_exec_profile",
"claude_code_exec_use_sdk": "model.claude_code_exec_use_sdk",
"claude_code_exec_effort": "model.claude_code_exec_effort",
"claude_code_exec_max_thinking_tokens": "model.claude_code_exec_max_thinking_tokens",
"seed": "train.seed",
"test_env_num": "evaluation.test_env_num",
"env": "env.name",
"out_root": "env.out_root",
}
mapped = []
for k, v in cli.items():
dotted = _MAP.get(k)
if dotted:
mapped.append(f"{dotted}={v}")
else:
mapped.append(f"env.{k}={v}")
apply_overrides(cfg, mapped)
else:
cfg.update(cli)
cfg = flatten_config(cfg) if structured else cfg
for new_key, old_key in (
("azure_openai_endpoint", "azure_endpoint"),
("azure_openai_api_version", "azure_api_version"),
("azure_openai_api_key", "azure_api_key"),
):
if cfg.get(new_key) in (None, "") and cfg.get(old_key) not in (None, ""):
cfg[new_key] = cfg[old_key]
explicit_backend = getattr(args, "backend", None)
if explicit_backend is None:
for option in args.cfg_options or []:
key = str(option).split("=", 1)[0].strip()
if key == "model.backend":
explicit_backend = str(option).split("=", 1)[1].strip()
break
backend = normalize_backend_name(cfg.get("model_backend") or cfg.get("student_backend") or "azure_openai")
def _has_model_override(dotted_key: str, legacy_key: str) -> bool:
if getattr(args, legacy_key, None) is not None:
return True
for option in args.cfg_options or []:
key = str(option).split("=", 1)[0].strip()
if key == dotted_key:
return True
return False
if explicit_backend is not None:
backend = normalize_backend_name(explicit_backend)
cfg["model_backend"] = backend
if backend in {"claude", "claude_chat"}:
cfg.setdefault("teacher_backend", "claude_chat")
cfg.setdefault("student_backend", "claude_chat")
elif backend in {"codex", "codex_exec"}:
cfg.setdefault("teacher_backend", "openai_chat")
cfg.setdefault("student_backend", "codex_exec")
elif backend == "claude_code_exec":
cfg.setdefault("teacher_backend", "openai_chat")
cfg.setdefault("student_backend", "claude_code_exec")
else:
cfg.setdefault("teacher_backend", "openai_chat")
cfg.setdefault("student_backend", "openai_chat")
else:
cfg.setdefault("teacher_backend", "openai_chat")
cfg.setdefault("student_backend", "openai_chat")
if cfg.get("teacher_backend") == "claude_chat":
if (
str(cfg.get("teacher_model", "") or "").strip() in _OPENAI_DEFAULT_MODEL_SENTINELS
and not _has_model_override("model.teacher", "teacher_model")
):
cfg["teacher_model"] = default_model_for_backend("claude_chat")
if cfg.get("student_backend") == "claude_chat":
if (
str(cfg.get("student_model", "") or "").strip() in _OPENAI_DEFAULT_MODEL_SENTINELS
and not _has_model_override("model.student", "student_model")
):
cfg["student_model"] = default_model_for_backend("claude_chat")
if cfg.get("student_backend") == "claude_code_exec":
if (
str(cfg.get("student_model", "") or "").strip() in _OPENAI_DEFAULT_MODEL_SENTINELS
and not _has_model_override("model.student", "student_model")
):
cfg["student_model"] = default_model_for_backend("claude_chat")
if not cfg.get("out_root"):
env = cfg.get("env", "unknown")
model = cfg.get("student_model", "unknown").replace("/", "-")
ts = datetime.datetime.now().strftime("%Y%m%d_%H%M%S")
cfg["out_root"] = os.path.join("outputs", f"eval_{env}_{model}_{ts}")
cfg["out_root"] = os.path.abspath(cfg["out_root"])
out_root = cfg["out_root"]
os.makedirs(out_root, exist_ok=True)
# Load skill
skill_path = os.path.abspath(args.skill)
with open(skill_path) as f:
skill_content = f.read()
print(f" [skill] {skill_path} ({len(skill_content)} chars)")
# Configure models
configure_azure_openai(
endpoint=(cfg.get("azure_openai_endpoint") or cfg.get("azure_endpoint") or None),
api_version=(cfg.get("azure_openai_api_version") or cfg.get("azure_api_version") or None),
api_key=(cfg.get("azure_openai_api_key") or cfg.get("azure_api_key") or None),
auth_mode=cfg.get("azure_openai_auth_mode") or None,
ad_scope=cfg.get("azure_openai_ad_scope") or None,
managed_identity_client_id=cfg.get("azure_openai_managed_identity_client_id") or None,
teacher_endpoint=cfg.get("teacher_azure_openai_endpoint") or None,
teacher_api_version=cfg.get("teacher_azure_openai_api_version") or None,
teacher_api_key=cfg.get("teacher_azure_openai_api_key") or None,
teacher_auth_mode=cfg.get("teacher_azure_openai_auth_mode") or None,
teacher_ad_scope=cfg.get("teacher_azure_openai_ad_scope") or None,
teacher_managed_identity_client_id=(
cfg.get("teacher_azure_openai_managed_identity_client_id") or None
),
student_endpoint=cfg.get("student_azure_openai_endpoint") or None,
student_api_version=cfg.get("student_azure_openai_api_version") or None,
student_api_key=cfg.get("student_azure_openai_api_key") or None,
student_auth_mode=cfg.get("student_azure_openai_auth_mode") or None,
student_ad_scope=cfg.get("student_azure_openai_ad_scope") or None,
student_managed_identity_client_id=(
cfg.get("student_azure_openai_managed_identity_client_id") or None
),
)
set_teacher_backend(cfg.get("teacher_backend", "openai_chat"))
set_student_backend(cfg.get("student_backend", "openai_chat"))
set_teacher_deployment(cfg.get("teacher_model", default_model_for_backend(backend)))
set_student_deployment(cfg.get("student_model", default_model_for_backend(backend)))
configure_codex_exec(
path=cfg.get("codex_exec_path", "codex"),
sandbox=cfg.get("codex_exec_sandbox", "workspace-write"),
profile=cfg.get("codex_exec_profile", ""),
full_auto=cfg.get("codex_exec_full_auto", False),
reasoning_effort=cfg.get("codex_exec_reasoning_effort", "none"),
use_sdk=cfg.get("codex_exec_use_sdk", None),
network_access=cfg.get("codex_exec_network_access", False),
web_search=cfg.get("codex_exec_web_search", False),
approval_policy=cfg.get("codex_exec_approval_policy", "never"),
)
configure_claude_code_exec(
path=cfg.get("claude_code_exec_path", "claude"),
profile=cfg.get("claude_code_exec_profile", ""),
use_sdk=cfg.get("claude_code_exec_use_sdk", None),
effort=cfg.get("claude_code_exec_effort", cfg.get("reasoning_effort", "medium")),
max_thinking_tokens=cfg.get("claude_code_exec_max_thinking_tokens", 16384),
)
set_reasoning_effort(cfg.get("reasoning_effort", "") or None)
# Build adapter
adapter = get_adapter(cfg)
adapter.setup(cfg)
seed = cfg.get("seed", 42)
split = args.split or "all"
if split == "all":
items = (
adapter.build_eval_env(0, "train", seed)
+ adapter.build_eval_env(0, "valid_seen", seed)
+ adapter.build_eval_env(0, "valid_unseen", seed)
)
else:
env_num = cfg.get("test_env_num", 0)
items = adapter.build_eval_env(env_num, split, seed)
print(f"\n [eval] split={split} items={len(items)}")
print(f" [eval] out_root={out_root}")
print(f"{'='*60}")
# Run rollout
results = adapter.rollout(items, skill_content, out_root)
# Score
hard, soft = compute_score(results)
print(f"\n{'='*60}")
print(f" Results: hard={hard:.4f} soft={soft:.4f} (n={len(results)})")
print(f"{'='*60}")
# Save summary
summary = {
"skill": skill_path,
"split": split,
"n_items": len(results),
"hard": hard,
"soft": soft,
}
with open(os.path.join(out_root, "eval_summary.json"), "w") as f:
json.dump(summary, f, indent=2, ensure_ascii=False)
print(f" Saved to: {out_root}")
if __name__ == "__main__":
main()
scripts/run_alfworld.sh
Executable
+68
View File
@@ -0,0 +1,68 @@
#!/usr/bin/env bash
# ──────────────────────────────────────────────────────────────────────────────
# ReflACT — ALFWorld training launch script
#
# Usage:
# bash scripts/run_alfworld.sh
# bash scripts/run_alfworld.sh --num_epochs 2 --edit_budget 6
# ──────────────────────────────────────────────────────────────────────────────
set -euo pipefail
# ── Paths ────────────────────────────────────────────────────────────────────
WORKSPACE="${WORKSPACE:-$(cd "$(dirname "$0")/../.." && pwd)}"
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
PROJECT_ROOT="$(dirname "${SCRIPT_DIR}")"
# Activate conda environment
export PATH="${WORKSPACE}/miniconda3/envs/skillopt/bin:${WORKSPACE}/miniconda3/bin:${PATH}"
# ALFWorld data — uses ~/.cache/alfworld by default (standard alfworld location)
export ALFWORLD_DATA="${ALFWORLD_DATA:-${HOME}/.cache/alfworld}"
# Ensure ReflACT is importable
export PYTHONPATH="${PROJECT_ROOT}:${PYTHONPATH:-}"
# ── Verify ALFWorld data exists ──────────────────────────────────────────────
if [ ! -d "${ALFWORLD_DATA}/json_2.1.1" ]; then
echo "ERROR: ALFWorld data not found at ${ALFWORLD_DATA}/json_2.1.1"
echo ""
echo "To download ALFWorld data, run:"
echo " pip install alfworld[full]"
echo " alfworld-download"
echo ""
echo "Or set ALFWORLD_DATA to the directory containing json_2.1.1/"
exit 1
fi
# ── Azure OpenAI credentials ────────────────────────────────────────────────
export AZURE_OPENAI_ENDPOINT="${AZURE_OPENAI_ENDPOINT:?Set AZURE_OPENAI_ENDPOINT}"
export AZURE_OPENAI_API_KEY="${AZURE_OPENAI_API_KEY:?Set AZURE_OPENAI_API_KEY}"
export AZURE_OPENAI_API_VERSION="${AZURE_OPENAI_API_VERSION:-2025-04-01-preview}"
# ── Model configuration ─────────────────────────────────────────────────────
export TEACHER_DEPLOYMENT="${TEACHER_DEPLOYMENT:-gpt-5.5}"
export STUDENT_DEPLOYMENT="${STUDENT_DEPLOYMENT:-gpt-5.5}"
# ── Output directory ─────────────────────────────────────────────────────────
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
DEFAULT_OUT_ROOT="${PROJECT_ROOT}/outputs/skillopt_alfworld_${STUDENT_DEPLOYMENT}_${TIMESTAMP}"
# ── Run ──────────────────────────────────────────────────────────────────────
echo "============================================================"
echo " ReflACT — Reflective Agent Tuning (ALFWorld)"
echo "============================================================"
echo " Teacher: ${TEACHER_DEPLOYMENT}"
echo " Student: ${STUDENT_DEPLOYMENT}"
echo " ALFWORLD_DATA: ${ALFWORLD_DATA}"
echo " Output: ${DEFAULT_OUT_ROOT}"
echo "============================================================"
cd "${PROJECT_ROOT}"
python scripts/train.py \
--config configs/alfworld_default.yaml \
--out_root "${DEFAULT_OUT_ROOT}" \
"$@"
echo ""
echo "Done! Results saved to: ${DEFAULT_OUT_ROOT}"
scripts/run_searchqa.sh
Executable
+43
View File
@@ -0,0 +1,43 @@
#!/usr/bin/env bash
# ──────────────────────────────────────────────────────────────────────────────
# ReflACT — SearchQA training launch script
#
# Usage:
# bash scripts/run_searchqa.sh
# bash scripts/run_searchqa.sh --data_path data/searchqa_train_2000.json
# bash scripts/run_searchqa.sh --num_epochs 2 --edit_budget 6
# ──────────────────────────────────────────────────────────────────────────────
set -euo pipefail
# ── Paths ────────────────────────────────────────────────────────────────────
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
PROJECT_ROOT="$(dirname "${SCRIPT_DIR}")"
# Ensure ReflACT is importable
export PYTHONPATH="${PROJECT_ROOT}:${PYTHONPATH:-}"
# ── Model configuration ─────────────────────────────────────────────────────
export TEACHER_DEPLOYMENT="${TEACHER_DEPLOYMENT:-gpt-5.5}"
export STUDENT_DEPLOYMENT="${STUDENT_DEPLOYMENT:-gpt-5.5}"
# ── Output directory ─────────────────────────────────────────────────────────
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
DEFAULT_OUT_ROOT="${PROJECT_ROOT}/outputs/skillopt_searchqa_${STUDENT_DEPLOYMENT}_${TIMESTAMP}"
# ── Run ──────────────────────────────────────────────────────────────────────
echo "============================================================"
echo " ReflACT — Reflective Agent Tuning (SearchQA)"
echo "============================================================"
echo " Teacher: ${TEACHER_DEPLOYMENT}"
echo " Student: ${STUDENT_DEPLOYMENT}"
echo "============================================================"
cd "${PROJECT_ROOT}"
python scripts/train.py \
--config configs/searchqa_default.yaml \
--out_root "${DEFAULT_OUT_ROOT}" \
"$@"
echo ""
echo "Done! Results saved to: ${DEFAULT_OUT_ROOT}"
scripts/run_spreadsheetbench.sh
Executable
+48
View File
@@ -0,0 +1,48 @@
#!/usr/bin/env bash
# ──────────────────────────────────────────────────────────────────────────────
# ReflACT — SpreadsheetBench training launch script
#
# Usage:
# bash scripts/run_spreadsheetbench.sh \
# --data_root /path/to/data \
# --jsonl_path /path/to/benchmark.jsonl
#
# bash scripts/run_spreadsheetbench.sh \
# --data_root /path/to/data \
# --jsonl_path /path/to/benchmark.jsonl \
# --num_epochs 2 --edit_budget 6
# ──────────────────────────────────────────────────────────────────────────────
set -euo pipefail
# ── Paths ────────────────────────────────────────────────────────────────────
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
PROJECT_ROOT="$(dirname "${SCRIPT_DIR}")"
# Ensure ReflACT is importable
export PYTHONPATH="${PROJECT_ROOT}:${PYTHONPATH:-}"
# ── Model configuration ─────────────────────────────────────────────────────
export TEACHER_DEPLOYMENT="${TEACHER_DEPLOYMENT:-gpt-5.5}"
export STUDENT_DEPLOYMENT="${STUDENT_DEPLOYMENT:-gpt-5.5}"
# ── Output directory ─────────────────────────────────────────────────────────
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
DEFAULT_OUT_ROOT="${PROJECT_ROOT}/outputs/skillopt_spreadsheetbench_${STUDENT_DEPLOYMENT}_${TIMESTAMP}"
# ── Run ──────────────────────────────────────────────────────────────────────
echo "============================================================"
echo " ReflACT — Reflective Agent Tuning (SpreadsheetBench)"
echo "============================================================"
echo " Teacher: ${TEACHER_DEPLOYMENT}"
echo " Student: ${STUDENT_DEPLOYMENT}"
echo "============================================================"
cd "${PROJECT_ROOT}"
python scripts/train.py \
--config configs/spreadsheetbench_default.yaml \
--out_root "${DEFAULT_OUT_ROOT}" \
"$@"
echo ""
echo "Done! Results saved to: ${DEFAULT_OUT_ROOT}"
scripts/train.py
+504
View File
@@ -0,0 +1,504 @@
#!/usr/bin/env python3
"""ReflACT unified training entry point.
Usage
-----
python scripts/train.py --config configs/alfworld/default.yaml
Any YAML key can be overridden from the command line::
python scripts/train.py --config configs/alfworld/default.yaml \\
--batch_size 40 --num_epochs 2 --seed 123
Run ``python scripts/train.py --help`` for a full list of options.
"""
from __future__ import annotations
import argparse
import datetime
import os
import sys
# Ensure the project root is on sys.path so ``import skillopt`` works
# regardless of where the script is invoked from.
_SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__))
_PROJECT_ROOT = os.path.dirname(_SCRIPT_DIR)
if _PROJECT_ROOT not in sys.path:
sys.path.insert(0, _PROJECT_ROOT)
from skillopt.model.common import default_model_for_backend, normalize_backend_name
_OPENAI_DEFAULT_MODEL_SENTINELS = {"gpt-5.4", "gpt-5.5"}
# ── Environment registry ────────────────────────────────────────────────────
_ENV_REGISTRY: dict[str, type] = {}
def _register_builtins() -> None:
"""Lazy-import built-in adapters so we don't pull heavy deps at CLI parse time."""
try:
from skillopt.envs.alfworld.adapter import ALFWorldAdapter
_ENV_REGISTRY["alfworld"] = ALFWorldAdapter
except ImportError:
pass # ALFWorld deps not installed — skip
try:
from skillopt.envs.searchqa.adapter import SearchQAAdapter
_ENV_REGISTRY["searchqa"] = SearchQAAdapter
except ImportError:
pass
try:
from skillopt.envs.livemathematicianbench.adapter import LiveMathematicianBenchAdapter
_ENV_REGISTRY["livemathematicianbench"] = LiveMathematicianBenchAdapter
except ImportError:
pass
try:
from skillopt.envs.babyvision.adapter import BabyVisionAdapter
_ENV_REGISTRY["babyvision"] = BabyVisionAdapter
except ImportError:
pass
try:
from skillopt.envs.spreadsheetbench.adapter import SpreadsheetBenchAdapter
_ENV_REGISTRY["spreadsheetbench"] = SpreadsheetBenchAdapter
except ImportError:
pass
try:
from skillopt.envs.mmrb.adapter import MMRBAdapter
_ENV_REGISTRY["mmrb"] = MMRBAdapter
except ImportError:
pass
try:
from skillopt.envs.docvqa.adapter import DocVQAAdapter
_ENV_REGISTRY["docvqa"] = DocVQAAdapter
except ImportError:
pass
try:
from skillopt.envs.mathverse.adapter import MathVerseAdapter
_ENV_REGISTRY["mathverse"] = MathVerseAdapter
except ImportError:
pass
try:
from skillopt.envs.officeqa.adapter import OfficeQAAdapter
_ENV_REGISTRY["officeqa"] = OfficeQAAdapter
except ImportError:
pass
try:
from skillopt.envs.sealqa.adapter import SealQAAdapter
_ENV_REGISTRY["sealqa"] = SealQAAdapter
except ImportError:
pass
try:
from skillopt.envs.swebench.adapter import SWEBenchAdapter
_ENV_REGISTRY["swebench"] = SWEBenchAdapter
except ImportError:
pass
def get_adapter(cfg: dict):
"""Instantiate the environment adapter specified in ``cfg["env"]``."""
_register_builtins()
env_name = cfg.get("env", "alfworld")
if env_name not in _ENV_REGISTRY:
raise ValueError(
f"Unknown environment '{env_name}'. "
f"Available: {list(_ENV_REGISTRY.keys())}"
)
adapter_cls = _ENV_REGISTRY[env_name]
# Inspect adapter __init__ signature and only pass accepted kwargs
import inspect
sig = inspect.signature(adapter_cls.__init__)
accepted = set(sig.parameters.keys()) - {"self"}
adapter_kwargs: dict = {}
for key in accepted:
if key in cfg:
adapter_kwargs[key] = cfg[key]
return adapter_cls(**adapter_kwargs)
# ── CLI ──────────────────────────────────────────────────────────────────────
_BOOL = lambda x: x.lower() in ("true", "1", "yes") # noqa: E731
def parse_args() -> argparse.Namespace:
p = argparse.ArgumentParser(
description="ReflACT: Reflective Agent Tuning",
formatter_class=argparse.RawDescriptionHelpFormatter,
epilog=__doc__,
)
p.add_argument("--config", type=str, required=True,
help="Path to YAML config file")
p.add_argument("--cfg-options", nargs="+", default=[],
help="Override config: section.key=value (e.g. train.batch_size=40)")
# Legacy flat CLI overrides (still work, prefer --cfg-options for new usage)
p.add_argument("--env", type=str)
p.add_argument("--backend", type=str,
choices=["azure_openai", "codex", "codex_exec", "claude", "claude_chat", "claude_code_exec", "qwen", "qwen_chat"])
p.add_argument("--teacher_model", type=str)
p.add_argument("--student_model", type=str)
p.add_argument("--teacher_backend", type=str)
p.add_argument("--student_backend", type=str)
p.add_argument("--reasoning_effort", type=str,
choices=["", "low", "medium", "high", "xhigh", "max"])
p.add_argument("--rewrite_reasoning_effort", type=str)
p.add_argument("--rewrite_max_completion_tokens", type=int)
p.add_argument("--azure_endpoint", type=str)
p.add_argument("--azure_api_version", type=str)
p.add_argument("--azure_api_key", type=str)
p.add_argument("--azure_openai_endpoint", type=str)
p.add_argument("--azure_openai_api_version", type=str)
p.add_argument("--azure_openai_api_key", type=str)
p.add_argument("--azure_openai_auth_mode", type=str)
p.add_argument("--azure_openai_ad_scope", type=str)
p.add_argument("--azure_openai_managed_identity_client_id", type=str)
p.add_argument("--teacher_azure_openai_endpoint", type=str)
p.add_argument("--teacher_azure_openai_api_version", type=str)
p.add_argument("--teacher_azure_openai_api_key", type=str)
p.add_argument("--teacher_azure_openai_auth_mode", type=str)
p.add_argument("--teacher_azure_openai_ad_scope", type=str)
p.add_argument("--teacher_azure_openai_managed_identity_client_id", type=str)
p.add_argument("--student_azure_openai_endpoint", type=str)
p.add_argument("--student_azure_openai_api_version", type=str)
p.add_argument("--student_azure_openai_api_key", type=str)
p.add_argument("--student_azure_openai_auth_mode", type=str)
p.add_argument("--student_azure_openai_ad_scope", type=str)
p.add_argument("--student_azure_openai_managed_identity_client_id", type=str)
p.add_argument("--qwen_chat_base_url", type=str)
p.add_argument("--qwen_chat_api_key", type=str)
p.add_argument("--qwen_chat_temperature", type=float)
p.add_argument("--qwen_chat_timeout_seconds", type=float)
p.add_argument("--qwen_chat_max_tokens", type=int)
p.add_argument("--qwen_chat_enable_thinking", type=_BOOL)
p.add_argument("--codex_exec_path", type=str)
p.add_argument("--codex_exec_sandbox", type=str)
p.add_argument("--codex_exec_profile", type=str)
p.add_argument("--codex_exec_full_auto", type=_BOOL)
p.add_argument("--codex_exec_reasoning_effort", type=str)
p.add_argument("--codex_exec_use_sdk", type=str)
p.add_argument("--codex_exec_network_access", type=_BOOL)
p.add_argument("--codex_exec_web_search", type=_BOOL)
p.add_argument("--codex_exec_approval_policy", type=str)
p.add_argument("--claude_code_exec_path", type=str)
p.add_argument("--claude_code_exec_profile", type=str)
p.add_argument("--claude_code_exec_use_sdk", type=str)
p.add_argument("--claude_code_exec_effort", type=str)
p.add_argument("--claude_code_exec_max_thinking_tokens", type=int)
p.add_argument("--codex_trace_to_teacher", type=_BOOL)
p.add_argument("--skill_init", type=str)
p.add_argument("--num_epochs", type=int)
p.add_argument("--train_size", type=int)
p.add_argument("--steps_per_epoch", type=int)
p.add_argument("--batch_size", type=int)
p.add_argument("--accumulation", type=int)
p.add_argument("--seed", type=int)
p.add_argument("--edit_budget", type=int)
p.add_argument("--min_edit_budget", type=int)
p.add_argument("--lr_scheduler", type=str,
choices=["constant", "linear", "cosine", "autonomous"])
p.add_argument("--lr_control_mode", type=str,
choices=["fixed", "autonomous", "none"])
p.add_argument("--merge_batch_size", type=int)
p.add_argument("--max_analyst_rounds", type=int)
p.add_argument("--sel_env_num", type=int)
p.add_argument("--test_env_num", type=int)
p.add_argument("--eval_test", type=_BOOL)
p.add_argument("--use_gate", type=_BOOL)
p.add_argument("--max_steps", type=int)
p.add_argument("--max_api_workers", type=int)
p.add_argument("--analyst_workers", type=int)
p.add_argument("--failure_only", type=_BOOL)
p.add_argument("--minibatch_size", type=int)
p.add_argument("--use_meta_reflect", type=_BOOL)
p.add_argument("--meta_edit_budget", type=int)
p.add_argument("--skill_update_mode", type=str,
choices=[
"patch",
"rewrite_from_suggestions",
"rewrite",
"suggestions",
"full_rewrite",
"full_rewrite_minibatch",
"minibatch_full_rewrite",
])
p.add_argument("--use_deep_reflect", type=_BOOL)
p.add_argument("--deep_reflect_failures", type=int)
p.add_argument("--deep_reflect_successes", type=int)
p.add_argument("--use_slow_update", type=_BOOL)
p.add_argument("--slow_update_samples", type=int)
p.add_argument("--longitudinal_pair_policy", type=str,
choices=["mixed", "changed", "unchanged"])
p.add_argument("--use_meta_skill", type=_BOOL)
p.add_argument("--data_path", type=str)
p.add_argument("--split_mode", type=str,
choices=["ratio", "split_dir"])
p.add_argument("--split_ratio", type=str)
p.add_argument("--split_seed", type=int)
p.add_argument("--split_dir", type=str)
p.add_argument("--split_output_dir", type=str)
p.add_argument("--data_root", type=str)
p.add_argument("--max_turns", type=int)
p.add_argument("--workers", type=int)
p.add_argument("--limit", type=int)
p.add_argument("--shuffle_choices", type=_BOOL)
p.add_argument("--use_theorem", type=_BOOL)
p.add_argument("--use_sketch", type=_BOOL)
p.add_argument("--image_detail", type=str)
p.add_argument("--judge_model", type=str)
p.add_argument("--judge_max_completion_tokens", type=int)
p.add_argument("--judge_retries", type=int)
p.add_argument("--out_root", type=str)
p.add_argument("--mode", type=str)
return p.parse_args()
# ── Flat key → structured path mapping (for legacy CLI → structured config) ──
_LEGACY_TO_STRUCTURED: dict[str, str] = {
"backend": "model.backend",
"teacher_model": "model.teacher",
"student_model": "model.student",
"teacher_backend": "model.teacher_backend",
"student_backend": "model.student_backend",
"reasoning_effort": "model.reasoning_effort",
"rewrite_reasoning_effort": "model.rewrite_reasoning_effort",
"rewrite_max_completion_tokens": "model.rewrite_max_completion_tokens",
"azure_endpoint": "model.azure_endpoint",
"azure_api_version": "model.azure_api_version",
"azure_api_key": "model.azure_api_key",
"azure_openai_endpoint": "model.azure_openai_endpoint",
"azure_openai_api_version": "model.azure_openai_api_version",
"azure_openai_api_key": "model.azure_openai_api_key",
"azure_openai_auth_mode": "model.azure_openai_auth_mode",
"azure_openai_ad_scope": "model.azure_openai_ad_scope",
"azure_openai_managed_identity_client_id": "model.azure_openai_managed_identity_client_id",
"teacher_azure_openai_endpoint": "model.teacher_azure_openai_endpoint",
"teacher_azure_openai_api_version": "model.teacher_azure_openai_api_version",
"teacher_azure_openai_api_key": "model.teacher_azure_openai_api_key",
"teacher_azure_openai_auth_mode": "model.teacher_azure_openai_auth_mode",
"teacher_azure_openai_ad_scope": "model.teacher_azure_openai_ad_scope",
"teacher_azure_openai_managed_identity_client_id": "model.teacher_azure_openai_managed_identity_client_id",
"student_azure_openai_endpoint": "model.student_azure_openai_endpoint",
"student_azure_openai_api_version": "model.student_azure_openai_api_version",
"student_azure_openai_api_key": "model.student_azure_openai_api_key",
"student_azure_openai_auth_mode": "model.student_azure_openai_auth_mode",
"student_azure_openai_ad_scope": "model.student_azure_openai_ad_scope",
"student_azure_openai_managed_identity_client_id": "model.student_azure_openai_managed_identity_client_id",
"qwen_chat_base_url": "model.qwen_chat_base_url",
"qwen_chat_api_key": "model.qwen_chat_api_key",
"qwen_chat_temperature": "model.qwen_chat_temperature",
"qwen_chat_timeout_seconds": "model.qwen_chat_timeout_seconds",
"qwen_chat_max_tokens": "model.qwen_chat_max_tokens",
"qwen_chat_enable_thinking": "model.qwen_chat_enable_thinking",
"codex_exec_path": "model.codex_exec_path",
"codex_exec_sandbox": "model.codex_exec_sandbox",
"codex_exec_profile": "model.codex_exec_profile",
"codex_exec_full_auto": "model.codex_exec_full_auto",
"codex_exec_reasoning_effort": "model.codex_exec_reasoning_effort",
"codex_exec_use_sdk": "model.codex_exec_use_sdk",
"codex_exec_network_access": "model.codex_exec_network_access",
"codex_exec_web_search": "model.codex_exec_web_search",
"codex_exec_approval_policy": "model.codex_exec_approval_policy",
"claude_code_exec_path": "model.claude_code_exec_path",
"claude_code_exec_profile": "model.claude_code_exec_profile",
"claude_code_exec_use_sdk": "model.claude_code_exec_use_sdk",
"claude_code_exec_effort": "model.claude_code_exec_effort",
"claude_code_exec_max_thinking_tokens": "model.claude_code_exec_max_thinking_tokens",
"codex_trace_to_teacher": "model.codex_trace_to_teacher",
"num_epochs": "train.num_epochs",
"train_size": "train.train_size",
"steps_per_epoch": "train.steps_per_epoch",
"batch_size": "train.batch_size",
"accumulation": "train.accumulation",
"seed": "train.seed",
"minibatch_size": "gradient.minibatch_size",
"merge_batch_size": "gradient.merge_batch_size",
"analyst_workers": "gradient.analyst_workers",
"max_analyst_rounds": "gradient.max_analyst_rounds",
"failure_only": "gradient.failure_only",
"use_deep_reflect": "gradient.use_deep_reflect",
"deep_reflect_failures": "gradient.deep_reflect_failures",
"deep_reflect_successes": "gradient.deep_reflect_successes",
"edit_budget": "optimizer.learning_rate",
"min_edit_budget": "optimizer.min_learning_rate",
"lr_scheduler": "optimizer.lr_scheduler",
"lr_control_mode": "optimizer.lr_control_mode",
"skill_update_mode": "optimizer.skill_update_mode",
"use_meta_reflect": "optimizer.use_meta_reflect",
"meta_edit_budget": "optimizer.meta_learning_rate",
"use_slow_update": "optimizer.use_slow_update",
"slow_update_samples": "optimizer.slow_update_samples",
"longitudinal_pair_policy": "optimizer.longitudinal_pair_policy",
"use_meta_skill": "optimizer.use_meta_skill",
"use_gate": "evaluation.use_gate",
"sel_env_num": "evaluation.sel_env_num",
"test_env_num": "evaluation.test_env_num",
"eval_test": "evaluation.eval_test",
"env": "env.name",
"skill_init": "env.skill_init",
"out_root": "env.out_root",
}
def load_config(args: argparse.Namespace) -> dict:
"""Load config with _base_ inheritance, then apply CLI overrides."""
from skillopt.config import load_config as _load, flatten_config, is_structured
cfg = _load(args.config, overrides=args.cfg_options)
structured = is_structured(cfg)
# Apply legacy --key value overrides
cli = {k: v for k, v in vars(args).items()
if v is not None and k not in ("config", "cfg_options")}
if cli:
if structured:
from skillopt.config import apply_overrides
mapped = []
for k, v in cli.items():
dotted = _LEGACY_TO_STRUCTURED.get(k)
if dotted:
mapped.append(f"{dotted}={v}")
else:
mapped.append(f"env.{k}={v}")
apply_overrides(cfg, mapped)
else:
cfg.update(cli)
# Flatten structured config → flat dict for trainer/adapter
flat = flatten_config(cfg) if structured else cfg
for new_key, old_key in (
("azure_openai_endpoint", "azure_endpoint"),
("azure_openai_api_version", "azure_api_version"),
("azure_openai_api_key", "azure_api_key"),
):
if flat.get(new_key) in (None, "") and flat.get(old_key) not in (None, ""):
flat[new_key] = flat[old_key]
explicit_backend = getattr(args, "backend", None)
if explicit_backend is None:
for option in args.cfg_options or []:
key = str(option).split("=", 1)[0].strip()
if key == "model.backend":
explicit_backend = str(option).split("=", 1)[1].strip()
break
backend = normalize_backend_name(flat.get("model_backend") or flat.get("student_backend") or "azure_openai")
def _has_model_override(dotted_key: str, legacy_key: str) -> bool:
if getattr(args, legacy_key, None) is not None:
return True
for option in args.cfg_options or []:
key = str(option).split("=", 1)[0].strip()
if key == dotted_key:
return True
return False
if explicit_backend is not None:
backend = normalize_backend_name(explicit_backend)
flat["model_backend"] = backend
if backend in {"claude", "claude_chat"}:
flat.setdefault("teacher_backend", "claude_chat")
flat.setdefault("student_backend", "claude_chat")
elif backend in {"codex", "codex_exec"}:
flat.setdefault("teacher_backend", "openai_chat")
flat.setdefault("student_backend", "codex_exec")
elif backend == "claude_code_exec":
flat.setdefault("teacher_backend", "openai_chat")
flat.setdefault("student_backend", "claude_code_exec")
elif backend in {"qwen", "qwen_chat"}:
flat.setdefault("teacher_backend", "openai_chat")
flat.setdefault("student_backend", "qwen_chat")
else:
flat.setdefault("teacher_backend", "openai_chat")
flat.setdefault("student_backend", "openai_chat")
else:
flat.setdefault("teacher_backend", "openai_chat")
flat.setdefault("student_backend", "openai_chat")
if flat.get("teacher_backend") == "claude_chat":
if (
str(flat.get("teacher_model", "") or "").strip() in _OPENAI_DEFAULT_MODEL_SENTINELS
and not _has_model_override("model.teacher", "teacher_model")
):
flat["teacher_model"] = default_model_for_backend("claude_chat")
if flat.get("student_backend") == "claude_chat":
if (
str(flat.get("student_model", "") or "").strip() in _OPENAI_DEFAULT_MODEL_SENTINELS
and not _has_model_override("model.student", "student_model")
):
flat["student_model"] = default_model_for_backend("claude_chat")
if flat.get("student_backend") == "claude_code_exec":
if (
str(flat.get("student_model", "") or "").strip() in _OPENAI_DEFAULT_MODEL_SENTINELS
and not _has_model_override("model.student", "student_model")
):
flat["student_model"] = default_model_for_backend("claude_chat")
if flat.get("student_backend") == "qwen_chat":
if (
str(flat.get("student_model", "") or "").strip() in _OPENAI_DEFAULT_MODEL_SENTINELS
and not _has_model_override("model.student", "student_model")
):
flat["student_model"] = default_model_for_backend("qwen_chat")
# Auto-generate output root
if not flat.get("out_root"):
env = flat.get("env", "unknown")
model = flat.get("teacher_model", "unknown").replace("/", "-")
ts = datetime.datetime.now().strftime("%Y%m%d_%H%M%S")
flat["out_root"] = os.path.join("outputs", f"skillopt_{env}_{model}_{ts}")
flat["out_root"] = os.path.abspath(flat["out_root"])
return flat
# ── Main ─────────────────────────────────────────────────────────────────────
def main() -> None:
args = parse_args()
cfg = load_config(args)
print(f"\n{'='*60}")
print(f" ReflACT — Reflective Agent Tuning")
print(f"{'='*60}")
print(f" env: {cfg.get('env')}")
print(f" teacher_model: {cfg.get('teacher_model')}")
print(f" student_model: {cfg.get('student_model')}")
print(f" teacher_backend:{cfg.get('teacher_backend', 'openai_chat')}")
print(f" student_backend:{cfg.get('student_backend', 'openai_chat')}")
print(f" reasoning: {cfg.get('reasoning_effort') or 'off'}")
print(f" rewrite_effort: {cfg.get('rewrite_reasoning_effort') or 'off'}")
print(f" epochs: {cfg.get('num_epochs')}")
print(f" train_size: {cfg.get('train_size') or 'from dataset'}")
print(f" steps/epoch: auto")
print(f" batch_size: {cfg.get('batch_size')}")
print(f" edit_budget: {cfg.get('edit_budget')}")
print(f" lr_scheduler: {cfg.get('lr_scheduler', 'constant')}")
print(f" update_mode: {cfg.get('skill_update_mode', 'patch')}")
print(f" min_edit_budget:{cfg.get('min_edit_budget', 2)}")
print(f" minibatch_size: {cfg.get('minibatch_size')}")
print(f" seed: {cfg.get('seed')}")
print(f" meta_reflect: {cfg.get('use_meta_reflect', False)}")
print(f" meta_skill: {cfg.get('use_meta_skill', False)}")
print(f" out_root: {cfg.get('out_root')}")
print(f"{'='*60}\n")
# Build adapter
adapter = get_adapter(cfg)
# Build trainer and run
from skillopt.engine.trainer import ReflACTTrainer
trainer = ReflACTTrainer(cfg, adapter)
summary = trainer.train()
print(f"\n Output saved to: {cfg['out_root']}")
if summary.get("test_hard") is not None:
print(f" Final test: {summary['test_hard']:.4f}")
if __name__ == "__main__":
main()
skillopt/__init__.py
+29
View File
@@ -0,0 +1,29 @@
"""ReflACT: Reflective Agent Tuning.
A general-purpose framework for iteratively optimizing LLM agent skills
through structured reflection and self-improvement.
Pipeline stages:
1. Rollout — execute episodes with current skill
2. Reflect — analyze trajectories, generate patches
3. Aggregate — hierarchical merge of patches
4. Select — rank and select top edits
5. Update — apply edits to skill document
6. Evaluate — validate candidate skill, accept/reject
"""
__version__ = "0.1.0"
from skillopt.types import ( # noqa: F401
BatchSpec,
Edit,
EditOp,
FailureSummaryEntry,
GateAction,
GateResult,
MetaReflectResult,
Patch,
RawPatch,
RolloutResult,
SlowUpdateResult,
)
skillopt/config.py
+269
View File
@@ -0,0 +1,269 @@
"""ReflACT config loading engine — structured YAML with inheritance.
Supports two config formats:
1. **Structured** (new): sections like ``model``, ``train``, ``gradient``,
``optimizer``, ``evaluation``, ``env`` — with ``_base_`` inheritance.
2. **Flat** (legacy): all keys at top level — fully backward compatible.
Usage::
from skillopt.config import load_config, flatten_config
cfg = load_config("configs/searchqa_default.yaml")
flat = flatten_config(cfg) # always returns flat dict for trainer
"""
from __future__ import annotations
import copy
import os
from typing import Any
import yaml
# ── Section names that indicate a structured config ──────────────────────
_STRUCTURED_SECTIONS = frozenset({
"model", "train", "gradient", "optimizer", "evaluation", "env",
})
# ── Structured → flat key mapping ────────────────────────────────────────
_FLATTEN_MAP: dict[str, str] = {
"model.backend": "model_backend",
"model.teacher": "teacher_model",
"model.student": "student_model",
"model.teacher_backend": "teacher_backend",
"model.student_backend": "student_backend",
"model.reasoning_effort": "reasoning_effort",
"model.rewrite_reasoning_effort": "rewrite_reasoning_effort",
"model.rewrite_max_completion_tokens": "rewrite_max_completion_tokens",
"model.codex_exec_path": "codex_exec_path",
"model.codex_exec_sandbox": "codex_exec_sandbox",
"model.codex_exec_profile": "codex_exec_profile",
"model.codex_exec_full_auto": "codex_exec_full_auto",
"model.codex_exec_reasoning_effort": "codex_exec_reasoning_effort",
"model.codex_exec_use_sdk": "codex_exec_use_sdk",
"model.codex_exec_network_access": "codex_exec_network_access",
"model.codex_exec_web_search": "codex_exec_web_search",
"model.codex_exec_approval_policy": "codex_exec_approval_policy",
"model.claude_code_exec_path": "claude_code_exec_path",
"model.claude_code_exec_profile": "claude_code_exec_profile",
"model.claude_code_exec_use_sdk": "claude_code_exec_use_sdk",
"model.claude_code_exec_effort": "claude_code_exec_effort",
"model.claude_code_exec_max_thinking_tokens": "claude_code_exec_max_thinking_tokens",
"model.codex_trace_to_teacher": "codex_trace_to_teacher",
"model.azure_endpoint": "azure_endpoint",
"model.azure_api_version": "azure_api_version",
"model.azure_api_key": "azure_api_key",
"model.azure_openai_endpoint": "azure_openai_endpoint",
"model.azure_openai_api_version": "azure_openai_api_version",
"model.azure_openai_api_key": "azure_openai_api_key",
"model.azure_openai_auth_mode": "azure_openai_auth_mode",
"model.azure_openai_ad_scope": "azure_openai_ad_scope",
"model.azure_openai_managed_identity_client_id": "azure_openai_managed_identity_client_id",
"model.teacher_azure_openai_endpoint": "teacher_azure_openai_endpoint",
"model.teacher_azure_openai_api_version": "teacher_azure_openai_api_version",
"model.teacher_azure_openai_api_key": "teacher_azure_openai_api_key",
"model.teacher_azure_openai_auth_mode": "teacher_azure_openai_auth_mode",
"model.teacher_azure_openai_ad_scope": "teacher_azure_openai_ad_scope",
"model.teacher_azure_openai_managed_identity_client_id": "teacher_azure_openai_managed_identity_client_id",
"model.student_azure_openai_endpoint": "student_azure_openai_endpoint",
"model.student_azure_openai_api_version": "student_azure_openai_api_version",
"model.student_azure_openai_api_key": "student_azure_openai_api_key",
"model.student_azure_openai_auth_mode": "student_azure_openai_auth_mode",
"model.student_azure_openai_ad_scope": "student_azure_openai_ad_scope",
"model.student_azure_openai_managed_identity_client_id": "student_azure_openai_managed_identity_client_id",
"model.qwen_chat_base_url": "qwen_chat_base_url",
"model.qwen_chat_api_key": "qwen_chat_api_key",
"model.qwen_chat_temperature": "qwen_chat_temperature",
"model.qwen_chat_timeout_seconds": "qwen_chat_timeout_seconds",
"model.qwen_chat_max_tokens": "qwen_chat_max_tokens",
"model.qwen_chat_enable_thinking": "qwen_chat_enable_thinking",
"train.num_epochs": "num_epochs",
"train.train_size": "train_size",
"train.steps_per_epoch": "steps_per_epoch",
"train.batch_size": "batch_size",
"train.accumulation": "accumulation",
"train.seed": "seed",
"gradient.minibatch_size": "minibatch_size",
"gradient.merge_batch_size": "merge_batch_size",
"gradient.analyst_workers": "analyst_workers",
"gradient.failure_only": "failure_only",
"gradient.use_deep_reflect": "use_deep_reflect",
"gradient.deep_reflect_failures": "deep_reflect_failures",
"gradient.deep_reflect_successes": "deep_reflect_successes",
"gradient.max_analyst_rounds": "max_analyst_rounds",
"optimizer.learning_rate": "edit_budget",
"optimizer.min_learning_rate": "min_edit_budget",
"optimizer.lr_scheduler": "lr_scheduler",
"optimizer.lr_control_mode": "lr_control_mode",
"optimizer.skill_update_mode": "skill_update_mode",
"optimizer.use_meta_reflect": "use_meta_reflect",
"optimizer.meta_learning_rate": "meta_edit_budget",
"optimizer.use_slow_update": "use_slow_update",
"optimizer.slow_update_samples": "slow_update_samples",
"optimizer.longitudinal_pair_policy": "longitudinal_pair_policy",
"optimizer.use_meta_skill": "use_meta_skill",
"evaluation.use_gate": "use_gate",
"evaluation.sel_env_num": "sel_env_num",
"evaluation.test_env_num": "test_env_num",
"evaluation.eval_test": "eval_test",
"env.name": "env",
"env.skill_init": "skill_init",
"env.out_root": "out_root",
}
# ── Deep merge ───────────────────────────────────────────────────────────
def _deep_merge(base: dict, override: dict) -> dict:
"""Recursively merge *override* into *base* (returns new dict)."""
result = copy.deepcopy(base)
for key, val in override.items():
if key in result and isinstance(result[key], dict) and isinstance(val, dict):
result[key] = _deep_merge(result[key], val)
else:
result[key] = copy.deepcopy(val)
return result
# ── YAML loading with _base_ inheritance ─────────────────────────────────
def _load_yaml(path: str, _visited: set[str] | None = None) -> dict:
"""Load a YAML file, resolving ``_base_`` inheritance recursively."""
abs_path = os.path.abspath(path)
if _visited is None:
_visited = set()
if abs_path in _visited:
raise ValueError(f"Circular _base_ inheritance: {abs_path}")
_visited.add(abs_path)
with open(abs_path) as f:
cfg = yaml.safe_load(f) or {}
base_ref = cfg.pop("_base_", None)
if base_ref:
base_path = os.path.join(os.path.dirname(abs_path), base_ref)
base_cfg = _load_yaml(base_path, _visited)
cfg = _deep_merge(base_cfg, cfg)
return cfg
# ── Format detection ─────────────────────────────────────────────────────
def is_structured(cfg: dict) -> bool:
"""Return True if *cfg* uses the new structured section format."""
return any(
key in _STRUCTURED_SECTIONS and isinstance(cfg.get(key), dict)
for key in cfg
)
# ── Flatten ──────────────────────────────────────────────────────────────
def flatten_config(cfg: dict) -> dict:
"""Convert a structured config to the flat dict expected by the trainer.
If *cfg* is already flat, returns a shallow copy unchanged.
"""
if not is_structured(cfg):
return dict(cfg)
flat: dict[str, Any] = {}
evaluation_section = cfg.get("evaluation", {})
if isinstance(evaluation_section, dict) and evaluation_section.get("use_gate") is False:
raise ValueError(
"Gate validation is mandatory in this branch. Remove "
"`evaluation.use_gate: false` from the config."
)
# Apply the explicit mapping
for dotted, flat_key in _FLATTEN_MAP.items():
section, key = dotted.split(".", 1)
section_dict = cfg.get(section, {})
if isinstance(section_dict, dict) and key in section_dict:
flat[flat_key] = section_dict[key]
# Pass through env-specific keys not in the explicit mapping
env_section = cfg.get("env", {})
if isinstance(env_section, dict):
mapped_env_keys = {
k.split(".", 1)[1]
for k in _FLATTEN_MAP
if k.startswith("env.")
}
for key, val in env_section.items():
if key not in mapped_env_keys:
flat[key] = val
return flat
# ── Override application ─────────────────────────────────────────────────
def _cast_value(val_str: str) -> Any:
"""Auto-cast a CLI string value to int / float / bool / str."""
if val_str.lower() in ("true", "yes"):
return True
if val_str.lower() in ("false", "no"):
return False
try:
return int(val_str)
except ValueError:
pass
try:
return float(val_str)
except ValueError:
pass
return val_str
def apply_overrides(cfg: dict, overrides: list[str]) -> None:
"""Apply ``key=value`` overrides to a structured config (in place).
Supports both ``section.key=value`` (for structured configs) and
``key=value`` (for flat configs or flat keys in env section).
"""
for item in overrides:
if "=" not in item:
raise ValueError(f"Invalid override (expected key=value): {item!r}")
key, val_str = item.split("=", 1)
val = _cast_value(val_str)
if "." in key:
section, subkey = key.split(".", 1)
if section in cfg and isinstance(cfg[section], dict):
cfg[section][subkey] = val
else:
cfg.setdefault(section, {})[subkey] = val
else:
# Flat key — apply to top level (for legacy compat)
cfg[key] = val
# ── Public API ───────────────────────────────────────────────────────────
def load_config(
path: str,
overrides: list[str] | None = None,
) -> dict:
"""Load a config file with ``_base_`` inheritance and optional overrides.
Parameters
----------
path : str
Path to the YAML config file.
overrides : list[str] | None
``key=value`` strings from ``--cfg-options``.
Returns
-------
dict
The merged config (structured or flat depending on the YAML).
"""
cfg = _load_yaml(path)
if overrides:
apply_overrides(cfg, overrides)
return cfg
skillopt/datasets/__init__.py
+7
View File
@@ -0,0 +1,7 @@
"""ReflACT Datasets -- task batch planning and data loading.
Analogous to the datasets and dataloaders in neural network training:
provides batch sampling, epoch planning, and data management for the
ReflACT training pipeline.
"""
from skillopt.datasets.base import BaseDataLoader, BatchSpec, SplitDataLoader # noqa: F401
skillopt/datasets/base.py
+512
View File
@@ -0,0 +1,512 @@
"""Generic task dataloader abstractions for ReflACT.
ReflACT does not train model parameters directly. Instead, it iterates over
task batches, rolls out the current skill, reflects on failures/successes,
and updates the skill document. Because of that, the "dataloader" abstraction
here is closer to a batch sampler / episode planner than a tensor loader.
Class hierarchy::
BaseDataLoader # abstract — simulator-backed envs (e.g. ALFWorld)
└── SplitDataLoader # abstract — dataset-backed envs with split_dir
SplitDataLoader supports two dataset entry modes:
1. ``split_mode="split_dir"``: consume an existing split directory.
2. ``split_mode="ratio"``: build a deterministic split directory from a raw
dataset path using an explicit train:val:test ratio.
In either case, the standardised split layout is:
split_dir/
├── train/ # training items
├── val/ # validation / selection items (gate)
└── test/ # held-out test items
Each subdirectory's contents are benchmark-specific. Subclasses only need
to implement ``load_split_items(split_path)`` to teach the loader how to
read items from one of those directories.
"""
from __future__ import annotations
import glob
import json
import os
import random
from abc import ABC, abstractmethod
from dataclasses import dataclass, field
from typing import Any
@dataclass(slots=True)
class BatchSpec:
"""A concrete batch request consumed by the training loop.
Parameters
----------
phase : str
``"train"`` or ``"eval"``.
split : str
Dataset split name, typically ``"train"`` or an eval split.
seed : int
Random seed used to construct the batch deterministically.
batch_size : int
Requested number of items / episodes in this batch.
payload : object | None
Environment-specific batch payload. For dataset-backed environments
this is often a list of sampled items; for simulator-backed
environments this may be ``None`` and the seed alone can define the
batch.
metadata : dict[str, Any]
Optional structured metadata for logging, resume, or curriculum logic.
"""
phase: str
split: str
seed: int
batch_size: int
payload: object | None = None
metadata: dict[str, Any] = field(default_factory=dict)
class BaseDataLoader(ABC):
"""Abstract base class for task batch planning in ReflACT.
Subclasses are responsible for defining how a train or eval batch is
sampled. The default implementation here provides deterministic epoch seed
planning so all loaders share the same reproducibility behavior.
"""
def setup(self, cfg: dict) -> None:
"""Optional one-time initialization with the full trainer config."""
def set_out_root(self, out_root: str) -> None:
"""Optional hook for loaders that persist split files or state."""
def state_dict(self) -> dict[str, Any]:
"""Return serializable loader state for resume support."""
return {}
def load_state_dict(self, state: dict[str, Any]) -> None:
"""Restore loader state from :meth:`state_dict` output."""
def get_train_size(self) -> int | None:
"""Return the size of the training pool when known."""
return None
@staticmethod
def make_base_seeds(steps_per_epoch: int, accumulation: int, seed: int) -> list[int]:
"""Return the deterministic seed pool used to define train batches."""
batches_per_epoch = steps_per_epoch * accumulation
return [seed + i + 1 for i in range(batches_per_epoch)]
@staticmethod
def shuffle_epoch_seeds(base_seeds: list[int], epoch: int, seed: int) -> list[int]:
"""Return the per-epoch deterministic shuffle of *base_seeds*."""
epoch_rng = random.Random(seed + epoch * 1000)
shuffled = list(base_seeds)
epoch_rng.shuffle(shuffled)
return shuffled
def plan_train_epoch(
self,
*,
epoch: int,
steps_per_epoch: int,
accumulation: int,
batch_size: int,
seed: int,
**kwargs,
) -> list[BatchSpec]:
"""Build the full list of training batches for one epoch."""
base_seeds = self.make_base_seeds(
steps_per_epoch=steps_per_epoch,
accumulation=accumulation,
seed=seed,
)
shuffled_seeds = self.shuffle_epoch_seeds(base_seeds, epoch=epoch, seed=seed)
return [
self.build_train_batch(batch_size=batch_size, seed=batch_seed, **kwargs)
for batch_seed in shuffled_seeds
]
@abstractmethod
def build_train_batch(self, batch_size: int, seed: int, **kwargs) -> BatchSpec:
"""Construct one training batch specification."""
@abstractmethod
def build_eval_batch(
self,
env_num: int,
split: str,
seed: int,
**kwargs,
) -> BatchSpec:
"""Construct one evaluation batch specification."""
# ── Split-based dataloader for dataset-backed environments ──────────────
# Canonical split names expected under split_dir/
SPLIT_NAMES = ("train", "val", "test")
# Maps legacy / trainer split names → canonical directory names
_SPLIT_ALIAS: dict[str, str] = {
"train": "train",
"valid_seen": "val",
"selection": "val",
"val": "val",
"valid_unseen": "test",
"test": "test",
}
def _load_json_or_jsonl(path: str) -> list[dict]:
"""Load a list of items from a JSON or JSONL file."""
with open(path, encoding="utf-8") as f:
content = f.read().strip()
if not content:
return []
try:
data = json.loads(content)
except json.JSONDecodeError:
data = None
if isinstance(data, list):
return data
if isinstance(data, dict):
nested = data.get("data")
if isinstance(nested, list):
return nested
return list(data.values())
items: list[dict] = []
for line in content.splitlines():
line = line.strip()
if line:
items.append(json.loads(line))
return items
def _parse_split_ratio(text: str) -> tuple[int, int, int]:
parts = [part.strip() for part in str(text or "").split(":") if part.strip()]
if len(parts) != 3:
raise ValueError(
f"split_ratio must be in train:val:test form, got {text!r}"
)
try:
train, val, test = (int(part) for part in parts)
except ValueError as exc:
raise ValueError(
f"split_ratio must contain integers, got {text!r}"
) from exc
if min(train, val, test) <= 0:
raise ValueError(f"split_ratio parts must be positive, got {text!r}")
return train, val, test
def _compute_split_counts(total: int, ratio: tuple[int, int, int]) -> tuple[int, int, int]:
weights = list(ratio)
denom = sum(weights)
raw = [total * weight / denom for weight in weights]
counts = [int(value) for value in raw]
remaining = total - sum(counts)
order = sorted(
range(len(raw)),
key=lambda idx: (raw[idx] - counts[idx], weights[idx]),
reverse=True,
)
for idx in order[:remaining]:
counts[idx] += 1
return counts[0], counts[1], counts[2]
class SplitDataLoader(BaseDataLoader):
"""Base class for dataset-backed environments.
Supported modes:
- ``split_mode="split_dir"``: load an existing ``train/``, ``val/``,
``test/`` directory tree.
- ``split_mode="ratio"``: load raw items from ``data_path`` and materialize
a deterministic split directory with the requested ratio.
"""
def __init__(
self,
split_dir: str = "",
data_path: str = "",
split_mode: str = "ratio",
split_ratio: str = "2:1:7",
split_seed: int = 42,
split_output_dir: str = "",
seed: int = 42,
limit: int = 0,
**kwargs,
) -> None:
self.split_dir = split_dir
self.data_path = data_path
self.split_mode = split_mode
self.split_ratio = split_ratio
self.split_seed = int(split_seed)
self.split_output_dir = split_output_dir
self.seed = seed
self.limit = limit
self._splits: dict[str, list[dict]] = {}
# ── Setup ────────────────────────────────────────────────────────────
def setup(self, cfg: dict) -> None:
if not self.split_mode:
self.split_mode = str(cfg.get("split_mode", "ratio") or "ratio")
if not self.split_dir:
self.split_dir = cfg.get("split_dir", "")
if not self.data_path:
self.data_path = cfg.get("data_path", "")
if not self.split_output_dir:
self.split_output_dir = cfg.get("split_output_dir", "")
if "split_seed" in cfg and not self.split_seed:
self.split_seed = int(cfg.get("split_seed", 0) or 0)
if not self.split_seed:
self.split_seed = self.seed
if not self.split_ratio:
self.split_ratio = str(cfg.get("split_ratio", "2:1:7") or "2:1:7")
mode = str(self.split_mode or "ratio").strip().lower()
if mode not in {"ratio", "split_dir"}:
raise ValueError(
f"{type(self).__name__} split_mode must be 'ratio' or 'split_dir', "
f"got {self.split_mode!r}"
)
self.split_mode = mode
if self.split_mode == "ratio":
self.split_dir = self._materialize_ratio_split(cfg)
if not self.split_dir:
raise ValueError(
f"{type(self).__name__} requires either "
"`split_mode=ratio` with `data_path`, or `split_mode=split_dir` "
f"with `split_dir` pointing to {'/'.join(SPLIT_NAMES)}/."
)
self._load_all_splits()
def _resolve_split_output_dir(self, cfg: dict) -> str:
if self.split_output_dir:
return os.path.abspath(self.split_output_dir)
out_root = os.path.abspath(str(cfg.get("out_root") or os.getcwd()))
env_name = str(cfg.get("env") or type(self).__name__.replace("DataLoader", "").lower())
ratio_tag = str(self.split_ratio or "2:1:7").replace(":", "-")
return os.path.join(out_root, "_generated_splits", f"{env_name}_{ratio_tag}_seed{self.split_seed}")
def load_raw_items(self, data_path: str) -> list[dict]:
"""Load raw items from a dataset path before ratio splitting.
Subclasses can override when the raw dataset is not a single JSON/JSONL
file or when directory layouts require custom normalization.
"""
if os.path.isdir(data_path):
if any(os.path.isdir(os.path.join(data_path, name)) for name in SPLIT_NAMES):
raise ValueError(
f"{type(self).__name__} got a split directory as data_path. "
"Use split_mode=split_dir and pass it as split_dir instead."
)
candidates = sorted(glob.glob(os.path.join(data_path, "*.json")))
candidates += sorted(glob.glob(os.path.join(data_path, "*.jsonl")))
if len(candidates) != 1:
raise ValueError(
f"{type(self).__name__} expected data_path to be one JSON/JSONL file "
f"or a directory containing exactly one such file, got: {data_path}"
)
return _load_json_or_jsonl(candidates[0])
return _load_json_or_jsonl(data_path)
def write_split_items(self, split_path: str, items: list[dict]) -> None:
os.makedirs(split_path, exist_ok=True)
out_path = os.path.join(split_path, "items.json")
with open(out_path, "w", encoding="utf-8") as f:
json.dump(items, f, ensure_ascii=False, indent=2)
def _materialize_ratio_split(self, cfg: dict) -> str:
data_path = os.path.abspath(str(self.data_path or "").strip())
if not data_path:
raise ValueError(
f"{type(self).__name__} requires data_path when split_mode=ratio."
)
ratio = _parse_split_ratio(self.split_ratio)
items = self.load_raw_items(data_path)
if not isinstance(items, list) or not items:
raise ValueError(f"No raw items available for ratio split from {data_path}")
shuffled = list(items)
rng = random.Random(self.split_seed)
rng.shuffle(shuffled)
train_n, val_n, test_n = _compute_split_counts(len(shuffled), ratio)
train_items = shuffled[:train_n]
val_items = shuffled[train_n: train_n + val_n]
test_items = shuffled[train_n + val_n: train_n + val_n + test_n]
split_dir = self._resolve_split_output_dir(cfg)
manifest = {
"source_data_path": data_path,
"split_mode": "ratio",
"split_ratio": self.split_ratio,
"split_seed": self.split_seed,
"counts": {
"train": len(train_items),
"val": len(val_items),
"test": len(test_items),
},
}
os.makedirs(split_dir, exist_ok=True)
self.write_split_items(os.path.join(split_dir, "train"), train_items)
self.write_split_items(os.path.join(split_dir, "val"), val_items)
self.write_split_items(os.path.join(split_dir, "test"), test_items)
with open(os.path.join(split_dir, "split_manifest.json"), "w", encoding="utf-8") as f:
json.dump(manifest, f, ensure_ascii=False, indent=2)
print(
f" [{type(self).__name__}] generated ratio split {self.split_ratio} "
f"at {split_dir} from {data_path}"
)
return split_dir
def _load_all_splits(self) -> None:
for name in SPLIT_NAMES:
split_path = os.path.join(self.split_dir, name)
if not os.path.isdir(split_path):
raise ValueError(
f"Missing '{name}/' subdirectory in split_dir: {self.split_dir}"
)
items = self.load_split_items(split_path)
if self.limit:
items = items[: self.limit]
self._splits[name] = items
counts = " ".join(f"{k}={len(v)}" for k, v in self._splits.items())
print(f" [{type(self).__name__}] {counts} (from {self.split_dir})")
def load_split_items(self, split_path: str) -> list[dict]:
"""Load items from one split directory (e.g. ``split_dir/train/``).
Default: finds the first ``.json`` file in the directory and loads it
as a JSON array. Subclasses can override for custom formats.
"""
json_files = sorted(glob.glob(os.path.join(split_path, "*.json")))
if not json_files:
raise FileNotFoundError(
f"No .json file found in {split_path}"
)
with open(json_files[0], encoding="utf-8") as f:
items = json.load(f)
if not isinstance(items, list):
raise ValueError(
f"Expected JSON array in {json_files[0]}, got {type(items).__name__}"
)
return items
# ── Accessors ────────────────────────────────────────────────────────
@property
def train_items(self) -> list[dict]:
return self._splits.get("train", [])
@property
def val_items(self) -> list[dict]:
return self._splits.get("val", [])
@property
def test_items(self) -> list[dict]:
return self._splits.get("test", [])
def get_split_items(self, split: str) -> list[dict]:
"""Resolve a split name (including legacy aliases) to its item list."""
canonical = _SPLIT_ALIAS.get(split, split)
return list(self._splits.get(canonical, self.val_items))
def get_train_size(self) -> int:
return len(self.train_items)
def plan_train_epoch(
self,
*,
epoch: int,
steps_per_epoch: int,
accumulation: int,
batch_size: int,
seed: int,
**kwargs,
) -> list[BatchSpec]:
"""Build one full epoch that covers the train split in shuffled order.
For split-backed datasets, an epoch should correspond to one pass over
the available training items rather than repeated independent sampling.
"""
epoch_rng = random.Random(seed + epoch * 1000)
items = list(self.train_items)
epoch_rng.shuffle(items)
total_batches = steps_per_epoch * accumulation
if total_batches <= 0:
return []
batches: list[BatchSpec] = []
cursor = 0
for batch_idx in range(total_batches):
batch_items = items[cursor: cursor + batch_size]
cursor += len(batch_items)
# Extremely small datasets can leave trailing empty microbatches
# when accumulation > 1. Reuse the shuffled prefix in that case so
# the trainer still receives the expected batch count.
if not batch_items and items:
refill_rng = random.Random(seed + epoch * 1000 + batch_idx + 1)
batch_items = list(items)
refill_rng.shuffle(batch_items)
batch_items = batch_items[:batch_size]
batches.append(
BatchSpec(
phase="train",
split="train",
seed=seed + epoch * 1000 + batch_idx + 1,
batch_size=len(batch_items),
payload=batch_items,
)
)
return batches
# ── Batch construction ───────────────────────────────────────────────
def build_train_batch(self, batch_size: int, seed: int, **kwargs) -> BatchSpec:
rng = random.Random(seed)
items = list(self.train_items)
rng.shuffle(items)
items = items[:batch_size]
return BatchSpec(
phase="train",
split="train",
seed=seed,
batch_size=len(items),
payload=items,
)
def build_eval_batch(
self,
env_num: int,
split: str,
seed: int,
**kwargs,
) -> BatchSpec:
items = self.get_split_items(split)
if env_num and env_num < len(items):
items = items[:env_num]
return BatchSpec(
phase="eval",
split=split,
seed=seed,
batch_size=len(items),
payload=items,
)
skillopt/engine/__init__.py
+9
View File
@@ -0,0 +1,9 @@
"""ReflACT Engine -- the training runner.
Analogous to the Runner in mmengine: orchestrates the full training pipeline
including rollout, gradient computation, aggregation, optimization, and
evaluation.
"""
from skillopt.engine.trainer import ReflACTTrainer # noqa: F401
__all__ = ["ReflACTTrainer"]
skillopt/engine/trainer.py
+2207
View File
File diff suppressed because it is too large Load Diff
skillopt/envs/__init__.py
+1
View File
@@ -0,0 +1 @@
"""ReflACT environment adapters."""
skillopt/envs/_template/README.md
+19
View File
@@ -0,0 +1,19 @@
# Benchmark Template
This directory provides scaffold files for adding a new benchmark to SkillOpt.
## Files
- `env_template.py` — Environment adapter template
- `loader_template.py` — Data loader template
- `config_template.yaml` — Config file template
## Usage
1. Copy this directory: `cp -r skillopt/envs/_template skillopt/envs/your_benchmark`
2. Rename files: remove `_template` suffix
3. Implement the `TODO` sections
4. Register in `skillopt/envs/__init__.py`
5. Create config at `configs/your_benchmark/default.yaml`
See the [documentation](../../docs/guide/new-benchmark.md) for the full guide.
skillopt/envs/_template/config_template.yaml
+45
View File
@@ -0,0 +1,45 @@
# ──────────────────────────────────────────────────
# SkillOpt Config Template — <Your Benchmark Name>
# ──────────────────────────────────────────────────
# Copy this file to configs/<your_benchmark>/default.yaml
# and customize the values below.
# Inherit global defaults
_base_: ['../_base_/default.yaml']
# ── Environment ──────────────────────────────────
env:
name: your_benchmark # Must match registry key
data_path: data/your_benchmark # Path to your data
split_mode: ratio # "ratio" or "split_dir"
split_ratio: "2:1:7" # train:val:test
exec_timeout: 120 # Per-task timeout (seconds)
# ── Training ─────────────────────────────────────
train:
num_epochs: 4 # Number of epochs
batch_size: 40 # Tasks per step (batch size)
seed: 42
# ── Gradient (Reflection) ───────────────────────
gradient:
analyst_workers: 16 # Parallel reflection workers
minibatch_size: 8
# ── Optimizer ────────────────────────────────────
optimizer:
learning_rate: 4 # Max edits per step (edit budget)
lr_scheduler: cosine # cosine | linear | constant | autonomous
use_slow_update: true # Epoch-boundary momentum
use_meta_skill: true # Cross-epoch teacher memory
# ── Evaluation ───────────────────────────────────
evaluation:
use_gate: true # Validation gating
eval_test: true # Run test eval after training
# ── Model ────────────────────────────────────────
model:
backend: azure_openai # azure_openai | openai_chat | claude_code_exec | qwen
teacher: gpt-5.5
student: gpt-5.5
skillopt/envs/_template/env_template.py
+92
View File
@@ -0,0 +1,92 @@
"""
Benchmark Environment Template
===============================
Copy this file and implement the TODO sections to add a new benchmark.
The EnvAdapter is responsible for:
1. Executing tasks using the student model + current skill document
2. Evaluating predictions against ground truth
3. Returning structured results for the training loop
"""
from skillopt.envs.base import EnvAdapter
class TemplateBenchmarkEnv(EnvAdapter):
"""
Environment adapter for <Your Benchmark Name>.
Rename this class and implement the abstract methods below.
"""
def __init__(self, cfg: dict):
super().__init__(cfg)
# TODO: Initialize benchmark-specific state
# Example: self.tools = load_tools(cfg)
async def execute(self, item, skill: str, model):
"""
Execute a single task with the student model.
Args:
item: DataItem with .id, .input, .ground_truth, .metadata
skill: Current skill document content (Markdown string)
model: Student model backend instance
Returns:
TaskResult with prediction, score, and trajectory
"""
# Step 1: Build the prompt combining skill + task input
prompt = self.build_prompt(item, skill)
# Step 2: Call the student model
# TODO: Customize the message format for your benchmark
messages = [
{"role": "system", "content": skill},
{"role": "user", "content": item.input},
]
response = await model.generate(messages)
# Step 3: Parse the model response into a prediction
prediction = self.parse_response(response.content)
# Step 4: Score the prediction
score = self.evaluate(prediction, item.ground_truth)
# Step 5: Return structured result
return {
"item_id": item.id,
"prediction": prediction,
"score": score,
"trajectory": messages + [{"role": "assistant", "content": response.content}],
}
def evaluate(self, prediction: str, ground_truth: str) -> float:
"""
Score a prediction against the ground truth.
Returns:
Float between 0.0 (wrong) and 1.0 (correct)
TODO: Implement your scoring metric. Common options:
- Exact match: float(pred.strip().lower() == gt.strip().lower())
- F1 score: compute token overlap
- ANLS: for document QA tasks
- Custom: any float in [0, 1]
"""
# Placeholder — exact match
return float(prediction.strip().lower() == ground_truth.strip().lower())
def build_prompt(self, item, skill: str) -> str:
"""Combine skill document with task input."""
return f"{skill}\n\n---\n\nQuestion: {item.input}"
def parse_response(self, response: str) -> str:
"""
Extract the answer from the model's raw response.
TODO: Implement extraction logic. For example:
- Extract text after "Answer:"
- Parse JSON output
- Extract from code blocks
"""
return response.strip()
skillopt/envs/_template/loader_template.py
+103
View File
@@ -0,0 +1,103 @@
"""
Benchmark Data Loader Template
================================
Copy this file and implement the TODO sections to load your benchmark data.
The DataLoader is responsible for:
1. Loading raw data from disk
2. Splitting into train / validation / test sets
3. Providing DataItem objects to the training loop
"""
from pathlib import Path
class TemplateBenchmarkLoader:
"""
Data loader for <Your Benchmark Name>.
Rename this class and implement the methods below.
"""
def __init__(self, data_dir: str = "data/your_benchmark", **kwargs):
self.data_dir = Path(data_dir)
self.items = []
self.splits = {}
def setup(self, cfg: dict):
"""
Initialize the loader with config.
Called once before training starts.
Args:
cfg: Dict with keys like 'split_mode', 'train_ratio', 'val_ratio', etc.
"""
# Step 1: Load raw data
self.items = self._load_items()
# Step 2: Create splits
split_mode = cfg.get("split_mode", "ratio")
if split_mode == "ratio":
self._split_by_ratio(
train_ratio=cfg.get("train_ratio", 0.7),
val_ratio=cfg.get("val_ratio", 0.15),
)
elif split_mode == "split_dir":
self._load_predefined_splits(cfg.get("split_dir", self.data_dir))
def _load_items(self) -> list:
"""
Load raw data into structured items.
TODO: Implement data loading. Each item should have at minimum:
- id: unique identifier
- input: the task input (question, instruction, etc.)
- ground_truth: the expected answer
- metadata: optional dict with extra info
Example:
items = []
for path in self.data_dir.glob("*.json"):
data = json.loads(path.read_text())
for entry in data:
items.append({
"id": entry["id"],
"input": entry["question"],
"ground_truth": entry["answer"],
"metadata": {"source": path.name},
})
return items
"""
raise NotImplementedError("Implement _load_items() for your benchmark")
def _split_by_ratio(self, train_ratio: float, val_ratio: float):
"""Split items by ratio."""
import random
random.shuffle(self.items)
n = len(self.items)
n_train = int(n * train_ratio)
n_val = int(n * val_ratio)
self.splits = {
"train": self.items[:n_train],
"valid": self.items[n_train:n_train + n_val],
"test": self.items[n_train + n_val:],
}
def _load_predefined_splits(self, split_dir):
"""Load from pre-split directories."""
# TODO: Implement if your benchmark has pre-defined splits
raise NotImplementedError
def get_split_items(self, split: str) -> list:
"""
Return items for a given split.
Args:
split: One of "train", "valid", "test"
Returns:
List of data items for the requested split
"""
if split not in self.splits:
raise ValueError(f"Unknown split '{split}'. Available: {list(self.splits.keys())}")
return self.splits[split]
skillopt/envs/alfworld/__init__.py
+5
View File
@@ -0,0 +1,5 @@
"""ALFWorld environment adapter for ReflACT."""
from skillopt.envs.alfworld.adapter import ALFWorldAdapter
__all__ = ["ALFWorldAdapter"]
skillopt/envs/alfworld/adapter.py
+585
View File
@@ -0,0 +1,585 @@
"""ALFWorld environment adapter for ReflACT.
Connects the ReflACT training loop to ALFWorld by implementing
:class:`~skillopt.envs.base.EnvAdapter`.
"""
from __future__ import annotations
from dataclasses import dataclass
import json
import os
from skillopt.gradient.deep_probe import generate_deep_probe_instruction
from skillopt.datasets.base import BatchSpec
from skillopt.envs.base import EnvAdapter
from skillopt.envs.alfworld.dataloader import ALFWorldDataLoader
from skillopt.envs.alfworld.rollout import (
build_alfworld_env,
run_alfworld_batch,
TASKS,
)
from skillopt.gradient.reflect import run_minibatch_reflect
from skillopt.utils import compute_score
@dataclass(frozen=True)
class ALFWorldBatchRun:
"""Lazy ALFWorld batch description.
The adapter materializes this in rollout chunks so a large evaluation set
does not keep every ALFWorld simulator open at once.
"""
env_num: int
eval_dataset: str
seed: int
is_train: bool
workers: int
specific_gamefiles: list[str] | None = None
result_ids: list[str] | None = None
items: list[dict] | None = None
def __iter__(self):
return iter(self.items or [])
def __len__(self) -> int:
return int(self.env_num or 0)
class ALFWorldAdapter(EnvAdapter):
"""ALFWorld environment adapter.
Parameters
----------
max_steps : int
Maximum steps per ALFWorld episode (default 50).
max_api_workers : int
Maximum concurrent API calls during rollout (default 8).
analyst_workers : int
Parallel workers for analyst stage (default 16).
failure_only : bool
If True, only run error analyst (skip success analyst).
minibatch_size : int
Trajectories per analyst group, M (default 8).
edit_budget : int
Maximum edits per minibatch, L (default 4).
"""
def __init__(
self,
split_dir: str = "",
data_path: str = "",
split_mode: str = "split_dir",
split_ratio: str = "2:1:7",
split_seed: int = 42,
split_output_dir: str = "",
seed: int = 42,
limit: int = 0,
train_size: int = 0,
max_steps: int = 50,
workers: int = 8,
max_api_workers: int = 8,
analyst_workers: int = 16,
failure_only: bool = False,
minibatch_size: int = 8,
edit_budget: int = 4,
use_deep_reflect: bool = False,
deep_reflect_failures: int = 4,
deep_reflect_successes: int = 2,
) -> None:
self.max_steps = max_steps
self.workers = max(int(workers or 1), 1)
self.max_api_workers = max_api_workers
self.analyst_workers = analyst_workers
self.failure_only = failure_only
self.minibatch_size = minibatch_size
self.edit_budget = edit_budget
self.use_deep_reflect = use_deep_reflect
self.deep_reflect_failures = deep_reflect_failures
self.deep_reflect_successes = deep_reflect_successes
self.dataloader = ALFWorldDataLoader(
split_dir=split_dir,
data_path=data_path,
split_mode=split_mode,
split_ratio=split_ratio,
split_seed=split_seed,
split_output_dir=split_output_dir,
seed=seed,
limit=limit,
train_size=train_size,
)
self._traj_cache: dict[str, dict | None] = {}
def setup(self, cfg: dict) -> None:
super().setup(cfg)
self.dataloader.setup(cfg)
def _load_traj_data(self, item: dict) -> dict | None:
gamefile = str(item.get("gamefile") or "").strip()
if not gamefile:
return None
if gamefile in self._traj_cache:
return self._traj_cache[gamefile]
traj_path = os.path.join(os.path.dirname(gamefile), "traj_data.json")
try:
with open(traj_path, encoding="utf-8") as f:
data = json.load(f)
except Exception:
data = None
self._traj_cache[gamefile] = data
return data
@staticmethod
def _unique_lines(values: list[str], *, limit: int = 0) -> list[str]:
lines: list[str] = []
seen: set[str] = set()
for raw in values:
line = str(raw or "").strip()
if not line or line in seen:
continue
seen.add(line)
lines.append(line)
if limit > 0 and len(lines) >= limit:
break
return lines
@staticmethod
def _format_high_pddl(high_pddl: list[dict]) -> list[str]:
steps: list[str] = []
for idx, step in enumerate(high_pddl or [], start=1):
discrete = step.get("discrete_action") or {}
action = str(discrete.get("action") or "").strip()
args = [str(arg).strip() for arg in (discrete.get("args") or []) if str(arg).strip()]
if action and args:
text = f"{action}({', '.join(args)})"
elif action:
text = action
else:
planner_action = step.get("planner_action") or {}
text = str(planner_action.get("action") or "").strip()
if text:
steps.append(f"{idx}. {text}")
return steps
def _build_reference_bundle(self, item: dict) -> dict:
data = self._load_traj_data(item)
if not data:
return {}
anns = ((data.get("turk_annotations") or {}).get("anns") or [])
task_descs = self._unique_lines(
[ann.get("task_desc", "") for ann in anns],
limit=3,
)
high_descs = self._unique_lines(
[step for ann in anns for step in (ann.get("high_descs") or [])],
limit=12,
)
pddl_params = {
key: value
for key, value in (data.get("pddl_params") or {}).items()
if value not in ("", None, [], {})
}
scene = data.get("scene") or {}
scene_summary = {
key: scene.get(key)
for key in ("floor_plan", "scene_num", "dirty_and_empty")
if scene.get(key) not in ("", None, [], {})
}
high_pddl = self._format_high_pddl((data.get("plan") or {}).get("high_pddl") or [])
task_type = str(data.get("task_type") or item.get("task_type") or "").strip()
return {
"task_type": task_type,
"task_descs": task_descs,
"high_descs": high_descs,
"pddl_params": pddl_params,
"high_pddl": high_pddl,
"scene_summary": scene_summary,
}
def build_reference_text(self, item: dict) -> str:
bundle = self._build_reference_bundle(item)
if not bundle:
return ""
parts: list[str] = []
if bundle["task_type"]:
parts.append(f"## Reference Task Type\n{bundle['task_type']}")
if bundle["task_descs"]:
parts.append(
"## Reference Human Task Descriptions\n"
+ "\n".join(f"- {line}" for line in bundle["task_descs"])
)
if bundle["high_descs"]:
parts.append(
"## Reference Human High-Level Steps\n"
+ "\n".join(f"{idx}. {line}" for idx, line in enumerate(bundle["high_descs"], start=1))
)
if bundle["pddl_params"]:
parts.append(
"## Reference PDDL Params\n"
+ "\n".join(f"- {key}: {value}" for key, value in bundle["pddl_params"].items())
)
if bundle["high_pddl"]:
parts.append(
"## Reference Planner High-Level Plan\n" + "\n".join(bundle["high_pddl"])
)
if bundle["scene_summary"]:
parts.append(
"## Reference Scene Summary\n"
+ "\n".join(f"- {key}: {value}" for key, value in bundle["scene_summary"].items())
)
return "\n\n".join(parts)
def get_reference_metadata(self, item: dict) -> dict:
bundle = self._build_reference_bundle(item)
if not bundle:
return {"fields": [], "preview": ""}
fields: list[str] = []
previews: list[str] = []
if bundle["task_type"]:
fields.append("task_type")
previews.append(f"[task_type] {bundle['task_type']}")
if bundle["task_descs"]:
fields.append("task_desc")
previews.append("[task_desc]\n" + "\n".join(bundle["task_descs"][:2]))
if bundle["high_descs"]:
fields.append("high_descs")
previews.append("[high_descs]\n" + "\n".join(bundle["high_descs"][:3]))
if bundle["pddl_params"]:
fields.append("pddl_params")
previews.append(
"[pddl_params]\n"
+ "\n".join(
f"{key}: {value}" for key, value in list(bundle["pddl_params"].items())[:4]
)
)
if bundle["high_pddl"]:
fields.append("plan.high_pddl")
previews.append("[plan.high_pddl]\n" + "\n".join(bundle["high_pddl"][:3]))
if bundle["scene_summary"]:
fields.append("scene")
previews.append(
"[scene]\n"
+ "\n".join(
f"{key}: {value}" for key, value in bundle["scene_summary"].items()
)
)
return {
"fields": fields,
"preview": "\n\n".join(previews)[:600],
}
@staticmethod
def _infer_dataset_from_gamefile(gamefile: str) -> tuple[str, bool]:
path = str(gamefile or "")
if "/valid_seen/" in path:
return "eval_in_distribution", False
if "/valid_unseen/" in path:
return "eval_out_of_distribution", False
return "train", True
def get_dataloader(self):
return self.dataloader
def _comparison_items(self, items: list[dict]) -> list[dict]:
enriched: list[dict] = []
for item in items:
row = dict(item)
bundle = self._build_reference_bundle(row)
if bundle.get("task_descs"):
row["task_description"] = bundle["task_descs"][0]
elif bundle.get("task_type"):
row["task_description"] = bundle["task_type"]
enriched.append(row)
return enriched
def requires_ray(self) -> bool:
return False
def build_env_from_batch(self, batch: BatchSpec, **kwargs):
gamefiles = list(batch.metadata.get("gamefiles") or [])
result_ids = list(batch.metadata.get("result_ids") or [])
items = self._comparison_items(list(batch.payload or []))
return ALFWorldBatchRun(
env_num=batch.batch_size,
eval_dataset=batch.metadata.get("eval_dataset", batch.split),
seed=batch.seed,
is_train=batch.metadata.get("is_train", batch.phase == "train"),
specific_gamefiles=gamefiles or None,
result_ids=result_ids or None,
items=items,
workers=self.workers,
)
def build_train_env(self, batch_size: int, seed: int, **kwargs):
batch = self.dataloader.build_train_batch(batch_size=batch_size, seed=seed, **kwargs)
return self.build_env_from_batch(batch, **kwargs)
def build_eval_env(self, env_num: int, split: str, seed: int, **kwargs):
batch = self.dataloader.build_eval_batch(env_num=env_num, split=split, seed=seed, **kwargs)
return self.build_env_from_batch(batch, **kwargs)
def rollout(
self,
env_manager,
skill_content: str,
out_dir: str,
**kwargs,
) -> list[dict]:
results_path = os.path.join(out_dir, "results.jsonl")
os.makedirs(out_dir, exist_ok=True)
# Resume support
if os.path.exists(results_path):
existing: list[dict] = []
with open(results_path) as f:
for line in f:
try:
existing.append(json.loads(line))
except Exception:
pass
if existing:
return existing
if isinstance(env_manager, ALFWorldBatchRun):
results = self._run_batch(
env_manager,
skill_content=skill_content,
out_dir=out_dir,
)
else:
results = run_alfworld_batch(
env_manager=env_manager,
skill_content=skill_content,
max_steps=self.max_steps,
out_root=out_dir,
max_api_workers=self.max_api_workers,
result_ids=getattr(env_manager, "_skillopt_result_ids", None),
)
with open(results_path, "w") as f:
for r in results:
f.write(json.dumps(r, ensure_ascii=False) + "\n")
return results
@staticmethod
def _close_env(env_manager) -> None:
close = getattr(env_manager, "close", None)
if callable(close):
close()
def _run_batch(
self,
batch: ALFWorldBatchRun,
skill_content: str,
out_dir: str,
*,
diagnostic_mode: bool = False,
diagnostic_instruction: str = "",
) -> list[dict]:
total = int(batch.env_num or 0)
if total <= 0:
return []
workers = max(1, min(int(batch.workers or self.workers), total))
if total > workers:
print(
f" [alfworld rollout] episodes={total} "
f"env_workers={workers} chunks={(total + workers - 1) // workers}"
)
all_results: list[dict] = []
for start in range(0, total, workers):
chunk_size = min(workers, total - start)
chunk_gamefiles = (
batch.specific_gamefiles[start:start + chunk_size]
if batch.specific_gamefiles
else None
)
chunk_ids = (
batch.result_ids[start:start + chunk_size]
if batch.result_ids
else [f"env_{idx:03d}" for idx in range(start, start + chunk_size)]
)
chunk_env = build_alfworld_env(
env_num=chunk_size,
eval_dataset=batch.eval_dataset,
seed=batch.seed + start,
is_train=batch.is_train,
specific_gamefiles=chunk_gamefiles,
)
try:
chunk_results = run_alfworld_batch(
env_manager=chunk_env,
skill_content=skill_content,
max_steps=self.max_steps,
out_root=out_dir,
max_api_workers=min(self.max_api_workers, chunk_size),
diagnostic_mode=diagnostic_mode,
diagnostic_instruction=diagnostic_instruction,
result_ids=chunk_ids,
)
finally:
self._close_env(chunk_env)
all_results.extend(chunk_results)
return all_results
def reflect(
self,
results: list[dict],
skill_content: str,
out_dir: str,
**kwargs,
) -> list[dict | None]:
prediction_dir = kwargs.get("prediction_dir", os.path.join(out_dir, "predictions"))
patches_dir = kwargs.get("patches_dir", os.path.join(out_dir, "patches"))
random_seed = kwargs.get("random_seed")
step_buffer_context = kwargs.get("step_buffer_context", "")
meta_skill_context = kwargs.get("meta_skill_context", "")
return run_minibatch_reflect(
results=results,
skill_content=skill_content,
prediction_dir=prediction_dir,
patches_dir=patches_dir,
workers=self.analyst_workers,
failure_only=self.failure_only,
minibatch_size=self.minibatch_size,
edit_budget=self.edit_budget,
random_seed=random_seed,
error_system=self.get_error_minibatch_prompt(),
success_system=self.get_success_minibatch_prompt(),
step_buffer_context=step_buffer_context,
meta_skill_context=meta_skill_context,
)
def deep_reflect(
self,
results: list[dict],
skill_content: str,
out_dir: str,
**kwargs,
) -> list[dict | None]:
if not self.use_deep_reflect:
return []
prediction_dir = kwargs.get("prediction_dir", os.path.join(out_dir, "predictions"))
random_seed = kwargs.get("random_seed")
step_buffer_context = kwargs.get("step_buffer_context", "")
meta_skill_context = kwargs.get("meta_skill_context", "")
selected_items = self.select_representative_items(
results,
results,
n_failures=self.deep_reflect_failures,
n_successes=self.deep_reflect_successes,
seed=random_seed,
)
if not selected_items:
return []
selected_ids = {str(item["id"]) for item in selected_items}
selected_results = [row for row in results if str(row.get("id")) in selected_ids]
selected_examples = self.attach_reference_context(selected_results, selected_items)
field_counts: dict[str, int] = {}
selected_metadata: list[dict] = []
for item in selected_items:
meta = self.get_reference_metadata(item)
for field in meta["fields"]:
field_counts[field] = field_counts.get(field, 0) + 1
selected_metadata.append({
"id": str(item["id"]),
"task_type": str(item.get("task_type") or "alfworld"),
"gamefile": str(item.get("gamefile") or ""),
"reference_fields": meta["fields"],
"reference_preview": meta["preview"],
})
deep_dir = os.path.join(out_dir, "deep_reflect")
rollout_dir = os.path.join(deep_dir, "rollout")
patches_dir = os.path.join(deep_dir, "patches")
os.makedirs(deep_dir, exist_ok=True)
field_summary = ", ".join(
f"{field}({count}/{len(selected_items)})"
for field, count in sorted(field_counts.items())
) or "none"
print(
f" [2b/6 DEEP REFLECT setup] selected={len(selected_items)} "
f"reference_fields={field_summary}"
)
probe = generate_deep_probe_instruction(
skill_content=skill_content,
items=selected_examples,
prediction_dir=prediction_dir,
system_prompt=self.get_deep_probe_prompt(),
step_buffer_context=step_buffer_context,
meta_skill_context=meta_skill_context,
output_requirements=[
"- Some trajectories may include a hidden Reference block. Use it to target the student's latent subgoal, missing precondition, or next-step intent, but do not reveal or paraphrase that reference to the student.",
"- The instruction must request a brief diagnostic readout inside the existing <think>...</think> block.",
"- The student must still output exactly one admissible action inside <action>...</action>.",
"- Do not ask for exhaustive inventories, full plans, or long chain-of-thought.",
"- The instruction text should be ready to append directly to the student's prompt.",
],
)
if not probe:
return []
with open(os.path.join(deep_dir, "probe.json"), "w", encoding="utf-8") as f:
json.dump(
{
**probe,
"reference_summary": {
"selected_count": len(selected_items),
"field_counts": field_counts,
},
"selected_examples": selected_metadata,
},
f,
ensure_ascii=False,
indent=2,
)
gamefiles = [str(item.get("gamefile") or "") for item in selected_items]
if any(not gamefile for gamefile in gamefiles):
return []
eval_dataset, is_train = self._infer_dataset_from_gamefile(gamefiles[0])
deep_env = ALFWorldBatchRun(
env_num=len(selected_items),
eval_dataset=eval_dataset,
seed=random_seed or 42,
is_train=is_train,
specific_gamefiles=gamefiles,
workers=min(self.workers, max(len(selected_items), 1)),
result_ids=[str(item["id"]) for item in selected_items],
)
deep_results = self._run_batch(
deep_env,
skill_content=skill_content,
out_dir=rollout_dir,
diagnostic_mode=True,
diagnostic_instruction=probe["probe_instruction"],
)
deep_results = self.attach_reference_context(deep_results, selected_items)
return run_minibatch_reflect(
results=deep_results,
skill_content=skill_content,
prediction_dir=os.path.join(rollout_dir, "predictions"),
patches_dir=patches_dir,
workers=self.analyst_workers,
failure_only=self.failure_only,
minibatch_size=self.minibatch_size,
edit_budget=self.edit_budget,
random_seed=random_seed,
error_system=self.get_error_minibatch_prompt(),
success_system=self.get_success_minibatch_prompt(),
step_buffer_context=step_buffer_context,
meta_skill_context=meta_skill_context,
)
def get_task_types(self) -> list[str]:
return list(TASKS)
skillopt/envs/alfworld/dataloader.py
+123
View File
@@ -0,0 +1,123 @@
"""ALFWorld task dataloader."""
from __future__ import annotations
from skillopt.datasets.base import BatchSpec, SplitDataLoader
class ALFWorldDataLoader(SplitDataLoader):
"""ALFWorld batch planner.
In split_dir mode, batches are fixed gamefile items so ablations differ
only in how the same training set is batched.
"""
def __init__(
self,
split_dir: str = "",
data_path: str = "",
split_mode: str = "split_dir",
split_ratio: str = "2:1:7",
split_seed: int = 42,
split_output_dir: str = "",
seed: int = 42,
limit: int = 0,
train_size: int = 0,
**kwargs,
) -> None:
super().__init__(
split_dir=split_dir,
data_path=data_path,
split_mode=split_mode,
split_ratio=split_ratio,
split_seed=split_seed,
split_output_dir=split_output_dir,
seed=seed,
limit=limit,
)
self.train_size_override = int(train_size or 0)
@staticmethod
def _metadata_for_items(items: list[dict], split: str, phase: str) -> dict:
gamefiles = [str(item.get("gamefile") or "") for item in items]
if any(not gamefile for gamefile in gamefiles):
raise ValueError("ALFWorld split items must contain non-empty gamefile paths.")
eval_dataset = "train"
is_train = phase == "train"
first = gamefiles[0] if gamefiles else ""
if "/valid_seen/" in first:
eval_dataset = "eval_in_distribution"
is_train = False
elif "/valid_unseen/" in first:
eval_dataset = "eval_out_of_distribution"
is_train = False
return {
"eval_dataset": eval_dataset,
"is_train": is_train,
"gamefiles": gamefiles,
"result_ids": [str(item.get("id") or idx) for idx, item in enumerate(items)],
}
def get_train_size(self) -> int:
if self.train_size_override > 0:
return self.train_size_override
return super().get_train_size()
def build_train_batch(self, batch_size: int, seed: int, **kwargs) -> BatchSpec:
batch = super().build_train_batch(batch_size=batch_size, seed=seed, **kwargs)
items = list(batch.payload or [])
batch.metadata.update(self._metadata_for_items(items, "train", "train"))
return BatchSpec(
phase="train",
split="train",
seed=seed,
batch_size=len(items),
payload=items,
metadata=batch.metadata,
)
def plan_train_epoch(
self,
*,
epoch: int,
steps_per_epoch: int,
accumulation: int,
batch_size: int,
seed: int,
**kwargs,
) -> list[BatchSpec]:
batches = super().plan_train_epoch(
epoch=epoch,
steps_per_epoch=steps_per_epoch,
accumulation=accumulation,
batch_size=batch_size,
seed=seed,
**kwargs,
)
for batch in batches:
items = list(batch.payload or [])
batch.metadata.update(self._metadata_for_items(items, "train", "train"))
return batches
def build_eval_batch(
self,
env_num: int,
split: str,
seed: int,
**kwargs,
) -> BatchSpec:
batch = super().build_eval_batch(
env_num=env_num,
split=split,
seed=seed,
**kwargs,
)
items = list(batch.payload or [])
batch.metadata.update(self._metadata_for_items(items, split, "eval"))
return BatchSpec(
phase="eval",
split=split,
seed=seed,
batch_size=len(items),
payload=items,
metadata=batch.metadata,
)
skillopt/envs/alfworld/prompts/analyst_error.md
+55
View File
@@ -0,0 +1,55 @@
You are an expert failure-analysis agent for ALFWorld embodied household tasks.
You will be given MULTIPLE failed agent trajectories from a single minibatch
and the current skill document.
Your job is to identify the most important COMMON failure patterns across
the batch and propose a concise set of skill edits.
## ALFWorld Task Types
- pick_and_place: Put object in/on a receptacle
- pick_two_obj_and_place: Put two instances of an object in/on a receptacle
- look_at_obj_in_light: Examine an object under a desklamp
- pick_heat_then_place_in_recep: Heat an object and put it in/on a receptacle
- pick_cool_then_place_in_recep: Cool an object and put it in/on a receptacle
- pick_clean_then_place_in_recep: Clean an object and put it in/on a receptacle
## Failure Type Categories
- **navigation_loop**: the agent revisits the same locations repeatedly without progress
- **missed_object**: the agent fails to pick up a visible/reachable goal object
- **wrong_sequence**: the agent performs actions in the wrong order (e.g., placing before transforming)
- **premature_stop**: the agent stops or gets stuck before completing all goal conditions
- **action_loop**: the agent repeats the same action without advancing
- **appliance_error**: the agent misuses or skips an appliance (microwave, fridge, sink)
- **rule_missing**: the skill lacks a relevant rule for this situation
- **rule_wrong**: an existing skill rule is misleading or incorrect
- **rule_ignored**: the skill has the right rule but the agent did not follow it
- **other**: none of the above
## Analysis Process
1. Read ALL trajectories in the minibatch.
2. Identify the most prevalent, systematic failure patterns across them.
3. For each pattern, classify its failure type.
4. Propose skill edits that address the COMMON patterns — not individual edge cases.
5. Edits must be generalizable; do not hardcode task-specific values.
6. Only patch gaps in the skill — do not duplicate existing content.
You will be told the maximum number of edits (the budget L). Produce AT MOST L edits,
focusing on the highest-impact patterns. You may produce fewer if warranted.
Respond ONLY with a valid JSON object (no markdown fences, no extra text):
{
"batch_size": <number of trajectories analysed>,
"failure_summary": [
{"failure_type": "<type>", "count": <int>, "description": "<one-line>"}
],
"patch": {
"reasoning": "<why these edits address the batch's common failures>",
"edits": [
{"op": "append", "content": "<markdown to add at end of skill>"},
{"op": "insert_after", "target": "<exact heading/text to insert after>", "content": "<markdown>"},
{"op": "replace", "target": "<exact text to replace>", "content": "<replacement>"},
{"op": "delete", "target": "<exact text to remove>"}
]
}
}
Only include edits that are needed. "edits" can be an empty list if no patch is warranted.
skillopt/envs/alfworld/prompts/analyst_success.md
+33
View File
@@ -0,0 +1,33 @@
You are an expert success-pattern analyst for AI agents operating in ALFWorld,
a text-based embodied household environment.
You will be given MULTIPLE successful agent trajectories from a single minibatch
and the current skill document. Your job is to identify generalizable behavior
patterns that are COMMON across the batch and worth encoding in the skill.
## Rules
- Only propose patches for patterns NOT already covered in the skill.
- Focus on patterns that appear across MULTIPLE trajectories in the batch.
- Be concise. Patterns must generalize beyond specific tasks.
- Prefer reinforcing existing sections over adding new top-level sections.
- If the agents' success involved efficient exploration or smart appliance usage,
consider reinforcing that in the patch.
You will be told the maximum number of edits (the budget L). Produce AT MOST L edits,
focusing on the most broadly applicable patterns. You may produce fewer if warranted.
Respond ONLY with a valid JSON object:
{
"batch_size": <number of trajectories analysed>,
"success_patterns": ["<pattern 1>", "<pattern 2>"],
"patch": {
"reasoning": "<why these patterns are worth encoding>",
"edits": [
{"op": "append", "content": "<markdown>"},
{"op": "insert_after", "target": "<heading/text>", "content": "<markdown>"},
{"op": "replace", "target": "<old text>", "content": "<new text>"},
{"op": "delete", "target": "<exact text to remove>"}
]
}
}
"edits" may be empty if the skill already covers all observed patterns.
skillopt/envs/alfworld/prompts/deep_probe.md
+35
View File
@@ -0,0 +1,35 @@
You are an expert diagnostic-probe designer for ALFWorld embodied tasks.
You will design one short diagnostic instruction to append to the student's prompt
for a handful of representative ALFWorld trajectories.
The goal is to expose whether the student has the right intermediate subgoal,
object/receptacle state, and next-step intention without substantially changing
the current scaffold.
## Hard Constraints
1. Do NOT substantially change the student's existing action-selection scaffold.
2. Do NOT prescribe a brand-new planner or long multi-step policy.
3. Do NOT ask for exhaustive search over all objects or all admissible actions.
4. Keep the diagnostic readout brief and place it inside the existing <think>...</think> block.
5. The student must still output exactly one admissible action inside <action>...</action>.
6. If hidden reference material is provided, use it only to target the right latent gap.
7. Never copy hidden reference content into the student-facing probe.
## Good Probe Targets
- current subgoal
- target object / target receptacle / target state
- decisive missing precondition
- why one candidate action is better than a tempting alternative
- whether the current step should explore, transform an object, or place it
## Bad Probe Targets
- a full optimal plan from start to finish
- exhaustive object inventories
- a new theorem-like or planner-like protocol
Respond ONLY with a valid JSON object:
{
"reasoning": "<why this probe reveals the latent skill gap>",
"probe_instruction": "<the exact instruction text to append to the student prompt>"
}
skillopt/envs/alfworld/prompts/rollout_no_history.md
+8
View File
@@ -0,0 +1,8 @@
You are an expert agent operating in the ALFRED Embodied Environment.
Your current observation is: {current_observation}
Your admissible actions of the current situation are: [{admissible_actions}].
Now it's your turn to take an action.
You should first reason step-by-step about the current situation. This reasoning process MUST be enclosed within <think> </think> tags.
Once you've finished your reasoning, you should choose an admissible action for current step and present it within <action> </action> tags.
skillopt/envs/alfworld/prompts/rollout_with_history.md
+9
View File
@@ -0,0 +1,9 @@
You are an expert agent operating in the ALFRED Embodied Environment. Your task is to: {task_description}
Prior to this step, you have already taken {step_count} step(s). Below are the most recent {history_length} observations and the corresponding actions you took: {action_history}
You are now at step {current_step} and your current observation is: {current_observation}
Your admissible actions of the current situation are: [{admissible_actions}].
Now it's your turn to take an action.
You should first reason step-by-step about the current situation. This reasoning process MUST be enclosed within <think> </think> tags.
Once you've finished your reasoning, you should choose an admissible action for current step and present it within <action> </action> tags.
skillopt/envs/alfworld/prompts/rollout_with_memory.md
+16
View File
@@ -0,0 +1,16 @@
You are an expert agent operating in the ALFRED Embodied Environment. Your task is to: {task_description}
## Retrieved Relevant Experience
{retrieved_memories}
## Current Progress
Prior to this step, you have already taken {step_count} step(s). Below are the most recent {history_length} observations and the corresponding actions you took: {action_history}
You are now at step {current_step} and your current observation is: {current_observation}
Your admissible actions of the current situation are: [{admissible_actions}].
Now it's your turn to take an action.
You should first reason step-by-step about the current situation. This reasoning process MUST be enclosed within <think> </think> tags.
Once you've finished your reasoning, you should choose an admissible action for current step and present it within <action> </action> tags.
skillopt/envs/alfworld/reflect.py
+4
View File
@@ -0,0 +1,4 @@
"""ALFWorld Reflect stage.
Prompts are now loaded from .md files by the base adapter.
"""
skillopt/envs/alfworld/rollout.py
+359
View File
@@ -0,0 +1,359 @@
"""ALFWorld rollout module for ReflACT.
Provides:
- build_alfworld_env(): build ALFWorld environment (wraps vendored SkillRL env)
- run_alfworld_batch(): run a batch of ALFWorld episodes in parallel
- TASKS: list of ALFWorld task types
"""
from __future__ import annotations
import json
import os
import re
import sys
import time
import concurrent.futures
import numpy as np
from skillopt.model import chat_student
# ── Constants ─────────────────────────────────────────────────────────────────
TASKS = [
"pick_and_place",
"pick_two_obj_and_place",
"look_at_obj_in_light",
"pick_heat_then_place_in_recep",
"pick_cool_then_place_in_recep",
"pick_clean_then_place_in_recep",
]
# ── Helpers ───────────────────────────────────────────────────────────────────
def _get_task_type(gamefile: str) -> str:
for task in TASKS:
if task in gamefile:
return task
return "other"
def _extract_action(model_response: str) -> str | None:
match = re.search(r"<action>(.*?)</action>", model_response, re.DOTALL)
return match.group(1).strip() if match else None
def _extract_think(model_response: str) -> str | None:
match = re.search(r"<think>(.*?)</think>", model_response, re.DOTALL)
return match.group(1).strip() if match else None
def _build_skill_prompt(skill_content: str) -> str:
"""Build the skill section to inject into the agent's system prompt."""
if not skill_content or not skill_content.strip():
return ""
return (
"\n\n## Skill Knowledge\n"
"Below is a skill document with learned strategies. "
"Use these guidelines to inform your decisions:\n\n"
f"{skill_content}\n"
)
def _append_diagnostic_instruction(prompt: str, diagnostic_instruction: str) -> str:
if not diagnostic_instruction or not diagnostic_instruction.strip():
return prompt
return f"{prompt}\n\n## Training Readout\n{diagnostic_instruction.strip()}\n"
# ── Environment builder ──────────────────────────────────────────────────────
def build_alfworld_env(
env_num: int,
eval_dataset: str = "eval_out_of_distribution",
seed: int = 42,
is_train: bool = False,
specific_gamefiles: list[str] | None = None,
):
"""Build ALFWorld environment manager.
Args:
env_num: number of parallel environments
eval_dataset: 'eval_in_distribution' or 'eval_out_of_distribution' or train
seed: random seed
is_train: whether to use training set
Returns:
env_manager: AlfWorldEnvironmentManager instance
"""
from omegaconf import OmegaConf
from functools import partial
from skillopt.envs.alfworld.vendor.alfworld_envs import build_alfworld_envs
from skillopt.envs.alfworld.vendor.alfworld_projection import alfworld_projection
from skillopt.envs.alfworld.vendor.env_manager import AlfWorldEnvironmentManager
HERE = os.path.dirname(os.path.abspath(__file__))
alf_config_path = os.path.join(HERE, "vendor", "config_tw.yaml")
env_kwargs = {"eval_dataset": eval_dataset}
envs = build_alfworld_envs(
alf_config_path,
seed=seed,
env_num=env_num,
group_n=1,
is_train=is_train,
env_kwargs=env_kwargs,
resources_per_worker=None,
gamefiles=specific_gamefiles,
)
config = OmegaConf.create(
{
"env": {
"history_length": 2,
"env_name": "alfworld/AlfredTWEnv",
}
}
)
projection_f = partial(alfworld_projection)
env_manager = AlfWorldEnvironmentManager(envs, projection_f, config)
return env_manager
# ── Batch rollout ─────────────────────────────────────────────────────────────
def run_alfworld_batch(
env_manager,
skill_content: str,
max_steps: int = 50,
out_root: str = "",
max_api_workers: int = 8,
temperature: float = 0.4,
max_completion_tokens: int = 2048,
diagnostic_mode: bool = False,
diagnostic_instruction: str = "",
result_ids: list[str] | None = None,
) -> list[dict]:
"""Run a batch of ALFWorld episodes.
Returns a list of result dicts compatible with SkillOpt pipeline:
[
{
"id": "<env_idx>_<gamefile_hash>",
"hard": 0 or 1,
"soft": 0.0 or 1.0,
"n_turns": <int>,
"fail_reason": "<str>",
"agent_ok": True,
"task_type": "<str>",
"gamefile": "<str>",
"task_description": "<str>",
},
...
]
Also saves conversation.json per environment in out_root/predictions/<task_id>/
"""
skill_prompt = _build_skill_prompt(skill_content)
obs, infos = env_manager.reset({})
env_num = len(obs["text"])
env_dones = [False] * env_num
overall_success = [False] * env_num
# Build per-env metadata
env_meta: list[dict] = []
for i in range(env_num):
gamefile = infos[i].get("extra.gamefile", "") if isinstance(infos[i], dict) else ""
task_type = _get_task_type(gamefile)
# Extract task description from initial observation
task_desc = ""
anchor_text = obs["anchor"][i] if "anchor" in obs else ""
task_start = anchor_text.find("Your task is to: ")
if task_start != -1:
task_desc = anchor_text[task_start + len("Your task is to: "):].strip()
env_meta.append({
"gamefile": gamefile,
"task_type": task_type,
"task_description": task_desc,
})
# Per-env conversation records
conversations: list[list[dict]] = [[] for _ in range(env_num)]
for step_idx in range(max_steps):
if all(env_dones):
break
active_indices = [i for i in range(env_num) if not env_dones[i]]
# Build prompts with skill injection
prompts: dict[int, str] = {}
for i in active_indices:
prompt = obs["text"][i]
if skill_prompt:
# Inject skill before the action instruction
prompt = skill_prompt + "\n" + prompt
if diagnostic_mode and diagnostic_instruction.strip():
prompt = _append_diagnostic_instruction(prompt, diagnostic_instruction)
prompts[i] = prompt
# Call API in parallel
actions = ["None"] * env_num
action_timeout = 180
def call_api(idx):
try:
response, _ = chat_student(
system="You are an expert agent operating in the ALFRED Embodied Environment.",
user=prompts[idx],
max_completion_tokens=max_completion_tokens,
retries=5,
stage="rollout",
timeout=120,
)
response = (response or "").strip()
if not response:
return idx, "<think>empty model response</think><action>look</action>"
if _extract_action(response) is None:
return idx, "<think>missing action tag</think><action>look</action>"
return idx, response
except Exception as e:
return idx, "<think>error</think><action>look</action>"
executor = concurrent.futures.ThreadPoolExecutor(max_workers=max_api_workers)
try:
futures = {executor.submit(call_api, i): i for i in active_indices}
started_at = {future: time.time() for future in futures}
pending_futs = set(futures)
while pending_futs:
done, _ = concurrent.futures.wait(
pending_futs,
timeout=5,
return_when=concurrent.futures.FIRST_COMPLETED,
)
now = time.time()
timed_out = [
future for future in pending_futs - done
if now - started_at[future] >= action_timeout
]
for future in done:
pending_futs.remove(future)
try:
idx, response = future.result()
except Exception: # noqa: BLE001
idx = futures[future]
response = "<think>error</think><action>look</action>"
actions[idx] = response
for future in timed_out:
pending_futs.remove(future)
idx = futures[future]
actions[idx] = "<think>api timeout</think><action>look</action>"
finally:
executor.shutdown(wait=False, cancel_futures=True)
# Save model responses before stepping
model_responses = {i: actions[i] for i in active_indices}
# Step environment
obs, rewards, dones, infos = env_manager.step(actions)
# Record trajectory
for i in active_indices:
step_record = {
"step": step_idx,
"action": _extract_action(model_responses[i]),
"reasoning": _extract_think(model_responses[i]),
"model_response": model_responses[i],
"env_feedback": obs["anchor"][i] if "anchor" in obs else "",
"reward": float(rewards[i]),
"done": bool(dones[i]),
}
conversations[i].append(step_record)
# Update done status
for i in range(env_num):
if env_dones[i]:
continue
if dones[i]:
env_dones[i] = True
won = bool(infos[i].get("won", False))
overall_success[i] = won
# Build results and save conversations
results: list[dict] = []
pred_dir = os.path.join(out_root, "predictions") if out_root else ""
for i in range(env_num):
gamefile = env_meta[i]["gamefile"]
task_type = env_meta[i]["task_type"]
task_desc = env_meta[i]["task_description"]
n_turns = len(conversations[i])
won = overall_success[i]
# Generate stable task ID from env index and gamefile
task_id = str(result_ids[i]) if result_ids and i < len(result_ids) else f"env_{i:03d}"
fail_reason = ""
if not won:
if not env_dones[i]:
fail_reason = f"Timeout after {max_steps} steps"
else:
fail_reason = "Episode ended without completing the task"
result = {
"id": task_id,
"hard": 1 if won else 0,
"soft": 1.0 if won else 0.0,
"n_turns": n_turns,
"fail_reason": fail_reason,
"agent_ok": True, # ALFWorld agent always runs OK (no crash)
"task_type": task_type,
"gamefile": gamefile,
"task_description": task_desc,
"instruction_type": task_type, # for compatibility with v2 pipeline
}
results.append(result)
# Save conversation
if pred_dir:
conv_dir = os.path.join(pred_dir, task_id)
os.makedirs(conv_dir, exist_ok=True)
with open(os.path.join(conv_dir, "conversation.json"), "w") as f:
json.dump(conversations[i], f, ensure_ascii=False, indent=2)
return results
# ── Item loading (for compatibility with split_three_way) ────────────────────
def load_alfworld_items(
eval_dataset: str,
env_num: int,
seed: int = 42,
is_train: bool = False,
) -> list[dict]:
"""Create pseudo-item dicts for ALFWorld environments.
Since ALFWorld doesn't have a static JSON dataset like SpreadsheetBench,
we create lightweight item dicts that carry enough metadata for the pipeline.
The actual environment is built dynamically.
Returns:
List of dicts with "id" keys, one per environment slot.
"""
items = []
for i in range(env_num):
items.append({
"id": f"env_{i:03d}",
"eval_dataset": eval_dataset,
"env_index": i,
})
return items
skillopt/envs/alfworld/skills/initial.md
+45
View File
@@ -0,0 +1,45 @@
# ALFWorld Embodied Agent Skill
## Overview
This skill guides agents operating in the ALFWorld text-based embodied environment.
The agent must complete household tasks by navigating rooms, interacting with objects,
and using appliances. Actions must be chosen from the admissible action list provided
at each step.
**Output format**: Always output `<think>...</think>` for reasoning, then `<action>...</action>` for the chosen action.
---
## Task Types
| Type | Goal | Key Steps |
|------|------|-----------|
| Pick & Place | Put object X in/on receptacle Y | Find X -> take X -> go to Y -> put X in/on Y |
| Pick Two & Place | Put two instances of X in/on Y | Find X1 -> take -> place -> find X2 -> take -> place |
| Examine in Light | Examine object X under desklamp | Find X -> take X -> find desklamp -> use desklamp |
| Clean & Place | Clean object X and put in/on Y | Find X -> take X -> go to sink -> clean X -> go to Y -> put X |
| Heat & Place | Heat object X and put in/on Y | Find X -> take X -> go to microwave -> heat X -> go to Y -> put X |
| Cool & Place | Cool object X and put in/on Y | Find X -> take X -> go to fridge -> cool X -> go to Y -> put X |
---
## General Principles
1. **Decompose the task**: Parse the goal into ordered sub-goals (locate, acquire, transform, deliver). Complete each before moving to the next.
2. **Systematic exploration**: Search each surface and container exactly once before revisiting. Open closed containers (drawers, cabinets, fridge) before judging them empty.
3. **Grab immediately**: When a required object is visible and reachable, take it right away before moving elsewhere.
4. **Transform before placing**: If the task requires cleaning, heating, or cooling, perform the state change at the appropriate appliance before heading to the final destination.
5. **Direct delivery**: Once holding the transformed (or untransformed) goal object, navigate straight to the target receptacle and place it.
6. **Track progress**: Maintain an internal count of how many objects still need to be found and placed. Only stop searching when the count reaches zero.
7. **Avoid loops**: Never repeat the same action more than twice in a row. If stuck, move to a different unexplored location.
8. **Only choose admissible actions**: Always pick an action from the admissible action list. Do not invent actions.
---
## Common Mistakes to Avoid
- **Revisiting searched locations**: Keep track of which surfaces/containers have been checked; do not re-examine them.
- **Ignoring visible objects**: If the target object appears in the observation, pick it up immediately.
- **Skipping state changes**: Do not place an object at the destination without first cleaning/heating/cooling it when required.
- **Premature termination**: Do not stop the episode until all goal conditions are verified as met.
- **Action loops**: Repeatedly toggling or examining the same object wastes steps. Move on to new locations instead.
skillopt/envs/alfworld/vendor/__init__.py
Vendored
+9
View File
@@ -0,0 +1,9 @@
"""Vendored ALFWorld environment runtime.
Minimal subset of SkillRL's agent_system package needed to run
ALFWorld environments with ReflACT. Original source:
https://github.com/NTU-LANTERN/SkillRL (Apache-2.0 License)
"""
from .alfworld_envs import AlfworldEnvs, build_alfworld_envs
from .alfworld_projection import alfworld_projection
from .env_manager import AlfWorldEnvironmentManager
skillopt/envs/alfworld/vendor/alfworld_envs.py
Vendored
+221
View File
@@ -0,0 +1,221 @@
# Vendored from SkillRL (Apache-2.0 License)
# Original: agent_system/environments/env_package/alfworld/envs.py
# Modified: imports use pip-installed alfworld package instead of vendored copy.
import os
import multiprocessing as mp
import traceback
import yaml
import gymnasium as gym
import numpy as np
from alfworld.agents.environment import get_environment
def load_config_file(path):
assert os.path.exists(path), f"Invalid config file: {path}"
with open(path) as reader:
config = yaml.safe_load(reader)
return config
def compute_reward(info, multi_modal=False):
if multi_modal:
reward = 10.0 * float(info['won']) + float(info['goal_condition_success_rate'])
else:
reward = 10.0 * float(info['won'])
return reward
class AlfworldWorker:
"""Stateful worker that holds one ALFWorld sub-environment."""
def __init__(self, config, seed, base_env, gamefile=None):
if gamefile:
base_env.game_files = [gamefile]
if hasattr(base_env, "num_games"):
base_env.num_games = 1
self.env = base_env.init_env(batch_size=1)
self.env.seed(seed)
def step(self, action):
actions = [action]
obs, scores, dones, infos = self.env.step(actions)
infos['observation_text'] = obs
return obs, scores, dones, infos
def reset(self):
obs, infos = self.env.reset()
infos['observation_text'] = obs
return obs, infos
def _worker_loop(cmd_q, result_q, config, seed, is_train, eval_dataset, gamefile):
"""Run one ALFWorld environment in a child process."""
try:
env_type = config['env']['type']
base_env = get_environment(env_type)(
config,
train_eval='train' if is_train else eval_dataset,
)
worker = AlfworldWorker(config, seed, base_env, gamefile)
result_q.put((True, "ready"))
except BaseException:
result_q.put((False, traceback.format_exc()))
return
while True:
cmd, payload = cmd_q.get()
if cmd == "close":
result_q.put((True, None))
return
try:
if cmd == "reset":
result = worker.reset()
elif cmd == "step":
result = worker.step(payload)
else:
raise ValueError(f"Unknown ALFWorld worker command: {cmd}")
result_q.put((True, result))
except BaseException:
result_q.put((False, traceback.format_exc()))
class _ProcessWorker:
"""Small stdlib actor wrapper for one environment process."""
def __init__(self, ctx, config, seed, is_train, eval_dataset, gamefile=None):
self.cmd_q = ctx.Queue(maxsize=1)
self.result_q = ctx.Queue(maxsize=1)
self.process = ctx.Process(
target=_worker_loop,
args=(self.cmd_q, self.result_q, config, seed, is_train, eval_dataset, gamefile),
)
self.process.start()
ok, payload = self.result_q.get()
if not ok:
self.close(kill=True)
raise RuntimeError(f"Failed to start ALFWorld worker:\n{payload}")
def send(self, cmd, payload=None):
self.cmd_q.put((cmd, payload))
def recv(self):
ok, payload = self.result_q.get()
if not ok:
raise RuntimeError(f"ALFWorld worker failed:\n{payload}")
return payload
def close(self, kill=False):
if self.process.is_alive() and not kill:
try:
self.send("close")
self.recv()
except Exception:
kill = True
if kill and self.process.is_alive():
self.process.terminate()
self.process.join(timeout=5)
if self.process.is_alive():
self.process.kill()
self.process.join(timeout=1)
self.cmd_q.close()
self.result_q.close()
class AlfworldEnvs(gym.Env):
"""Vectorized ALFWorld environment using local process workers."""
def __init__(self, alf_config_path, seed, env_num, group_n,
resources_per_worker, is_train=True, env_kwargs=None, gamefiles=None):
super().__init__()
if env_kwargs is None:
env_kwargs = {}
eval_dataset = env_kwargs.get('eval_dataset', 'eval_in_distribution')
config = load_config_file(alf_config_path)
env_type = config['env']['type']
self.multi_modal = (env_type == 'AlfredThorEnv')
self.num_processes = env_num * group_n
self.group_n = group_n
self.gamefiles = list(gamefiles or [])
if self.gamefiles and len(self.gamefiles) != self.num_processes:
raise ValueError(
f"Expected {self.num_processes} gamefiles, got {len(self.gamefiles)}"
)
start_method = os.environ.get("ALFWORLD_WORKER_START_METHOD") or None
ctx = mp.get_context(start_method) if start_method else mp.get_context()
self.workers = []
for i in range(self.num_processes):
worker_gamefile = self.gamefiles[i] if self.gamefiles else None
worker = _ProcessWorker(
ctx,
config,
seed + (i // self.group_n),
is_train,
eval_dataset,
worker_gamefile,
)
self.workers.append(worker)
self.prev_admissible_commands = [None for _ in range(self.num_processes)]
def step(self, actions):
assert len(actions) == self.num_processes
for i, worker in enumerate(self.workers):
worker.send("step", actions[i])
results = [worker.recv() for worker in self.workers]
text_obs_list = []
rewards_list = []
dones_list = []
info_list = []
for i, (obs, scores, dones, info) in enumerate(results):
for k in info.keys():
info[k] = info[k][0]
text_obs_list.append(obs[0])
dones_list.append(dones[0])
info_list.append(info)
self.prev_admissible_commands[i] = info['admissible_commands']
rewards_list.append(compute_reward(info, self.multi_modal))
image_obs_list = None
return text_obs_list, image_obs_list, rewards_list, dones_list, info_list
def reset(self):
for worker in self.workers:
worker.send("reset")
results = [worker.recv() for worker in self.workers]
text_obs_list = []
info_list = []
for i, (obs, info) in enumerate(results):
for k in info.keys():
info[k] = info[k][0]
text_obs_list.append(obs[0])
self.prev_admissible_commands[i] = info['admissible_commands']
info_list.append(info)
image_obs_list = None
return text_obs_list, image_obs_list, info_list
@property
def get_admissible_commands(self):
return self.prev_admissible_commands
def close(self):
for worker in self.workers:
worker.close()
def build_alfworld_envs(alf_config_path, seed, env_num, group_n,
resources_per_worker, is_train=True, env_kwargs=None, gamefiles=None):
"""Build vectorized ALFWorld environments."""
return AlfworldEnvs(
alf_config_path, seed, env_num, group_n,
resources_per_worker, is_train, env_kwargs, gamefiles,
)
skillopt/envs/alfworld/vendor/alfworld_projection.py
Vendored
+60
View File
@@ -0,0 +1,60 @@
# Vendored from SkillRL (Apache-2.0 License)
# Original: agent_system/environments/env_package/alfworld/projection.py
from typing import List
import re
def alfworld_projection(actions: List[str], action_pools: List[List[str]]):
"""Process raw model outputs into valid ALFWorld actions.
Extracts text from ``<action>...</action>`` tags and validates that
the response also contains ``<think>...</think>`` tags.
Parameters
----------
actions : list[str]
Raw model outputs, one per environment.
action_pools : list[list[str]]
Admissible action lists per environment (unused but kept for API compat).
Returns
-------
actions : list[str]
Cleaned action strings.
valids : list[int]
1 if the action was successfully parsed, 0 otherwise.
"""
valids = [0] * len(actions)
for i in range(len(actions)):
original_str = actions[i]
actions[i] = actions[i].lower()
start_tag = "<action>"
end_tag = "</action>"
start_idx = actions[i].find(start_tag)
end_idx = actions[i].find(end_tag)
try:
if start_idx == -1 or end_idx == -1:
actions[i] = actions[i][-30:]
continue
extracted_action = actions[i][start_idx + len(start_tag):end_idx].strip().lower()
actions[i] = extracted_action
valids[i] = 1
except Exception:
actions[i] = actions[i][-30:]
# Require <think>...</think>
think_start_idx = original_str.find("<think>")
think_end_idx = original_str.find("</think>")
if think_start_idx == -1 or think_end_idx == -1:
valids[i] = 0
# Reject responses containing Chinese characters
if re.search(r'[\u4e00-\u9fff]', original_str):
valids[i] = 0
return actions, valids
skillopt/envs/alfworld/vendor/alfworld_prompts.py
Vendored
+8
View File
@@ -0,0 +1,8 @@
# Vendored from SkillRL (Apache-2.0 License)
# Original: agent_system/environments/prompts/alfworld.py
from skillopt.prompts import load_prompt
ALFWORLD_TEMPLATE_NO_HIS = load_prompt("rollout_no_history", env="alfworld")
ALFWORLD_TEMPLATE = load_prompt("rollout_with_history", env="alfworld")
ALFWORLD_TEMPLATE_WITH_MEMORY = load_prompt("rollout_with_memory", env="alfworld")
skillopt/envs/alfworld/vendor/config_tw.yaml
Vendored
+145
View File
@@ -0,0 +1,145 @@
dataset:
data_path: '$ALFWORLD_DATA/json_2.1.1/train'
eval_id_data_path: '$ALFWORLD_DATA/json_2.1.1/valid_seen' # null/None to disable
eval_ood_data_path: '$ALFWORLD_DATA/json_2.1.1/valid_unseen' # null/None to disable
num_train_games: -1 # max training games (<=0 indicates full dataset)
num_eval_games: -1 # max evaluation games (<=0 indicates full dataset)
logic:
domain: '$ALFWORLD_DATA/logic/alfred.pddl' # PDDL domain file that defines the world dynamics
grammar: '$ALFWORLD_DATA/logic/alfred.twl2' # Grammar file that defines the text feedbacks
env:
type: 'AlfredTWEnv' # 'AlfredTWEnv' or 'AlfredThorEnv' or 'AlfredHybrid'
# regen_game_files: False # check if game is solvable by expert and save to game.tw-pddl file
domain_randomization: False # shuffle Textworld print order and object id nums
task_types: [1, 2, 3, 4, 5, 6] # task-type ids: 1 - Pick & Place, 2 - Examine in Light, 3 - Clean & Place, 4 - Heat & Place, 5 - Cool & Place, 6 - Pick Two & Place
expert_timeout_steps: 150 # max steps before timeout for expert to solve the task
expert_type: "handcoded" # 'handcoded' or 'planner'. Note: the planner is very slow for real-time use
goal_desc_human_anns_prob: 0.0 # prob of using human-annotated goal language instead of templated goals (1.0 indicates all human annotations from ALFRED)
hybrid:
start_eps: 100000 # starting episode of hybrid training, tw-only training upto this point
thor_prob: 0.5 # prob of AlfredThorEnv during hybrid training
eval_mode: "tw" # 'tw' or 'thor' - env used for evaluation during hybrid training
thor:
screen_width: 300 # width of THOR window
screen_height: 300 # height of THOR window
smooth_nav: False # smooth rotations, looks, and translations during navigation (very slow)
save_frames_to_disk: False # save frame PNGs to disk (useful for making videos)
save_frames_path: './videos/' # path to save frame PNGs
controller:
type: 'oracle' # 'oracle' or 'oracle_astar' or 'mrcnn' or 'mrcnn_astar' (aka BUTLER)
debug: False
load_receps: True # load receptacle locations from precomputed dict (if available)
mask_rcnn:
pretrained_model_path: '$ALFWORLD_DATA/detectors/mrcnn.pth'
general:
random_seed: 42
use_cuda: True # disable this when running on machine without cuda
visdom: False # plot training/eval curves, run with visdom server
task: 'alfred'
training_method: 'dagger' # 'dqn' or 'dagger'
save_path: './training/' # path to save pytorch models
observation_pool_capacity: 3 # k-size queue, 0 indicates no observation
hide_init_receptacles: False # remove initial observation containing navigable receptacles
training:
batch_size: 10
max_episode: 50000
smoothing_eps: 0.1
optimizer:
learning_rate: 0.001
clip_grad_norm: 5
evaluate:
run_eval: True
batch_size: 10
env:
type: "AlfredTWEnv"
checkpoint:
report_frequency: 1000 # report every N episode
experiment_tag: 'test' # name of experiment
load_pretrained: False # during test, enable this so that the agent load your pretrained model
load_from_tag: 'not loading anything' # name of pre-trained model to load in save_path
model:
encoder_layers: 1
decoder_layers: 1
encoder_conv_num: 5
block_hidden_dim: 64
n_heads: 1
dropout: 0.1
block_dropout: 0.1
recurrent: True
rl:
action_space: "admissible" # 'admissible' (candidates from text engine) or 'generation' (seq2seq-style generation) or 'beam_search_choice' or 'exhaustive' (not working)
max_target_length: 20 # max token length for seq2seq generation
beam_width: 10 # 1 means greedy
generate_top_k: 3
training:
max_nb_steps_per_episode: 50 # terminate after this many steps
learn_start_from_this_episode: 0 # delay updates until this epsiode
target_net_update_frequency: 500 # sync target net with online net per this many epochs
replay:
accumulate_reward_from_final: True
count_reward_lambda: 0.0 # 0 to disable
novel_object_reward_lambda: 0.0 # 0 to disable
discount_gamma_game_reward: 0.9
discount_gamma_count_reward: 0.5
discount_gamma_novel_object_reward: 0.5
replay_memory_capacity: 500000 # adjust this depending on your RAM size
replay_memory_priority_fraction: 0.5
update_per_k_game_steps: 5
replay_batch_size: 64
multi_step: 3
replay_sample_history_length: 4
replay_sample_update_from: 2
epsilon_greedy:
noisy_net: False # if this is true, then epsilon greedy is disabled
epsilon_anneal_episodes: 1000 # -1 if not annealing
epsilon_anneal_from: 0.3
epsilon_anneal_to: 0.1
dagger:
action_space: "generation" # 'admissible' (candidates from text engine) or 'generation' (seq2seq-style generation) or 'exhaustive' (not working)
max_target_length: 20 # max token length for seq2seq generation
beam_width: 10 # 1 means greedy
generate_top_k: 5
unstick_by_beam_search: False # use beam-search for failed actions, set True during evaluation
training:
max_nb_steps_per_episode: 50 # terminate after this many steps
fraction_assist:
fraction_assist_anneal_episodes: 50000
fraction_assist_anneal_from: 1.0
fraction_assist_anneal_to: 0.01
fraction_random:
fraction_random_anneal_episodes: 0
fraction_random_anneal_from: 0.0
fraction_random_anneal_to: 0.0
replay:
replay_memory_capacity: 500000
update_per_k_game_steps: 5
replay_batch_size: 64
replay_sample_history_length: 4
replay_sample_update_from: 2
vision_dagger:
model_type: "resnet" # 'resnet' (whole image features) or 'maskrcnn_whole' (whole image MaskRCNN feats) or 'maskrcnn' (top k MaskRCNN detection feats) or 'no_vision' (zero vision input)
resnet_fc_dim: 64
maskrcnn_top_k_boxes: 10 # top k box features
use_exploration_frame_feats: False # append feats from initial exploration (memory intensive!)
sequence_aggregation_method: "average" # 'sum' or 'average' or 'rnn'
skillopt/envs/alfworld/vendor/env_base.py
Vendored
+84
View File
@@ -0,0 +1,84 @@
# Vendored from SkillRL (Apache-2.0 License)
# Original: agent_system/environments/base.py
# Trimmed to only include what ALFWorld needs.
from typing import List, Tuple, Dict, Any
import numpy as np
from collections import defaultdict
def to_numpy(data):
"""Convert data to numpy array."""
# Lazy-check for torch.Tensor to avoid hard dependency on torch
_torch_tensor = None
try:
import torch
_torch_tensor = torch.Tensor
except ImportError:
pass
if _torch_tensor is not None and isinstance(data, _torch_tensor):
data = data.detach().cpu().numpy()
elif isinstance(data, np.ndarray):
pass
elif isinstance(data, (int, float, bool, Tuple, List)):
data = np.array(data)
else:
raise ValueError(f"Unsupported type: {type(data)})")
return data
class EnvironmentManagerBase:
"""Base class for vectorized environment managers.
Manages a set of parallel environments, handles action projection,
observation post-processing, and history tracking.
"""
def __init__(self, envs, projection_f, config):
self.envs = envs
self.projection_f = projection_f
self.config = config
def reset(self, kwargs) -> Dict[str, Any]:
obs, infos = self.envs.reset()
return {'text': None, 'image': obs, 'anchor': None}, infos
def step(self, text_actions: List[str]):
actions, valids = self.projection_f(text_actions)
next_obs, rewards, dones, infos = self.envs.step(actions)
next_observations = {
'text': None,
'image': next_obs,
'anchor': None,
}
for i, info in enumerate(infos):
info['is_action_valid'] = to_numpy(valids[i])
rewards = to_numpy(rewards)
dones = to_numpy(dones)
return next_observations, rewards, dones, infos
def close(self) -> None:
self.envs.close()
def success_evaluator(self, *args, **kwargs) -> Dict[str, np.ndarray]:
total_infos = kwargs['total_infos']
total_batch_list = kwargs['total_batch_list']
batch_size = len(total_batch_list)
success = defaultdict(list)
for bs in range(batch_size):
self._process_batch(bs, total_batch_list, total_infos, success)
assert len(success['success_rate']) == batch_size
return {key: np.array(value) for key, value in success.items()}
def _process_batch(self, batch_idx, total_batch_list, total_infos, success):
for i in reversed(range(len(total_batch_list[batch_idx]))):
batch_item = total_batch_list[batch_idx][i]
if batch_item['active_masks']:
info = total_infos[batch_idx][i]
won_value = float(info['won'])
success['success_rate'].append(won_value)
return
skillopt/envs/alfworld/vendor/env_manager.py
Vendored
+139
View File
@@ -0,0 +1,139 @@
# Vendored from SkillRL (Apache-2.0 License)
# Original: agent_system/environments/env_manager.py
# Trimmed to only include AlfWorldEnvironmentManager and its helpers.
from typing import List, Dict, Any
from collections import defaultdict
import numpy as np
from skillopt.envs.alfworld.vendor.env_base import EnvironmentManagerBase, to_numpy
from skillopt.envs.alfworld.vendor.alfworld_prompts import (
ALFWORLD_TEMPLATE,
ALFWORLD_TEMPLATE_NO_HIS,
ALFWORLD_TEMPLATE_WITH_MEMORY,
)
from skillopt.envs.alfworld.vendor.memory import SimpleMemory
def parse_gamefile(infos):
gamefile = []
for info in infos:
if 'extra.gamefile' in info:
gamefile.append(info['extra.gamefile'])
else:
gamefile.append(None)
return gamefile
def set_gamefile(infos, gamefile):
for i in range(len(infos)):
if 'extra.gamefile' in infos[i]:
infos[i]['extra.gamefile'] = gamefile[i]
else:
infos[i]['extra.gamefile'] = None
return infos
class AlfWorldEnvironmentManager(EnvironmentManagerBase):
"""Manages parallel ALFWorld environments with observation templating."""
def __init__(self, envs, projection_f, config):
self.memory = SimpleMemory()
self.retrieval_memory = None
super().__init__(envs, projection_f, config)
def reset(self, kwargs):
text_obs, image_obs, infos = self.envs.reset()
self.gamefile = parse_gamefile(infos)
self.memory.reset(batch_size=len(text_obs))
self.tasks = []
self.pre_text_obs = text_obs
self.extract_task(text_obs)
full_text_obs = self.build_text_obs(text_obs, self.envs.get_admissible_commands, init=True)
return {'text': full_text_obs, 'image': image_obs, 'anchor': text_obs}, infos
def step(self, text_actions: List[str]):
actions, valids = self.projection_f(text_actions, self.envs.get_admissible_commands)
text_obs, image_obs, rewards, dones, infos = self.envs.step(actions)
self.memory.store({'text_obs': self.pre_text_obs, 'action': actions})
self.pre_text_obs = text_obs
full_text_obs = self.build_text_obs(text_obs, self.envs.get_admissible_commands)
if infos[0].get("extra.gamefile") is None:
infos = set_gamefile(infos, self.gamefile)
for i, info in enumerate(infos):
info['is_action_valid'] = to_numpy(valids[i])
next_observations = {'text': full_text_obs, 'image': image_obs, 'anchor': text_obs}
rewards = to_numpy(rewards)
dones = to_numpy(dones)
return next_observations, rewards, dones, infos
def extract_task(self, text_obs: List[str]):
for obs in text_obs:
task_start = obs.find('Your task is to: ')
if task_start != -1:
self.tasks.append(obs[task_start + len('Your task is to: '):].strip())
else:
raise ValueError("Task description not found in text observation.")
def build_text_obs(self, text_obs: List[str], admissible_actions: List[List[str]], init: bool = False) -> List[str]:
postprocess_text_obs = []
if not init and self.config.env.history_length > 0:
memory_contexts, valid_lens = self.memory.fetch(
self.config.env.history_length,
obs_key="text_obs",
action_key="action",
)
for i in range(len(text_obs)):
reformatted_admissible_actions = "\n ".join(
f"'{s}'" for s in admissible_actions[i] if s != 'help'
)
if init or self.config.env.history_length <= 0:
obs = ALFWORLD_TEMPLATE_NO_HIS.format(
current_observation=text_obs[i],
admissible_actions=reformatted_admissible_actions,
)
else:
obs = ALFWORLD_TEMPLATE.format(
task_description=self.tasks[i],
step_count=len(self.memory[i]),
history_length=valid_lens[i],
action_history=memory_contexts[i],
current_step=len(self.memory[i]) + 1,
current_observation=text_obs[i],
admissible_actions=reformatted_admissible_actions,
)
postprocess_text_obs.append(obs)
return postprocess_text_obs
def _process_batch(self, batch_idx, total_batch_list, total_infos, success):
for i in reversed(range(len(total_batch_list[batch_idx]))):
batch_item = total_batch_list[batch_idx][i]
if batch_item['active_masks']:
info = total_infos[batch_idx][i]
won_value = float(info['won'])
success['success_rate'].append(won_value)
gamefile = info.get("extra.gamefile")
if gamefile:
self._process_gamefile(gamefile, won_value, success)
return
def _process_gamefile(self, gamefile, won_value, success):
tasks = [
"pick_and_place",
"pick_two_obj_and_place",
"look_at_obj_in_light",
"pick_heat_then_place_in_recep",
"pick_cool_then_place_in_recep",
"pick_clean_then_place_in_recep",
]
for task in tasks:
if task in gamefile:
success[f"{task}_success_rate"].append(won_value)
break
skillopt/envs/alfworld/vendor/memory.py
Vendored
+87
View File
@@ -0,0 +1,87 @@
# Vendored from SkillRL (Apache-2.0 License)
# Original: agent_system/memory/base.py + agent_system/memory/memory.py
# Merged into a single file for simplicity.
from abc import ABC, abstractmethod
from typing import List, Dict, Any, Tuple
class BaseMemory(ABC):
"""Base class for memory management."""
@abstractmethod
def __len__(self):
pass
@abstractmethod
def __getitem__(self, idx: int):
pass
@abstractmethod
def reset(self, batch_size: int):
pass
@abstractmethod
def store(self, record: Dict[str, List[Any]]):
pass
@abstractmethod
def fetch(self, step: int):
pass
class SimpleMemory(BaseMemory):
"""Per-environment history buffer for storing observations and actions."""
def __init__(self):
self._data = None
self.keys = None
self.batch_size = 0
def __len__(self):
return len(self._data)
def __getitem__(self, idx):
return self._data[idx]
def reset(self, batch_size: int):
if self._data is not None:
self._data.clear()
self._data = [[] for _ in range(batch_size)]
self.batch_size = batch_size
self.keys = None
def store(self, record: Dict[str, List[Any]]):
if self.keys is None:
self.keys = list(record.keys())
assert self.keys == list(record.keys())
for env_idx in range(self.batch_size):
self._data[env_idx].append({k: record[k][env_idx] for k in self.keys})
def fetch(
self,
history_length: int,
obs_key: str = "text_obs",
action_key: str = "action",
) -> Tuple[List[str], List[int]]:
memory_contexts, valid_lengths = [], []
for env_idx in range(self.batch_size):
recent = self._data[env_idx][-history_length:]
valid_len = len(recent)
start_idx = len(self._data[env_idx]) - valid_len
lines = []
for j, rec in enumerate(recent):
step_num = start_idx + j + 1
act = rec[action_key]
obs = rec[obs_key]
lines.append(
f"[Observation {step_num}: '{obs}', Action {step_num}: '{act}']"
)
memory_contexts.append("\n".join(lines))
valid_lengths.append(valid_len)
return memory_contexts, valid_lengths
skillopt/envs/babyvision/__init__.py
+1
View File
@@ -0,0 +1 @@
"""BabyVision environment package for ReflACT."""
skillopt/envs/babyvision/adapter.py
+267
View File
@@ -0,0 +1,267 @@
"""BabyVision environment adapter for ReflACT."""
from __future__ import annotations
import json
import os
from skillopt.gradient.deep_probe import generate_deep_probe_instruction
from skillopt.datasets.base import BatchSpec
from skillopt.gradient.reflect import run_minibatch_reflect
from skillopt.envs.base import EnvAdapter
from skillopt.envs.babyvision.dataloader import BabyVisionDataLoader
from skillopt.envs.babyvision.rollout import run_batch
from skillopt.model import get_student_backend
class BabyVisionAdapter(EnvAdapter):
"""BabyVision adapter."""
def build_reference_text(self, item: dict) -> str:
cot = str(item.get("cot") or "").strip()
if not cot:
return ""
return f"## Reference CoT\n{cot}"
def get_reference_metadata(self, item: dict) -> dict:
cot = str(item.get("cot") or "").strip()
if not cot:
return {"fields": [], "preview": ""}
return {
"fields": ["cot"],
"preview": cot[:400],
}
def __init__(
self,
split_dir: str = "",
data_path: str = "",
split_mode: str = "ratio",
split_ratio: str = "2:1:7",
split_seed: int = 42,
split_output_dir: str = "",
max_turns: int = 1,
workers: int = 32,
analyst_workers: int = 16,
failure_only: bool = False,
minibatch_size: int = 8,
edit_budget: int = 4,
seed: int = 42,
limit: int = 0,
image_detail: str = "auto",
judge_model: str = "gpt-5.4",
judge_max_completion_tokens: int = 256,
judge_retries: int = 5,
use_deep_reflect: bool = False,
deep_reflect_failures: int = 4,
deep_reflect_successes: int = 2,
) -> None:
self.max_turns = max_turns
self.workers = workers
self.analyst_workers = analyst_workers
self.failure_only = failure_only
self.minibatch_size = minibatch_size
self.edit_budget = edit_budget
self.image_detail = image_detail
self.judge_model = judge_model
self.judge_max_completion_tokens = judge_max_completion_tokens
self.judge_retries = judge_retries
self.use_deep_reflect = use_deep_reflect
self.deep_reflect_failures = deep_reflect_failures
self.deep_reflect_successes = deep_reflect_successes
self.dataloader = BabyVisionDataLoader(
split_dir=split_dir,
data_path=data_path,
split_mode=split_mode,
split_ratio=split_ratio,
split_seed=split_seed,
split_output_dir=split_output_dir,
seed=seed,
limit=limit,
)
def setup(self, cfg: dict) -> None:
super().setup(cfg)
self.dataloader.setup(cfg)
def get_dataloader(self):
return self.dataloader
def build_env_from_batch(self, batch: BatchSpec, **kwargs):
return list(batch.payload or [])
def build_train_env(self, batch_size: int, seed: int, **kwargs):
batch = self.dataloader.build_train_batch(batch_size=batch_size, seed=seed, **kwargs)
return self.build_env_from_batch(batch, **kwargs)
def build_eval_env(self, env_num: int, split: str, seed: int, **kwargs):
batch = self.dataloader.build_eval_batch(env_num=env_num, split=split, seed=seed, **kwargs)
return self.build_env_from_batch(batch, **kwargs)
def rollout(
self,
env_manager,
skill_content: str,
out_dir: str,
**kwargs,
) -> list[dict]:
items: list[dict] = env_manager
return run_batch(
items=items,
out_root=out_dir,
skill_content=skill_content,
max_turns=self.max_turns,
workers=self.workers,
image_detail=self.image_detail,
judge_model=self.judge_model,
judge_max_completion_tokens=self.judge_max_completion_tokens,
judge_retries=self.judge_retries,
diagnostic_mode=kwargs.get("diagnostic_mode", False),
diagnostic_instruction=kwargs.get("diagnostic_instruction", ""),
diagnostic_trace_context_by_id=kwargs.get("diagnostic_trace_context_by_id"),
)
def reflect(
self,
results: list[dict],
skill_content: str,
out_dir: str,
**kwargs,
) -> list[dict | None]:
prediction_dir = kwargs.get("prediction_dir", os.path.join(out_dir, "predictions"))
patches_dir = kwargs.get("patches_dir", os.path.join(out_dir, "patches"))
random_seed = kwargs.get("random_seed")
step_buffer_context = kwargs.get("step_buffer_context", "")
meta_skill_context = kwargs.get("meta_skill_context", "")
return run_minibatch_reflect(
results=results,
skill_content=skill_content,
prediction_dir=prediction_dir,
patches_dir=patches_dir,
workers=self.analyst_workers,
failure_only=self.failure_only,
minibatch_size=self.minibatch_size,
edit_budget=self.edit_budget,
random_seed=random_seed,
error_system=self.get_error_minibatch_prompt(),
success_system=self.get_success_minibatch_prompt(),
step_buffer_context=step_buffer_context,
meta_skill_context=meta_skill_context,
update_mode=getattr(self, "_cfg", {}).get("skill_update_mode", "patch"),
)
def deep_reflect(
self,
results: list[dict],
skill_content: str,
out_dir: str,
**kwargs,
) -> list[dict | None]:
if not self.use_deep_reflect:
return []
env_manager = kwargs.get("env_manager")
prediction_dir = kwargs.get("prediction_dir", os.path.join(out_dir, "predictions"))
random_seed = kwargs.get("random_seed")
step_buffer_context = kwargs.get("step_buffer_context", "")
meta_skill_context = kwargs.get("meta_skill_context", "")
codex_backend = get_student_backend() == "codex_exec"
selected_items = self.select_representative_items(
results,
env_manager if isinstance(env_manager, list) else None,
n_failures=self.deep_reflect_failures,
n_successes=self.deep_reflect_successes,
seed=random_seed,
)
if not selected_items:
return []
selected_ids = {str(item["id"]) for item in selected_items}
selected_results = [row for row in results if str(row.get("id")) in selected_ids]
selected_examples = self.attach_reference_context(selected_results, selected_items)
if codex_backend:
selected_examples = self.attach_codex_probe_context(selected_examples, prediction_dir)
selected_metadata = []
cot_count = 0
for item in selected_items:
meta = self.get_reference_metadata(item)
if meta["fields"]:
cot_count += 1
selected_metadata.append({
"id": str(item["id"]),
"task_type": str(item.get("subtype") or item.get("task_type") or "babyvision"),
"reference_fields": meta["fields"],
"reference_preview": meta["preview"],
})
deep_dir = os.path.join(out_dir, "deep_reflect")
rollout_dir = os.path.join(deep_dir, "rollout")
patches_dir = os.path.join(deep_dir, "patches")
os.makedirs(deep_dir, exist_ok=True)
print(
f" [2b/6 DEEP REFLECT setup] selected={len(selected_items)} "
f"reference_fields=cot({cot_count}/{len(selected_items)})"
)
probe = generate_deep_probe_instruction(
skill_content=skill_content,
items=selected_examples,
prediction_dir=prediction_dir,
system_prompt=self.get_codex_deep_probe_prompt() if codex_backend else self.get_deep_probe_prompt(),
step_buffer_context=step_buffer_context,
meta_skill_context=meta_skill_context,
)
if not probe:
return []
diagnostic_trace_context_by_id = None
if codex_backend:
selected_items, diagnostic_trace_context_by_id, probe = self.resolve_codex_probe_target(
selected_items=selected_items,
selected_examples=selected_examples,
prediction_dir=prediction_dir,
probe=probe,
)
probe_record = {
**probe,
"reference_summary": {
"selected_count": len(selected_items),
"field_counts": {
"cot": cot_count,
},
},
"selected_examples": selected_metadata,
}
with open(os.path.join(deep_dir, "probe.json"), "w", encoding="utf-8") as f:
json.dump(probe_record, f, ensure_ascii=False, indent=2)
deep_results = run_batch(
items=selected_items,
out_root=rollout_dir,
skill_content=skill_content,
max_turns=self.max_turns,
workers=min(self.workers, max(len(selected_items), 1)),
image_detail=self.image_detail,
judge_model=self.judge_model,
judge_max_completion_tokens=self.judge_max_completion_tokens,
judge_retries=self.judge_retries,
diagnostic_mode=True,
diagnostic_instruction=probe["probe_instruction"],
diagnostic_trace_context_by_id=diagnostic_trace_context_by_id,
)
deep_results = self.attach_reference_context(deep_results, selected_items)
return run_minibatch_reflect(
results=deep_results,
skill_content=skill_content,
prediction_dir=os.path.join(rollout_dir, "predictions"),
patches_dir=patches_dir,
workers=self.analyst_workers,
failure_only=self.failure_only,
minibatch_size=self.minibatch_size,
edit_budget=self.edit_budget,
random_seed=random_seed,
error_system=self.get_error_minibatch_prompt(),
success_system=self.get_success_minibatch_prompt(),
step_buffer_context=step_buffer_context,
meta_skill_context=meta_skill_context,
update_mode=getattr(self, "_cfg", {}).get("skill_update_mode", "patch"),
)
def get_task_types(self) -> list[str]:
return self.dataloader.get_task_types()
skillopt/envs/babyvision/dataloader.py
+214
View File
@@ -0,0 +1,214 @@
"""BabyVision task dataloader."""
from __future__ import annotations
import json
import os
from typing import Any
from skillopt.datasets.base import SplitDataLoader
# ── Raw data loading utilities (for preprocessing / standalone eval) ─────
_CHOICE_LABELS = ["A", "B", "C", "D", "E", "F", "G"]
def _iter_jsonl(path: str) -> list[dict]:
items: list[dict] = []
with open(path, encoding="utf-8") as f:
for line in f:
line = line.strip()
if not line:
continue
items.append(json.loads(line))
return items
def _normalize_ans_type(raw: Any, options: list[dict], choice_answer: Any) -> str:
text = str(raw or "").strip().lower()
if text in {"choice", "multiple_choice", "mcq", "option"}:
return "choice"
if text in {"blank", "open", "open_ended", "fill_blank", "short_answer"}:
return "blank"
if options or choice_answer not in (None, "", []):
return "choice"
return "blank"
def _coerce_options(raw: Any) -> list[dict]:
options: list[dict] = []
if isinstance(raw, list):
for idx, item in enumerate(raw):
if isinstance(item, dict):
text = str(item.get("text") or item.get("content") or item.get("option") or "").strip()
label = str(item.get("label") or _CHOICE_LABELS[idx]).strip()
else:
text = str(item).strip()
label = _CHOICE_LABELS[idx]
if text:
options.append({"label": label, "text": text})
elif isinstance(raw, dict):
for idx, (key, value) in enumerate(raw.items()):
text = str(value).strip()
if text:
options.append({"label": str(key).strip() or _CHOICE_LABELS[idx], "text": text})
return options
def _normalize_choice_answer(choice_answer: Any, options: list[dict]) -> dict[str, str]:
if not options:
return {"label": "", "text": ""}
if isinstance(choice_answer, dict):
label = str(choice_answer.get("label") or "").strip().upper()
text = str(choice_answer.get("text") or "").strip()
for option in options:
if label and option["label"].strip().upper() == label:
return {"label": option["label"], "text": option["text"]}
if text and option["text"] == text:
return {"label": option["label"], "text": option["text"]}
if isinstance(choice_answer, int):
idx = choice_answer
if 0 <= idx < len(options):
return dict(options[idx])
if 1 <= idx <= len(options):
return dict(options[idx - 1])
text = str(choice_answer or "").strip()
label = text.upper().rstrip(".):")
for option in options:
if option["label"].strip().upper() == label:
return dict(option)
if option["text"] == text:
return dict(option)
return {"label": "", "text": ""}
def _coerce_blank_answers(raw: Any) -> list[str]:
if isinstance(raw, list):
return [str(item).strip() for item in raw if str(item).strip()]
if raw is None:
return []
text = str(raw).strip()
return [text] if text else []
def load_items(data_path: str) -> list[dict]:
"""Load and normalise BabyVision items from a directory or JSONL file."""
if not data_path:
raise ValueError("BabyVision requires data_path pointing to a local dataset directory or meta_data.jsonl.")
if os.path.isdir(data_path):
meta_path = os.path.join(data_path, "meta_data.jsonl")
image_root = os.path.join(data_path, "images")
else:
meta_path = data_path
image_root = os.path.join(os.path.dirname(data_path), "images")
if not os.path.exists(meta_path):
raise ValueError(
"BabyVision expected a meta_data.jsonl file. "
f"Could not find: {meta_path}"
)
raw_items = _iter_jsonl(meta_path)
items: list[dict] = []
for idx, raw in enumerate(raw_items):
options = _coerce_options(raw.get("options") or raw.get("choices") or raw.get("choiceOptions"))
ans_type = _normalize_ans_type(raw.get("ansType"), options, raw.get("choiceAns"))
correct_choice = _normalize_choice_answer(raw.get("choiceAns"), options)
blank_answers = _coerce_blank_answers(raw.get("blankAns"))
image_name = str(
raw.get("image")
or raw.get("image_path")
or raw.get("image_file")
or raw.get("img")
or ""
).strip()
if not image_name:
continue
image_path = image_name if os.path.isabs(image_name) else os.path.join(image_root, image_name)
if not os.path.exists(image_path):
alt = os.path.join(os.path.dirname(meta_path), image_name)
if os.path.exists(alt):
image_path = alt
else:
continue
task_id = str(raw.get("taskId") or raw.get("id") or idx + 1)
task_type = str(raw.get("type") or raw.get("taskType") or "unknown").strip() or "unknown"
subtype = str(raw.get("subtype") or raw.get("subType") or task_type).strip() or task_type
question = str(raw.get("question") or raw.get("query") or "").strip()
if not question:
continue
if ans_type == "choice" and not correct_choice["label"]:
continue
if ans_type != "choice" and not blank_answers:
continue
items.append({
"id": task_id,
"task_type": task_type,
"subtype": subtype,
"question": question,
"image_path": os.path.abspath(image_path),
"ans_type": ans_type,
"choices": options,
"correct_choice": correct_choice,
"blank_answers": blank_answers,
"cot": str(raw.get("coT") or raw.get("cot") or "").strip(),
"source_path": os.path.abspath(meta_path),
})
if not items:
raise ValueError(f"No valid BabyVision items loaded from {data_path}")
return items
# ── Dataloader ───────────────────────────────────────────────────────────
class BabyVisionDataLoader(SplitDataLoader):
"""BabyVision dataloader."""
def __init__(
self,
split_dir: str = "",
data_path: str = "",
split_mode: str = "ratio",
split_ratio: str = "2:1:7",
split_seed: int = 42,
split_output_dir: str = "",
seed: int = 42,
limit: int = 0,
**kwargs,
) -> None:
super().__init__(
split_dir=split_dir,
data_path=data_path,
split_mode=split_mode,
split_ratio=split_ratio,
split_seed=split_seed,
split_output_dir=split_output_dir,
seed=seed,
limit=limit,
)
self._task_types: list[str] = []
def load_raw_items(self, data_path: str) -> list[dict]:
return load_items(data_path)
def setup(self, cfg: dict) -> None:
super().setup(cfg)
all_items = self.train_items + self.val_items + self.test_items
task_types = {
item.get("subtype") or item.get("task_type") or "unknown"
for item in all_items
}
self._task_types = sorted(task_types)
def get_task_types(self) -> list[str]:
return list(self._task_types)
skillopt/envs/babyvision/evaluator.py
+160
View File
@@ -0,0 +1,160 @@
"""BabyVision evaluation helpers using the official-style LLM judge."""
from __future__ import annotations
import re
import string
import regex
from skillopt.model import chat_with_deployment
from skillopt.prompts import load_prompt
_EVAL_MODE = "babyvision_judge_v2_official_style"
def normalize_text(text: str) -> str:
text = str(text).strip().lower()
text = "".join(ch for ch in text if ch not in string.punctuation)
return " ".join(text.split())
def extract_boxed_answer(text: str | None) -> str | None:
"""Extract the final answer using the official BabyVision rule."""
if text is None:
return None
pattern = r'\\boxed\{((?:[^{}]|{(?:[^{}]|{.*})*})*)\}'
matches = regex.findall(pattern, text)
if matches:
return matches[-1]
pattern_alt = r'<\|begin_of_box\|>(.*?)<\|end_of_box\|>'
matches_alt = regex.findall(pattern_alt, text)
if matches_alt:
return matches_alt[-1].strip()
return None
def _token_f1(prediction: str, gold: str) -> float:
pred_tokens = normalize_text(prediction).split()
gold_tokens = normalize_text(gold).split()
if not pred_tokens and not gold_tokens:
return 1.0
if not pred_tokens or not gold_tokens:
return 0.0
pred_set = {}
gold_set = {}
for tok in pred_tokens:
pred_set[tok] = pred_set.get(tok, 0) + 1
for tok in gold_tokens:
gold_set[tok] = gold_set.get(tok, 0) + 1
common = 0
for tok, count in pred_set.items():
common += min(count, gold_set.get(tok, 0))
if common == 0:
return 0.0
precision = common / len(pred_tokens)
recall = common / len(gold_tokens)
return 2 * precision * recall / (precision + recall)
def _format_choices(choices: list[dict]) -> str:
return "\n".join(f"{choice['label']}. {choice['text']}" for choice in choices)
def _judge_answer(
*,
item: dict,
prediction_text: str,
extracted_answer: str,
judge_model: str,
max_completion_tokens: int,
retries: int,
) -> dict:
if item["ans_type"] == "choice":
ground_truth = str(item["correct_choice"]["label"])
else:
if len(item["blank_answers"]) == 1:
ground_truth = item["blank_answers"][0]
else:
ground_truth = " | ".join(item["blank_answers"])
question = str(item["question"])
if item["ans_type"] == "choice" and item.get("choices"):
question = f"{question}\nChoices:\n{_format_choices(item['choices'])}"
raw, _ = chat_with_deployment(
deployment=judge_model,
system="You are a careful and strict evaluator.",
user=load_prompt("judge", env="babyvision").format(
question=question,
groundtruth=ground_truth,
modeloutput=extracted_answer,
),
max_completion_tokens=max_completion_tokens,
retries=retries,
stage="babyvision_judge",
)
judge_response_clean = str(raw).strip().lower()
if "true" in judge_response_clean:
correct = True
elif "false" in judge_response_clean:
correct = False
else:
correct = False
return {
"raw": raw,
"correct": correct,
"reason": judge_response_clean,
"matched_gold": ground_truth if correct else "",
}
def evaluate_item(
*,
item: dict,
prediction_text: str,
judge_model: str,
max_completion_tokens: int = 256,
retries: int = 5,
) -> dict:
answer = extract_boxed_answer(prediction_text)
judge = _judge_answer(
item=item,
prediction_text=prediction_text,
extracted_answer=answer,
judge_model=judge_model,
max_completion_tokens=max_completion_tokens,
retries=retries,
)
hard = 1.0 if judge["correct"] else 0.0
result = {
"evaluation_mode": _EVAL_MODE,
"predicted_answer": answer,
"em": hard,
"f1": hard,
"sub_em": hard,
"judge_model": judge_model,
"judge_raw": judge["raw"],
"judge_reason": judge["reason"],
"matched_gold": judge["matched_gold"],
}
if item["ans_type"] == "choice":
result["predicted_label"] = str(answer or "").strip().upper().rstrip(".):")
result["predicted_text"] = ""
result["correct_label"] = str(item["correct_choice"].get("label") or "")
result["correct_text"] = str(item["correct_choice"].get("text") or "")
else:
result["gold_answers"] = list(item["blank_answers"])
best_f1 = 0.0
for gold in item["blank_answers"]:
best_f1 = max(best_f1, _token_f1(str(answer or ""), gold))
result["string_f1"] = best_f1
return result
def evaluation_mode() -> str:
return _EVAL_MODE
skillopt/envs/babyvision/prompts/analyst_error.md
+36
View File
@@ -0,0 +1,36 @@
You are an expert failure-analysis agent for child-level visual reasoning tasks.
You will be given MULTIPLE failed BabyVision trajectories from a minibatch and the current skill document.
Each trajectory includes the text prompt, the model answer, and the evaluation result.
You do not have direct access to raw pixel content during reflection, so focus on general reasoning,
option-selection, and visual-question-answering behaviors that can be improved through prompting.
## Failure Type Categories
- **visual_detail_miss**: the agent likely overlooked a salient visual attribute, relation, count, or object state
- **option_mismatch**: the agent selected the wrong option despite relevant evidence likely being present
- **instruction_slip**: the agent ignored output format or answered too vaguely
- **answer_granularity**: the agent gave an answer that was too broad, too narrow, or mismatched the expected specificity
- **other**: none of the above
## Rules
1. Focus on patterns recurring across the minibatch.
2. Prefer reusable behaviors for inspecting images and grounding answers in visible evidence.
3. Do not memorize dataset-specific answers.
4. Only patch gaps not already covered by the current skill.
Respond ONLY with a valid JSON object:
{
"batch_size": <number>,
"failure_summary": [
{"failure_type": "<type>", "count": <int>, "description": "<one-line>"}
],
"patch": {
"reasoning": "<why these edits address the common failures>",
"edits": [
{"op": "append", "content": "<markdown>"},
{"op": "insert_after", "target": "<heading/text>", "content": "<markdown>"},
{"op": "replace", "target": "<old text>", "content": "<new text>"},
{"op": "delete", "target": "<exact text to remove>"}
]
}
}
skillopt/envs/babyvision/prompts/analyst_success.md
+25
View File
@@ -0,0 +1,25 @@
You are an expert success-pattern analyst for child-level visual reasoning tasks.
You will be given MULTIPLE successful BabyVision trajectories from a minibatch and the current skill document.
Identify generalizable behavior patterns that help the agent inspect the image carefully and answer at the right level of specificity.
## Rules
- Focus on broadly useful visual QA behaviors.
- Prefer patterns about systematic image inspection, comparing options, and concise grounded answers.
- Do not add dataset-specific facts.
- "edits" may be empty if the skill already captures the useful patterns.
Respond ONLY with a valid JSON object:
{
"batch_size": <number>,
"success_patterns": ["<pattern 1>", "<pattern 2>"],
"patch": {
"reasoning": "<why these patterns matter>",
"edits": [
{"op": "append", "content": "<markdown>"},
{"op": "insert_after", "target": "<heading/text>", "content": "<markdown>"},
{"op": "replace", "target": "<old text>", "content": "<new text>"},
{"op": "delete", "target": "<exact text to remove>"}
]
}
}
skillopt/envs/babyvision/prompts/deep_probe.md
+25
View File
@@ -0,0 +1,25 @@
You are an expert diagnostic-probe designer for BabyVision-style visual reasoning tasks.
You will be shown representative trajectories, the current student skill, and the student's original prompt context.
Design one SMALL diagnostic instruction that exposes the student's intermediate visual judgment without materially changing the original scaffold.
## Hard Constraints
1. Do NOT substantially change the original scaffold.
2. Do NOT prescribe a new step-by-step solving method.
3. You MAY ask for a short structured list of a few intermediate conclusions, candidate cues, or counted units, as long as it stays close to the original scaffold.
4. Do NOT ask for exhaustive listing of all cells, all objects, or a full chain-of-thought.
5. Ask only for a short readout that reveals the student's current latent state.
6. Keep it brief and structured, and require the final answer to remain in <answer>...</answer>.
## Good Probe Targets
- top answer and runner-up
- decisive visual cue
- suspicious region or compared objects
- counting unit or formatting interpretation
- 2-4 short intermediate conclusions that directly support the final answer
Respond ONLY with a valid JSON object:
{
"reasoning": "<why this probe is informative>",
"probe_instruction": "<the exact instruction text to append to the student prompt>"
}
skillopt/envs/babyvision/prompts/judge.md
+35
View File
@@ -0,0 +1,35 @@
You are a careful and strict evaluator. You will be given:
1. **Question**
2. **Ground Truth Answer** (correct answer)
3. **Model Output** (answer from another model)
**Your goal:** Determine if the Model Output **accurately matches** the Ground Truth Answer in meaning.
* Matching means: the facts, entities, and key details are equivalent, even if phrasing differs.
* Not matching means: the Model Output is wrong, incomplete, contains extra incorrect facts, or changes the meaning.
**Process (internal reasoning):**
1. Read and understand the Question, Ground Truth Answer, and Model Output.
2. Ignore small wording differences, formatting, or synonyms.
3. If all factual content matches, conclude `1`. Otherwise, conclude `0`.
**Important:**
* Think through your decision step-by-step **internally** before responding.
* In your final output, return **only** True or False, with no extra text or explanation.
**Output format:**
True
or
False
**Input:**
Question: {question},
Ground Truth Answer: {groundtruth},
Model Output: {modeloutput}
skillopt/envs/babyvision/prompts/rollout_system.md
+13
View File
@@ -0,0 +1,13 @@
You are an expert visual reasoning agent solving child-level image understanding tasks.
{skill_section}## Task Format
You will receive one image and one question about it.
Inspect the image carefully before answering. Ground the answer in visible evidence.
## Answer Format
Think step by step, then provide your final answer in \boxed{{Answer}} format.
- For multiple-choice questions, output only the single choice label, such as \boxed{{A}}.
- For open questions, output only a short final answer inside \boxed{{...}}.
Example:
\boxed{{B}}
skillopt/envs/babyvision/reflect.py
+4
View File
@@ -0,0 +1,4 @@
"""BabyVision Reflect stage.
Prompts are now loaded from .md files by the base adapter.
"""
skillopt/envs/babyvision/rollout.py
+483
View File
@@ -0,0 +1,483 @@
"""BabyVision rollout — multimodal visual QA with image input."""
from __future__ import annotations
import base64
import json
import mimetypes
import os
from concurrent.futures import ThreadPoolExecutor, as_completed
from skillopt.envs.babyvision.evaluator import evaluate_item, evaluation_mode, extract_boxed_answer
from skillopt.model import chat_student_messages, get_student_backend, is_student_exec_backend
from skillopt.model.codex_harness import prepare_workspace, render_skill_md, run_student_exec
from skillopt.prompts import load_prompt
def _build_system(skill_content: str) -> str:
if skill_content.strip():
skill_section = f"## Skill\n{skill_content.strip()}\n\n"
else:
skill_section = ""
return load_prompt("rollout_system", env="babyvision").format(skill_section=skill_section)
def _format_choices(choices: list[dict]) -> str:
return "\n".join(f"{choice['label']}. {choice['text']}" for choice in choices)
def _build_user_text(
item: dict,
*,
diagnostic_mode: bool = False,
diagnostic_instruction: str = "",
diagnostic_trace_context: str = "",
) -> str:
parts = []
if diagnostic_trace_context.strip():
parts.append(
"## Previous Codex Trace Snapshot\n"
"This is a partial transcript from an earlier attempt. Use it as your current reasoning context.\n\n"
f"{diagnostic_trace_context.strip()}"
)
parts.append(f"## Question\n{item['question']}")
if item["ans_type"] == "choice":
parts.append(f"## Choices\n{_format_choices(item['choices'])}")
parts.append("Answer using the single correct option label in \\boxed{...}.")
else:
parts.append("Answer with a short phrase in \\boxed{...}.")
if diagnostic_mode and diagnostic_instruction.strip():
parts.append(f"## Training Readout\n{diagnostic_instruction.strip()}")
return "\n\n".join(parts)
def _image_to_data_uri(path: str) -> str:
mime = mimetypes.guess_type(path)[0] or "image/png"
with open(path, "rb") as f:
encoded = base64.b64encode(f.read()).decode("ascii")
return f"data:{mime};base64,{encoded}"
def _build_messages(
item: dict,
skill_content: str,
image_detail: str,
*,
diagnostic_mode: bool = False,
diagnostic_instruction: str = "",
diagnostic_trace_context: str = "",
) -> tuple[list[dict], str, str]:
system = _build_system(skill_content)
user_text = _build_user_text(
item,
diagnostic_mode=diagnostic_mode,
diagnostic_instruction=diagnostic_instruction,
diagnostic_trace_context=diagnostic_trace_context,
)
image_url = {
"url": _image_to_data_uri(item["image_path"]),
}
if image_detail and image_detail != "auto":
image_url["detail"] = image_detail
messages = [
{"role": "system", "content": system},
{
"role": "user",
"content": [
{"type": "text", "text": user_text},
{"type": "image_url", "image_url": image_url},
],
},
]
return messages, system, user_text
def _build_codex_skill(skill_content: str) -> str:
return render_skill_md(
skill_content,
description="Dynamic ReflACT skill for solving the current BabyVision visual reasoning question.",
preamble=(
"Use this skill when answering the current visual reasoning question.\n"
"Inspect the attached image carefully and return the final answer in \\boxed{...}."
),
)
def _run_codex_once(
*,
pred_dir: str,
item: dict,
skill_content: str,
model: str,
timeout: int,
image_detail: str,
diagnostic_mode: bool = False,
diagnostic_instruction: str = "",
diagnostic_trace_context: str = "",
previous_response: str = "",
) -> tuple[str, str, str, str]:
user_text = _build_user_text(
item,
diagnostic_mode=diagnostic_mode,
diagnostic_instruction=diagnostic_instruction,
diagnostic_trace_context=diagnostic_trace_context,
)
task_parts = [user_text]
if previous_response:
task_parts.append(
"## Previous Attempt\n"
f"{previous_response}\n\n"
"Review the same image and question carefully. If needed, correct the answer."
)
task_text = "\n\n".join(task_parts)
skill_md = _build_codex_skill(skill_content)
work_dir = os.path.join(pred_dir, "codex_exec")
prepare_workspace(
work_dir=work_dir,
skill_md=skill_md,
task_text=task_text,
images=[item["image_path"]],
)
prompt = (
"Use the `skillopt-student` skill available in this workspace.\n"
"Read `task.md`, inspect the attached image, and answer the question.\n"
"Return the final answer in \\boxed{...}."
)
final_message, raw = run_student_exec(
work_dir=work_dir,
prompt=prompt,
model=model,
timeout=timeout,
images=[item["image_path"]],
)
return final_message or raw, raw, skill_md, task_text
def process_one(
item: dict,
out_root: str,
skill_content: str,
*,
max_turns: int = 1,
image_detail: str = "auto",
judge_model: str = "gpt-5.4",
judge_max_completion_tokens: int = 256,
judge_retries: int = 5,
diagnostic_mode: bool = False,
diagnostic_instruction: str = "",
diagnostic_trace_context: str = "",
) -> dict:
item_id = str(item["id"])
result = {
"id": item_id,
"question": item["question"],
"task_type": item.get("subtype") or item.get("task_type") or "babyvision",
"task_description": item["question"],
"hard": 0,
"soft": 0.0,
"predicted_answer": "",
"predicted_label": "",
"predicted_text": "",
"response": "",
"fail_reason": "",
"agent_ok": False,
"n_turns": 0,
"image_path": item["image_path"],
"ans_type": item["ans_type"],
"evaluation_mode": evaluation_mode(),
"judge_model": judge_model,
}
if item["ans_type"] == "choice":
result["correct_label"] = item["correct_choice"]["label"]
result["correct_text"] = item["correct_choice"]["text"]
else:
result["gold_answers"] = item["blank_answers"]
try:
pred_dir = os.path.join(out_root, "predictions", item_id)
os.makedirs(pred_dir, exist_ok=True)
if is_student_exec_backend():
from skillopt.model import azure_openai as _llm
response = ""
conversation: list[dict] = [
{"role": "user", "content": f"{item['question']}\n\n[image] {os.path.basename(item['image_path'])}"}
]
system_prompt = ""
user_text = ""
for turn in range(max_turns):
response, raw, system_prompt, user_text = _run_codex_once(
pred_dir=pred_dir,
item=item,
skill_content=skill_content,
model=_llm.STUDENT_DEPLOYMENT,
timeout=120,
image_detail=image_detail,
diagnostic_mode=diagnostic_mode if turn == 0 else False,
diagnostic_instruction=diagnostic_instruction if turn == 0 else "",
diagnostic_trace_context=diagnostic_trace_context if turn == 0 else "",
previous_response=response if turn > 0 else "",
)
conversation.append({"type": "message", "turn": turn + 1, "content": response})
if extract_boxed_answer(response) is not None:
break
result["response"] = response
result["agent_ok"] = True
result["n_turns"] = len(conversation) - 1
with open(os.path.join(pred_dir, "student_system_prompt.txt"), "w", encoding="utf-8") as f:
f.write(system_prompt)
with open(os.path.join(pred_dir, "student_user_prompt.txt"), "w", encoding="utf-8") as f:
f.write(user_text)
eval_result = evaluate_item(
item=item,
prediction_text=response,
judge_model=judge_model,
max_completion_tokens=judge_max_completion_tokens,
retries=judge_retries,
)
result["evaluation_mode"] = eval_result["evaluation_mode"]
result["judge_raw"] = eval_result["judge_raw"]
result["judge_reason"] = eval_result["judge_reason"]
result["matched_gold"] = eval_result["matched_gold"]
if item["ans_type"] == "choice":
result["predicted_label"] = eval_result["predicted_label"]
result["predicted_text"] = eval_result["predicted_text"]
result["predicted_answer"] = eval_result["predicted_answer"]
result["hard"] = int(eval_result["em"])
result["soft"] = eval_result["f1"]
if not result["hard"]:
result["fail_reason"] = (
f"judge=0: predicted '{eval_result['predicted_label'] or eval_result['predicted_answer']}' "
f"but expected '{eval_result['correct_label']}' ({eval_result['judge_reason']})"
)
eval_detail = (
f"[EVALUATION RESULT]\n"
f"Question: {item['question']}\n"
f"Predicted label: {eval_result['predicted_label']!r}\n"
f"Predicted text: {eval_result['predicted_text']!r}\n"
f"Correct label: {eval_result['correct_label']!r}\n"
f"Correct text: {eval_result['correct_text']!r}\n"
f"Judge correct: {eval_result['em']}\n"
f"Judge reason: {eval_result['judge_reason']}"
)
else:
result["predicted_answer"] = eval_result["predicted_answer"]
result["hard"] = int(eval_result["em"])
result["soft"] = eval_result["f1"]
if not result["hard"]:
result["fail_reason"] = (
f"judge=0: predicted '{eval_result['predicted_answer']}' "
f"but expected {item['blank_answers']} ({eval_result['judge_reason']})"
)
eval_detail = (
f"[EVALUATION RESULT]\n"
f"Question: {item['question']}\n"
f"Predicted answer: {eval_result['predicted_answer']!r}\n"
f"Gold answers: {item['blank_answers']!r}\n"
f"Judge correct: {eval_result['em']}\n"
f"Judge reason: {eval_result['judge_reason']}\n"
f"String F1: {eval_result.get('string_f1', 0.0):.4f}"
)
conversation.append({"role": "system", "content": eval_detail})
with open(os.path.join(pred_dir, "conversation.json"), "w", encoding="utf-8") as f:
json.dump(conversation, f, ensure_ascii=False, indent=2)
return result
messages, system_prompt, user_text = _build_messages(
item,
skill_content,
image_detail,
diagnostic_mode=diagnostic_mode,
diagnostic_instruction=diagnostic_instruction,
diagnostic_trace_context=diagnostic_trace_context,
)
response = ""
conversation: list[dict] = [
{"role": "user", "content": f"{user_text}\n\n[image] {os.path.basename(item['image_path'])}"}
]
for turn in range(max_turns):
if turn == 0:
resp_text, _ = chat_student_messages(
messages=messages,
max_completion_tokens=768,
retries=5,
stage="rollout",
)
else:
refinement_text = (
f"Your previous answer was:\n{response}\n\n"
"Review the same image and question carefully. "
"If needed, correct your answer. Output the final answer in \\boxed{...}."
)
refinement_messages = [
messages[0],
messages[1],
{"role": "assistant", "content": response},
{"role": "user", "content": refinement_text},
]
resp_text, _ = chat_student_messages(
messages=refinement_messages,
max_completion_tokens=512,
retries=5,
stage="rollout",
)
response = resp_text
conversation.append({"type": "message", "turn": turn + 1, "content": resp_text})
if extract_boxed_answer(resp_text) is not None:
break
result["response"] = response
result["agent_ok"] = True
result["n_turns"] = len(conversation) - 1
with open(os.path.join(pred_dir, "student_system_prompt.txt"), "w", encoding="utf-8") as f:
f.write(system_prompt)
with open(os.path.join(pred_dir, "student_user_prompt.txt"), "w", encoding="utf-8") as f:
f.write(user_text)
eval_result = evaluate_item(
item=item,
prediction_text=response,
judge_model=judge_model,
max_completion_tokens=judge_max_completion_tokens,
retries=judge_retries,
)
result["evaluation_mode"] = eval_result["evaluation_mode"]
result["judge_raw"] = eval_result["judge_raw"]
result["judge_reason"] = eval_result["judge_reason"]
result["matched_gold"] = eval_result["matched_gold"]
if item["ans_type"] == "choice":
result["predicted_label"] = eval_result["predicted_label"]
result["predicted_text"] = eval_result["predicted_text"]
result["predicted_answer"] = eval_result["predicted_answer"]
result["hard"] = int(eval_result["em"])
result["soft"] = eval_result["f1"]
if not result["hard"]:
result["fail_reason"] = (
f"judge=0: predicted '{eval_result['predicted_label'] or eval_result['predicted_answer']}' "
f"but expected '{eval_result['correct_label']}' ({eval_result['judge_reason']})"
)
eval_detail = (
f"[EVALUATION RESULT]\n"
f"Question: {item['question']}\n"
f"Predicted label: {eval_result['predicted_label']!r}\n"
f"Predicted text: {eval_result['predicted_text']!r}\n"
f"Correct label: {eval_result['correct_label']!r}\n"
f"Correct text: {eval_result['correct_text']!r}\n"
f"Judge correct: {eval_result['em']}\n"
f"Judge reason: {eval_result['judge_reason']}"
)
else:
result["predicted_answer"] = eval_result["predicted_answer"]
result["hard"] = int(eval_result["em"])
result["soft"] = eval_result["f1"]
if not result["hard"]:
result["fail_reason"] = (
f"judge=0: predicted '{eval_result['predicted_answer']}' "
f"but expected {item['blank_answers']} ({eval_result['judge_reason']})"
)
eval_detail = (
f"[EVALUATION RESULT]\n"
f"Question: {item['question']}\n"
f"Predicted answer: {eval_result['predicted_answer']!r}\n"
f"Gold answers: {item['blank_answers']!r}\n"
f"Judge correct: {eval_result['em']}\n"
f"Judge reason: {eval_result['judge_reason']}\n"
f"String F1: {eval_result.get('string_f1', 0.0):.4f}"
)
conversation.append({"role": "system", "content": eval_detail})
with open(os.path.join(pred_dir, "conversation.json"), "w", encoding="utf-8") as f:
json.dump(conversation, f, ensure_ascii=False, indent=2)
except Exception as e: # noqa: BLE001
result["fail_reason"] = f"error: {e}"
return result
def run_batch(
items: list[dict],
out_root: str,
skill_content: str,
*,
max_turns: int = 1,
workers: int = 32,
image_detail: str = "auto",
judge_model: str = "gpt-5.4",
judge_max_completion_tokens: int = 256,
judge_retries: int = 5,
diagnostic_mode: bool = False,
diagnostic_instruction: str = "",
diagnostic_trace_context_by_id: dict[str, str] | None = None,
) -> list[dict]:
results_path = os.path.join(out_root, "results.jsonl")
os.makedirs(out_root, exist_ok=True)
expected_eval_mode = evaluation_mode()
done_ids: set[str] = set()
existing: list[dict] = []
rewrite_results = False
if os.path.exists(results_path):
with open(results_path, encoding="utf-8") as f:
for line in f:
try:
row = json.loads(line)
if row.get("evaluation_mode") != expected_eval_mode:
rewrite_results = True
continue
done_ids.add(str(row["id"]))
existing.append(row)
except Exception:
rewrite_results = True
pending = [item for item in items if str(item["id"]) not in done_ids]
if not pending and not rewrite_results:
return existing
total = len(existing) + len(pending)
completed = len(existing)
correct_count = sum(1 for r in existing if r.get("hard", 0))
if existing:
print(f" [rollout] resuming: {completed}/{total} already done", flush=True)
results = list(existing)
file_mode = "w" if rewrite_results else "a"
with open(results_path, file_mode, encoding="utf-8") as outf, ThreadPoolExecutor(max_workers=workers) as ex:
if rewrite_results:
for row in existing:
outf.write(json.dumps(row, ensure_ascii=False) + "\n")
futs = {
ex.submit(
process_one,
item,
out_root,
skill_content,
max_turns=max_turns,
image_detail=image_detail,
judge_model=judge_model,
judge_max_completion_tokens=judge_max_completion_tokens,
judge_retries=judge_retries,
diagnostic_mode=diagnostic_mode,
diagnostic_instruction=diagnostic_instruction,
diagnostic_trace_context=(diagnostic_trace_context_by_id or {}).get(str(item["id"]), ""),
): item
for item in pending
}
for fut in as_completed(futs):
row = fut.result()
results.append(row)
completed += 1
if row.get("hard", 0):
correct_count += 1
acc = correct_count / completed if completed else 0
print(
f" [rollout] {completed}/{total} "
f"(acc={acc:.3f}) id={row.get('id', '?')} "
f"hard={row.get('hard', '?')}",
flush=True,
)
outf.write(json.dumps(row, ensure_ascii=False) + "\n")
outf.flush()
return results
skillopt/envs/babyvision/skills/initial.md
+18
View File
@@ -0,0 +1,18 @@
# BabyVision Visual QA Heuristics
## Image Inspection
- First identify the main objects, their attributes, and their spatial relations before answering.
- If the question involves counting, compare all relevant instances carefully instead of stopping after the first match.
- If the question asks about color, size, position, or action, verify the specific visible evidence for that attribute.
## Multiple Choice
- Compare every option against the visible image evidence before deciding.
- Prefer the option that matches the image exactly; reject options that are only partially true or too vague.
- When two options are close, check the smallest discriminating visual detail.
## Open Answers
- Answer with the shortest phrase that is fully supported by the image.
- Match the expected level of specificity: not broader than the image evidence, not narrower than the question asks.
## Final Answer
- Output only the final answer inside <answer>...</answer>.
skillopt/envs/base.py
+396
View File
@@ -0,0 +1,396 @@
"""ReflACT environment adapter — abstract interface.
To connect ReflACT to a new environment (benchmark, simulator, etc.),
implement a subclass of :class:`EnvAdapter` with environment-specific
rollout and reflection logic.
Example::
class MyBenchAdapter(EnvAdapter):
def build_train_env(self, batch_size, seed, **kw):
return MyEnvManager(split="train", n=batch_size, seed=seed)
def build_eval_env(self, env_num, split, seed, **kw):
return MyEnvManager(split=split, n=env_num, seed=seed)
def rollout(self, env_manager, skill_content, out_dir, **kw):
# Run episodes, return [{"id": ..., "hard": 0/1, "soft": 0.0-1.0, ...}]
...
def reflect(self, results, skill_content, out_dir, **kw):
# Analyze trajectories, return list of patch dicts
...
def get_task_types(self):
return ["task_a", "task_b"]
"""
from __future__ import annotations
from abc import ABC, abstractmethod
import os
import random
from skillopt.datasets.base import BaseDataLoader, BatchSpec
from skillopt.model.codex_harness import extract_codex_trace_prefix, format_codex_trace_steps, parse_codex_raw
from skillopt.prompts import load_prompt
class EnvAdapter(ABC):
"""Abstract adapter for connecting ReflACT to any environment.
Subclasses must implement all abstract methods. The ReflACT trainer
calls these methods at the appropriate pipeline stages.
"""
# ── Lifecycle hooks ────────────────────────────────────────────────────
def setup(self, cfg: dict) -> None:
"""Called once by the trainer before the training loop begins.
Override to perform one-time initialization that requires the full
config (e.g., data loading, split creation). Default is a no-op.
"""
self._cfg = dict(cfg)
def get_dataloader(self) -> BaseDataLoader | None:
"""Return the task dataloader used by this adapter, if any."""
return None
def requires_ray(self) -> bool:
"""Return whether this adapter requires Ray runtime initialization."""
return False
def deep_reflect(
self,
results: list[dict],
skill_content: str,
out_dir: str,
**kwargs,
) -> list[dict | None]:
"""Optional deeper diagnostic reflection pass.
Default behavior is a no-op. Dataset-backed adapters may override this
to re-query the student on a small representative subset of the current
batch using minimally-perturbed diagnostic prompts that expose
intermediate reasoning state.
"""
return []
def build_reference_text(self, item: dict) -> str:
"""Return hidden reference material for deep reflection, if any."""
return str(item.get("reference_text") or "").strip()
def get_reference_metadata(self, item: dict) -> dict:
"""Return structured metadata about hidden reference material."""
reference_text = self.build_reference_text(item)
if not reference_text:
return {"fields": [], "preview": ""}
return {
"fields": ["reference_text"],
"preview": reference_text[:400],
}
def get_codex_deep_probe_prompt(self) -> str | None:
env_name = getattr(self, "_cfg", {}).get("env_name")
return load_prompt("deep_probe_codex", env=env_name)
def attach_codex_probe_context(
self,
results: list[dict],
prediction_dir: str,
) -> list[dict]:
"""Attach compact Codex step metadata for codex-aware deep reflection."""
enriched: list[dict] = []
for row in results:
merged = dict(row)
tid = str(row.get("id"))
raw_path = os.path.join(prediction_dir, tid, "codex_raw.txt")
if os.path.exists(raw_path):
with open(raw_path, encoding="utf-8") as f:
raw = f.read()
parsed = parse_codex_raw(raw)
merged["codex_probe_trace_steps"] = format_codex_trace_steps(raw)
merged["codex_probe_step_count"] = len(parsed["steps"])
enriched.append(merged)
return enriched
def resolve_codex_probe_target(
self,
*,
selected_items: list[dict],
selected_examples: list[dict],
prediction_dir: str,
probe: dict,
) -> tuple[list[dict], dict[str, str] | None, dict]:
"""Resolve the teacher-selected codex probe target and raw trace prefix."""
target_id = str(probe.get("probe_target_id", "")).strip()
selected_id_set = {str(item["id"]) for item in selected_items}
if target_id not in selected_id_set:
target_id = str(selected_items[0]["id"])
target_item = next(item for item in selected_items if str(item["id"]) == target_id)
target_result = next(
(row for row in selected_examples if str(row.get("id")) == target_id),
None,
)
max_probe_step = int((target_result or {}).get("codex_probe_step_count", 0))
default_probe_step = max_probe_step - 1 if max_probe_step > 1 else max_probe_step
probe_after_step = int(probe.get("probe_after_step", default_probe_step))
if max_probe_step > 0:
probe_after_step = max(0, min(probe_after_step, max_probe_step))
else:
probe_after_step = 0
raw_path = os.path.join(prediction_dir, target_id, "codex_raw.txt")
trace_prefix = ""
if os.path.exists(raw_path):
with open(raw_path, encoding="utf-8") as f:
trace_prefix = extract_codex_trace_prefix(f.read(), after_step=probe_after_step)
updated_probe = dict(probe)
updated_probe["probe_target_id"] = target_id
updated_probe["probe_after_step"] = probe_after_step
return [target_item], {target_id: trace_prefix}, updated_probe
def attach_reference_context(
self,
results: list[dict],
items: list[dict] | None,
) -> list[dict]:
"""Attach environment-specific hidden reference text to result dicts."""
if not results or not items:
return list(results)
item_by_id = {
str(item.get("id")): item
for item in items
if isinstance(item, dict) and item.get("id") is not None
}
enriched: list[dict] = []
for row in results:
merged = dict(row)
item = item_by_id.get(str(row.get("id")))
if item:
reference_text = self.build_reference_text(item)
if reference_text:
merged["reference_text"] = reference_text
enriched.append(merged)
return enriched
def select_representative_items(
self,
results: list[dict],
items: list[dict] | None,
*,
n_failures: int,
n_successes: int,
seed: int | None = None,
) -> list[dict]:
"""Select a small diverse subset of current-batch items by outcome."""
if not items:
return []
item_by_id = {
str(item.get("id")): item
for item in items
if isinstance(item, dict) and item.get("id") is not None
}
failures = [
(result, item_by_id[str(result.get("id"))])
for result in results
if not result.get("hard") and str(result.get("id")) in item_by_id
]
successes = [
(result, item_by_id[str(result.get("id"))])
for result in results
if result.get("hard") and str(result.get("id")) in item_by_id
]
rng = random.Random(seed)
def _pick(pool: list[tuple[dict, dict]], quota: int) -> list[dict]:
if quota <= 0 or not pool:
return []
shuffled = list(pool)
rng.shuffle(shuffled)
picked_ids: set[str] = set()
picked: list[dict] = []
seen_types: set[str] = set()
for result, item in shuffled:
task_type = str(result.get("task_type") or item.get("task_type") or item.get("subtype") or "unknown")
item_id = str(item["id"])
if task_type in seen_types or item_id in picked_ids:
continue
picked.append(item)
picked_ids.add(item_id)
seen_types.add(task_type)
if len(picked) >= quota:
return picked
for _, item in shuffled:
item_id = str(item["id"])
if item_id in picked_ids:
continue
picked.append(item)
picked_ids.add(item_id)
if len(picked) >= quota:
break
return picked
selected = _pick(failures, n_failures)
selected_ids = {str(item["id"]) for item in selected}
selected.extend(
item for item in _pick(successes, n_successes)
if str(item["id"]) not in selected_ids
)
return selected
def build_env_from_batch(self, batch: BatchSpec, **kwargs):
"""Build an environment manager or item list from a :class:`BatchSpec`.
Default behavior preserves the legacy adapter API by routing training
batches through :meth:`build_train_env` and evaluation batches through
:meth:`build_eval_env`.
"""
if batch.phase == "train":
return self.build_train_env(batch_size=batch.batch_size, seed=batch.seed, **kwargs)
return self.build_eval_env(
env_num=batch.batch_size,
split=batch.split,
seed=batch.seed,
**kwargs,
)
@abstractmethod
def build_train_env(self, batch_size: int, seed: int, **kwargs):
"""Build a training environment manager.
Returns
-------
object
An environment manager that can be passed to :meth:`rollout`.
"""
@abstractmethod
def build_eval_env(self, env_num: int, split: str, seed: int, **kwargs):
"""Build an evaluation environment manager.
Parameters
----------
env_num : int
Number of evaluation environments.
split : str
Dataset split (e.g. ``"valid_seen"``, ``"valid_unseen"``).
seed : int
Random seed for reproducibility.
Returns
-------
object
An environment manager that can be passed to :meth:`rollout`.
"""
@abstractmethod
def rollout(
self,
env_manager,
skill_content: str,
out_dir: str,
**kwargs,
) -> list[dict]:
"""Run a batch of episodes using the current skill.
Returns
-------
list[dict]
Each dict conforms to :class:`~skillopt.types.RolloutResult`:
must have ``"id"`` (str), ``"hard"`` (0/1), ``"soft"``
(float 0-1). May include env-specific fields.
"""
@abstractmethod
def reflect(
self,
results: list[dict],
skill_content: str,
out_dir: str,
**kwargs,
) -> list[dict | None]:
"""Analyze rollout results and produce patches.
Each returned dict conforms to :class:`~skillopt.types.RawPatch`:
``"patch"`` (with ``"edits"`` list) + ``"source_type"``
(``"failure"`` or ``"success"``).
Returns
-------
list[dict | None]
Raw analyst outputs; ``None`` entries are filtered out.
"""
@abstractmethod
def get_task_types(self) -> list[str]:
"""Return the list of task type names for this environment."""
# ── Prompt configuration (two-level priority) ────────────────────────
#
# Priority: env-specific prompt file > generic default prompt file.
#
# Prompts are loaded from ``.md`` files via ``load_prompt(name, env)``:
# 1. ``skillopt/envs/<env>/prompts/<name>.md`` (env-specific)
# 2. ``skillopt/prompts/<name>.md`` (generic fallback)
#
# Subclasses can still override ``get_*_prompt()`` for full control.
@property
def _env_name(self) -> str:
"""Derive the env directory name from this adapter's module path."""
# e.g. "skillopt.envs.searchqa.adapter" → "searchqa"
module = type(self).__module__
parts = module.split(".")
if len(parts) >= 3 and parts[-3] == "envs":
return parts[-2]
return ""
def _load_env_prompt(self, name: str) -> str | None:
"""Load a prompt with env-specific override. Returns None if not found."""
try:
return load_prompt(name, env=self._env_name)
except FileNotFoundError:
return None
def get_error_minibatch_prompt(self) -> str | None:
update_mode = getattr(self, "_cfg", {}).get("skill_update_mode", "patch")
raw_mode = str(update_mode).strip().lower()
if raw_mode in {"full_rewrite", "full_rewrite_minibatch", "minibatch_full_rewrite", "skill_rewrite_minibatch"}:
prompt = self._load_env_prompt("analyst_error_full_rewrite")
if prompt is not None:
return prompt
if raw_mode in {"rewrite", "rewrite_from_suggestions", "suggestions", "rewrite_suggestions"}:
prompt = self._load_env_prompt("analyst_error_rewrite")
if prompt is not None:
return prompt
return self._load_env_prompt("analyst_error")
def get_success_minibatch_prompt(self) -> str | None:
update_mode = getattr(self, "_cfg", {}).get("skill_update_mode", "patch")
raw_mode = str(update_mode).strip().lower()
if raw_mode in {"full_rewrite", "full_rewrite_minibatch", "minibatch_full_rewrite", "skill_rewrite_minibatch"}:
prompt = self._load_env_prompt("analyst_success_full_rewrite")
if prompt is not None:
return prompt
if raw_mode in {"rewrite", "rewrite_from_suggestions", "suggestions", "rewrite_suggestions"}:
prompt = self._load_env_prompt("analyst_success_rewrite")
if prompt is not None:
return prompt
return self._load_env_prompt("analyst_success")
def get_deep_probe_prompt(self) -> str | None:
return self._load_env_prompt("deep_probe")
def get_meta_reflect_prompt(self) -> str | None:
update_mode = getattr(self, "_cfg", {}).get("skill_update_mode", "patch")
if str(update_mode).strip().lower() == "rewrite_from_suggestions":
prompt = self._load_env_prompt("meta_reflect_rewrite")
if prompt is not None:
return prompt
return self._load_env_prompt("meta_reflect")
skillopt/envs/deep_reflect.py
+114
View File
@@ -0,0 +1,114 @@
from __future__ import annotations
import json
import os
from typing import Any, Callable
from skillopt.gradient.deep_probe import generate_deep_probe_instruction
from skillopt.gradient.reflect import run_minibatch_reflect
def run_no_reference_deep_reflect(
adapter: Any,
results: list[dict],
skill_content: str,
out_dir: str,
*,
env_manager: Any = None,
prediction_dir: str | None = None,
random_seed: int | None = None,
step_buffer_context: str = "",
output_requirements: list[str] | None = None,
metadata_builder: Callable[[dict], dict] | None = None,
) -> list[dict | None]:
"""Run teacher-designed diagnostic probing without hidden references."""
if not getattr(adapter, "use_deep_reflect", False):
return []
if not isinstance(env_manager, list):
return []
prediction_dir = prediction_dir or os.path.join(out_dir, "predictions")
selected_items = adapter.select_representative_items(
results,
env_manager,
n_failures=getattr(adapter, "deep_reflect_failures", 4),
n_successes=getattr(adapter, "deep_reflect_successes", 2),
seed=random_seed,
)
if not selected_items:
return []
selected_ids = {str(item["id"]) for item in selected_items}
selected_results = [row for row in results if str(row.get("id")) in selected_ids]
if metadata_builder is None:
selected_metadata = [
{
"id": str(item.get("id")),
"task_type": str(item.get("task_type") or item.get("topic") or "unknown"),
"question_preview": str(item.get("question") or "")[:200],
}
for item in selected_items
]
else:
selected_metadata = [metadata_builder(item) for item in selected_items]
deep_dir = os.path.join(out_dir, "deep_reflect")
rollout_dir = os.path.join(deep_dir, "rollout")
patches_dir = os.path.join(deep_dir, "patches")
os.makedirs(deep_dir, exist_ok=True)
print(
f" [2b/6 DEEP REFLECT setup] selected={len(selected_items)} "
"mode=no_reference_probe"
)
probe = generate_deep_probe_instruction(
skill_content=skill_content,
items=selected_results,
prediction_dir=prediction_dir,
system_prompt=adapter.get_deep_probe_prompt(),
step_buffer_context=step_buffer_context,
output_requirements=output_requirements,
)
if not probe:
return []
with open(os.path.join(deep_dir, "probe.json"), "w", encoding="utf-8") as f:
json.dump(
{
**probe,
"reference_summary": {
"mode": "no_reference_probe",
"selected_count": len(selected_items),
},
"selected_examples": selected_metadata,
},
f,
ensure_ascii=False,
indent=2,
)
deep_results = adapter.rollout(
selected_items,
skill_content,
rollout_dir,
diagnostic_mode=True,
diagnostic_instruction=probe["probe_instruction"],
)
return run_minibatch_reflect(
results=deep_results,
skill_content=skill_content,
prediction_dir=os.path.join(rollout_dir, "predictions"),
patches_dir=patches_dir,
workers=getattr(adapter, "analyst_workers", 8),
failure_only=getattr(adapter, "failure_only", False),
minibatch_size=getattr(adapter, "minibatch_size", 8),
edit_budget=getattr(adapter, "edit_budget", 4),
random_seed=random_seed,
error_system=adapter.get_error_minibatch_prompt(),
success_system=adapter.get_success_minibatch_prompt(),
step_buffer_context=step_buffer_context,
update_mode=getattr(getattr(adapter, "_cfg", {}), "get", lambda *_: "patch")(
"skill_update_mode",
"patch",
),
)
skillopt/envs/docvqa/__init__.py
+1
View File
@@ -0,0 +1 @@
"""DocVQA environment package for ReflACT."""
skillopt/envs/docvqa/adapter.py
+151
View File
@@ -0,0 +1,151 @@
from __future__ import annotations
import os
from skillopt.datasets.base import BatchSpec
from skillopt.envs.base import EnvAdapter
from skillopt.envs.deep_reflect import run_no_reference_deep_reflect
from skillopt.envs.docvqa.dataloader import DocVQADataLoader
from skillopt.envs.docvqa.rollout import run_batch
from skillopt.gradient.reflect import run_minibatch_reflect
class DocVQAAdapter(EnvAdapter):
def __init__(
self,
split_dir: str = "",
data_path: str = "",
split_mode: str = "split_dir",
split_ratio: str = "2:1:7",
split_seed: int = 42,
split_output_dir: str = "",
max_turns: int = 1,
exec_timeout: int = 120,
workers: int = 16,
analyst_workers: int = 16,
failure_only: bool = False,
minibatch_size: int = 8,
edit_budget: int = 4,
seed: int = 42,
limit: int = 0,
image_detail: str = "auto",
use_deep_reflect: bool = False,
deep_reflect_failures: int = 4,
deep_reflect_successes: int = 2,
) -> None:
self.max_turns = max_turns
self.exec_timeout = exec_timeout
self.workers = workers
self.analyst_workers = analyst_workers
self.failure_only = failure_only
self.minibatch_size = minibatch_size
self.edit_budget = edit_budget
self.image_detail = image_detail
self.use_deep_reflect = use_deep_reflect
self.deep_reflect_failures = deep_reflect_failures
self.deep_reflect_successes = deep_reflect_successes
self.dataloader = DocVQADataLoader(
split_dir=split_dir,
data_path=data_path,
split_mode=split_mode,
split_ratio=split_ratio,
split_seed=split_seed,
split_output_dir=split_output_dir,
seed=seed,
limit=limit,
)
def setup(self, cfg: dict) -> None:
super().setup(cfg)
self.dataloader.setup(cfg)
def get_dataloader(self):
return self.dataloader
def build_env_from_batch(self, batch: BatchSpec, **kwargs):
return list(batch.payload or [])
def build_train_env(self, batch_size: int, seed: int, **kwargs):
batch = self.dataloader.build_train_batch(batch_size=batch_size, seed=seed, **kwargs)
return self.build_env_from_batch(batch, **kwargs)
def build_eval_env(self, env_num: int, split: str, seed: int, **kwargs):
batch = self.dataloader.build_eval_batch(env_num=env_num, split=split, seed=seed, **kwargs)
return self.build_env_from_batch(batch, **kwargs)
def rollout(self, env_manager, skill_content: str, out_dir: str, **kwargs) -> list[dict]:
items: list[dict] = env_manager
return run_batch(
items=items,
out_root=out_dir,
skill_content=skill_content,
max_turns=self.max_turns,
exec_timeout=self.exec_timeout,
workers=self.workers,
image_detail=self.image_detail,
diagnostic_mode=kwargs.get("diagnostic_mode", False),
diagnostic_instruction=kwargs.get("diagnostic_instruction", ""),
task_timeout=self.exec_timeout,
)
def reflect(self, results: list[dict], skill_content: str, out_dir: str, **kwargs) -> list[dict | None]:
prediction_dir = kwargs.get("prediction_dir", os.path.join(out_dir, "predictions"))
patches_dir = kwargs.get("patches_dir", os.path.join(out_dir, "patches"))
random_seed = kwargs.get("random_seed")
step_buffer_context = kwargs.get("step_buffer_context", "")
return run_minibatch_reflect(
results=results,
skill_content=skill_content,
prediction_dir=prediction_dir,
patches_dir=patches_dir,
workers=self.analyst_workers,
failure_only=self.failure_only,
minibatch_size=self.minibatch_size,
edit_budget=self.edit_budget,
random_seed=random_seed,
error_system=self.get_error_minibatch_prompt(),
success_system=self.get_success_minibatch_prompt(),
step_buffer_context=step_buffer_context,
update_mode=getattr(self, "_cfg", {}).get("skill_update_mode", "patch"),
)
def deep_reflect(
self,
results: list[dict],
skill_content: str,
out_dir: str,
**kwargs,
) -> list[dict | None]:
return run_no_reference_deep_reflect(
self,
results,
skill_content,
out_dir,
env_manager=kwargs.get("env_manager"),
prediction_dir=kwargs.get("prediction_dir"),
random_seed=kwargs.get("random_seed"),
step_buffer_context=kwargs.get("step_buffer_context", ""),
output_requirements=[
"- There is no hidden reference block. Use only the document image prompt, student output, and evaluation result to infer what intermediate state is worth probing.",
"- The instruction must explicitly request a short <analysis>...</analysis> block before the final <answer>...</answer>.",
"- The readout should focus on visual region, field/table/figure label, OCR text read, candidate answer, and answer-format normalization.",
"- Do not ask for exhaustive transcription or a full chain-of-thought.",
"- The instruction text should be ready to append directly to the student's prompt.",
],
metadata_builder=lambda item: {
"id": str(item.get("id")),
"task_type": str(item.get("task_type") or "docvqa"),
"question_preview": str(item.get("question") or "")[:200],
"image_path": item.get("image_path", ""),
"docId": item.get("docId", ""),
"page": item.get("ucsf_document_page_no", ""),
},
)
def get_task_types(self) -> list[str]:
seen: list[str] = []
for item in self.dataloader.train_items + self.dataloader.val_items + self.dataloader.test_items:
task_type = str(item.get("task_type") or "docvqa")
if task_type not in seen:
seen.append(task_type)
return seen or ["docvqa"]
skillopt/envs/docvqa/dataloader.py
+61
View File
@@ -0,0 +1,61 @@
from __future__ import annotations
import ast
import csv
from pathlib import Path
from skillopt.datasets.base import SplitDataLoader
def _parse_answers(raw: str) -> list[str]:
text = str(raw or "").strip()
if not text:
return []
try:
parsed = ast.literal_eval(text)
except Exception:
return [text]
if isinstance(parsed, list):
return [str(item).strip() for item in parsed if str(item).strip()]
return [str(parsed).strip()]
def _extract_document_path(question: str) -> tuple[str, str]:
marker = "document_path:"
if marker not in question:
return question.strip(), ""
main, tail = question.split(marker, 1)
return main.strip(), tail.strip()
def _normalize_row(row: dict[str, str]) -> dict:
question_text, document_path = _extract_document_path(str(row.get("question") or ""))
answers = _parse_answers(row.get("answer") or row.get("ground_truth") or "")
image_path = str(row.get("image_path") or document_path or "").strip()
task_type = str(row.get("topic") or row.get("category") or "docvqa").strip() or "docvqa"
return {
"id": str(row.get("questionId") or row.get("id") or "").strip(),
"question": question_text,
"answer": answers[0] if answers else "",
"answers": answers,
"task_type": task_type,
"subtask": task_type,
"image_paths": [image_path] if image_path else [],
"image_path": image_path,
"questionId": str(row.get("questionId") or "").strip(),
"docId": str(row.get("docId") or "").strip(),
"ucsf_document_id": str(row.get("ucsf_document_id") or "").strip(),
"ucsf_document_page_no": str(row.get("ucsf_document_page_no") or "").strip(),
"source_split": str(row.get("source_split") or "").strip(),
}
class DocVQADataLoader(SplitDataLoader):
def load_split_items(self, split_path: str) -> list[dict]:
path = Path(split_path)
csv_files = sorted(path.glob("*.csv"))
if not csv_files:
raise FileNotFoundError(f"No .csv file found in {split_path}")
with csv_files[0].open(encoding="utf-8", newline="") as f:
reader = csv.DictReader(f)
return [_normalize_row(row) for row in reader]
skillopt/envs/docvqa/evaluator.py
+113
View File
@@ -0,0 +1,113 @@
from __future__ import annotations
import ast
import json
from collections.abc import Iterable
from typing import Any
DEFAULT_ANLS_THRESHOLD = 0.5
def _normalize_text(value: Any) -> str:
if value is None:
return ""
text = str(value).strip().lower()
return " ".join(text.split())
def _levenshtein_distance(a: str, b: str) -> int:
if a == b:
return 0
if not a:
return len(b)
if not b:
return len(a)
if len(a) > len(b):
a, b = b, a
previous = list(range(len(b) + 1))
for i, char_a in enumerate(a, start=1):
current = [i]
for j, char_b in enumerate(b, start=1):
insert_cost = current[j - 1] + 1
delete_cost = previous[j] + 1
replace_cost = previous[j - 1] + (char_a != char_b)
current.append(min(insert_cost, delete_cost, replace_cost))
previous = current
return previous[-1]
def _score_single_answer(predicted: Any, target: Any, threshold: float) -> float:
predicted_norm = _normalize_text(predicted)
target_norm = _normalize_text(target)
if not predicted_norm and not target_norm:
return 1.0
if not predicted_norm or not target_norm:
return 0.0
distance = _levenshtein_distance(predicted_norm, target_norm)
normalized_distance = distance / max(len(predicted_norm), len(target_norm))
if normalized_distance >= threshold:
return 0.0
return 1.0 - normalized_distance
def _extract_answer_strings(raw: Any) -> list[str]:
if raw is None:
return [""]
if isinstance(raw, str):
text = raw.strip()
if not text:
return [""]
parsed = None
if text[0] in "[{":
try:
parsed = json.loads(text)
except json.JSONDecodeError:
try:
parsed = ast.literal_eval(text)
except (ValueError, SyntaxError):
parsed = None
if parsed is None:
return [text]
return _extract_answer_strings(parsed)
if isinstance(raw, dict):
for key in ("answers", "ground_truth", "answer"):
if key in raw:
return _extract_answer_strings(raw[key])
return [str(raw)]
if isinstance(raw, Iterable) and not isinstance(raw, (bytes, bytearray)):
answers: list[str] = []
for item in raw:
if isinstance(item, dict):
for key in ("text", "answer", "value"):
if key in item:
answers.extend(_extract_answer_strings(item[key]))
break
else:
answers.append(str(item))
continue
answers.append(str(item))
return answers or [""]
return [str(raw)]
def extract_answer(text: str) -> str:
lower = text.lower()
start = lower.rfind("<answer>")
end = lower.rfind("</answer>")
if start != -1 and end != -1 and end > start:
return text[start + len("<answer>"):end].strip()
lines = [line.strip() for line in text.splitlines() if line.strip()]
return lines[-1] if lines else text.strip()
def evaluate(prediction_text: str, gold_answers: Any) -> dict:
answer = extract_answer(prediction_text)
answers = _extract_answer_strings(gold_answers)
score = 0.0
for target in answers:
score = max(score, _score_single_answer(answer, target, DEFAULT_ANLS_THRESHOLD))
return {
"anls": score,
"predicted_answer": answer,
"gold_answers": answers,
}
skillopt/envs/docvqa/prompts/analyst_error.md
+35
View File
@@ -0,0 +1,35 @@
You are an expert failure-analysis agent for visual document question answering tasks.
You will be given MULTIPLE failed DocVQA trajectories from a single minibatch and the current skill document. Each trajectory includes the model response and an evaluation result scored with ANLS against one or more acceptable answers.
Your job is to identify the most important COMMON failure patterns across the batch and propose concise skill edits.
## Failure Type Categories
- evidence_miss: the model overlooked the relevant visible region or line
- near_match_confusion: the model selected a nearby but incorrect text span
- normalization_error: the answer differed mainly in formatting, spacing, punctuation, or minor text normalization
- reading_error: the model misread the document content
- other: none of the above
## Rules
- Focus on common, reusable reading and extraction behaviors.
- Do not hardcode image-specific answers.
- Prefer concise edits that improve evidence selection and exact span extraction.
Respond ONLY with a valid JSON object (no markdown fences, no extra text):
{
"batch_size": <number of trajectories analysed>,
"failure_summary": [
{"failure_type": "<type>", "count": <int>, "description": "<one-line>"}
],
"patch": {
"reasoning": "<why these edits address the batch's common failures>",
"edits": [
{"op": "append", "content": "<markdown to add at end of skill>"},
{"op": "insert_after", "target": "<exact heading/text to insert after>", "content": "<markdown>"},
{"op": "replace", "target": "<exact text to replace>", "content": "<replacement>"},
{"op": "delete", "target": "<exact text to remove>"}
]
}
}
Only include edits that are needed. "edits" can be an empty list if no patch is warranted.
skillopt/envs/docvqa/prompts/analyst_success.md
+24
View File
@@ -0,0 +1,24 @@
You are an expert success-pattern analyst for visual document question answering tasks.
You will be given MULTIPLE successful DocVQA trajectories from a single minibatch and the current skill document. Your job is to identify common visual reading and exact-answer extraction behaviors worth encoding in the skill.
## Rules
- Focus on patterns shared across multiple successful trajectories.
- Reinforce reusable behaviors like locating the right region, copying exact spans, and preferring the shortest exact answer over paraphrase.
- Only propose patches for patterns not already captured by the current skill.
Respond ONLY with a valid JSON object:
{
"batch_size": <number of trajectories analysed>,
"success_patterns": ["<pattern 1>", "<pattern 2>"],
"patch": {
"reasoning": "<why these patterns are worth encoding>",
"edits": [
{"op": "append", "content": "<markdown>"},
{"op": "insert_after", "target": "<heading/text>", "content": "<markdown>"},
{"op": "replace", "target": "<old text>", "content": "<new text>"},
{"op": "delete", "target": "<exact text to remove>"}
]
}
}
"edits" may be empty if the skill already covers all observed patterns.
skillopt/envs/docvqa/prompts/rollout_system.md
+12
View File
@@ -0,0 +1,12 @@
You are an expert visual document question answering agent.
{skill_section}You will receive a document image and a question about the document.
Read the visual evidence carefully and answer concisely.
Rules:
- Ground the answer in the visible document content.
- Prefer exact spans, numbers, dates, and names from the document.
- Do not invent content that is not visible.
- If multiple near-matches exist, choose the one best supported by the document.
Return the final answer inside <answer>...</answer>.
skillopt/envs/docvqa/rollout.py
+388
View File
@@ -0,0 +1,388 @@
from __future__ import annotations
import json
import os
import time
from concurrent.futures import FIRST_COMPLETED, ThreadPoolExecutor, wait
from skillopt.envs.docvqa.evaluator import evaluate
from skillopt.model import chat_student_messages, get_student_backend, is_student_exec_backend
from skillopt.model.codex_harness import prepare_workspace, render_skill_md, run_student_exec
from skillopt.prompts import load_prompt
def _build_system(skill_content: str) -> str:
if skill_content.strip():
skill_section = f"## Skill\n{skill_content.strip()}\n\n"
else:
skill_section = ""
return load_prompt("rollout_system", env="docvqa").format(skill_section=skill_section)
def _image_to_data_uri(path: str) -> str:
import base64
import mimetypes
mime = mimetypes.guess_type(path)[0] or "image/png"
with open(path, "rb") as f:
encoded = base64.b64encode(f.read()).decode("ascii")
return f"data:{mime};base64,{encoded}"
def _build_messages(
item: dict,
skill_content: str,
image_detail: str,
*,
diagnostic_mode: bool = False,
diagnostic_instruction: str = "",
) -> tuple[list[dict], str, str]:
system = _build_system(skill_content)
user_text = item["question"] + "\n\nReturn the final answer inside <answer>...</answer>."
if diagnostic_mode and diagnostic_instruction.strip():
user_text += f"\n\n## Training Readout\n{diagnostic_instruction.strip()}"
image_url = {"url": _image_to_data_uri(item["image_path"])}
if image_detail and image_detail != "auto":
image_url["detail"] = image_detail
messages = [
{"role": "system", "content": system},
{
"role": "user",
"content": [
{"type": "text", "text": user_text},
{"type": "image_url", "image_url": image_url},
],
},
]
return messages, system, user_text
def _build_codex_skill(skill_content: str) -> str:
return render_skill_md(
skill_content,
description="Dynamic ReflACT skill for solving the current DocVQA document-image question.",
preamble=(
"Use this skill when answering the current DocVQA question.\n"
"Inspect the attached document image carefully and return the final answer inside <answer>...</answer>."
),
)
def _run_codex_once(
*,
pred_dir: str,
item: dict,
skill_content: str,
model: str,
timeout: int,
image_detail: str,
diagnostic_mode: bool = False,
diagnostic_instruction: str = "",
previous_response: str = "",
) -> tuple[str, str, str, str]:
_ = image_detail
_messages, _system, user_text = _build_messages(
item,
skill_content,
image_detail,
diagnostic_mode=diagnostic_mode,
diagnostic_instruction=diagnostic_instruction,
)
task_parts = [user_text]
image_abs = os.path.abspath(item["image_path"])
task_parts.append(
"## Document Image\n"
"The document image is available in this workspace via `ATTACHMENTS.md`.\n"
f"Original image path: `{image_abs}`\n"
"Open or inspect that image before answering; do not answer from memory."
)
if previous_response:
task_parts.append(
"## Previous Attempt\n"
f"{previous_response}\n\n"
"Review the same document image carefully and correct the answer if needed."
)
task_text = "\n\n".join(task_parts)
skill_md = _build_codex_skill(skill_content)
work_dir = os.path.join(pred_dir, "codex_exec")
prepare_workspace(
work_dir=work_dir,
skill_md=skill_md,
task_text=task_text,
images=[item["image_path"]],
)
prompt = (
"Use the `skillopt-student` skill available in this workspace.\n"
"Read `task.md`, inspect the attached document image, and answer the DocVQA question.\n"
"Return the final answer inside <answer>...</answer>."
)
final_message, raw = run_student_exec(
work_dir=work_dir,
prompt=prompt,
model=model,
timeout=timeout,
images=[item["image_path"]],
)
return final_message or raw, raw, skill_md, task_text
def process_one(
item: dict,
out_root: str,
skill_content: str,
*,
max_turns: int = 1,
exec_timeout: int = 120,
image_detail: str = "auto",
diagnostic_mode: bool = False,
diagnostic_instruction: str = "",
) -> dict:
item_id = str(item["id"])
result = {
"id": item_id,
"question": item["question"],
"task_type": item.get("subtask") or item.get("task_type") or "docvqa",
"task_description": item["question"],
"hard": 0,
"soft": 0.0,
"predicted_answer": "",
"response": "",
"fail_reason": "",
"agent_ok": False,
"n_turns": 0,
"image_paths": item.get("image_paths", []),
"gold_answer": item.get("answers", []),
}
try:
response = ""
system_prompt = ""
user_text = ""
conversation: list[dict] = []
if is_student_exec_backend():
from skillopt.model import azure_openai as _llm
conversation = [
{
"role": "user",
"content": item["question"] + "\n\n" + f"[image] {os.path.basename(item['image_path'])}",
}
]
for turn in range(max_turns):
response, _raw, system_prompt, user_text = _run_codex_once(
pred_dir=os.path.join(out_root, "predictions", item_id),
item=item,
skill_content=skill_content,
model=_llm.STUDENT_DEPLOYMENT,
timeout=exec_timeout,
image_detail=image_detail,
diagnostic_mode=diagnostic_mode if turn == 0 else False,
diagnostic_instruction=diagnostic_instruction if turn == 0 else "",
previous_response=response if turn > 0 else "",
)
conversation.append({"type": "message", "turn": turn + 1, "content": response})
if "<answer>" in response.lower():
break
else:
messages, system_prompt, user_text = _build_messages(
item,
skill_content,
image_detail,
diagnostic_mode=diagnostic_mode,
diagnostic_instruction=diagnostic_instruction,
)
conversation = [
{
"role": "user",
"content": user_text + "\n\n" + f"[image] {os.path.basename(item['image_path'])}",
}
]
for turn in range(max_turns):
if turn == 0:
resp_text, _ = chat_student_messages(
messages=messages,
max_completion_tokens=768,
retries=5,
stage="rollout",
timeout=exec_timeout,
)
else:
refinement_messages = [
messages[0],
messages[1],
{"role": "assistant", "content": response},
{"role": "user", "content": "Review the same image carefully and answer again. Keep the final answer inside <answer>...</answer>."},
]
resp_text, _ = chat_student_messages(
messages=refinement_messages,
max_completion_tokens=512,
retries=5,
stage="rollout",
timeout=exec_timeout,
)
response = resp_text
conversation.append({"type": "message", "turn": turn + 1, "content": resp_text})
if "<answer>" in resp_text.lower():
break
result["response"] = response
result["agent_ok"] = True
result["n_turns"] = len(conversation) - 1
pred_dir = os.path.join(out_root, "predictions", item_id)
os.makedirs(pred_dir, exist_ok=True)
with open(os.path.join(pred_dir, "student_system_prompt.txt"), "w", encoding="utf-8") as f:
f.write(system_prompt)
with open(os.path.join(pred_dir, "student_user_prompt.txt"), "w", encoding="utf-8") as f:
f.write(user_text)
eval_result = evaluate(response, item.get("answers", []))
result["predicted_answer"] = eval_result["predicted_answer"]
result["hard"] = int(eval_result["anls"] >= 0.999)
result["soft"] = eval_result["anls"]
if result["soft"] <= 0.0:
result["fail_reason"] = f"predicted '{eval_result['predicted_answer']}' but expected one of {item.get('answers', [])}"
eval_detail = (
"[EVALUATION RESULT]\n"
f"Question: {item['question']}\n"
f"Predicted answer: {eval_result['predicted_answer']!r}\n"
f"Gold answers: {item.get('answers', [])!r}\n"
f"ANLS: {eval_result['anls']:.4f}"
)
conversation.append({"role": "system", "content": eval_detail})
with open(os.path.join(pred_dir, "conversation.json"), "w", encoding="utf-8") as f:
json.dump(conversation, f, ensure_ascii=False, indent=2)
except Exception as e: # noqa: BLE001
result["fail_reason"] = f"error: {e}"
return result
def run_batch(
items: list[dict],
out_root: str,
skill_content: str,
*,
max_turns: int = 1,
exec_timeout: int = 120,
workers: int = 16,
image_detail: str = "auto",
diagnostic_mode: bool = False,
diagnostic_instruction: str = "",
task_timeout: int = 600,
) -> list[dict]:
task_timeout = max(int(task_timeout), int(exec_timeout) + 60)
results_path = os.path.join(out_root, "results.jsonl")
os.makedirs(out_root, exist_ok=True)
done_ids: set[str] = set()
existing: list[dict] = []
if os.path.exists(results_path):
with open(results_path, encoding="utf-8") as f:
for line in f:
try:
row = json.loads(line)
except Exception:
continue
done_ids.add(str(row["id"]))
existing.append(row)
pending = [item for item in items if str(item["id"]) not in done_ids]
if not pending:
return existing
def _timeout_result(item: dict) -> dict:
return {
"id": str(item["id"]),
"question": item.get("question", ""),
"task_type": item.get("subtask") or item.get("task_type") or "docvqa",
"task_description": item.get("question", ""),
"hard": 0,
"soft": 0.0,
"predicted_answer": "",
"response": "",
"fail_reason": f"task-timeout-{task_timeout}s",
"agent_ok": False,
"n_turns": 0,
"image_paths": item.get("image_paths", []),
"gold_answer": item.get("answers", []),
"phase": "timeout",
}
def _error_result(item: dict, exc: Exception) -> dict:
row = _timeout_result(item)
row["phase"] = "error"
row["fail_reason"] = f"unexpected: {type(exc).__name__}: {exc}"
return row
started_at: dict[str, float] = {}
def _run_one(item: dict) -> dict:
started_at[str(item["id"])] = time.time()
return process_one(
item,
out_root,
skill_content,
max_turns=max_turns,
exec_timeout=exec_timeout,
image_detail=image_detail,
diagnostic_mode=diagnostic_mode,
diagnostic_instruction=diagnostic_instruction,
)
total = len(existing) + len(pending)
completed = len(existing)
correct = sum(1 for r in existing if r.get("hard", 0))
if existing:
print(f" [rollout] resuming: {completed}/{total} already done", flush=True)
results = list(existing)
with open(results_path, "a", encoding="utf-8") as outf:
ex = ThreadPoolExecutor(max_workers=workers)
try:
futs = {ex.submit(_run_one, item): item for item in pending}
pending_futs = set(futs)
while pending_futs:
done, _ = wait(pending_futs, timeout=5, return_when=FIRST_COMPLETED)
now = time.time()
timed_out = [
fut for fut in pending_futs - done
if str(futs[fut]["id"]) in started_at
and now - started_at[str(futs[fut]["id"])] >= task_timeout
]
for fut in done:
pending_futs.remove(fut)
item = futs[fut]
try:
res = fut.result()
except Exception as exc: # noqa: BLE001
res = _error_result(item, exc)
results.append(res)
completed += 1
if res.get("hard", 0):
correct += 1
acc = correct / completed if completed else 0
print(
f" [rollout] {completed}/{total} "
f"(acc={acc:.3f}) id={res['id']} "
f"hard={res.get('hard', '?')}",
flush=True,
)
outf.write(json.dumps(res, ensure_ascii=False) + "\n")
outf.flush()
for fut in timed_out:
pending_futs.remove(fut)
fut.cancel()
res = _timeout_result(futs[fut])
results.append(res)
completed += 1
acc = correct / completed if completed else 0
print(
f" [rollout] {completed}/{total} "
f"(acc={acc:.3f}) id={res['id']} TIMEOUT",
flush=True,
)
outf.write(json.dumps(res, ensure_ascii=False) + "\n")
outf.flush()
finally:
ex.shutdown(wait=False, cancel_futures=True)
return results
skillopt/envs/docvqa/skills/initial.md
+11
View File
@@ -0,0 +1,11 @@
# DocVQA Skill
## Visual Evidence Discipline
- Read the document carefully before answering.
- Prefer the smallest exact text span that answers the question.
- When several nearby strings look similar, choose the one whose surrounding labels or layout best match the question.
## Exact Answer Discipline
- Copy names, numbers, and dates exactly from the document whenever possible.
- Prefer direct extraction over paraphrase.
- Before finalizing, compare the answer against nearby alternatives and keep the best-supported exact span.
skillopt/envs/livemathematicianbench/__init__.py
+1
View File
@@ -0,0 +1 @@
"""LiveMathematicianBench environment package for ReflACT."""
skillopt/envs/livemathematicianbench/adapter.py
+282
View File
@@ -0,0 +1,282 @@
"""LiveMathematicianBench environment adapter for ReflACT."""
from __future__ import annotations
import json
import os
from skillopt.gradient.deep_probe import generate_deep_probe_instruction
from skillopt.datasets.base import BatchSpec
from skillopt.gradient.reflect import run_minibatch_reflect
from skillopt.envs.base import EnvAdapter
from skillopt.envs.livemathematicianbench.dataloader import LiveMathematicianBenchDataLoader
from skillopt.envs.livemathematicianbench.rollout import run_batch
from skillopt.model import get_student_backend
class LiveMathematicianBenchAdapter(EnvAdapter):
"""LiveMathematicianBench adapter."""
def build_reference_text(self, item: dict) -> str:
parts: list[str] = []
theorem = str(item.get("theorem") or "").strip()
sketch = str(item.get("sketch") or "").strip()
if theorem:
parts.append(f"## Reference Theorem\n{theorem}")
if sketch:
parts.append(f"## Reference Sketch\n{sketch}")
return "\n\n".join(parts)
def get_reference_metadata(self, item: dict) -> dict:
fields: list[str] = []
previews: list[str] = []
theorem = str(item.get("theorem") or "").strip()
sketch = str(item.get("sketch") or "").strip()
if theorem:
fields.append("theorem")
previews.append(f"[theorem]\n{theorem[:220]}")
if sketch:
fields.append("sketch")
previews.append(f"[sketch]\n{sketch[:220]}")
return {
"fields": fields,
"preview": "\n\n".join(previews)[:500],
}
def __init__(
self,
split_dir: str = "",
data_path: str = "",
split_mode: str = "ratio",
split_ratio: str = "2:1:7",
split_seed: int = 42,
split_output_dir: str = "",
max_turns: int = 1,
exec_timeout: int = 600,
workers: int = 64,
analyst_workers: int = 16,
failure_only: bool = False,
minibatch_size: int = 8,
edit_budget: int = 4,
seed: int = 42,
limit: int = 0,
shuffle_choices: bool = True,
use_theorem: bool = False,
use_sketch: bool = False,
use_deep_reflect: bool = False,
deep_reflect_failures: int = 4,
deep_reflect_successes: int = 2,
) -> None:
self.max_turns = max_turns
self.exec_timeout = exec_timeout
self.workers = workers
self.analyst_workers = analyst_workers
self.failure_only = failure_only
self.minibatch_size = minibatch_size
self.edit_budget = edit_budget
self.use_theorem = use_theorem
self.use_sketch = use_sketch
self.use_deep_reflect = use_deep_reflect
self.deep_reflect_failures = deep_reflect_failures
self.deep_reflect_successes = deep_reflect_successes
self.dataloader = LiveMathematicianBenchDataLoader(
split_dir=split_dir,
data_path=data_path,
split_mode=split_mode,
split_ratio=split_ratio,
split_seed=split_seed,
split_output_dir=split_output_dir,
seed=seed,
limit=limit,
shuffle_choices=shuffle_choices,
)
def setup(self, cfg: dict) -> None:
super().setup(cfg)
self.dataloader.setup(cfg)
def get_dataloader(self):
return self.dataloader
def build_env_from_batch(self, batch: BatchSpec, **kwargs):
return list(batch.payload or [])
def build_train_env(self, batch_size: int, seed: int, **kwargs):
batch = self.dataloader.build_train_batch(batch_size=batch_size, seed=seed, **kwargs)
return self.build_env_from_batch(batch, **kwargs)
def build_eval_env(self, env_num: int, split: str, seed: int, **kwargs):
batch = self.dataloader.build_eval_batch(env_num=env_num, split=split, seed=seed, **kwargs)
return self.build_env_from_batch(batch, **kwargs)
def rollout(
self,
env_manager,
skill_content: str,
out_dir: str,
**kwargs,
) -> list[dict]:
items: list[dict] = env_manager
return run_batch(
items=items,
out_root=out_dir,
skill_content=skill_content,
max_turns=self.max_turns,
exec_timeout=self.exec_timeout,
workers=self.workers,
use_theorem=self.use_theorem,
use_sketch=self.use_sketch,
diagnostic_mode=kwargs.get("diagnostic_mode", False),
diagnostic_instruction=kwargs.get("diagnostic_instruction", ""),
diagnostic_trace_context_by_id=kwargs.get("diagnostic_trace_context_by_id"),
task_timeout=self.exec_timeout,
)
def reflect(
self,
results: list[dict],
skill_content: str,
out_dir: str,
**kwargs,
) -> list[dict | None]:
prediction_dir = kwargs.get("prediction_dir", os.path.join(out_dir, "predictions"))
patches_dir = kwargs.get("patches_dir", os.path.join(out_dir, "patches"))
random_seed = kwargs.get("random_seed")
step_buffer_context = kwargs.get("step_buffer_context", "")
meta_skill_context = kwargs.get("meta_skill_context", "")
return run_minibatch_reflect(
results=results,
skill_content=skill_content,
prediction_dir=prediction_dir,
patches_dir=patches_dir,
workers=self.analyst_workers,
failure_only=self.failure_only,
minibatch_size=self.minibatch_size,
edit_budget=self.edit_budget,
random_seed=random_seed,
error_system=self.get_error_minibatch_prompt(),
success_system=self.get_success_minibatch_prompt(),
step_buffer_context=step_buffer_context,
meta_skill_context=meta_skill_context,
update_mode=getattr(self, "_cfg", {}).get("skill_update_mode", "patch"),
)
def deep_reflect(
self,
results: list[dict],
skill_content: str,
out_dir: str,
**kwargs,
) -> list[dict | None]:
if not self.use_deep_reflect:
return []
env_manager = kwargs.get("env_manager")
prediction_dir = kwargs.get("prediction_dir", os.path.join(out_dir, "predictions"))
random_seed = kwargs.get("random_seed")
step_buffer_context = kwargs.get("step_buffer_context", "")
meta_skill_context = kwargs.get("meta_skill_context", "")
codex_backend = get_student_backend() == "codex_exec"
selected_items = self.select_representative_items(
results,
env_manager if isinstance(env_manager, list) else None,
n_failures=self.deep_reflect_failures,
n_successes=self.deep_reflect_successes,
seed=random_seed,
)
if not selected_items:
return []
selected_ids = {str(item["id"]) for item in selected_items}
selected_results = [row for row in results if str(row.get("id")) in selected_ids]
selected_examples = self.attach_reference_context(selected_results, selected_items)
if codex_backend:
selected_examples = self.attach_codex_probe_context(selected_examples, prediction_dir)
selected_metadata = []
theorem_count = 0
sketch_count = 0
for item in selected_items:
meta = self.get_reference_metadata(item)
if "theorem" in meta["fields"]:
theorem_count += 1
if "sketch" in meta["fields"]:
sketch_count += 1
selected_metadata.append({
"id": str(item["id"]),
"task_type": str(item.get("theorem_type", ["math_mcq"])[0] if item.get("theorem_type") else "math_mcq"),
"reference_fields": meta["fields"],
"reference_preview": meta["preview"],
})
deep_dir = os.path.join(out_dir, "deep_reflect")
rollout_dir = os.path.join(deep_dir, "rollout")
patches_dir = os.path.join(deep_dir, "patches")
os.makedirs(deep_dir, exist_ok=True)
print(
f" [2b/6 DEEP REFLECT setup] selected={len(selected_items)} "
f"reference_fields=theorem({theorem_count}/{len(selected_items)}),"
f"sketch({sketch_count}/{len(selected_items)})"
)
probe = generate_deep_probe_instruction(
skill_content=skill_content,
items=selected_examples,
prediction_dir=prediction_dir,
system_prompt=self.get_codex_deep_probe_prompt() if codex_backend else self.get_deep_probe_prompt(),
step_buffer_context=step_buffer_context,
meta_skill_context=meta_skill_context,
)
if not probe:
return []
diagnostic_trace_context_by_id = None
if codex_backend:
selected_items, diagnostic_trace_context_by_id, probe = self.resolve_codex_probe_target(
selected_items=selected_items,
selected_examples=selected_examples,
prediction_dir=prediction_dir,
probe=probe,
)
probe_record = {
**probe,
"reference_summary": {
"selected_count": len(selected_items),
"field_counts": {
"theorem": theorem_count,
"sketch": sketch_count,
},
},
"selected_examples": selected_metadata,
}
with open(os.path.join(deep_dir, "probe.json"), "w", encoding="utf-8") as f:
json.dump(probe_record, f, ensure_ascii=False, indent=2)
deep_results = run_batch(
items=selected_items,
out_root=rollout_dir,
skill_content=skill_content,
max_turns=self.max_turns,
workers=min(self.workers, max(len(selected_items), 1)),
use_theorem=self.use_theorem,
use_sketch=self.use_sketch,
diagnostic_mode=True,
diagnostic_instruction=probe["probe_instruction"],
diagnostic_trace_context_by_id=diagnostic_trace_context_by_id,
task_timeout=self.exec_timeout,
)
deep_results = self.attach_reference_context(deep_results, selected_items)
return run_minibatch_reflect(
results=deep_results,
skill_content=skill_content,
prediction_dir=os.path.join(rollout_dir, "predictions"),
patches_dir=patches_dir,
workers=self.analyst_workers,
failure_only=self.failure_only,
minibatch_size=self.minibatch_size,
edit_budget=self.edit_budget,
random_seed=random_seed,
error_system=self.get_error_minibatch_prompt(),
success_system=self.get_success_minibatch_prompt(),
step_buffer_context=step_buffer_context,
meta_skill_context=meta_skill_context,
update_mode=getattr(self, "_cfg", {}).get("skill_update_mode", "patch"),
)
def get_task_types(self) -> list[str]:
return self.dataloader.get_task_types()
skillopt/envs/livemathematicianbench/dataloader.py
+308
View File
@@ -0,0 +1,308 @@
"""LiveMathematicianBench task dataloader."""
from __future__ import annotations
import glob
import hashlib
import json
import os
import random
from typing import Any
from skillopt.datasets.base import BatchSpec, SplitDataLoader
# ── Raw data loading utilities (for preprocessing / standalone eval) ─────
_CHOICE_LABELS = ["A", "B", "C", "D", "E", "F", "G"]
def _load_json(path: str) -> Any:
with open(path) as f:
return json.load(f)
def _iter_monthly_files(data_path: str) -> list[str]:
if not data_path:
return []
if os.path.isfile(data_path):
return [data_path]
if os.path.isdir(data_path):
nested = glob.glob(
os.path.join(data_path, "**", "qa_*_final.json"),
recursive=True,
)
flat = glob.glob(os.path.join(data_path, "qa_*_final.json"))
return sorted(set(nested + flat))
return []
def _coerce_choices(raw_choices: Any) -> list[dict]:
if isinstance(raw_choices, list):
choices: list[dict] = []
for idx, item in enumerate(raw_choices):
if isinstance(item, dict):
label = str(item.get("label") or _CHOICE_LABELS[idx]).strip()
text = str(item.get("text") or item.get("content") or "").strip()
else:
label = _CHOICE_LABELS[idx]
text = str(item).strip()
if text:
choices.append({"label": label, "text": text})
return choices
if isinstance(raw_choices, dict):
labels = sorted(raw_choices.keys())
return [
{"label": str(label).strip(), "text": str(raw_choices[label]).strip()}
for label in labels
if str(raw_choices[label]).strip()
]
return []
def _coerce_theorem_types(raw: Any) -> list[str]:
if isinstance(raw, list):
return [str(x).strip() for x in raw if str(x).strip()]
if raw is None:
return []
text = str(raw).strip()
return [text] if text else []
def _normalize_label(text: str) -> str:
return str(text).strip().upper().rstrip(".):")
def _normalize_item(item: dict, row_idx: int, source_path: str) -> dict:
mcq = item.get("mcq", {}) if isinstance(item.get("mcq"), dict) else {}
question = str(mcq.get("question") or item.get("question") or "").strip()
choices = _coerce_choices(mcq.get("choices") or item.get("choices") or [])
correct = mcq.get("correct_choice") or item.get("correct_choice") or {}
if isinstance(correct, dict):
correct_label = _normalize_label(correct.get("label", ""))
correct_text = str(correct.get("text") or "").strip()
else:
correct_label = _normalize_label(correct)
correct_text = ""
choice_by_label = {
_normalize_label(choice["label"]): choice["text"]
for choice in choices
}
if correct_label and not correct_text:
correct_text = choice_by_label.get(correct_label, "")
if correct_label and correct_text and correct_label not in choice_by_label:
choices.append({"label": correct_label, "text": correct_text})
choices.sort(key=lambda choice: _CHOICE_LABELS.index(choice["label"]) if choice["label"] in _CHOICE_LABELS else len(_CHOICE_LABELS))
choice_by_label[correct_label] = correct_text
month = str(item.get("month") or "").strip()
item_no = item.get("no", row_idx + 1)
item_id = f"{month}:{item_no}" if month else str(item_no)
return {
"id": item_id,
"month": month,
"no": item_no,
"paper_link": str(item.get("paper_link") or "").strip(),
"theorem": str(item.get("theorem") or "").strip(),
"sketch": str(item.get("sketch") or "").strip(),
"theorem_type": _coerce_theorem_types(item.get("theorem_type")),
"question": question,
"choices": choices,
"correct_choice": {
"label": correct_label,
"text": correct_text,
},
"source_path": source_path,
}
def load_items(data_path: str) -> list[dict]:
"""Load and normalise LiveMathematicianBench items from JSON files."""
files = _iter_monthly_files(data_path)
if not files:
raise ValueError(
"LiveMathematicianBench requires data_path to be a qa_*_final.json file "
"or a directory containing monthly qa_*_final.json files."
)
items: list[dict] = []
for path in files:
raw = _load_json(path)
if not isinstance(raw, list):
raise ValueError(f"Expected JSON array in {path}, got {type(raw).__name__}")
for row_idx, item in enumerate(raw):
norm = _normalize_item(item, row_idx=row_idx, source_path=path)
if norm["question"] and norm["choices"] and norm["correct_choice"]["label"]:
items.append(norm)
if not items:
raise ValueError(f"No valid LiveMathematicianBench items loaded from {data_path}")
return items
# ── Dataloader ───────────────────────────────────────────────────────────
class LiveMathematicianBenchDataLoader(SplitDataLoader):
"""LiveMathematicianBench dataloader with per-seed choice shuffling."""
def __init__(
self,
split_dir: str = "",
data_path: str = "",
split_mode: str = "ratio",
split_ratio: str = "2:1:7",
split_seed: int = 42,
split_output_dir: str = "",
seed: int = 42,
limit: int = 0,
shuffle_choices: bool = True,
**kwargs,
) -> None:
super().__init__(
split_dir=split_dir,
data_path=data_path,
split_mode=split_mode,
split_ratio=split_ratio,
split_seed=split_seed,
split_output_dir=split_output_dir,
seed=seed,
limit=limit,
)
self.shuffle_choices = shuffle_choices
self._task_types: list[str] = []
def load_raw_items(self, data_path: str) -> list[dict]:
return load_items(data_path)
def setup(self, cfg: dict) -> None:
super().setup(cfg)
all_items = self.train_items + self.val_items + self.test_items
task_types: set[str] = set()
for item in all_items:
for name in item.get("theorem_type", []):
if name:
task_types.add(name)
self._task_types = sorted(task_types)
def get_task_types(self) -> list[str]:
return list(self._task_types)
# ── Choice shuffling ─────────────────────────────────────────────────
@staticmethod
def _item_shuffle_seed(item_id: str, seed: int) -> int:
digest = hashlib.sha256(f"{seed}:{item_id}".encode("utf-8")).hexdigest()
return int(digest[:16], 16)
def _shuffle_item_choices(self, item: dict, seed: int) -> dict:
if not self.shuffle_choices:
return {
**item,
"choices": [dict(c) for c in item["choices"]],
"correct_choice": dict(item["correct_choice"]),
}
shuffled_choices = [dict(c) for c in item["choices"]]
rng = random.Random(self._item_shuffle_seed(str(item["id"]), seed))
rng.shuffle(shuffled_choices)
original_correct = _normalize_label(item["correct_choice"]["label"])
remapped_choices: list[dict] = []
new_correct_choice = dict(item["correct_choice"])
for idx, choice in enumerate(shuffled_choices):
new_label = _CHOICE_LABELS[idx]
old_label = _normalize_label(choice["label"])
remapped_choices.append({"label": new_label, "text": choice["text"]})
if old_label == original_correct:
new_correct_choice = {"label": new_label, "text": choice["text"]}
transformed = dict(item)
transformed["choices"] = remapped_choices
transformed["correct_choice"] = new_correct_choice
return transformed
def _materialize_batch(self, items: list[dict], seed: int) -> list[dict]:
return [self._shuffle_item_choices(item, seed) for item in items]
# ── Batch construction (override for choice shuffling) ───────────────
def plan_train_epoch(
self,
*,
epoch: int,
steps_per_epoch: int,
accumulation: int,
batch_size: int,
seed: int,
**kwargs,
) -> list[BatchSpec]:
"""Build a shuffled epoch while preserving per-batch choice shuffling."""
epoch_rng = random.Random(seed + epoch * 1000)
items = list(self.train_items)
epoch_rng.shuffle(items)
total_batches = steps_per_epoch * accumulation
if total_batches <= 0:
return []
batches: list[BatchSpec] = []
cursor = 0
for batch_idx in range(total_batches):
batch_seed = seed + epoch * 1000 + batch_idx + 1
batch_items = items[cursor: cursor + batch_size]
cursor += len(batch_items)
if not batch_items and items:
refill_rng = random.Random(batch_seed)
batch_items = list(items)
refill_rng.shuffle(batch_items)
batch_items = batch_items[:batch_size]
batch_items = self._materialize_batch(batch_items, batch_seed)
batches.append(
BatchSpec(
phase="train",
split="train",
seed=batch_seed,
batch_size=len(batch_items),
payload=batch_items,
)
)
return batches
def build_train_batch(self, batch_size: int, seed: int, **kwargs) -> BatchSpec:
rng = random.Random(seed)
items = list(self.train_items)
rng.shuffle(items)
items = self._materialize_batch(items[:batch_size], seed)
return BatchSpec(
phase="train",
split="train",
seed=seed,
batch_size=len(items),
payload=items,
)
def build_eval_batch(
self,
env_num: int,
split: str,
seed: int,
**kwargs,
) -> BatchSpec:
items = self.get_split_items(split)
if env_num and env_num < len(items):
items = items[:env_num]
items = self._materialize_batch(items, seed)
return BatchSpec(
phase="eval",
split=split,
seed=seed,
batch_size=len(items),
payload=items,
)
skillopt/envs/livemathematicianbench/evaluator.py
+62
View File
@@ -0,0 +1,62 @@
"""LiveMathematicianBench evaluation helpers."""
from __future__ import annotations
import re
def extract_answer(text: str) -> str:
matches = re.findall(r"<answer>(.*?)</answer>", text, re.DOTALL | re.IGNORECASE)
if matches:
return matches[-1].strip()
lines = [ln.strip() for ln in text.strip().splitlines() if ln.strip()]
if lines:
return lines[-1]
return text.strip()
def normalize_label(text: str) -> str:
return str(text).strip().upper().rstrip(".):")
def parse_choice_label(prediction_text: str, choices: list[dict]) -> str:
answer = extract_answer(prediction_text)
label = normalize_label(answer)
valid_labels = {normalize_label(choice.get("label", "")) for choice in choices}
if label in valid_labels:
return label
answer_lower = answer.lower()
for choice in choices:
choice_label = normalize_label(choice.get("label", ""))
choice_text = str(choice.get("text", "")).strip()
if choice_text and choice_text.lower() == answer_lower:
return choice_label
first_token = normalize_label(answer.split()[0]) if answer.split() else ""
if first_token in valid_labels:
return first_token
return label
def evaluate(prediction_text: str, correct_choice: dict, choices: list[dict]) -> dict:
predicted_label = parse_choice_label(prediction_text, choices)
correct_label = normalize_label(correct_choice.get("label", ""))
predicted_text = ""
correct_text = str(correct_choice.get("text", "")).strip()
for choice in choices:
if normalize_label(choice.get("label", "")) == predicted_label:
predicted_text = str(choice.get("text", "")).strip()
break
is_correct = float(predicted_label == correct_label)
return {
"em": is_correct,
"f1": is_correct,
"sub_em": is_correct,
"predicted_answer": predicted_label or extract_answer(prediction_text),
"predicted_label": predicted_label,
"predicted_text": predicted_text,
"correct_label": correct_label,
"correct_text": correct_text,
}
skillopt/envs/livemathematicianbench/prompts/analyst_error.md
+37
View File
@@ -0,0 +1,37 @@
You are an expert failure-analysis agent for theorem-grounded mathematical multiple-choice questions.
You will be given MULTIPLE failed trajectories from a single minibatch and the current skill document.
Each trajectory includes the student's response and an evaluation result showing the predicted option
versus the correct option.
Your job is to identify COMMON reasoning failures across the batch and propose concise skill edits.
## Failure Type Categories
- **quantifier_miss**: the agent missed exact quantifiers, scope, or existence/uniqueness conditions
- **strength_mismatch**: the agent preferred a weaker or stronger statement than what was proved
- **condition_miss**: the agent ignored hypotheses, equality cases, or domain restrictions
- **option_confusion**: the agent confused similar answer choices or failed to compare them exactly
- **other**: none of the above
## Rules
1. Focus on patterns that recur across the minibatch.
2. Prefer edits that improve exact choice discrimination, not theorem-specific memorization.
3. Do not hardcode paper-specific content.
4. Only patch gaps not already covered by the skill.
Respond ONLY with a valid JSON object:
{
"batch_size": <number>,
"failure_summary": [
{"failure_type": "<type>", "count": <int>, "description": "<one-line>"}
],
"patch": {
"reasoning": "<why these edits address the common failures>",
"edits": [
{"op": "append", "content": "<markdown>"},
{"op": "insert_after", "target": "<heading/text>", "content": "<markdown>"},
{"op": "replace", "target": "<old text>", "content": "<new text>"},
{"op": "delete", "target": "<exact text to remove>"}
]
}
}
skillopt/envs/livemathematicianbench/prompts/analyst_success.md
+25
View File
@@ -0,0 +1,25 @@
You are an expert success-pattern analyst for theorem-grounded mathematical multiple-choice questions.
You will be given MULTIPLE successful trajectories from a minibatch and the current skill document.
Identify generalizable behavior patterns that are genuinely helping the agent choose the exact correct option.
## Rules
- Focus on broadly useful reasoning behaviors.
- Prefer patterns about exact comparison of options, quantifiers, and equality conditions.
- Do not add theorem-specific facts.
- "edits" may be empty if the skill already captures the useful patterns.
Respond ONLY with a valid JSON object:
{
"batch_size": <number>,
"success_patterns": ["<pattern 1>", "<pattern 2>"],
"patch": {
"reasoning": "<why these patterns matter>",
"edits": [
{"op": "append", "content": "<markdown>"},
{"op": "insert_after", "target": "<heading/text>", "content": "<markdown>"},
{"op": "replace", "target": "<old text>", "content": "<new text>"},
{"op": "delete", "target": "<exact text to remove>"}
]
}
}

Some files were not shown because too many files have changed in this diff Show More