SkillOpt v0.1.0: initial release
- Skill optimization framework with training loop analogy - 11 benchmarks, 4 model backends (Azure OpenAI, Claude, Codex, Qwen) - WebUI for browser-based training control - Pluggable architecture for extending benchmarks and backends
This commit is contained in:
CharlesYang030
@@ -0,0 +1,24 @@
|
|||||||
|
# SkillOpt Environment Variables
|
||||||
|
# Copy this file to .env and fill in your values.
|
||||||
|
|
||||||
|
# ── Azure OpenAI (required for openai_chat backend) ──────────────────
|
||||||
|
AZURE_OPENAI_ENDPOINT=https://your-resource.openai.azure.com/
|
||||||
|
AZURE_OPENAI_API_VERSION=2024-12-01-preview
|
||||||
|
# Authentication: choose one method
|
||||||
|
# Option 1: API Key
|
||||||
|
AZURE_OPENAI_API_KEY=
|
||||||
|
# Option 2: Azure CLI (set auth_mode=azure_cli in config)
|
||||||
|
# Option 3: Managed Identity (set auth_mode=managed_identity + client_id in config)
|
||||||
|
|
||||||
|
# ── OpenAI (alternative to Azure) ────────────────────────────────────
|
||||||
|
# OPENAI_API_KEY=sk-...
|
||||||
|
|
||||||
|
# ── Anthropic / Claude (for claude_chat backend) ─────────────────────
|
||||||
|
# ANTHROPIC_API_KEY=sk-ant-...
|
||||||
|
|
||||||
|
# ── Qwen Local Model (for qwen_chat backend) ────────────────────────
|
||||||
|
# QWEN_CHAT_BASE_URL=http://localhost:8000/v1
|
||||||
|
# QWEN_CHAT_MODEL=Qwen/Qwen3.5-4B
|
||||||
|
|
||||||
|
# ── Ray (optional, for distributed rollout) ──────────────────────────
|
||||||
|
# RAY_ADDRESS=auto
|
||||||
+41
@@ -0,0 +1,41 @@
|
|||||||
|
__pycache__/
|
||||||
|
*.pyc
|
||||||
|
*.egg-info/
|
||||||
|
build/
|
||||||
|
dist/
|
||||||
|
site/
|
||||||
|
|
||||||
|
data/
|
||||||
|
outputs/
|
||||||
|
logs/
|
||||||
|
external/
|
||||||
|
|
||||||
|
/BabyVision/
|
||||||
|
/MMRB/
|
||||||
|
/SpreadsheetBench/
|
||||||
|
/dl4ir-searchQA/
|
||||||
|
|
||||||
|
configs/local/
|
||||||
|
configs/**/*.local.yaml
|
||||||
|
*.local.md
|
||||||
|
*.secret.md
|
||||||
|
*.bak
|
||||||
|
|
||||||
|
.env
|
||||||
|
.secrets/
|
||||||
|
.codex_azure*/
|
||||||
|
|
||||||
|
# Internal docs (not for open-source release)
|
||||||
|
docs/ablation_plan.md
|
||||||
|
docs/ablation_paper_tables.md
|
||||||
|
docs/ablation_paper_tables.html
|
||||||
|
docs/experiment_commands.md
|
||||||
|
docs/slow_update_flowchart.md
|
||||||
|
docs/session_memory.md
|
||||||
|
docs/harness_fresh_machine_handoff.md
|
||||||
|
docs/harness_monitoring_memory.md
|
||||||
|
docs/harness_reproduction_secrets.secret.md
|
||||||
|
docs/reflact_conda_env_export.yml
|
||||||
|
docs/reflact_overview.html
|
||||||
|
docs/render_ablation_paper_tables.py
|
||||||
|
docs/让*
|
||||||
@@ -0,0 +1,43 @@
|
|||||||
|
# Contributing to SkillOpt
|
||||||
|
|
||||||
|
Thank you for your interest in contributing! SkillOpt welcomes contributions of all kinds.
|
||||||
|
|
||||||
|
## Getting Started
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git clone https://github.com/microsoft/SkillOpt.git
|
||||||
|
cd SkillOpt
|
||||||
|
pip install -e ".[dev]"
|
||||||
|
```
|
||||||
|
|
||||||
|
## How to Contribute
|
||||||
|
|
||||||
|
### 🐛 Bug Reports
|
||||||
|
Open a GitHub issue with reproduction steps, expected/actual behavior, and your config file (remove API keys).
|
||||||
|
|
||||||
|
### 🔧 Add a Benchmark
|
||||||
|
See the [guide](docs/guide/new-benchmark.md) and use the scaffold at `skillopt/envs/_template/`.
|
||||||
|
|
||||||
|
### 🤖 Add a Model Backend
|
||||||
|
See the [guide](docs/guide/new-backend.md).
|
||||||
|
|
||||||
|
### 📝 Improve Documentation
|
||||||
|
```bash
|
||||||
|
pip install -e ".[docs]"
|
||||||
|
mkdocs serve # Preview at http://localhost:8000
|
||||||
|
```
|
||||||
|
|
||||||
|
## Pull Request Process
|
||||||
|
|
||||||
|
1. Fork the repo and create a feature branch
|
||||||
|
2. Make changes and test with an existing benchmark
|
||||||
|
3. Submit a PR with a clear description
|
||||||
|
4. Ensure CI passes
|
||||||
|
|
||||||
|
## Code Style
|
||||||
|
- Follow existing patterns in the codebase
|
||||||
|
- Use type hints for function signatures
|
||||||
|
- Keep docstrings concise
|
||||||
|
|
||||||
|
## License
|
||||||
|
By contributing, you agree your contributions are licensed under the [MIT License](LICENSE).
|
||||||
@@ -0,0 +1,21 @@
|
|||||||
|
MIT License
|
||||||
|
|
||||||
|
Copyright (c) 2026 Microsoft Corporation
|
||||||
|
|
||||||
|
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||||
|
of this software and associated documentation files (the "Software"), to deal
|
||||||
|
in the Software without restriction, including without limitation the rights
|
||||||
|
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||||
|
copies of the Software, and to permit persons to whom the Software is
|
||||||
|
furnished to do so, subject to the following conditions:
|
||||||
|
|
||||||
|
The above copyright notice and this permission notice shall be included in all
|
||||||
|
copies or substantial portions of the Software.
|
||||||
|
|
||||||
|
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||||
|
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||||
|
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||||
|
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||||
|
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||||
|
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||||
|
SOFTWARE.
|
||||||
@@ -0,0 +1,421 @@
|
|||||||
|
# SkillOpt
|
||||||
|
|
||||||
|
**Executive Strategy for Self-Evolving Agent Skills**
|
||||||
|
|
||||||
|
[](LICENSE)
|
||||||
|
[](https://www.python.org/)
|
||||||
|
|
||||||
|
*Train agent skills like you train neural networks — with epochs, learning rates, and validation gates — but without touching model weights.*
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## What is SkillOpt?
|
||||||
|
|
||||||
|
SkillOpt is a framework for optimizing a natural-language **skill document** through iterative rollout, reflection, editing, and gated validation.
|
||||||
|
|
||||||
|
It does **not** fine-tune model parameters. Instead, it treats the skill document as the optimization target:
|
||||||
|
|
||||||
|
- The **student** model executes tasks with the current skill
|
||||||
|
- The **teacher** model analyzes trajectories and proposes edits
|
||||||
|
- The framework merges, ranks, applies, and validates those edits
|
||||||
|
- Only validated skill updates are kept
|
||||||
|
|
||||||
|
| Deep Learning | SkillOpt |
|
||||||
|
|---|---|
|
||||||
|
| Model weights | Skill document (Markdown) |
|
||||||
|
| Forward pass | Rollout (student executes tasks) |
|
||||||
|
| Loss computation | Reflect (teacher analyzes trajectories) |
|
||||||
|
| Gradient | Edit patches (proposed skill improvements) |
|
||||||
|
| Gradient clipping | Edit ranking & selection (`learning_rate`) |
|
||||||
|
| Weight update | Patch application to skill document |
|
||||||
|
| Validation | Gated evaluation on held-out split |
|
||||||
|
| Learning rate schedule | `lr_scheduler`: cosine, linear decay |
|
||||||
|
| Epochs | Multi-epoch training with slow update & meta skill |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Method Overview
|
||||||
|
|
||||||
|
### Optimization Target
|
||||||
|
|
||||||
|
Each run maintains a mutable markdown skill document. The framework repeatedly improves that document instead of changing model parameters.
|
||||||
|
|
||||||
|
This gives a training-style loop for prompt / policy optimization:
|
||||||
|
|
||||||
|
1. Roll out the current skill on a batch of tasks.
|
||||||
|
2. Reflect on failures and successes.
|
||||||
|
3. Merge patch proposals into a coherent candidate update.
|
||||||
|
4. Rank and select a bounded number of edits.
|
||||||
|
5. Apply those edits to produce a candidate skill.
|
||||||
|
6. Validate the candidate skill on a held-out selection split.
|
||||||
|
7. Keep the update only if the gate accepts it.
|
||||||
|
|
||||||
|
### Per-Step Pipeline
|
||||||
|
|
||||||
|
Every training step executes the following pipeline in `skillopt/engine/trainer.py`:
|
||||||
|
|
||||||
|
1. **Rollout**
|
||||||
|
The student model runs a batch of tasks using the current skill.
|
||||||
|
|
||||||
|
2. **Reflect**
|
||||||
|
The teacher analyzes minibatches of trajectories and emits raw patches.
|
||||||
|
Failure-driven and success-driven patches are tracked separately.
|
||||||
|
|
||||||
|
3. **Aggregate**
|
||||||
|
Raw patches are merged hierarchically. Metadata such as `support_count` and `source_type` is carried into the merged patch so later ranking can use it.
|
||||||
|
|
||||||
|
4. **Select**
|
||||||
|
The teacher ranks the merged edit pool and keeps up to `edit_budget` edits.
|
||||||
|
|
||||||
|
5. **Update**
|
||||||
|
The selected edits are applied to the skill document. The framework records an `edit_apply_report.json` so you can see which edits actually landed, which were skipped, and why.
|
||||||
|
|
||||||
|
6. **Evaluate / Gate**
|
||||||
|
The candidate skill is evaluated on the selection split. A candidate update is accepted only if it improves over the current selection score; a new global best is tracked separately.
|
||||||
|
|
||||||
|
### Within-Epoch Memory
|
||||||
|
|
||||||
|
Inside an epoch, the trainer maintains a step buffer containing:
|
||||||
|
|
||||||
|
- Compact failure-pattern summaries from previous steps
|
||||||
|
- Rejected edits and their score deltas
|
||||||
|
|
||||||
|
That context is fed back into later reflection calls so the teacher can avoid repeating ineffective edits and can focus on unsolved error patterns.
|
||||||
|
|
||||||
|
### Epoch-Level Mechanisms
|
||||||
|
|
||||||
|
#### Slow Update
|
||||||
|
|
||||||
|
At the end of each epoch, `slow_update` compares the previous epoch's terminal skill and current epoch's terminal skill on a sampled train subset. It then writes longitudinal guidance into a protected slow-update region inside the skill document.
|
||||||
|
|
||||||
|
This guidance is **not** blindly written through — it is converted into a candidate skill and sent through the same selection gate as step-level updates.
|
||||||
|
|
||||||
|
#### Meta Skill
|
||||||
|
|
||||||
|
`meta_skill` is teacher-side cross-epoch memory. It does not directly edit the current skill. Instead, it writes a compact memory artifact describing longer-term patterns across adjacent epochs. That memory is loaded into later reflection / merge / ranking calls as extra context.
|
||||||
|
|
||||||
|
#### Meta Reflect
|
||||||
|
|
||||||
|
`meta_reflect` runs at epoch end over the step history of the current epoch. It looks at accepted and rejected directions from the whole epoch, proposes higher-level patch edits, applies them to a meta candidate, and then sends that candidate through the same selection gate.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Quick Start
|
||||||
|
|
||||||
|
### Install
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git clone https://github.com/AgenticOpt/SkillOpt.git
|
||||||
|
cd SkillOpt
|
||||||
|
pip install -e .
|
||||||
|
```
|
||||||
|
|
||||||
|
### Configure API Credentials
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cp .env.example .env
|
||||||
|
# Edit .env with your API credentials, then:
|
||||||
|
source .env
|
||||||
|
```
|
||||||
|
|
||||||
|
**Azure OpenAI** (API key or managed identity):
|
||||||
|
```bash
|
||||||
|
export AZURE_OPENAI_ENDPOINT="https://your-resource.openai.azure.com/"
|
||||||
|
export AZURE_OPENAI_API_KEY="your-key"
|
||||||
|
# Or use managed identity: set azure_openai_auth_mode=managed_identity in config
|
||||||
|
```
|
||||||
|
|
||||||
|
**OpenAI** directly:
|
||||||
|
```bash
|
||||||
|
export OPENAI_API_KEY="sk-..."
|
||||||
|
```
|
||||||
|
|
||||||
|
**Anthropic Claude**:
|
||||||
|
```bash
|
||||||
|
export ANTHROPIC_API_KEY="sk-ant-..."
|
||||||
|
```
|
||||||
|
|
||||||
|
**Qwen (local vLLM)**:
|
||||||
|
```bash
|
||||||
|
export QWEN_CHAT_BASE_URL="http://localhost:8000/v1"
|
||||||
|
export QWEN_CHAT_MODEL="Qwen/Qwen3.5-4B"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Run Training
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python scripts/train.py --config configs/searchqa/default.yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Configuration
|
||||||
|
|
||||||
|
SkillOpt uses a hierarchical YAML configuration system. Each benchmark config inherits from `configs/_base_/default.yaml`.
|
||||||
|
|
||||||
|
### Configuration Structure
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
model:
|
||||||
|
teacher_backend: openai_chat # openai_chat | claude_chat | qwen_chat
|
||||||
|
student_backend: openai_chat # openai_chat | claude_chat | codex_exec | qwen_chat
|
||||||
|
teacher: gpt-5.5 # teacher model deployment name
|
||||||
|
student: gpt-5.5 # student model deployment name
|
||||||
|
reasoning_effort: medium # low | medium | high
|
||||||
|
|
||||||
|
train:
|
||||||
|
num_epochs: 4
|
||||||
|
batch_size: 40
|
||||||
|
seed: 42
|
||||||
|
|
||||||
|
gradient:
|
||||||
|
minibatch_size: 8 # trajectories per reflection call
|
||||||
|
analyst_workers: 16 # parallel reflection workers
|
||||||
|
use_deep_reflect: false # deep multi-turn probing
|
||||||
|
deep_reflect_failures: 4
|
||||||
|
deep_reflect_successes: 2
|
||||||
|
|
||||||
|
optimizer:
|
||||||
|
learning_rate: 4 # max edits per step (edit_budget)
|
||||||
|
min_learning_rate: 2 # min edits for decay schedulers
|
||||||
|
lr_scheduler: cosine # constant | linear | cosine | autonomous
|
||||||
|
skill_update_mode: patch # patch | rewrite_from_suggestions | full_rewrite_minibatch
|
||||||
|
use_slow_update: true
|
||||||
|
use_meta_skill: true
|
||||||
|
use_meta_reflect: false
|
||||||
|
|
||||||
|
evaluation:
|
||||||
|
use_gate: true # gated validation (always recommended)
|
||||||
|
|
||||||
|
env:
|
||||||
|
name: "" # benchmark name
|
||||||
|
skill_init: "" # path to initial skill document
|
||||||
|
split_mode: ratio # ratio | split_dir
|
||||||
|
split_ratio: "2:1:7" # train:val:test
|
||||||
|
```
|
||||||
|
|
||||||
|
### CLI Overrides
|
||||||
|
|
||||||
|
Override any config key from the command line:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python scripts/train.py \
|
||||||
|
--config configs/searchqa/default.yaml \
|
||||||
|
--cfg-options model.teacher_backend=openai_chat \
|
||||||
|
model.student_backend=codex_exec \
|
||||||
|
train.batch_size=40 \
|
||||||
|
optimizer.learning_rate=4
|
||||||
|
|
||||||
|
# Legacy flat overrides also work for common keys:
|
||||||
|
python scripts/train.py \
|
||||||
|
--config configs/searchqa/default.yaml \
|
||||||
|
--backend azure_openai \
|
||||||
|
--teacher_model gpt-5.5 \
|
||||||
|
--student_model gpt-5.5 \
|
||||||
|
--reasoning_effort medium
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Model Backends
|
||||||
|
|
||||||
|
All model access goes through the unified backend router in `skillopt/model/`.
|
||||||
|
|
||||||
|
| Backend | Use case | Config key |
|
||||||
|
|---|---|---|
|
||||||
|
| `openai_chat` | Azure OpenAI / OpenAI API | teacher / student |
|
||||||
|
| `claude_chat` | Anthropic Claude | teacher / student |
|
||||||
|
| `codex_exec` | Codex execution harness | student only |
|
||||||
|
| `qwen_chat` | Local Qwen via vLLM | teacher / student |
|
||||||
|
|
||||||
|
Separate teacher/student endpoints are supported:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
model:
|
||||||
|
teacher_backend: openai_chat
|
||||||
|
student_backend: codex_exec
|
||||||
|
teacher: gpt-5.5
|
||||||
|
student: gpt-5.5-codex
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Data Splits
|
||||||
|
|
||||||
|
SkillOpt supports two split modes:
|
||||||
|
|
||||||
|
**Ratio split** — auto-generate from raw data:
|
||||||
|
```bash
|
||||||
|
python scripts/train.py \
|
||||||
|
--config configs/searchqa/default.yaml \
|
||||||
|
--split_mode ratio \
|
||||||
|
--data_path /path/to/searchqa_data.json
|
||||||
|
```
|
||||||
|
|
||||||
|
**Pre-split directory** — consume prepared splits:
|
||||||
|
```bash
|
||||||
|
python scripts/train.py \
|
||||||
|
--config configs/searchqa/default.yaml \
|
||||||
|
--split_mode split_dir \
|
||||||
|
--split_dir /path/to/searchqa_split
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Supported Benchmarks
|
||||||
|
|
||||||
|
| Benchmark | Type | Config |
|
||||||
|
|---|---|---|
|
||||||
|
| SearchQA | QA | `configs/searchqa/default.yaml` |
|
||||||
|
| SpreadsheetBench | Code generation | `configs/spreadsheetbench/default.yaml` |
|
||||||
|
| ALFWorld | Embodied agent | `configs/alfworld/default.yaml` |
|
||||||
|
| DocVQA | Document QA | `configs/docvqa/default.yaml` |
|
||||||
|
| OfficeQA | Tool-augmented QA | `configs/officeqa/default.yaml` |
|
||||||
|
| SealQA | Tool-augmented QA | `configs/sealqa/default.yaml` |
|
||||||
|
| BabyVision | Vision QA | `configs/babyvision/default.yaml` |
|
||||||
|
| LiveMathematicianBench | Math | `configs/livemathematicianbench/default.yaml` |
|
||||||
|
| MathVerse | Multimodal math | `configs/mathverse/default.yaml` |
|
||||||
|
| MMRB | Multimodal reasoning | `configs/mmrb/default.yaml` |
|
||||||
|
| SWEBench | Software engineering | `configs/swebench/default.yaml` |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Running Training
|
||||||
|
|
||||||
|
Basic training:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python scripts/train.py --config configs/searchqa/default.yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
Exec harness (Codex student):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python scripts/train.py \
|
||||||
|
--config configs/searchqa/default.yaml \
|
||||||
|
--teacher_backend openai_chat \
|
||||||
|
--student_backend codex_exec \
|
||||||
|
--teacher_model gpt-5.5 \
|
||||||
|
--student_model gpt-5.5-codex \
|
||||||
|
--use_deep_reflect true \
|
||||||
|
--skill_update_mode rewrite_from_suggestions
|
||||||
|
```
|
||||||
|
|
||||||
|
SWEBench:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python scripts/train.py \
|
||||||
|
--config configs/swebench/default.yaml \
|
||||||
|
--cfg-options env.dataset_name=lite env.split_ratio=2:1:7
|
||||||
|
```
|
||||||
|
|
||||||
|
### Eval Only
|
||||||
|
|
||||||
|
Evaluate a specific skill without training:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python scripts/eval_only.py \
|
||||||
|
--config configs/searchqa/default.yaml \
|
||||||
|
--skill skillopt/envs/searchqa/skills/initial.md
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Output Structure
|
||||||
|
|
||||||
|
Each run writes a structured output directory:
|
||||||
|
|
||||||
|
```
|
||||||
|
outputs/<run_name>/
|
||||||
|
├── config.json # Flattened runtime config
|
||||||
|
├── history.json # Per-step history records
|
||||||
|
├── runtime_state.json # Resume state (for auto-resume)
|
||||||
|
├── best_skill.md # Current best validated skill
|
||||||
|
├── skills/skill_vXXXX.md # Skill snapshot per step
|
||||||
|
├── steps/step_XXXX/ # Per-step artifacts
|
||||||
|
│ ├── merged_patch.json
|
||||||
|
│ ├── ranked_edits.json
|
||||||
|
│ ├── candidate_skill.md
|
||||||
|
│ ├── edit_apply_report.json
|
||||||
|
│ ├── rewrite_result.json # when rewrite mode is enabled
|
||||||
|
│ └── selection_eval/
|
||||||
|
├── slow_update/epoch_XX/
|
||||||
|
├── meta_skill/epoch_XX/
|
||||||
|
└── meta_reflect/epoch_XX/
|
||||||
|
```
|
||||||
|
|
||||||
|
### Resume Behavior
|
||||||
|
|
||||||
|
The trainer resumes from `runtime_state.json` when present. That state tracks:
|
||||||
|
|
||||||
|
- Last completed step
|
||||||
|
- Current skill path and score
|
||||||
|
- Best skill path and score
|
||||||
|
- Origin tags for current and best skill
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Extending SkillOpt
|
||||||
|
|
||||||
|
### Add a New Benchmark
|
||||||
|
|
||||||
|
1. Create `skillopt/envs/<your_env>/` with:
|
||||||
|
- `adapter.py` — implements `EnvAdapter`
|
||||||
|
- `dataloader.py` — data loading logic
|
||||||
|
- `rollout.py` — student execution logic
|
||||||
|
- `skills/initial.md` — initial skill document
|
||||||
|
2. Add a config at `configs/<your_env>/default.yaml`
|
||||||
|
3. Register in `skillopt/envs/__init__.py`
|
||||||
|
|
||||||
|
See `skillopt/envs/_template/` for a scaffold.
|
||||||
|
|
||||||
|
### Add a New Model Backend
|
||||||
|
|
||||||
|
Implement a backend in `skillopt/model/` following the interface in `skillopt/model/common.py`, then register it in `skillopt/model/router.py`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## WebUI
|
||||||
|
|
||||||
|
Launch the monitoring dashboard (optional):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pip install -e ".[webui]"
|
||||||
|
python -m skillopt_webui.app
|
||||||
|
```
|
||||||
|
|
||||||
|
Provides browser-based config selection, training launch, and real-time log monitoring.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Minimal Setup
|
||||||
|
|
||||||
|
```bash
|
||||||
|
conda create -n skillopt python=3.11
|
||||||
|
conda activate skillopt
|
||||||
|
pip install -e .
|
||||||
|
```
|
||||||
|
|
||||||
|
Depending on the benchmark, you may also need:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pip install datasets gymnasium numpy
|
||||||
|
```
|
||||||
|
|
||||||
|
For SWEBench, you also need a working Docker environment plus the SWE-bench harness dependencies.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Citation
|
||||||
|
|
||||||
|
```bibtex
|
||||||
|
@article{skillopt2026,
|
||||||
|
title={SkillOpt: Executive Strategy for Self-Evolving Agent Skills},
|
||||||
|
author={SkillOpt Team},
|
||||||
|
year={2026}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## License
|
||||||
|
|
||||||
|
MIT — see [LICENSE](LICENSE).
|
||||||
@@ -0,0 +1,93 @@
|
|||||||
|
# SkillOpt default configuration — base for all environments.
|
||||||
|
# Environment configs should inherit via: _base_: default.yaml
|
||||||
|
|
||||||
|
model:
|
||||||
|
backend: azure_openai
|
||||||
|
teacher: gpt-5.5
|
||||||
|
student: gpt-5.5
|
||||||
|
teacher_backend: openai_chat
|
||||||
|
student_backend: openai_chat
|
||||||
|
reasoning_effort: medium
|
||||||
|
rewrite_reasoning_effort: ""
|
||||||
|
rewrite_max_completion_tokens: 64000
|
||||||
|
codex_exec_path: codex
|
||||||
|
codex_exec_sandbox: workspace-write
|
||||||
|
codex_exec_profile: ""
|
||||||
|
codex_exec_full_auto: false
|
||||||
|
codex_exec_reasoning_effort: none
|
||||||
|
codex_exec_use_sdk: auto
|
||||||
|
codex_exec_network_access: false
|
||||||
|
codex_exec_web_search: false
|
||||||
|
codex_exec_approval_policy: never
|
||||||
|
claude_code_exec_path: claude
|
||||||
|
claude_code_exec_profile: ""
|
||||||
|
claude_code_exec_use_sdk: auto
|
||||||
|
claude_code_exec_effort: medium
|
||||||
|
claude_code_exec_max_thinking_tokens: 16384
|
||||||
|
codex_trace_to_teacher: true
|
||||||
|
azure_openai_endpoint: "" # e.g. "https://your-resource.openai.azure.com/"
|
||||||
|
azure_openai_api_version: "2024-12-01-preview"
|
||||||
|
azure_openai_api_key: "" # Fill locally if you do not export AZURE_OPENAI_API_KEY
|
||||||
|
azure_openai_auth_mode: azure_cli
|
||||||
|
azure_openai_ad_scope: "https://cognitiveservices.azure.com/.default"
|
||||||
|
azure_openai_managed_identity_client_id: ""
|
||||||
|
teacher_azure_openai_endpoint: "" # e.g. "https://your-resource.openai.azure.com/"
|
||||||
|
teacher_azure_openai_api_version: "2024-12-01-preview"
|
||||||
|
teacher_azure_openai_api_key: ""
|
||||||
|
teacher_azure_openai_auth_mode: azure_cli
|
||||||
|
teacher_azure_openai_ad_scope: "https://cognitiveservices.azure.com/.default"
|
||||||
|
teacher_azure_openai_managed_identity_client_id: ""
|
||||||
|
student_azure_openai_endpoint: "" # e.g. "https://your-resource.openai.azure.com/"
|
||||||
|
student_azure_openai_api_version: "2024-12-01-preview"
|
||||||
|
student_azure_openai_api_key: ""
|
||||||
|
student_azure_openai_auth_mode: azure_cli
|
||||||
|
student_azure_openai_ad_scope: "https://cognitiveservices.azure.com/.default"
|
||||||
|
student_azure_openai_managed_identity_client_id: ""
|
||||||
|
|
||||||
|
train:
|
||||||
|
num_epochs: 4
|
||||||
|
train_size: 0 # 0 = derive from dataset split when available
|
||||||
|
batch_size: 40
|
||||||
|
accumulation: 1
|
||||||
|
seed: 42
|
||||||
|
|
||||||
|
gradient:
|
||||||
|
minibatch_size: 8
|
||||||
|
merge_batch_size: 8
|
||||||
|
analyst_workers: 16
|
||||||
|
max_analyst_rounds: 3
|
||||||
|
failure_only: false
|
||||||
|
use_deep_reflect: false
|
||||||
|
deep_reflect_failures: 4
|
||||||
|
deep_reflect_successes: 2
|
||||||
|
|
||||||
|
optimizer:
|
||||||
|
learning_rate: 4 # max edits per step (edit_budget)
|
||||||
|
min_learning_rate: 2 # min edits for decay schedulers
|
||||||
|
lr_scheduler: cosine # constant / linear / cosine / autonomous
|
||||||
|
lr_control_mode: fixed # fixed / autonomous / none
|
||||||
|
skill_update_mode: patch # patch / rewrite_from_suggestions / full_rewrite_minibatch
|
||||||
|
use_meta_reflect: false
|
||||||
|
meta_learning_rate: 4 # max edits per epoch-level meta-reflect
|
||||||
|
use_slow_update: true
|
||||||
|
slow_update_samples: 20
|
||||||
|
longitudinal_pair_policy: mixed # mixed / changed / unchanged
|
||||||
|
use_meta_skill: true
|
||||||
|
|
||||||
|
evaluation:
|
||||||
|
use_gate: true
|
||||||
|
sel_env_num: 0
|
||||||
|
test_env_num: 0
|
||||||
|
eval_test: true
|
||||||
|
|
||||||
|
env:
|
||||||
|
name: ""
|
||||||
|
skill_init: ""
|
||||||
|
split_mode: ratio # ratio = build deterministic split from data_path; split_dir = use pre-split train/val/test
|
||||||
|
split_ratio: "2:1:7" # explicit default for dataset-backed benchmarks: train:val:test
|
||||||
|
split_seed: 42
|
||||||
|
split_dir: ""
|
||||||
|
data_path: ""
|
||||||
|
split_output_dir: ""
|
||||||
|
exec_timeout: 120 # per student model/code-agent call timeout in seconds
|
||||||
|
out_root: ""
|
||||||
@@ -0,0 +1,30 @@
|
|||||||
|
_base_: ../_base_/default.yaml
|
||||||
|
|
||||||
|
train:
|
||||||
|
train_size: 0
|
||||||
|
accumulation: 1
|
||||||
|
|
||||||
|
gradient:
|
||||||
|
minibatch_size: 8
|
||||||
|
merge_batch_size: 8
|
||||||
|
|
||||||
|
optimizer:
|
||||||
|
learning_rate: 4
|
||||||
|
use_meta_reflect: false
|
||||||
|
|
||||||
|
evaluation:
|
||||||
|
sel_env_num: 0
|
||||||
|
test_env_num: 0
|
||||||
|
|
||||||
|
env:
|
||||||
|
name: alfworld
|
||||||
|
skill_init: skillopt/envs/alfworld/skills/initial.md
|
||||||
|
split_mode: split_dir
|
||||||
|
split_ratio: "2:1:7"
|
||||||
|
split_dir: data/ablation_splits/alfworld/2-1-7_seed42
|
||||||
|
data_path: ""
|
||||||
|
split_output_dir: ""
|
||||||
|
max_steps: 50
|
||||||
|
workers: 8
|
||||||
|
max_api_workers: 8
|
||||||
|
limit: 0
|
||||||
@@ -0,0 +1,21 @@
|
|||||||
|
_base_: ../_base_/default.yaml
|
||||||
|
|
||||||
|
train:
|
||||||
|
batch_size: 64
|
||||||
|
accumulation: 1
|
||||||
|
|
||||||
|
env:
|
||||||
|
name: babyvision
|
||||||
|
skill_init: skillopt/envs/babyvision/skills/initial.md
|
||||||
|
split_mode: ratio
|
||||||
|
split_ratio: "2:1:7"
|
||||||
|
split_dir: ""
|
||||||
|
data_path: ""
|
||||||
|
split_output_dir: ""
|
||||||
|
max_turns: 1
|
||||||
|
workers: 16
|
||||||
|
limit: 0
|
||||||
|
image_detail: auto
|
||||||
|
judge_model: gpt-5.4
|
||||||
|
judge_max_completion_tokens: 256
|
||||||
|
judge_retries: 5
|
||||||
@@ -0,0 +1,28 @@
|
|||||||
|
_base_: ../_base_/default.yaml
|
||||||
|
|
||||||
|
model:
|
||||||
|
reasoning_effort: medium
|
||||||
|
|
||||||
|
train:
|
||||||
|
batch_size: 40
|
||||||
|
accumulation: 1
|
||||||
|
|
||||||
|
gradient:
|
||||||
|
minibatch_size: 8
|
||||||
|
merge_batch_size: 8
|
||||||
|
|
||||||
|
optimizer:
|
||||||
|
learning_rate: 4
|
||||||
|
|
||||||
|
env:
|
||||||
|
name: docvqa
|
||||||
|
skill_init: skillopt/envs/docvqa/skills/initial.md
|
||||||
|
split_mode: split_dir
|
||||||
|
split_ratio: "2:1:7"
|
||||||
|
split_dir: data/docvqa/splits
|
||||||
|
data_path: ""
|
||||||
|
split_output_dir: ""
|
||||||
|
max_turns: 1
|
||||||
|
workers: 16
|
||||||
|
image_detail: auto
|
||||||
|
limit: 0
|
||||||
@@ -0,0 +1,22 @@
|
|||||||
|
_base_: ../_base_/default.yaml
|
||||||
|
|
||||||
|
train:
|
||||||
|
train_size: 0
|
||||||
|
batch_size: 40
|
||||||
|
accumulation: 1
|
||||||
|
|
||||||
|
env:
|
||||||
|
name: livemathematicianbench
|
||||||
|
skill_init: skillopt/envs/livemathematicianbench/skills/initial.md
|
||||||
|
split_mode: split_dir
|
||||||
|
split_ratio: "2:1:7"
|
||||||
|
split_dir: data/ablation_splits/livemathematicianbench/2-1-7_seed42
|
||||||
|
data_path: ""
|
||||||
|
split_output_dir: ""
|
||||||
|
max_turns: 1
|
||||||
|
exec_timeout: 300
|
||||||
|
workers: 64
|
||||||
|
limit: 0
|
||||||
|
shuffle_choices: true
|
||||||
|
use_theorem: false
|
||||||
|
use_sketch: false
|
||||||
@@ -0,0 +1,23 @@
|
|||||||
|
_base_: ../_base_/default.yaml
|
||||||
|
|
||||||
|
model:
|
||||||
|
codex_exec_sandbox: danger-full-access
|
||||||
|
|
||||||
|
train:
|
||||||
|
batch_size: 64
|
||||||
|
accumulation: 1
|
||||||
|
|
||||||
|
env:
|
||||||
|
name: mathverse
|
||||||
|
skill_init: skillopt/envs/mathverse/skills/initial.md
|
||||||
|
split_dir: ""
|
||||||
|
data_root: data/MathVerse
|
||||||
|
problem_version: Text Lite
|
||||||
|
use_text_dominant_reference: false
|
||||||
|
max_turns: 1
|
||||||
|
workers: 16
|
||||||
|
limit: 0
|
||||||
|
image_detail: auto
|
||||||
|
judge_model: gpt-5.4
|
||||||
|
judge_max_completion_tokens: 256
|
||||||
|
judge_retries: 5
|
||||||
@@ -0,0 +1,18 @@
|
|||||||
|
_base_: ../_base_/default.yaml
|
||||||
|
|
||||||
|
train:
|
||||||
|
batch_size: 128
|
||||||
|
accumulation: 1
|
||||||
|
|
||||||
|
env:
|
||||||
|
name: mmrb
|
||||||
|
skill_init: skillopt/envs/mmrb/skills/initial.md
|
||||||
|
split_mode: ratio
|
||||||
|
split_ratio: "2:1:7"
|
||||||
|
split_dir: ""
|
||||||
|
data_path: ""
|
||||||
|
split_output_dir: ""
|
||||||
|
max_turns: 1
|
||||||
|
workers: 16
|
||||||
|
limit: 0
|
||||||
|
image_detail: auto
|
||||||
@@ -0,0 +1,34 @@
|
|||||||
|
_base_: ../_base_/default.yaml
|
||||||
|
|
||||||
|
model:
|
||||||
|
reasoning_effort: medium
|
||||||
|
|
||||||
|
train:
|
||||||
|
batch_size: 40
|
||||||
|
accumulation: 1
|
||||||
|
|
||||||
|
gradient:
|
||||||
|
minibatch_size: 8
|
||||||
|
merge_batch_size: 8
|
||||||
|
|
||||||
|
optimizer:
|
||||||
|
learning_rate: 4
|
||||||
|
|
||||||
|
env:
|
||||||
|
name: officeqa
|
||||||
|
skill_init: skillopt/envs/officeqa/skills/initial.md
|
||||||
|
split_mode: split_dir
|
||||||
|
split_dir: data/officeqa_split
|
||||||
|
data_dirs:
|
||||||
|
- data/officeqa_docs_official
|
||||||
|
workers: 4
|
||||||
|
max_tool_turns: 24
|
||||||
|
max_completion_tokens: 10000
|
||||||
|
search_mode: offline
|
||||||
|
max_queries_per_turn: 4
|
||||||
|
search_api_url: http://apisix.westus2.cloudapp.azure.com/search_tool/search
|
||||||
|
search_auth_env: OFFICEQA_CUSTOM_SEARCH_AUTH
|
||||||
|
search_provider: duckduckgo
|
||||||
|
search_max_num_results: 4
|
||||||
|
search_timeout_seconds: 20
|
||||||
|
limit: 0
|
||||||
@@ -0,0 +1,23 @@
|
|||||||
|
_base_: ../_base_/default.yaml
|
||||||
|
|
||||||
|
model:
|
||||||
|
reasoning_effort: medium
|
||||||
|
|
||||||
|
train:
|
||||||
|
batch_size: 10
|
||||||
|
accumulation: 1
|
||||||
|
|
||||||
|
gradient:
|
||||||
|
minibatch_size: 8
|
||||||
|
merge_batch_size: 8
|
||||||
|
|
||||||
|
optimizer:
|
||||||
|
learning_rate: 4
|
||||||
|
|
||||||
|
env:
|
||||||
|
name: sealqa
|
||||||
|
skill_init: skillopt/envs/sealqa/skills/initial.md
|
||||||
|
split_dir: data/sealqa_split
|
||||||
|
workers: 4
|
||||||
|
max_tool_turns: 12
|
||||||
|
limit: 0
|
||||||
@@ -0,0 +1,32 @@
|
|||||||
|
_base_: ../_base_/default.yaml
|
||||||
|
|
||||||
|
model:
|
||||||
|
reasoning_effort: medium
|
||||||
|
|
||||||
|
train:
|
||||||
|
train_size: 400
|
||||||
|
batch_size: 40
|
||||||
|
accumulation: 1
|
||||||
|
|
||||||
|
gradient:
|
||||||
|
minibatch_size: 8
|
||||||
|
merge_batch_size: 8
|
||||||
|
|
||||||
|
optimizer:
|
||||||
|
learning_rate: 4
|
||||||
|
|
||||||
|
evaluation:
|
||||||
|
sel_env_num: 0
|
||||||
|
test_env_num: 0
|
||||||
|
|
||||||
|
env:
|
||||||
|
name: searchqa
|
||||||
|
skill_init: skillopt/envs/searchqa/skills/initial.md
|
||||||
|
split_mode: split_dir
|
||||||
|
split_ratio: "2:1:7"
|
||||||
|
split_dir: data/searchqa_split
|
||||||
|
data_path: ""
|
||||||
|
split_output_dir: ""
|
||||||
|
max_turns: 1
|
||||||
|
workers: 24
|
||||||
|
limit: 0
|
||||||
@@ -0,0 +1,34 @@
|
|||||||
|
_base_: ../_base_/default.yaml
|
||||||
|
|
||||||
|
model:
|
||||||
|
reasoning_effort: medium
|
||||||
|
|
||||||
|
train:
|
||||||
|
train_size: 80
|
||||||
|
batch_size: 40
|
||||||
|
accumulation: 1
|
||||||
|
|
||||||
|
gradient:
|
||||||
|
minibatch_size: 8
|
||||||
|
merge_batch_size: 8
|
||||||
|
|
||||||
|
optimizer:
|
||||||
|
learning_rate: 4
|
||||||
|
|
||||||
|
evaluation:
|
||||||
|
sel_env_num: 0
|
||||||
|
test_env_num: 0
|
||||||
|
|
||||||
|
env:
|
||||||
|
name: spreadsheetbench
|
||||||
|
skill_init: skillopt/envs/spreadsheetbench/skills/initial.md
|
||||||
|
split_mode: split_dir
|
||||||
|
split_ratio: "2:1:7"
|
||||||
|
split_dir: data/spreadsheetbench_split
|
||||||
|
data_path: ""
|
||||||
|
split_output_dir: ""
|
||||||
|
data_root: data/spreadsheetbench_verified_400
|
||||||
|
mode: multi
|
||||||
|
max_turns: 30
|
||||||
|
exec_timeout: 600
|
||||||
|
workers: 24
|
||||||
@@ -0,0 +1,36 @@
|
|||||||
|
_base_: ../_base_/default.yaml
|
||||||
|
|
||||||
|
model:
|
||||||
|
reasoning_effort: medium
|
||||||
|
|
||||||
|
train:
|
||||||
|
batch_size: 20
|
||||||
|
accumulation: 1
|
||||||
|
|
||||||
|
gradient:
|
||||||
|
minibatch_size: 4
|
||||||
|
merge_batch_size: 8
|
||||||
|
|
||||||
|
optimizer:
|
||||||
|
learning_rate: 4
|
||||||
|
|
||||||
|
evaluation:
|
||||||
|
sel_env_num: 0
|
||||||
|
test_env_num: 0
|
||||||
|
|
||||||
|
env:
|
||||||
|
name: swebench
|
||||||
|
skill_init: skillopt/envs/swebench/skills/initial.md
|
||||||
|
split_mode: ratio
|
||||||
|
split_ratio: "2:1:7"
|
||||||
|
split_dir: ""
|
||||||
|
data_path: ""
|
||||||
|
split_output_dir: ""
|
||||||
|
dataset_name: lite
|
||||||
|
hf_split: test
|
||||||
|
workers: 8
|
||||||
|
eval_workers: 8
|
||||||
|
step_limit: 50
|
||||||
|
cost_limit: 3.0
|
||||||
|
timeout_per_instance: 600
|
||||||
|
limit: 0
|
||||||
@@ -0,0 +1,69 @@
|
|||||||
|
# Contributing to SkillOpt
|
||||||
|
|
||||||
|
Thank you for your interest in contributing to SkillOpt! This guide covers how to get started.
|
||||||
|
|
||||||
|
## Development Setup
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git clone https://github.com/microsoft/SkillOpt.git
|
||||||
|
cd SkillOpt
|
||||||
|
pip install -e ".[dev]"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Ways to Contribute
|
||||||
|
|
||||||
|
### 🐛 Bug Reports
|
||||||
|
|
||||||
|
Open an issue with:
|
||||||
|
- Steps to reproduce
|
||||||
|
- Expected vs actual behavior
|
||||||
|
- Config file used (sanitize API keys)
|
||||||
|
- Python version and OS
|
||||||
|
|
||||||
|
### 🔧 New Benchmark
|
||||||
|
|
||||||
|
See [Add a New Benchmark](guide/new-benchmark.md) for the implementation guide.
|
||||||
|
|
||||||
|
**Checklist:**
|
||||||
|
- [ ] Data loader in `skillopt/envs/<benchmark>/loader.py`
|
||||||
|
- [ ] Environment adapter in `skillopt/envs/<benchmark>/env.py`
|
||||||
|
- [ ] Config file in `configs/<benchmark>/default.yaml`
|
||||||
|
- [ ] Registration in `skillopt/envs/__init__.py`
|
||||||
|
- [ ] Documentation page in `docs/`
|
||||||
|
|
||||||
|
### 🤖 New Model Backend
|
||||||
|
|
||||||
|
See [Add a New Model Backend](guide/new-backend.md) for the implementation guide.
|
||||||
|
|
||||||
|
**Checklist:**
|
||||||
|
- [ ] Backend in `skillopt/model/<backend>.py`
|
||||||
|
- [ ] Registration in `skillopt/model/__init__.py`
|
||||||
|
- [ ] API key entry in `.env.example`
|
||||||
|
- [ ] Documentation update
|
||||||
|
|
||||||
|
### 📝 Documentation
|
||||||
|
|
||||||
|
Documentation is built with MkDocs Material:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pip install -e ".[docs]"
|
||||||
|
mkdocs serve # Preview at http://localhost:8000
|
||||||
|
```
|
||||||
|
|
||||||
|
## Code Style
|
||||||
|
|
||||||
|
- Follow existing patterns in the codebase
|
||||||
|
- Use type hints for function signatures
|
||||||
|
- Keep docstrings concise
|
||||||
|
|
||||||
|
## Pull Request Process
|
||||||
|
|
||||||
|
1. Fork the repository
|
||||||
|
2. Create a feature branch: `git checkout -b feature/my-benchmark`
|
||||||
|
3. Make your changes
|
||||||
|
4. Test with an existing benchmark config
|
||||||
|
5. Submit a PR with a clear description
|
||||||
|
|
||||||
|
## License
|
||||||
|
|
||||||
|
By contributing, you agree that your contributions will be licensed under the MIT License.
|
||||||
@@ -0,0 +1,109 @@
|
|||||||
|
# Configuration Guide
|
||||||
|
|
||||||
|
SkillOpt uses YAML configuration files with a hierarchical override system.
|
||||||
|
|
||||||
|
## Config Structure
|
||||||
|
|
||||||
|
```
|
||||||
|
configs/
|
||||||
|
├── _base_/
|
||||||
|
│ └── default.yaml # Global defaults
|
||||||
|
├── searchqa/
|
||||||
|
│ └── default.yaml # SearchQA overrides
|
||||||
|
├── docvqa/
|
||||||
|
│ └── default.yaml # DocVQA overrides
|
||||||
|
└── alfworld/
|
||||||
|
└── default.yaml # ALFWorld overrides
|
||||||
|
```
|
||||||
|
|
||||||
|
Benchmark configs inherit from `_base_/default.yaml` and override specific values.
|
||||||
|
|
||||||
|
## Key Parameters
|
||||||
|
|
||||||
|
### Model
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
model:
|
||||||
|
backend: azure_openai # azure_openai | openai_chat | claude_code_exec | qwen
|
||||||
|
teacher: gpt-5.5 # Teacher model (for reflection)
|
||||||
|
student: gpt-5.5 # Student model (for rollout)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Training
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
train:
|
||||||
|
num_epochs: 4 # Number of training epochs
|
||||||
|
batch_size: 40 # Tasks per step (batch size)
|
||||||
|
accumulation: 1 # Gradient accumulation
|
||||||
|
seed: 42
|
||||||
|
```
|
||||||
|
|
||||||
|
### Gradient (Reflection)
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
gradient:
|
||||||
|
minibatch_size: 8 # Reflect minibatch size
|
||||||
|
analyst_workers: 16 # Parallel reflection workers
|
||||||
|
max_analyst_rounds: 3 # Max rounds of analyst reflection
|
||||||
|
failure_only: false # Only reflect on failures
|
||||||
|
```
|
||||||
|
|
||||||
|
### Optimizer
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
optimizer:
|
||||||
|
learning_rate: 4 # Max edits per step (edit budget)
|
||||||
|
min_learning_rate: 2 # Min edits for decay schedulers
|
||||||
|
lr_scheduler: cosine # constant | linear | cosine | autonomous
|
||||||
|
use_slow_update: true # Momentum-like blending at epoch boundary
|
||||||
|
slow_update_samples: 20 # Samples for slow update evaluation
|
||||||
|
use_meta_skill: true # Cross-epoch strategy memory
|
||||||
|
```
|
||||||
|
|
||||||
|
### Evaluation
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
evaluation:
|
||||||
|
use_gate: true # Validation gating (accept/reject updates)
|
||||||
|
eval_test: true # Run test evaluation after training
|
||||||
|
```
|
||||||
|
|
||||||
|
### Environment (Data)
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
env:
|
||||||
|
name: searchqa # Benchmark name
|
||||||
|
split_mode: ratio # ratio | split_dir
|
||||||
|
split_ratio: "2:1:7" # train:val:test ratio
|
||||||
|
data_path: "" # Path to dataset
|
||||||
|
exec_timeout: 120 # Per-task timeout (seconds)
|
||||||
|
```
|
||||||
|
|
||||||
|
## CLI Overrides
|
||||||
|
|
||||||
|
Override any config value from the command line:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python scripts/train.py \
|
||||||
|
--config configs/searchqa/default.yaml \
|
||||||
|
optimizer.learning_rate=16 \
|
||||||
|
optimizer.lr_scheduler=linear \
|
||||||
|
gradient.analyst_workers=8
|
||||||
|
```
|
||||||
|
|
||||||
|
## Environment Variables
|
||||||
|
|
||||||
|
Model credentials are loaded from environment variables:
|
||||||
|
|
||||||
|
| Variable | Backend | Description |
|
||||||
|
|---|---|---|
|
||||||
|
| `AZURE_OPENAI_ENDPOINT` | azure_openai | Azure resource endpoint |
|
||||||
|
| `AZURE_OPENAI_API_KEY` | azure_openai | Azure API key |
|
||||||
|
| `OPENAI_API_KEY` | openai | OpenAI API key |
|
||||||
|
| `ANTHROPIC_API_KEY` | claude | Anthropic API key |
|
||||||
|
| `QWEN_API_BASE` | qwen | Local Qwen vLLM endpoint |
|
||||||
|
|
||||||
|
## Full Reference
|
||||||
|
|
||||||
|
See [Configuration Reference](../reference/config.md) for the complete parameter list.
|
||||||
@@ -0,0 +1,51 @@
|
|||||||
|
# Deep Learning ↔ SkillOpt Analogy
|
||||||
|
|
||||||
|
SkillOpt is designed around a core insight: **optimizing natural-language prompts follows the same structure as training neural networks**. This page maps every DL concept to its SkillOpt counterpart.
|
||||||
|
|
||||||
|
## Complete Mapping
|
||||||
|
|
||||||
|
| Deep Learning | SkillOpt | Description |
|
||||||
|
|---|---|---|
|
||||||
|
| **Model weights** | Skill document (Markdown) | The thing being optimized |
|
||||||
|
| **Forward pass** | Rollout | Student executes tasks using current skill |
|
||||||
|
| **Loss function** | Task evaluator | Scores task execution quality |
|
||||||
|
| **Backpropagation** | Reflect | Teacher analyzes failures → edit patches |
|
||||||
|
| **Gradients** | Edit patches | Proposed changes to the skill |
|
||||||
|
| **Gradient aggregation** | Patch aggregation | Merge similar edits |
|
||||||
|
| **Gradient clipping** | Edit selection | Cap max edits per step |
|
||||||
|
| **Learning rate** | `learning_rate` | Max number of edits applied per step |
|
||||||
|
| **LR scheduler** | `lr_scheduler` | Decay schedule: cosine, linear, constant |
|
||||||
|
| **SGD step** | Skill update | Apply selected patches to document |
|
||||||
|
| **Validation set** | Selection split | Gate checks improvement before accepting |
|
||||||
|
| **Early stopping** | Gate patience | Reject updates that don't improve |
|
||||||
|
| **Training step** | Step | One rollout → reflect → update cycle |
|
||||||
|
| **Epoch** | Epoch | Full pass with slow update + meta memory |
|
||||||
|
| **Momentum** | Slow update | Longitudinal comparison at epoch boundary |
|
||||||
|
| **Meta-learning** | Meta skill | Cross-epoch teacher strategy memory |
|
||||||
|
| **Batch size** | `batch_size` | Tasks sampled per rollout |
|
||||||
|
| **Data parallelism** | `analyst_workers` | Parallel reflection workers |
|
||||||
|
| **Training set** | Train split | Items used for rollout |
|
||||||
|
| **Test set** | Test split | Held-out final evaluation |
|
||||||
|
| **Warm-up** | (implicit) | High LR early steps explore broadly |
|
||||||
|
| **Checkpointing** | Skill snapshots | Saved after each accepted step |
|
||||||
|
| **Transfer learning** | Seed skill / cross-benchmark init | Start from pre-trained skill |
|
||||||
|
|
||||||
|
## Why This Analogy Matters
|
||||||
|
|
||||||
|
1. **Familiar mental model**: ML practitioners immediately understand how to tune SkillOpt
|
||||||
|
2. **Principled hyperparameter search**: Grid search over `learning_rate` × `lr_scheduler` works just like in DL
|
||||||
|
3. **Proven mechanisms**: Gating ≈ validation-based selection, patience ≈ early stopping, slow update ≈ momentum — all with strong theoretical motivation
|
||||||
|
|
||||||
|
## Hyperparameter Transfer Rules
|
||||||
|
|
||||||
|
From our experiments, these DL intuitions transfer well:
|
||||||
|
|
||||||
|
!!! success "What transfers"
|
||||||
|
- **Cosine schedule > constant** — same as in DL, cosine annealing helps convergence
|
||||||
|
- **Moderate LR (4-16) > very high/low** — too few edits = slow learning, too many = noisy
|
||||||
|
- **Slow update helps** — longitudinal comparison prevents catastrophic forgetting across epochs
|
||||||
|
- **Meta skill memory improves reflection** — teacher benefits from cross-epoch strategy notes
|
||||||
|
|
||||||
|
!!! warning "What doesn't transfer"
|
||||||
|
- **Batch size ≠ better** — larger rollout batches have diminishing returns due to API costs
|
||||||
|
- **More epochs ≠ better** — skills converge faster than neural networks (2-4 epochs usually enough)
|
||||||
@@ -0,0 +1,110 @@
|
|||||||
|
# Your First Experiment
|
||||||
|
|
||||||
|
This guide walks through running a complete SkillOpt training on SearchQA.
|
||||||
|
|
||||||
|
## 1. Choose a Benchmark
|
||||||
|
|
||||||
|
SkillOpt includes ready-to-use configs for several benchmarks:
|
||||||
|
|
||||||
|
| Benchmark | Difficulty | Typical Runtime |
|
||||||
|
|---|---|---|
|
||||||
|
| SearchQA | ⭐ Easy | ~30 min |
|
||||||
|
| DocVQA | ⭐⭐ Medium | ~2 hours |
|
||||||
|
| ALFWorld | ⭐⭐⭐ Hard | ~3 hours |
|
||||||
|
|
||||||
|
We'll use **SearchQA** as it's the fastest to complete.
|
||||||
|
|
||||||
|
## 2. Configure
|
||||||
|
|
||||||
|
Review the config file:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cat configs/searchqa/default.yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
Key parameters (deep learning analogy in parentheses):
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
train:
|
||||||
|
num_epochs: 4 # (epochs)
|
||||||
|
batch_size: 40 # (batch size)
|
||||||
|
|
||||||
|
optimizer:
|
||||||
|
learning_rate: 4 # (max edits per step)
|
||||||
|
lr_scheduler: cosine # (learning rate schedule)
|
||||||
|
use_slow_update: true # (momentum at epoch boundary)
|
||||||
|
use_meta_skill: true # (cross-epoch teacher memory)
|
||||||
|
|
||||||
|
gradient:
|
||||||
|
analyst_workers: 16 # (parallel reflection workers)
|
||||||
|
|
||||||
|
evaluation:
|
||||||
|
use_gate: true # (validation gating)
|
||||||
|
```
|
||||||
|
|
||||||
|
## 3. Train
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python scripts/train.py --config configs/searchqa/default.yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
You'll see output like:
|
||||||
|
|
||||||
|
```
|
||||||
|
[Step 1/8] Rollout: 20 items, 4 workers...
|
||||||
|
[Step 1/8] Score: 0.65 → Reflect...
|
||||||
|
[Step 1/8] 6 edit patches generated
|
||||||
|
[Step 1/8] Selected 4 edits (lr=8, cosine → 7.7)
|
||||||
|
[Step 1/8] Gate: val score 0.68 > 0.65 ✓ ACCEPT
|
||||||
|
[Step 2/8] ...
|
||||||
|
```
|
||||||
|
|
||||||
|
## 4. Monitor
|
||||||
|
|
||||||
|
Training outputs are saved to `outputs/<benchmark>/<run_id>/`:
|
||||||
|
|
||||||
|
```
|
||||||
|
outputs/searchqa/2024-01-15_10-30-00/
|
||||||
|
├── steps/
|
||||||
|
│ ├── step_0001/
|
||||||
|
│ │ ├── candidate_skill.md
|
||||||
|
│ │ ├── step_record.json
|
||||||
|
│ │ └── trajectory_digest.json
|
||||||
|
│ └── step_0002/
|
||||||
|
├── slow_update/
|
||||||
|
│ └── epoch_02/
|
||||||
|
├── meta_skill/
|
||||||
|
│ └── epoch_02/
|
||||||
|
├── skills/
|
||||||
|
│ └── step_0001.md
|
||||||
|
├── best_skill.md
|
||||||
|
├── history.json
|
||||||
|
└── config.yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
## 5. Evaluate
|
||||||
|
|
||||||
|
Evaluate the best skill on the test split:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python scripts/eval_only.py \
|
||||||
|
--config configs/searchqa/default.yaml \
|
||||||
|
--skill outputs/searchqa/<run_id>/skills/best_skill.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## WebUI
|
||||||
|
|
||||||
|
Prefer a graphical interface? Launch the WebUI:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pip install -e ".[webui]"
|
||||||
|
python -m skillopt_webui.app
|
||||||
|
```
|
||||||
|
|
||||||
|
Then open `http://localhost:7860` in your browser to configure parameters and launch training.
|
||||||
|
|
||||||
|
## Next Steps
|
||||||
|
|
||||||
|
- [Understand the training loop](training-loop.md)
|
||||||
|
- [Configuration reference](../reference/config.md)
|
||||||
|
- [Add a new benchmark](new-benchmark.md)
|
||||||
@@ -0,0 +1,89 @@
|
|||||||
|
# Installation
|
||||||
|
|
||||||
|
## Requirements
|
||||||
|
|
||||||
|
- Python ≥ 3.10
|
||||||
|
- At least one model API key (Azure OpenAI, OpenAI, Anthropic, or local Qwen)
|
||||||
|
|
||||||
|
## Quick Install
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git clone https://github.com/microsoft/SkillOpt.git
|
||||||
|
cd SkillOpt
|
||||||
|
pip install -e .
|
||||||
|
```
|
||||||
|
|
||||||
|
## Optional Dependencies
|
||||||
|
|
||||||
|
Install extras for specific benchmarks or backends:
|
||||||
|
|
||||||
|
=== "ALFWorld"
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pip install -e ".[alfworld]"
|
||||||
|
```
|
||||||
|
|
||||||
|
=== "Claude Backend"
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pip install -e ".[claude]"
|
||||||
|
```
|
||||||
|
|
||||||
|
=== "Qwen (Local)"
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pip install -e ".[qwen]"
|
||||||
|
```
|
||||||
|
|
||||||
|
=== "WebUI"
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pip install -e ".[webui]"
|
||||||
|
```
|
||||||
|
|
||||||
|
=== "Development"
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pip install -e ".[dev]"
|
||||||
|
```
|
||||||
|
|
||||||
|
=== "All"
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pip install -e ".[alfworld,claude,qwen,webui,dev]"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Environment Variables
|
||||||
|
|
||||||
|
Copy the example `.env` file and fill in your credentials:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cp .env.example .env
|
||||||
|
```
|
||||||
|
|
||||||
|
Edit `.env` with your API keys:
|
||||||
|
|
||||||
|
```ini
|
||||||
|
# Azure OpenAI (default backend)
|
||||||
|
AZURE_OPENAI_ENDPOINT=https://your-resource.openai.azure.com/
|
||||||
|
AZURE_OPENAI_API_KEY=your-key
|
||||||
|
|
||||||
|
# Or use OpenAI directly
|
||||||
|
OPENAI_API_KEY=sk-...
|
||||||
|
|
||||||
|
# Or Anthropic Claude
|
||||||
|
ANTHROPIC_API_KEY=sk-ant-...
|
||||||
|
```
|
||||||
|
|
||||||
|
!!! tip
|
||||||
|
You only need credentials for the backend you plan to use. Azure OpenAI is the default.
|
||||||
|
|
||||||
|
## Verify Installation
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python -c "import skillopt; print('SkillOpt ready!')"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Next Steps
|
||||||
|
|
||||||
|
→ [Run your first experiment](first-experiment.md)
|
||||||
@@ -0,0 +1,130 @@
|
|||||||
|
# Add a New Model Backend
|
||||||
|
|
||||||
|
SkillOpt supports multiple LLM backends. This guide shows how to add your own.
|
||||||
|
|
||||||
|
## Backend Architecture
|
||||||
|
|
||||||
|
```
|
||||||
|
skillopt/model/
|
||||||
|
├── base.py # Abstract base class
|
||||||
|
├── azure_openai.py # Azure OpenAI backend
|
||||||
|
├── openai_model.py # Direct OpenAI backend
|
||||||
|
├── claude.py # Anthropic Claude backend
|
||||||
|
├── qwen.py # Local Qwen (vLLM) backend
|
||||||
|
└── your_backend.py # Your new backend
|
||||||
|
```
|
||||||
|
|
||||||
|
## Step 1: Create the Backend
|
||||||
|
|
||||||
|
Create `skillopt/model/your_backend.py`:
|
||||||
|
|
||||||
|
```python
|
||||||
|
from skillopt.model.base import ModelBackend, ModelResponse
|
||||||
|
|
||||||
|
class YourBackend(ModelBackend):
|
||||||
|
"""Your custom model backend."""
|
||||||
|
|
||||||
|
def __init__(self, cfg: dict):
|
||||||
|
super().__init__(cfg)
|
||||||
|
self.model_name = cfg.get('model_name', 'your-default-model')
|
||||||
|
self.api_key = os.environ.get('YOUR_API_KEY', '')
|
||||||
|
self.client = self._init_client()
|
||||||
|
|
||||||
|
def _init_client(self):
|
||||||
|
"""Initialize API client."""
|
||||||
|
# TODO: Set up your API client
|
||||||
|
pass
|
||||||
|
|
||||||
|
async def generate(
|
||||||
|
self,
|
||||||
|
messages: list[dict],
|
||||||
|
temperature: float = 0.7,
|
||||||
|
max_tokens: int = 4096,
|
||||||
|
**kwargs
|
||||||
|
) -> ModelResponse:
|
||||||
|
"""
|
||||||
|
Generate a completion.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
messages: Chat messages [{"role": "...", "content": "..."}]
|
||||||
|
temperature: Sampling temperature
|
||||||
|
max_tokens: Maximum tokens in response
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
ModelResponse with content, usage, and metadata
|
||||||
|
"""
|
||||||
|
response = await self.client.chat(
|
||||||
|
model=self.model_name,
|
||||||
|
messages=messages,
|
||||||
|
temperature=temperature,
|
||||||
|
max_tokens=max_tokens,
|
||||||
|
)
|
||||||
|
|
||||||
|
return ModelResponse(
|
||||||
|
content=response.text,
|
||||||
|
usage={
|
||||||
|
'prompt_tokens': response.usage.input,
|
||||||
|
'completion_tokens': response.usage.output,
|
||||||
|
},
|
||||||
|
model=self.model_name,
|
||||||
|
)
|
||||||
|
|
||||||
|
async def generate_with_tools(
|
||||||
|
self,
|
||||||
|
messages: list[dict],
|
||||||
|
tools: list[dict],
|
||||||
|
**kwargs
|
||||||
|
) -> ModelResponse:
|
||||||
|
"""Generate with tool/function calling support."""
|
||||||
|
# Optional: implement if your model supports tool use
|
||||||
|
raise NotImplementedError("Tool use not supported")
|
||||||
|
```
|
||||||
|
|
||||||
|
## Step 2: Register the Backend
|
||||||
|
|
||||||
|
Add to `skillopt/model/__init__.py`:
|
||||||
|
|
||||||
|
```python
|
||||||
|
from .your_backend import YourBackend
|
||||||
|
|
||||||
|
BACKEND_REGISTRY = {
|
||||||
|
# ... existing backends ...
|
||||||
|
'your_backend': YourBackend,
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Step 3: Configure
|
||||||
|
|
||||||
|
Use your backend in any config:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
model:
|
||||||
|
backend: your_backend
|
||||||
|
model_name: your-model-id
|
||||||
|
temperature: 0.7
|
||||||
|
max_tokens: 4096
|
||||||
|
```
|
||||||
|
|
||||||
|
Set credentials via environment variable:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
export YOUR_API_KEY="your-key"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Required Interface
|
||||||
|
|
||||||
|
Your backend must implement these methods:
|
||||||
|
|
||||||
|
| Method | Required | Description |
|
||||||
|
|---|---|---|
|
||||||
|
| `generate()` | ✅ | Basic text generation |
|
||||||
|
| `generate_with_tools()` | Optional | Tool/function calling |
|
||||||
|
| `count_tokens()` | Optional | Token counting for context management |
|
||||||
|
|
||||||
|
## Tips
|
||||||
|
|
||||||
|
!!! tip
|
||||||
|
- Test your backend with `python -c "from skillopt.model.your_backend import YourBackend"` first
|
||||||
|
- Use `async` methods for all API calls — SkillOpt uses asyncio throughout
|
||||||
|
- Implement retry logic with exponential backoff for production use
|
||||||
|
- Add your API key to `.env.example` when submitting a PR
|
||||||
@@ -0,0 +1,181 @@
|
|||||||
|
# Add a New Benchmark
|
||||||
|
|
||||||
|
Extend SkillOpt with your own benchmark in ~100 lines of code.
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
To add a benchmark, you need:
|
||||||
|
|
||||||
|
1. **Data Loader** — Loads and splits your dataset
|
||||||
|
2. **Environment Adapter** — Executes tasks and returns scores
|
||||||
|
3. **Config** — YAML configuration file
|
||||||
|
|
||||||
|
## Step 1: Create the Benchmark Package
|
||||||
|
|
||||||
|
```bash
|
||||||
|
mkdir -p skillopt/envs/my_benchmark
|
||||||
|
touch skillopt/envs/my_benchmark/__init__.py
|
||||||
|
```
|
||||||
|
|
||||||
|
## Step 2: Implement the Data Loader
|
||||||
|
|
||||||
|
Create `skillopt/envs/my_benchmark/loader.py`:
|
||||||
|
|
||||||
|
```python
|
||||||
|
from skillopt.data.base import DataLoader, DataItem
|
||||||
|
|
||||||
|
class MyBenchmarkDataLoader(DataLoader):
|
||||||
|
"""Load and split your benchmark data."""
|
||||||
|
|
||||||
|
def __init__(self, data_dir: str, **kwargs):
|
||||||
|
super().__init__(**kwargs)
|
||||||
|
self.data_dir = data_dir
|
||||||
|
|
||||||
|
def setup(self, cfg: dict):
|
||||||
|
"""Initialize splits based on config."""
|
||||||
|
self.split_mode = cfg.get('split_mode', 'ratio')
|
||||||
|
# Load your data here
|
||||||
|
self.items = self._load_items()
|
||||||
|
self._create_splits(cfg)
|
||||||
|
|
||||||
|
def _load_items(self) -> list[DataItem]:
|
||||||
|
"""Load raw data into DataItem objects."""
|
||||||
|
items = []
|
||||||
|
# TODO: Load your data
|
||||||
|
for entry in your_data:
|
||||||
|
items.append(DataItem(
|
||||||
|
id=entry['id'],
|
||||||
|
input=entry['question'],
|
||||||
|
ground_truth=entry['answer'],
|
||||||
|
metadata=entry.get('metadata', {})
|
||||||
|
))
|
||||||
|
return items
|
||||||
|
|
||||||
|
def get_split_items(self, split: str) -> list[DataItem]:
|
||||||
|
"""Return items for a given split (train/valid/test)."""
|
||||||
|
return self.splits[split]
|
||||||
|
```
|
||||||
|
|
||||||
|
## Step 3: Implement the Environment Adapter
|
||||||
|
|
||||||
|
Create `skillopt/envs/my_benchmark/env.py`:
|
||||||
|
|
||||||
|
```python
|
||||||
|
from skillopt.envs.base import EnvAdapter, TaskResult
|
||||||
|
|
||||||
|
class MyBenchmarkEnv(EnvAdapter):
|
||||||
|
"""Execute tasks and evaluate results."""
|
||||||
|
|
||||||
|
def __init__(self, cfg: dict):
|
||||||
|
super().__init__(cfg)
|
||||||
|
|
||||||
|
async def execute(self, item: DataItem, skill: str, model) -> TaskResult:
|
||||||
|
"""
|
||||||
|
Execute a single task.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
item: The data item to process
|
||||||
|
skill: Current skill document content
|
||||||
|
model: The student model instance
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
TaskResult with prediction, score, and trajectory
|
||||||
|
"""
|
||||||
|
# Build prompt with skill document
|
||||||
|
prompt = self.build_prompt(item, skill)
|
||||||
|
|
||||||
|
# Get model response
|
||||||
|
response = await model.generate(prompt)
|
||||||
|
|
||||||
|
# Extract prediction
|
||||||
|
prediction = self.parse_response(response)
|
||||||
|
|
||||||
|
# Score against ground truth
|
||||||
|
score = self.evaluate(prediction, item.ground_truth)
|
||||||
|
|
||||||
|
return TaskResult(
|
||||||
|
item_id=item.id,
|
||||||
|
prediction=prediction,
|
||||||
|
score=score,
|
||||||
|
trajectory=[
|
||||||
|
{"role": "system", "content": skill},
|
||||||
|
{"role": "user", "content": item.input},
|
||||||
|
{"role": "assistant", "content": response}
|
||||||
|
]
|
||||||
|
)
|
||||||
|
|
||||||
|
def evaluate(self, prediction: str, ground_truth: str) -> float:
|
||||||
|
"""
|
||||||
|
Score a prediction against ground truth.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Float between 0.0 and 1.0
|
||||||
|
"""
|
||||||
|
# TODO: Implement your scoring logic
|
||||||
|
# Examples: exact match, F1, ANLS, etc.
|
||||||
|
return float(prediction.strip() == ground_truth.strip())
|
||||||
|
|
||||||
|
def build_prompt(self, item, skill: str) -> str:
|
||||||
|
"""Combine skill document with task input."""
|
||||||
|
return f"{skill}\n\n---\n\nQuestion: {item.input}"
|
||||||
|
|
||||||
|
def parse_response(self, response: str) -> str:
|
||||||
|
"""Extract the answer from model response."""
|
||||||
|
return response.strip()
|
||||||
|
```
|
||||||
|
|
||||||
|
## Step 4: Register the Benchmark
|
||||||
|
|
||||||
|
Add to `skillopt/envs/__init__.py`:
|
||||||
|
|
||||||
|
```python
|
||||||
|
from .my_benchmark.env import MyBenchmarkEnv
|
||||||
|
from .my_benchmark.loader import MyBenchmarkDataLoader
|
||||||
|
|
||||||
|
BENCHMARK_REGISTRY = {
|
||||||
|
# ... existing benchmarks ...
|
||||||
|
'my_benchmark': {
|
||||||
|
'env': MyBenchmarkEnv,
|
||||||
|
'loader': MyBenchmarkDataLoader,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Step 5: Create Config
|
||||||
|
|
||||||
|
Create `configs/my_benchmark/default.yaml`:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
_base_: ['../_base_/default.yaml']
|
||||||
|
|
||||||
|
env:
|
||||||
|
name: my_benchmark
|
||||||
|
data_path: data/my_benchmark
|
||||||
|
split_mode: ratio
|
||||||
|
split_ratio: "2:1:7"
|
||||||
|
|
||||||
|
train:
|
||||||
|
num_epochs: 4
|
||||||
|
batch_size: 40
|
||||||
|
|
||||||
|
optimizer:
|
||||||
|
learning_rate: 4
|
||||||
|
lr_scheduler: cosine
|
||||||
|
use_slow_update: true
|
||||||
|
use_meta_skill: true
|
||||||
|
|
||||||
|
gradient:
|
||||||
|
analyst_workers: 16
|
||||||
|
```
|
||||||
|
|
||||||
|
## Step 6: Run
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python scripts/train.py --config configs/my_benchmark/default.yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
## Tips
|
||||||
|
|
||||||
|
!!! tip
|
||||||
|
- Use a small `batch_size` (10-20) for initial testing
|
||||||
|
- The `evaluate()` method is critical — a noisy metric will confuse the optimizer
|
||||||
@@ -0,0 +1,78 @@
|
|||||||
|
# Skill Document
|
||||||
|
|
||||||
|
A **skill document** is a Markdown file that serves as the "prompt weights" of your agent. SkillOpt trains this document through iterative optimization.
|
||||||
|
|
||||||
|
## What is a Skill Document?
|
||||||
|
|
||||||
|
A skill document is a structured set of instructions that tells a language model **how** to approach a specific type of task. It's analogous to learned weights in a neural network — encoding task-specific knowledge in natural language rather than floating-point parameters.
|
||||||
|
|
||||||
|
## Structure
|
||||||
|
|
||||||
|
A typical skill document contains:
|
||||||
|
|
||||||
|
```markdown
|
||||||
|
# Task Strategy
|
||||||
|
|
||||||
|
## General Approach
|
||||||
|
- Break complex problems into sub-steps
|
||||||
|
- Always verify intermediate results
|
||||||
|
|
||||||
|
## Common Patterns
|
||||||
|
- When you see X, try approach Y
|
||||||
|
- Avoid Z because it leads to errors
|
||||||
|
|
||||||
|
## Edge Cases
|
||||||
|
- If the input contains A, handle it specially by...
|
||||||
|
- Watch out for B — it requires C
|
||||||
|
|
||||||
|
## Output Format
|
||||||
|
- Always include reasoning before the answer
|
||||||
|
- Format numbers with proper units
|
||||||
|
```
|
||||||
|
|
||||||
|
## How It Evolves
|
||||||
|
|
||||||
|
During training, the skill document is modified by **edit patches**:
|
||||||
|
|
||||||
|
1. **Additions**: New rules or strategies discovered from failed trajectories
|
||||||
|
2. **Modifications**: Refining existing rules that are partially correct
|
||||||
|
3. **Deletions**: Removing rules that consistently lead to errors
|
||||||
|
|
||||||
|
Each edit is validated through the **gate** mechanism before being permanently accepted.
|
||||||
|
|
||||||
|
## Initial Skill
|
||||||
|
|
||||||
|
You can start training with:
|
||||||
|
|
||||||
|
- **Empty skill**: The system learns everything from scratch
|
||||||
|
- **Seed skill**: Provide initial instructions to bootstrap training
|
||||||
|
- **Pre-trained skill**: Transfer a skill from a related benchmark
|
||||||
|
|
||||||
|
Configure the initial skill in your YAML:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
train:
|
||||||
|
init_skill: "path/to/initial_skill.md" # or omit for empty
|
||||||
|
```
|
||||||
|
|
||||||
|
## Skill Quality Metrics
|
||||||
|
|
||||||
|
Track your skill's evolution through:
|
||||||
|
|
||||||
|
- **Validation score**: Primary metric on the selection split
|
||||||
|
- **Test score**: Final metric on held-out test data
|
||||||
|
- **Skill length**: Total tokens in the document
|
||||||
|
- **Edit acceptance rate**: Fraction of proposed edits that pass gating
|
||||||
|
|
||||||
|
## Best Practices
|
||||||
|
|
||||||
|
!!! tip "Tips for better skills"
|
||||||
|
1. **Start with a seed skill** (`env.skill_init`) if you have domain knowledge — it converges faster
|
||||||
|
2. **Use cosine LR schedule** — aggressive early exploration + careful late refinement
|
||||||
|
3. **Enable slow update** (`use_slow_update: true`) to prevent forgetting across epochs
|
||||||
|
4. **Enable meta skill** (`use_meta_skill: true`) so the teacher accumulates strategy memory
|
||||||
|
|
||||||
|
## Next Steps
|
||||||
|
|
||||||
|
- [Deep Learning Analogy](dl-analogy.md)
|
||||||
|
- [Configuration Reference](../reference/config.md)
|
||||||
@@ -0,0 +1,92 @@
|
|||||||
|
# The Training Loop
|
||||||
|
|
||||||
|
SkillOpt's core insight: **optimizing natural-language skill documents follows the same structure as training neural networks**.
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
```
|
||||||
|
┌─────────────────────────────────────────────────────────┐
|
||||||
|
│ Training Loop │
|
||||||
|
│ │
|
||||||
|
│ for epoch in epochs: │
|
||||||
|
│ for step in steps: │
|
||||||
|
│ 1. Rollout — Student executes tasks │
|
||||||
|
│ 2. Reflect — Teacher analyzes trajectories │
|
||||||
|
│ 3. Aggregate — Hierarchical merge of patches │
|
||||||
|
│ 4. Select — Rank & clip edits (learning rate) │
|
||||||
|
│ 5. Update — Apply patches to skill doc │
|
||||||
|
│ 6. Gate — Validate & accept/reject │
|
||||||
|
│ │
|
||||||
|
│ Epoch Boundary: │
|
||||||
|
│ • Slow Update (longitudinal comparison & guidance) │
|
||||||
|
│ • Meta Skill (cross-epoch strategy memory) │
|
||||||
|
└─────────────────────────────────────────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
## Stage Details
|
||||||
|
|
||||||
|
### 1. Rollout (Forward Pass)
|
||||||
|
|
||||||
|
The **student** model executes tasks using the current skill document as its prompt. Each task produces a trajectory and a score.
|
||||||
|
|
||||||
|
```python
|
||||||
|
# Analogy: forward pass through the network
|
||||||
|
predictions = model(input, skill_document)
|
||||||
|
scores = evaluate(predictions, ground_truth)
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2. Reflect (Backward Pass)
|
||||||
|
|
||||||
|
The **teacher** model analyzes failed trajectories and produces **edit patches** — structured suggestions for improving the skill document.
|
||||||
|
|
||||||
|
Two modes:
|
||||||
|
|
||||||
|
- **Shallow**: Analyze each trajectory independently
|
||||||
|
- **Deep**: Cross-reference multiple failures to find systemic issues
|
||||||
|
|
||||||
|
```python
|
||||||
|
# Analogy: computing gradients
|
||||||
|
gradients = loss.backward() # → edit patches
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. Aggregate
|
||||||
|
|
||||||
|
Semantically similar edit patches are merged to avoid redundant edits.
|
||||||
|
|
||||||
|
### 4. Select (Gradient Clipping)
|
||||||
|
|
||||||
|
Edits are ranked by relevance score. The `learning_rate` parameter caps how many edits are applied per step — just like gradient clipping prevents overshooting.
|
||||||
|
|
||||||
|
```python
|
||||||
|
# Analogy: gradient clipping + optimizer step size
|
||||||
|
selected = top_k(edits, k=learning_rate)
|
||||||
|
```
|
||||||
|
|
||||||
|
The `lr_scheduler` adjusts this over training:
|
||||||
|
|
||||||
|
- **cosine**: Start aggressive, taper smoothly
|
||||||
|
- **linear**: Linear decay
|
||||||
|
- **constant**: Fixed rate
|
||||||
|
|
||||||
|
### 5. Update (Parameter Update)
|
||||||
|
|
||||||
|
Selected edits are applied to the skill document, producing a new version.
|
||||||
|
|
||||||
|
### 6. Gate (Validation)
|
||||||
|
|
||||||
|
The updated skill is evaluated on a **selection split** (analogous to a validation set). The update is only accepted if performance improves.
|
||||||
|
|
||||||
|
## Epoch Boundary Mechanisms
|
||||||
|
|
||||||
|
### Slow Update
|
||||||
|
|
||||||
|
At the end of each epoch (starting from epoch 2), the system performs a **longitudinal comparison**: it rolls out both the previous epoch's skill and the current skill on the same samples, categorizes items as improved/regressed/persistent_fail/stable_success, then generates high-level **guidance** that is injected into the skill document. This prevents catastrophic forgetting of earlier improvements.
|
||||||
|
|
||||||
|
### Meta Skill
|
||||||
|
|
||||||
|
A **meta-skill memory** accumulates high-level strategy notes across the entire training run. At the end of each epoch, the teacher reflects on what changed between epochs and produces a compact memory that is provided as additional context during future reflection steps.
|
||||||
|
|
||||||
|
## Next Steps
|
||||||
|
|
||||||
|
- [Understand Skill Documents](skill-document.md)
|
||||||
|
- [DL ↔ SkillOpt analogy table](dl-analogy.md)
|
||||||
+170
@@ -0,0 +1,170 @@
|
|||||||
|
---
|
||||||
|
hide:
|
||||||
|
- navigation
|
||||||
|
---
|
||||||
|
|
||||||
|
<div class="hero" markdown>
|
||||||
|
|
||||||
|
# SkillOpt
|
||||||
|
|
||||||
|
### Train Agent Skills Like Neural Networks
|
||||||
|
|
||||||
|
*Optimize natural-language skill documents through iterative rollout, reflection, and gated validation — with epochs, learning rates, and validation gates — without touching model weights.*
|
||||||
|
|
||||||
|
[Get Started :material-rocket-launch:](guide/installation.md){ .md-button .md-button--primary }
|
||||||
|
[View on GitHub :material-github:](https://github.com/microsoft/SkillOpt){ .md-button }
|
||||||
|
|
||||||
|
</div>
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## How It Works
|
||||||
|
|
||||||
|
<div class="pipeline-container" markdown>
|
||||||
|
<div class="pipeline-wrapper">
|
||||||
|
|
||||||
|
<div class="pipeline-stage" id="stage-rollout">
|
||||||
|
<div class="stage-icon">🎯</div>
|
||||||
|
<div class="stage-label">Rollout</div>
|
||||||
|
<div class="stage-desc">Student executes tasks</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="pipeline-arrow"><div class="flow-line"></div></div>
|
||||||
|
|
||||||
|
<div class="pipeline-stage" id="stage-reflect">
|
||||||
|
<div class="stage-icon">🔍</div>
|
||||||
|
<div class="stage-label">Reflect</div>
|
||||||
|
<div class="stage-desc">Teacher analyzes trajectories</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="pipeline-arrow"><div class="flow-line"></div></div>
|
||||||
|
|
||||||
|
<div class="pipeline-stage" id="stage-aggregate">
|
||||||
|
<div class="stage-icon">🔗</div>
|
||||||
|
<div class="stage-label">Aggregate</div>
|
||||||
|
<div class="stage-desc">Merge edit patches</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="pipeline-arrow"><div class="flow-line"></div></div>
|
||||||
|
|
||||||
|
<div class="pipeline-stage" id="stage-select">
|
||||||
|
<div class="stage-icon">✂️</div>
|
||||||
|
<div class="stage-label">Select</div>
|
||||||
|
<div class="stage-desc">Rank & clip edits</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="pipeline-arrow"><div class="flow-line"></div></div>
|
||||||
|
|
||||||
|
<div class="pipeline-stage" id="stage-update">
|
||||||
|
<div class="stage-icon">📝</div>
|
||||||
|
<div class="stage-label">Update</div>
|
||||||
|
<div class="stage-desc">Apply to skill doc</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="pipeline-arrow"><div class="flow-line"></div></div>
|
||||||
|
|
||||||
|
<div class="pipeline-stage" id="stage-gate">
|
||||||
|
<div class="stage-icon">🚦</div>
|
||||||
|
<div class="stage-label">Gate</div>
|
||||||
|
<div class="stage-desc">Validate & accept</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="pipeline-epoch-bar">
|
||||||
|
<div class="epoch-mechanism">🔄 Slow Update</div>
|
||||||
|
<div class="epoch-mechanism">🧠 Meta Skill</div>
|
||||||
|
<div class="epoch-label">Epoch Boundary</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
</div>
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Deep Learning Analogy
|
||||||
|
|
||||||
|
SkillOpt brings the familiar deep-learning training paradigm to agentic prompt optimization:
|
||||||
|
|
||||||
|
| Deep Learning | SkillOpt |
|
||||||
|
|---|---|
|
||||||
|
| Model weights | Skill document (Markdown) |
|
||||||
|
| Forward pass | Rollout (student executes tasks) |
|
||||||
|
| Loss / gradient | Reflect (teacher produces edit patches) |
|
||||||
|
| Gradient clipping | Edit selection (`learning_rate` = max edits) |
|
||||||
|
| SGD step | Patch application to skill |
|
||||||
|
| Validation set | Gated evaluation on selection split |
|
||||||
|
| LR schedule | `lr_scheduler`: cosine, linear, constant |
|
||||||
|
| Epochs | Multi-epoch with slow update & meta skill memory |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Supported Benchmarks
|
||||||
|
|
||||||
|
| Benchmark | Type | Config |
|
||||||
|
|---|---|---|
|
||||||
|
| **DocVQA** | Document QA | `configs/docvqa/` |
|
||||||
|
| **ALFWorld** | Embodied AI | `configs/alfworld/` |
|
||||||
|
| **OfficeQA** | Enterprise QA | `configs/officeqa/` |
|
||||||
|
| **SearchQA** | Open-domain QA | `configs/searchqa/` |
|
||||||
|
| **LiveMathBench** | Math reasoning | `configs/livemathematicianbench/` |
|
||||||
|
| **SWEBench** | Software Engineering | `configs/swebench/` |
|
||||||
|
| + 5 more | Various | See [docs](guide/first-experiment.md) |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Quick Example
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Install
|
||||||
|
pip install -e .
|
||||||
|
|
||||||
|
# Configure credentials
|
||||||
|
export AZURE_OPENAI_ENDPOINT="https://your-resource.openai.azure.com/"
|
||||||
|
export AZURE_OPENAI_API_KEY="your-key"
|
||||||
|
|
||||||
|
# Train on SearchQA
|
||||||
|
python scripts/train.py --config configs/searchqa/default.yaml
|
||||||
|
|
||||||
|
# Evaluate best skill
|
||||||
|
python scripts/eval_only.py \
|
||||||
|
--config configs/searchqa/default.yaml \
|
||||||
|
--skill outputs/best_skill.md
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
<div class="grid cards" markdown>
|
||||||
|
|
||||||
|
- :material-book-open-variant:{ .lg .middle } **Getting Started**
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
Install SkillOpt, configure your API keys, and run your first experiment in 5 minutes.
|
||||||
|
|
||||||
|
[:octicons-arrow-right-24: Installation](guide/installation.md)
|
||||||
|
|
||||||
|
- :material-puzzle:{ .lg .middle } **Add a Benchmark**
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
Extend SkillOpt with your own benchmark in ~100 lines of code.
|
||||||
|
|
||||||
|
[:octicons-arrow-right-24: Extension Guide](guide/new-benchmark.md)
|
||||||
|
|
||||||
|
- :material-cog:{ .lg .middle } **Configuration**
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
Full reference for all hyperparameters with deep learning analogies.
|
||||||
|
|
||||||
|
[:octicons-arrow-right-24: Config Reference](reference/config.md)
|
||||||
|
|
||||||
|
- :material-monitor-dashboard:{ .lg .middle } **WebUI**
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
Configure, launch, and monitor training from your browser.
|
||||||
|
|
||||||
|
[:octicons-arrow-right-24: WebUI Guide](guide/first-experiment.md#webui)
|
||||||
|
|
||||||
|
</div>
|
||||||
@@ -0,0 +1,81 @@
|
|||||||
|
# API Reference
|
||||||
|
|
||||||
|
## Core Classes
|
||||||
|
|
||||||
|
### `EnvAdapter`
|
||||||
|
|
||||||
|
Abstract base class for benchmark environments.
|
||||||
|
|
||||||
|
```python
|
||||||
|
class EnvAdapter(ABC):
|
||||||
|
async def execute(self, item, skill, model) -> TaskResult
|
||||||
|
def evaluate(self, prediction, ground_truth) -> float
|
||||||
|
def build_prompt(self, item, skill) -> str
|
||||||
|
```
|
||||||
|
|
||||||
|
### `DataLoader`
|
||||||
|
|
||||||
|
Abstract base class for data loading and splitting.
|
||||||
|
|
||||||
|
```python
|
||||||
|
class DataLoader(ABC):
|
||||||
|
def setup(self, cfg: dict) -> None
|
||||||
|
def get_split_items(self, split: str) -> list[DataItem]
|
||||||
|
```
|
||||||
|
|
||||||
|
### `ModelBackend`
|
||||||
|
|
||||||
|
Abstract base class for LLM backends.
|
||||||
|
|
||||||
|
```python
|
||||||
|
class ModelBackend(ABC):
|
||||||
|
async def generate(self, messages, **kwargs) -> ModelResponse
|
||||||
|
async def generate_with_tools(self, messages, tools, **kwargs) -> ModelResponse
|
||||||
|
```
|
||||||
|
|
||||||
|
### `Trainer`
|
||||||
|
|
||||||
|
Main training loop orchestrator.
|
||||||
|
|
||||||
|
```python
|
||||||
|
class Trainer:
|
||||||
|
def __init__(self, cfg: dict)
|
||||||
|
async def train(self) -> TrainResult
|
||||||
|
async def evaluate(self, skill: str, split: str) -> EvalResult
|
||||||
|
```
|
||||||
|
|
||||||
|
## Data Classes
|
||||||
|
|
||||||
|
### `DataItem`
|
||||||
|
|
||||||
|
```python
|
||||||
|
@dataclass
|
||||||
|
class DataItem:
|
||||||
|
id: str
|
||||||
|
input: str
|
||||||
|
ground_truth: str
|
||||||
|
metadata: dict = field(default_factory=dict)
|
||||||
|
```
|
||||||
|
|
||||||
|
### `TaskResult`
|
||||||
|
|
||||||
|
```python
|
||||||
|
@dataclass
|
||||||
|
class TaskResult:
|
||||||
|
item_id: str
|
||||||
|
prediction: str
|
||||||
|
score: float
|
||||||
|
trajectory: list[dict]
|
||||||
|
```
|
||||||
|
|
||||||
|
### `ModelResponse`
|
||||||
|
|
||||||
|
```python
|
||||||
|
@dataclass
|
||||||
|
class ModelResponse:
|
||||||
|
content: str
|
||||||
|
usage: dict
|
||||||
|
model: str
|
||||||
|
```
|
||||||
|
|
||||||
|
For detailed source code, see the [`skillopt/`](https://github.com/microsoft/SkillOpt/tree/main/skillopt) directory.
|
||||||
@@ -0,0 +1,71 @@
|
|||||||
|
# CLI Reference
|
||||||
|
|
||||||
|
## Training
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python scripts/train.py --config <config.yaml> [overrides...]
|
||||||
|
```
|
||||||
|
|
||||||
|
### Arguments
|
||||||
|
|
||||||
|
| Argument | Description |
|
||||||
|
|---|---|
|
||||||
|
| `--config` | Path to YAML config file (required) |
|
||||||
|
| `key=value` | Override any config parameter |
|
||||||
|
|
||||||
|
### Examples
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Basic training
|
||||||
|
python scripts/train.py --config configs/searchqa/default.yaml
|
||||||
|
|
||||||
|
# With overrides
|
||||||
|
python scripts/train.py \
|
||||||
|
--config configs/searchqa/default.yaml \
|
||||||
|
--cfg-options optimizer.learning_rate=16 optimizer.lr_scheduler=linear
|
||||||
|
|
||||||
|
# With custom initial skill
|
||||||
|
python scripts/train.py \
|
||||||
|
--config configs/searchqa/default.yaml \
|
||||||
|
--cfg-options env.skill_init=skills/my_seed.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Evaluation
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python scripts/eval_only.py --config <config.yaml> --skill <skill.md>
|
||||||
|
```
|
||||||
|
|
||||||
|
### Arguments
|
||||||
|
|
||||||
|
| Argument | Description |
|
||||||
|
|---|---|
|
||||||
|
| `--config` | Path to YAML config file (required) |
|
||||||
|
| `--skill` | Path to skill document to evaluate (required) |
|
||||||
|
| `--split` | Evaluation split: `test` (default), `valid`, `train` |
|
||||||
|
|
||||||
|
### Examples
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Evaluate best skill on test set
|
||||||
|
python scripts/eval_only.py \
|
||||||
|
--config configs/searchqa/default.yaml \
|
||||||
|
--skill outputs/searchqa/run_001/skills/best_skill.md
|
||||||
|
|
||||||
|
# Evaluate on validation set
|
||||||
|
python scripts/eval_only.py \
|
||||||
|
--config configs/searchqa/default.yaml \
|
||||||
|
--skill outputs/searchqa/run_001/skills/best_skill.md \
|
||||||
|
--split valid
|
||||||
|
```
|
||||||
|
|
||||||
|
## WebUI
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python -m skillopt_webui.app [--port PORT] [--share]
|
||||||
|
```
|
||||||
|
|
||||||
|
| Argument | Default | Description |
|
||||||
|
|---|---|---|
|
||||||
|
| `--port` | 7860 | Port number |
|
||||||
|
| `--share` | false | Create public Gradio link |
|
||||||
@@ -0,0 +1,72 @@
|
|||||||
|
# Configuration Reference
|
||||||
|
|
||||||
|
Complete reference for all SkillOpt configuration parameters.
|
||||||
|
|
||||||
|
## Model
|
||||||
|
|
||||||
|
| Parameter | Type | Default | Description |
|
||||||
|
|---|---|---|---|
|
||||||
|
| `model.backend` | str | `azure_openai` | Backend: `azure_openai` / `openai_chat` / `claude_code_exec` / `qwen` |
|
||||||
|
| `model.teacher` | str | `gpt-5.5` | Teacher model (for reflection & slow update) |
|
||||||
|
| `model.student` | str | `gpt-5.5` | Student model (for rollout execution) |
|
||||||
|
| `model.reasoning_effort` | str | `medium` | Reasoning effort level |
|
||||||
|
|
||||||
|
## Training (`train`)
|
||||||
|
|
||||||
|
| Parameter | Type | Default | DL Analogy | Description |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| `train.num_epochs` | int | 4 | Epochs | Number of training epochs |
|
||||||
|
| `train.batch_size` | int | 40 | Batch size | Tasks sampled per step |
|
||||||
|
| `train.accumulation` | int | 1 | Gradient accumulation | Accumulation rounds per step |
|
||||||
|
| `train.seed` | int | 42 | Random seed | Reproducibility seed |
|
||||||
|
|
||||||
|
## Gradient / Reflection (`gradient`)
|
||||||
|
|
||||||
|
| Parameter | Type | Default | Description |
|
||||||
|
|---|---|---|---|
|
||||||
|
| `gradient.minibatch_size` | int | 8 | Reflect minibatch size |
|
||||||
|
| `gradient.merge_batch_size` | int | 8 | Patch merge batch size |
|
||||||
|
| `gradient.analyst_workers` | int | 16 | Parallel reflection workers |
|
||||||
|
| `gradient.max_analyst_rounds` | int | 3 | Max rounds of analyst reflection |
|
||||||
|
| `gradient.failure_only` | bool | `false` | Only reflect on failures |
|
||||||
|
|
||||||
|
## Optimizer (`optimizer`)
|
||||||
|
|
||||||
|
| Parameter | Type | Default | DL Analogy | Description |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| `optimizer.learning_rate` | int | 4 | Learning rate | Max edit patches per step (edit budget) |
|
||||||
|
| `optimizer.min_learning_rate` | int | 2 | Min LR | Min edits for decay schedulers |
|
||||||
|
| `optimizer.lr_scheduler` | str | `cosine` | LR schedule | `constant` / `linear` / `cosine` / `autonomous` |
|
||||||
|
| `optimizer.skill_update_mode` | str | `patch` | — | `patch` / `rewrite_from_suggestions` / `full_rewrite_minibatch` |
|
||||||
|
| `optimizer.use_slow_update` | bool | `true` | Momentum | Epoch-boundary longitudinal comparison & guidance |
|
||||||
|
| `optimizer.slow_update_samples` | int | 20 | — | Samples for slow update evaluation |
|
||||||
|
| `optimizer.use_meta_skill` | bool | `true` | Meta-learning | Cross-epoch teacher-side strategy memory |
|
||||||
|
| `optimizer.longitudinal_pair_policy` | str | `mixed` | — | `mixed` / `changed` / `unchanged` |
|
||||||
|
|
||||||
|
## Evaluation (`evaluation`)
|
||||||
|
|
||||||
|
| Parameter | Type | Default | Description |
|
||||||
|
|---|---|---|---|
|
||||||
|
| `evaluation.use_gate` | bool | `true` | Enable validation gating (accept/reject updates) |
|
||||||
|
| `evaluation.eval_test` | bool | `true` | Run test evaluation after training |
|
||||||
|
|
||||||
|
## Environment (`env`)
|
||||||
|
|
||||||
|
| Parameter | Type | Default | Description |
|
||||||
|
|---|---|---|---|
|
||||||
|
| `env.name` | str | — | Benchmark name (e.g., `searchqa`, `docvqa`) |
|
||||||
|
| `env.data_path` | str | — | Path to dataset |
|
||||||
|
| `env.skill_init` | str | — | Path to initial seed skill (optional) |
|
||||||
|
| `env.split_mode` | str | `ratio` | `ratio` or `split_dir` |
|
||||||
|
| `env.split_ratio` | str | `2:1:7` | Train:val:test ratio |
|
||||||
|
| `env.exec_timeout` | int | 120 | Per-task timeout in seconds |
|
||||||
|
| `env.out_root` | str | — | Output directory |
|
||||||
|
|
||||||
|
## Azure OpenAI Credentials
|
||||||
|
|
||||||
|
| Variable | Description |
|
||||||
|
|---|---|
|
||||||
|
| `AZURE_OPENAI_ENDPOINT` / `model.azure_openai_endpoint` | Azure resource endpoint |
|
||||||
|
| `AZURE_OPENAI_API_KEY` / `model.azure_openai_api_key` | Azure API key |
|
||||||
|
| `OPENAI_API_KEY` | OpenAI API key (for `openai_chat` backend) |
|
||||||
|
| `ANTHROPIC_API_KEY` | Anthropic API key (for `claude_code_exec` backend) |
|
||||||
+77
@@ -0,0 +1,77 @@
|
|||||||
|
site_name: SkillOpt Documentation
|
||||||
|
site_url: https://microsoft.github.io/SkillOpt
|
||||||
|
site_description: "SkillOpt: Agentic Skill Optimization via Reflective Training Loops"
|
||||||
|
repo_url: https://github.com/microsoft/SkillOpt
|
||||||
|
repo_name: microsoft/SkillOpt
|
||||||
|
|
||||||
|
theme:
|
||||||
|
name: material
|
||||||
|
palette:
|
||||||
|
- scheme: default
|
||||||
|
primary: indigo
|
||||||
|
accent: deep purple
|
||||||
|
toggle:
|
||||||
|
icon: material/brightness-7
|
||||||
|
name: Switch to dark mode
|
||||||
|
- scheme: slate
|
||||||
|
primary: indigo
|
||||||
|
accent: deep purple
|
||||||
|
toggle:
|
||||||
|
icon: material/brightness-4
|
||||||
|
name: Switch to light mode
|
||||||
|
features:
|
||||||
|
- navigation.instant
|
||||||
|
- navigation.tracking
|
||||||
|
- navigation.sections
|
||||||
|
- navigation.expand
|
||||||
|
- navigation.top
|
||||||
|
- content.code.copy
|
||||||
|
- content.tabs.link
|
||||||
|
- search.suggest
|
||||||
|
- search.highlight
|
||||||
|
icon:
|
||||||
|
repo: fontawesome/brands/github
|
||||||
|
font:
|
||||||
|
text: Inter
|
||||||
|
code: JetBrains Mono
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
nav:
|
||||||
|
- Home: index.md
|
||||||
|
- Getting Started:
|
||||||
|
- Installation: guide/installation.md
|
||||||
|
- First Experiment: guide/first-experiment.md
|
||||||
|
- Configuration: guide/configuration.md
|
||||||
|
- Core Concepts:
|
||||||
|
- Training Loop: guide/training-loop.md
|
||||||
|
- Skill Document: guide/skill-document.md
|
||||||
|
- Deep Learning Analogy: guide/dl-analogy.md
|
||||||
|
- Extension Guides:
|
||||||
|
- Add a New Benchmark: guide/new-benchmark.md
|
||||||
|
- Add a New Model Backend: guide/new-backend.md
|
||||||
|
- Reference:
|
||||||
|
- Configuration Reference: reference/config.md
|
||||||
|
- CLI Reference: reference/cli.md
|
||||||
|
- API Reference: reference/api.md
|
||||||
|
- Contributing: contributing.md
|
||||||
|
|
||||||
|
markdown_extensions:
|
||||||
|
- admonition
|
||||||
|
- pymdownx.details
|
||||||
|
- pymdownx.superfences
|
||||||
|
- pymdownx.tabbed:
|
||||||
|
alternate_style: true
|
||||||
|
- pymdownx.highlight:
|
||||||
|
anchor_linenums: true
|
||||||
|
- pymdownx.inlinehilite
|
||||||
|
- pymdownx.emoji:
|
||||||
|
emoji_index: !!python/name:material.extensions.emoji.twemoji
|
||||||
|
emoji_generator: !!python/name:material.extensions.emoji.to_svg
|
||||||
|
- attr_list
|
||||||
|
- md_in_html
|
||||||
|
- toc:
|
||||||
|
permalink: true
|
||||||
|
|
||||||
|
plugins:
|
||||||
|
- search
|
||||||
@@ -0,0 +1,75 @@
|
|||||||
|
[build-system]
|
||||||
|
requires = ["setuptools>=68.0", "wheel"]
|
||||||
|
build-backend = "setuptools.build_meta"
|
||||||
|
|
||||||
|
[project]
|
||||||
|
name = "skillopt"
|
||||||
|
version = "0.1.0"
|
||||||
|
description = "SkillOpt: Agentic Skill Optimization via Reflective Training Loops"
|
||||||
|
readme = "README.md"
|
||||||
|
license = {text = "MIT"}
|
||||||
|
requires-python = ">=3.10"
|
||||||
|
authors = [
|
||||||
|
{name = "SkillOpt Team"},
|
||||||
|
]
|
||||||
|
keywords = ["agent", "prompt-optimization", "skill-learning", "LLM", "agentic"]
|
||||||
|
classifiers = [
|
||||||
|
"Development Status :: 3 - Alpha",
|
||||||
|
"Intended Audience :: Science/Research",
|
||||||
|
"License :: OSI Approved :: MIT License",
|
||||||
|
"Programming Language :: Python :: 3",
|
||||||
|
"Programming Language :: Python :: 3.10",
|
||||||
|
"Programming Language :: Python :: 3.11",
|
||||||
|
"Programming Language :: Python :: 3.12",
|
||||||
|
"Topic :: Scientific/Engineering :: Artificial Intelligence",
|
||||||
|
]
|
||||||
|
dependencies = [
|
||||||
|
"openai>=1.30.0",
|
||||||
|
"pyyaml>=6.0",
|
||||||
|
"numpy>=1.24.0",
|
||||||
|
"openpyxl>=3.1.0",
|
||||||
|
"azure-identity>=1.15.0",
|
||||||
|
"azure-core>=1.30.0",
|
||||||
|
"httpx>=0.27.0",
|
||||||
|
]
|
||||||
|
|
||||||
|
[project.optional-dependencies]
|
||||||
|
# Benchmark-specific dependencies
|
||||||
|
alfworld = ["alfworld>=0.4.0", "gymnasium>=0.29.0"]
|
||||||
|
# Claude model backend
|
||||||
|
claude = ["claude-agent-sdk>=0.1.0"]
|
||||||
|
# Qwen local model backend (via vLLM)
|
||||||
|
qwen = ["vllm>=0.4.0"]
|
||||||
|
# Documentation site
|
||||||
|
docs = ["mkdocs-material>=9.5.0", "mkdocstrings[python]>=0.24.0"]
|
||||||
|
# WebUI dashboard
|
||||||
|
webui = ["gradio>=4.0.0"]
|
||||||
|
# Development tools
|
||||||
|
dev = ["ruff>=0.4.0", "pytest>=8.0.0"]
|
||||||
|
# All optional dependencies (except docs/dev/webui)
|
||||||
|
all = [
|
||||||
|
"alfworld>=0.4.0",
|
||||||
|
"gymnasium>=0.29.0",
|
||||||
|
"claude-agent-sdk>=0.1.0",
|
||||||
|
]
|
||||||
|
|
||||||
|
[project.scripts]
|
||||||
|
skillopt-train = "scripts.train:main"
|
||||||
|
skillopt-eval = "scripts.eval_only:main"
|
||||||
|
|
||||||
|
[project.urls]
|
||||||
|
Homepage = "https://github.com/microsoft/SkillOpt"
|
||||||
|
Documentation = "https://microsoft.github.io/SkillOpt"
|
||||||
|
Repository = "https://github.com/microsoft/SkillOpt"
|
||||||
|
Issues = "https://github.com/microsoft/SkillOpt/issues"
|
||||||
|
|
||||||
|
[tool.setuptools.packages.find]
|
||||||
|
include = ["skillopt*", "scripts*"]
|
||||||
|
|
||||||
|
[tool.ruff]
|
||||||
|
line-length = 120
|
||||||
|
target-version = "py310"
|
||||||
|
|
||||||
|
[tool.ruff.lint]
|
||||||
|
select = ["E", "F", "I", "W"]
|
||||||
|
ignore = ["E501"]
|
||||||
@@ -0,0 +1,25 @@
|
|||||||
|
# ── Core ──────────────────────────────────────────
|
||||||
|
openai>=1.30.0
|
||||||
|
pyyaml>=6.0
|
||||||
|
numpy>=1.24.0
|
||||||
|
openpyxl>=3.1.0
|
||||||
|
azure-identity>=1.15.0
|
||||||
|
azure-core>=1.30.0
|
||||||
|
httpx>=0.27.0
|
||||||
|
|
||||||
|
# ── Optional: ALFWorld benchmark ──────────────────
|
||||||
|
# alfworld>=0.4.0
|
||||||
|
# gymnasium>=0.29.0
|
||||||
|
|
||||||
|
# ── Optional: Claude model backend ────────────────
|
||||||
|
# claude-agent-sdk>=0.1.0
|
||||||
|
|
||||||
|
# ── Optional: Qwen local model (via vLLM) ────────
|
||||||
|
# vllm>=0.4.0
|
||||||
|
|
||||||
|
# ── Optional: WebUI dashboard ────────────────────
|
||||||
|
# gradio>=4.0.0
|
||||||
|
|
||||||
|
# ── Optional: Documentation site ─────────────────
|
||||||
|
# mkdocs-material>=9.5.0
|
||||||
|
# mkdocstrings[python]>=0.24.0
|
||||||
@@ -0,0 +1,53 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Download BabyVision from Hugging Face and convert it to local meta_data.jsonl + images/ format."""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
|
||||||
|
def parse_args() -> argparse.Namespace:
|
||||||
|
p = argparse.ArgumentParser(description=__doc__)
|
||||||
|
p.add_argument("--out_dir", type=str, required=True)
|
||||||
|
p.add_argument("--dataset", type=str, default="UnipatAI/BabyVision")
|
||||||
|
p.add_argument("--split", type=str, default="train")
|
||||||
|
return p.parse_args()
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> None:
|
||||||
|
args = parse_args()
|
||||||
|
|
||||||
|
try:
|
||||||
|
from datasets import load_dataset
|
||||||
|
except ImportError as exc: # pragma: no cover
|
||||||
|
raise SystemExit("Please install `datasets` first: pip install datasets pillow") from exc
|
||||||
|
|
||||||
|
out_dir = Path(args.out_dir).resolve()
|
||||||
|
images_dir = out_dir / "images"
|
||||||
|
meta_path = out_dir / "meta_data.jsonl"
|
||||||
|
images_dir.mkdir(parents=True, exist_ok=True)
|
||||||
|
|
||||||
|
dataset = load_dataset(args.dataset, split=args.split)
|
||||||
|
with open(meta_path, "w", encoding="utf-8") as outf:
|
||||||
|
for idx, row in enumerate(dataset):
|
||||||
|
image = row.get("image")
|
||||||
|
if image is None:
|
||||||
|
continue
|
||||||
|
task_id = str(row.get("taskId") or row.get("id") or idx + 1)
|
||||||
|
image_name = f"{task_id}.png"
|
||||||
|
image_path = images_dir / image_name
|
||||||
|
image.save(image_path)
|
||||||
|
|
||||||
|
record = dict(row)
|
||||||
|
record["image"] = image_name
|
||||||
|
outf.write(json.dumps(record, ensure_ascii=False) + "\n")
|
||||||
|
|
||||||
|
print(f"Saved BabyVision to {out_dir}")
|
||||||
|
print(f"Metadata: {meta_path}")
|
||||||
|
print(f"Images: {images_dir}")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,451 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""ReflACT eval-only: run a single skill on a dataset without training.
|
||||||
|
|
||||||
|
Usage
|
||||||
|
-----
|
||||||
|
python scripts/eval_only.py \
|
||||||
|
--config configs/spreadsheetbench/default.yaml \
|
||||||
|
--skill skillopt/envs/spreadsheetbench/skills/initial.md \
|
||||||
|
--split_dir /path/to/split \
|
||||||
|
--out_root outputs/eval_skill0
|
||||||
|
|
||||||
|
All YAML keys can be overridden from the CLI, same as train.py.
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import datetime
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
|
||||||
|
_SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__))
|
||||||
|
_PROJECT_ROOT = os.path.dirname(_SCRIPT_DIR)
|
||||||
|
if _PROJECT_ROOT not in sys.path:
|
||||||
|
sys.path.insert(0, _PROJECT_ROOT)
|
||||||
|
|
||||||
|
from skillopt.model import (
|
||||||
|
configure_azure_openai,
|
||||||
|
configure_claude_code_exec,
|
||||||
|
configure_codex_exec,
|
||||||
|
set_reasoning_effort,
|
||||||
|
set_student_backend,
|
||||||
|
set_student_deployment,
|
||||||
|
set_teacher_backend,
|
||||||
|
set_teacher_deployment,
|
||||||
|
)
|
||||||
|
from skillopt.model.common import default_model_for_backend, normalize_backend_name
|
||||||
|
|
||||||
|
_OPENAI_DEFAULT_MODEL_SENTINELS = {"gpt-5.4", "gpt-5.5"}
|
||||||
|
from skillopt.utils import compute_score
|
||||||
|
|
||||||
|
|
||||||
|
# ── Reuse registry from train.py ───────────────────────────────────────────
|
||||||
|
|
||||||
|
_ENV_REGISTRY: dict[str, type] = {}
|
||||||
|
|
||||||
|
|
||||||
|
def _register_builtins() -> None:
|
||||||
|
try:
|
||||||
|
from skillopt.envs.alfworld.adapter import ALFWorldAdapter
|
||||||
|
_ENV_REGISTRY["alfworld"] = ALFWorldAdapter
|
||||||
|
except ImportError:
|
||||||
|
pass
|
||||||
|
try:
|
||||||
|
from skillopt.envs.searchqa.adapter import SearchQAAdapter
|
||||||
|
_ENV_REGISTRY["searchqa"] = SearchQAAdapter
|
||||||
|
except ImportError:
|
||||||
|
pass
|
||||||
|
try:
|
||||||
|
from skillopt.envs.livemathematicianbench.adapter import LiveMathematicianBenchAdapter
|
||||||
|
_ENV_REGISTRY["livemathematicianbench"] = LiveMathematicianBenchAdapter
|
||||||
|
except ImportError:
|
||||||
|
pass
|
||||||
|
try:
|
||||||
|
from skillopt.envs.babyvision.adapter import BabyVisionAdapter
|
||||||
|
_ENV_REGISTRY["babyvision"] = BabyVisionAdapter
|
||||||
|
except ImportError:
|
||||||
|
pass
|
||||||
|
try:
|
||||||
|
from skillopt.envs.spreadsheetbench.adapter import SpreadsheetBenchAdapter
|
||||||
|
_ENV_REGISTRY["spreadsheetbench"] = SpreadsheetBenchAdapter
|
||||||
|
except ImportError:
|
||||||
|
pass
|
||||||
|
try:
|
||||||
|
from skillopt.envs.mmrb.adapter import MMRBAdapter
|
||||||
|
_ENV_REGISTRY["mmrb"] = MMRBAdapter
|
||||||
|
except ImportError:
|
||||||
|
pass
|
||||||
|
try:
|
||||||
|
from skillopt.envs.docvqa.adapter import DocVQAAdapter
|
||||||
|
_ENV_REGISTRY["docvqa"] = DocVQAAdapter
|
||||||
|
except ImportError:
|
||||||
|
pass
|
||||||
|
try:
|
||||||
|
from skillopt.envs.mathverse.adapter import MathVerseAdapter
|
||||||
|
_ENV_REGISTRY["mathverse"] = MathVerseAdapter
|
||||||
|
except ImportError:
|
||||||
|
pass
|
||||||
|
try:
|
||||||
|
from skillopt.envs.officeqa.adapter import OfficeQAAdapter
|
||||||
|
_ENV_REGISTRY["officeqa"] = OfficeQAAdapter
|
||||||
|
except ImportError:
|
||||||
|
pass
|
||||||
|
try:
|
||||||
|
from skillopt.envs.sealqa.adapter import SealQAAdapter
|
||||||
|
_ENV_REGISTRY["sealqa"] = SealQAAdapter
|
||||||
|
except ImportError:
|
||||||
|
pass
|
||||||
|
try:
|
||||||
|
from skillopt.envs.swebench.adapter import SWEBenchAdapter
|
||||||
|
_ENV_REGISTRY["swebench"] = SWEBenchAdapter
|
||||||
|
except ImportError:
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
def get_adapter(cfg: dict):
|
||||||
|
_register_builtins()
|
||||||
|
env_name = cfg.get("env", "alfworld")
|
||||||
|
if env_name not in _ENV_REGISTRY:
|
||||||
|
raise ValueError(
|
||||||
|
f"Unknown environment '{env_name}'. "
|
||||||
|
f"Available: {list(_ENV_REGISTRY.keys())}"
|
||||||
|
)
|
||||||
|
adapter_cls = _ENV_REGISTRY[env_name]
|
||||||
|
|
||||||
|
import inspect
|
||||||
|
sig = inspect.signature(adapter_cls.__init__)
|
||||||
|
accepted = set(sig.parameters.keys()) - {"self"}
|
||||||
|
adapter_kwargs = {k: cfg[k] for k in accepted if k in cfg}
|
||||||
|
return adapter_cls(**adapter_kwargs)
|
||||||
|
|
||||||
|
|
||||||
|
# ── CLI ────────────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
_BOOL = lambda x: str(x).lower() in ("true", "1", "yes") # noqa: E731
|
||||||
|
|
||||||
|
|
||||||
|
def parse_args() -> argparse.Namespace:
|
||||||
|
p = argparse.ArgumentParser(description="ReflACT eval-only")
|
||||||
|
p.add_argument("--config", type=str, required=True)
|
||||||
|
p.add_argument("--skill", type=str, required=True,
|
||||||
|
help="Path to skill .md file to evaluate")
|
||||||
|
p.add_argument("--split", type=str, default="all",
|
||||||
|
help="Which split to eval: train/valid_seen/valid_unseen/all (default: all)")
|
||||||
|
p.add_argument("--cfg-options", nargs="+", default=[],
|
||||||
|
help="Override config: section.key=value")
|
||||||
|
# Legacy flat overrides
|
||||||
|
p.add_argument("--env", type=str)
|
||||||
|
p.add_argument("--backend", type=str,
|
||||||
|
choices=["azure_openai", "codex", "codex_exec", "claude", "claude_chat", "claude_code_exec"])
|
||||||
|
p.add_argument("--teacher_model", type=str)
|
||||||
|
p.add_argument("--student_model", type=str)
|
||||||
|
p.add_argument("--teacher_backend", type=str)
|
||||||
|
p.add_argument("--student_backend", type=str)
|
||||||
|
p.add_argument("--reasoning_effort", type=str,
|
||||||
|
choices=["", "low", "medium", "high", "xhigh", "max"])
|
||||||
|
p.add_argument("--azure_endpoint", type=str)
|
||||||
|
p.add_argument("--azure_api_version", type=str)
|
||||||
|
p.add_argument("--azure_api_key", type=str)
|
||||||
|
p.add_argument("--azure_openai_endpoint", type=str)
|
||||||
|
p.add_argument("--azure_openai_api_version", type=str)
|
||||||
|
p.add_argument("--azure_openai_api_key", type=str)
|
||||||
|
p.add_argument("--azure_openai_auth_mode", type=str)
|
||||||
|
p.add_argument("--azure_openai_ad_scope", type=str)
|
||||||
|
p.add_argument("--azure_openai_managed_identity_client_id", type=str)
|
||||||
|
p.add_argument("--teacher_azure_openai_endpoint", type=str)
|
||||||
|
p.add_argument("--teacher_azure_openai_api_version", type=str)
|
||||||
|
p.add_argument("--teacher_azure_openai_api_key", type=str)
|
||||||
|
p.add_argument("--teacher_azure_openai_auth_mode", type=str)
|
||||||
|
p.add_argument("--teacher_azure_openai_ad_scope", type=str)
|
||||||
|
p.add_argument("--teacher_azure_openai_managed_identity_client_id", type=str)
|
||||||
|
p.add_argument("--student_azure_openai_endpoint", type=str)
|
||||||
|
p.add_argument("--student_azure_openai_api_version", type=str)
|
||||||
|
p.add_argument("--student_azure_openai_api_key", type=str)
|
||||||
|
p.add_argument("--student_azure_openai_auth_mode", type=str)
|
||||||
|
p.add_argument("--student_azure_openai_ad_scope", type=str)
|
||||||
|
p.add_argument("--student_azure_openai_managed_identity_client_id", type=str)
|
||||||
|
p.add_argument("--codex_exec_path", type=str)
|
||||||
|
p.add_argument("--codex_exec_sandbox", type=str)
|
||||||
|
p.add_argument("--codex_exec_profile", type=str)
|
||||||
|
p.add_argument("--codex_exec_full_auto", type=_BOOL)
|
||||||
|
p.add_argument("--codex_exec_reasoning_effort", type=str)
|
||||||
|
p.add_argument("--codex_exec_use_sdk", type=str)
|
||||||
|
p.add_argument("--codex_exec_network_access", type=_BOOL)
|
||||||
|
p.add_argument("--codex_exec_web_search", type=_BOOL)
|
||||||
|
p.add_argument("--codex_exec_approval_policy", type=str)
|
||||||
|
p.add_argument("--claude_code_exec_path", type=str)
|
||||||
|
p.add_argument("--claude_code_exec_profile", type=str)
|
||||||
|
p.add_argument("--claude_code_exec_use_sdk", type=str)
|
||||||
|
p.add_argument("--claude_code_exec_effort", type=str)
|
||||||
|
p.add_argument("--claude_code_exec_max_thinking_tokens", type=int)
|
||||||
|
p.add_argument("--out_root", type=str)
|
||||||
|
p.add_argument("--data_path", type=str)
|
||||||
|
p.add_argument("--split_mode", type=str,
|
||||||
|
choices=["ratio", "split_dir"])
|
||||||
|
p.add_argument("--split_ratio", type=str)
|
||||||
|
p.add_argument("--split_seed", type=int)
|
||||||
|
p.add_argument("--split_dir", type=str)
|
||||||
|
p.add_argument("--split_output_dir", type=str)
|
||||||
|
p.add_argument("--data_root", type=str)
|
||||||
|
p.add_argument("--max_turns", type=int)
|
||||||
|
p.add_argument("--workers", type=int)
|
||||||
|
p.add_argument("--max_api_workers", type=int)
|
||||||
|
p.add_argument("--seed", type=int)
|
||||||
|
p.add_argument("--test_env_num", type=int)
|
||||||
|
p.add_argument("--mode", type=str,
|
||||||
|
help="SpreadsheetBench: single/multi/react (default comes from config)")
|
||||||
|
return p.parse_args()
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> None:
|
||||||
|
args = parse_args()
|
||||||
|
|
||||||
|
from skillopt.config import load_config as _load, flatten_config, is_structured
|
||||||
|
|
||||||
|
cfg = _load(args.config, overrides=args.cfg_options)
|
||||||
|
structured = is_structured(cfg)
|
||||||
|
|
||||||
|
# Apply legacy --key value overrides
|
||||||
|
cli = {k: v for k, v in vars(args).items()
|
||||||
|
if v is not None and k not in ("config", "skill", "split", "cfg_options")}
|
||||||
|
if cli:
|
||||||
|
if structured:
|
||||||
|
from skillopt.config import apply_overrides
|
||||||
|
_MAP = {
|
||||||
|
"backend": "model.backend",
|
||||||
|
"teacher_model": "model.teacher",
|
||||||
|
"student_model": "model.student",
|
||||||
|
"teacher_backend": "model.teacher_backend",
|
||||||
|
"student_backend": "model.student_backend",
|
||||||
|
"reasoning_effort": "model.reasoning_effort",
|
||||||
|
"azure_endpoint": "model.azure_endpoint",
|
||||||
|
"azure_api_version": "model.azure_api_version",
|
||||||
|
"azure_api_key": "model.azure_api_key",
|
||||||
|
"azure_openai_endpoint": "model.azure_openai_endpoint",
|
||||||
|
"azure_openai_api_version": "model.azure_openai_api_version",
|
||||||
|
"azure_openai_api_key": "model.azure_openai_api_key",
|
||||||
|
"azure_openai_auth_mode": "model.azure_openai_auth_mode",
|
||||||
|
"azure_openai_ad_scope": "model.azure_openai_ad_scope",
|
||||||
|
"azure_openai_managed_identity_client_id": "model.azure_openai_managed_identity_client_id",
|
||||||
|
"teacher_azure_openai_endpoint": "model.teacher_azure_openai_endpoint",
|
||||||
|
"teacher_azure_openai_api_version": "model.teacher_azure_openai_api_version",
|
||||||
|
"teacher_azure_openai_api_key": "model.teacher_azure_openai_api_key",
|
||||||
|
"teacher_azure_openai_auth_mode": "model.teacher_azure_openai_auth_mode",
|
||||||
|
"teacher_azure_openai_ad_scope": "model.teacher_azure_openai_ad_scope",
|
||||||
|
"teacher_azure_openai_managed_identity_client_id": "model.teacher_azure_openai_managed_identity_client_id",
|
||||||
|
"student_azure_openai_endpoint": "model.student_azure_openai_endpoint",
|
||||||
|
"student_azure_openai_api_version": "model.student_azure_openai_api_version",
|
||||||
|
"student_azure_openai_api_key": "model.student_azure_openai_api_key",
|
||||||
|
"student_azure_openai_auth_mode": "model.student_azure_openai_auth_mode",
|
||||||
|
"student_azure_openai_ad_scope": "model.student_azure_openai_ad_scope",
|
||||||
|
"student_azure_openai_managed_identity_client_id": "model.student_azure_openai_managed_identity_client_id",
|
||||||
|
"codex_exec_path": "model.codex_exec_path",
|
||||||
|
"codex_exec_sandbox": "model.codex_exec_sandbox",
|
||||||
|
"codex_exec_profile": "model.codex_exec_profile",
|
||||||
|
"codex_exec_full_auto": "model.codex_exec_full_auto",
|
||||||
|
"codex_exec_reasoning_effort": "model.codex_exec_reasoning_effort",
|
||||||
|
"codex_exec_use_sdk": "model.codex_exec_use_sdk",
|
||||||
|
"codex_exec_network_access": "model.codex_exec_network_access",
|
||||||
|
"codex_exec_web_search": "model.codex_exec_web_search",
|
||||||
|
"codex_exec_approval_policy": "model.codex_exec_approval_policy",
|
||||||
|
"claude_code_exec_path": "model.claude_code_exec_path",
|
||||||
|
"claude_code_exec_profile": "model.claude_code_exec_profile",
|
||||||
|
"claude_code_exec_use_sdk": "model.claude_code_exec_use_sdk",
|
||||||
|
"claude_code_exec_effort": "model.claude_code_exec_effort",
|
||||||
|
"claude_code_exec_max_thinking_tokens": "model.claude_code_exec_max_thinking_tokens",
|
||||||
|
"seed": "train.seed",
|
||||||
|
"test_env_num": "evaluation.test_env_num",
|
||||||
|
"env": "env.name",
|
||||||
|
"out_root": "env.out_root",
|
||||||
|
}
|
||||||
|
mapped = []
|
||||||
|
for k, v in cli.items():
|
||||||
|
dotted = _MAP.get(k)
|
||||||
|
if dotted:
|
||||||
|
mapped.append(f"{dotted}={v}")
|
||||||
|
else:
|
||||||
|
mapped.append(f"env.{k}={v}")
|
||||||
|
apply_overrides(cfg, mapped)
|
||||||
|
else:
|
||||||
|
cfg.update(cli)
|
||||||
|
|
||||||
|
cfg = flatten_config(cfg) if structured else cfg
|
||||||
|
|
||||||
|
for new_key, old_key in (
|
||||||
|
("azure_openai_endpoint", "azure_endpoint"),
|
||||||
|
("azure_openai_api_version", "azure_api_version"),
|
||||||
|
("azure_openai_api_key", "azure_api_key"),
|
||||||
|
):
|
||||||
|
if cfg.get(new_key) in (None, "") and cfg.get(old_key) not in (None, ""):
|
||||||
|
cfg[new_key] = cfg[old_key]
|
||||||
|
|
||||||
|
explicit_backend = getattr(args, "backend", None)
|
||||||
|
if explicit_backend is None:
|
||||||
|
for option in args.cfg_options or []:
|
||||||
|
key = str(option).split("=", 1)[0].strip()
|
||||||
|
if key == "model.backend":
|
||||||
|
explicit_backend = str(option).split("=", 1)[1].strip()
|
||||||
|
break
|
||||||
|
|
||||||
|
backend = normalize_backend_name(cfg.get("model_backend") or cfg.get("student_backend") or "azure_openai")
|
||||||
|
|
||||||
|
def _has_model_override(dotted_key: str, legacy_key: str) -> bool:
|
||||||
|
if getattr(args, legacy_key, None) is not None:
|
||||||
|
return True
|
||||||
|
for option in args.cfg_options or []:
|
||||||
|
key = str(option).split("=", 1)[0].strip()
|
||||||
|
if key == dotted_key:
|
||||||
|
return True
|
||||||
|
return False
|
||||||
|
|
||||||
|
if explicit_backend is not None:
|
||||||
|
backend = normalize_backend_name(explicit_backend)
|
||||||
|
cfg["model_backend"] = backend
|
||||||
|
if backend in {"claude", "claude_chat"}:
|
||||||
|
cfg.setdefault("teacher_backend", "claude_chat")
|
||||||
|
cfg.setdefault("student_backend", "claude_chat")
|
||||||
|
elif backend in {"codex", "codex_exec"}:
|
||||||
|
cfg.setdefault("teacher_backend", "openai_chat")
|
||||||
|
cfg.setdefault("student_backend", "codex_exec")
|
||||||
|
elif backend == "claude_code_exec":
|
||||||
|
cfg.setdefault("teacher_backend", "openai_chat")
|
||||||
|
cfg.setdefault("student_backend", "claude_code_exec")
|
||||||
|
else:
|
||||||
|
cfg.setdefault("teacher_backend", "openai_chat")
|
||||||
|
cfg.setdefault("student_backend", "openai_chat")
|
||||||
|
else:
|
||||||
|
cfg.setdefault("teacher_backend", "openai_chat")
|
||||||
|
cfg.setdefault("student_backend", "openai_chat")
|
||||||
|
|
||||||
|
if cfg.get("teacher_backend") == "claude_chat":
|
||||||
|
if (
|
||||||
|
str(cfg.get("teacher_model", "") or "").strip() in _OPENAI_DEFAULT_MODEL_SENTINELS
|
||||||
|
and not _has_model_override("model.teacher", "teacher_model")
|
||||||
|
):
|
||||||
|
cfg["teacher_model"] = default_model_for_backend("claude_chat")
|
||||||
|
if cfg.get("student_backend") == "claude_chat":
|
||||||
|
if (
|
||||||
|
str(cfg.get("student_model", "") or "").strip() in _OPENAI_DEFAULT_MODEL_SENTINELS
|
||||||
|
and not _has_model_override("model.student", "student_model")
|
||||||
|
):
|
||||||
|
cfg["student_model"] = default_model_for_backend("claude_chat")
|
||||||
|
if cfg.get("student_backend") == "claude_code_exec":
|
||||||
|
if (
|
||||||
|
str(cfg.get("student_model", "") or "").strip() in _OPENAI_DEFAULT_MODEL_SENTINELS
|
||||||
|
and not _has_model_override("model.student", "student_model")
|
||||||
|
):
|
||||||
|
cfg["student_model"] = default_model_for_backend("claude_chat")
|
||||||
|
|
||||||
|
if not cfg.get("out_root"):
|
||||||
|
env = cfg.get("env", "unknown")
|
||||||
|
model = cfg.get("student_model", "unknown").replace("/", "-")
|
||||||
|
ts = datetime.datetime.now().strftime("%Y%m%d_%H%M%S")
|
||||||
|
cfg["out_root"] = os.path.join("outputs", f"eval_{env}_{model}_{ts}")
|
||||||
|
|
||||||
|
cfg["out_root"] = os.path.abspath(cfg["out_root"])
|
||||||
|
|
||||||
|
out_root = cfg["out_root"]
|
||||||
|
os.makedirs(out_root, exist_ok=True)
|
||||||
|
|
||||||
|
# Load skill
|
||||||
|
skill_path = os.path.abspath(args.skill)
|
||||||
|
with open(skill_path) as f:
|
||||||
|
skill_content = f.read()
|
||||||
|
print(f" [skill] {skill_path} ({len(skill_content)} chars)")
|
||||||
|
|
||||||
|
# Configure models
|
||||||
|
configure_azure_openai(
|
||||||
|
endpoint=(cfg.get("azure_openai_endpoint") or cfg.get("azure_endpoint") or None),
|
||||||
|
api_version=(cfg.get("azure_openai_api_version") or cfg.get("azure_api_version") or None),
|
||||||
|
api_key=(cfg.get("azure_openai_api_key") or cfg.get("azure_api_key") or None),
|
||||||
|
auth_mode=cfg.get("azure_openai_auth_mode") or None,
|
||||||
|
ad_scope=cfg.get("azure_openai_ad_scope") or None,
|
||||||
|
managed_identity_client_id=cfg.get("azure_openai_managed_identity_client_id") or None,
|
||||||
|
teacher_endpoint=cfg.get("teacher_azure_openai_endpoint") or None,
|
||||||
|
teacher_api_version=cfg.get("teacher_azure_openai_api_version") or None,
|
||||||
|
teacher_api_key=cfg.get("teacher_azure_openai_api_key") or None,
|
||||||
|
teacher_auth_mode=cfg.get("teacher_azure_openai_auth_mode") or None,
|
||||||
|
teacher_ad_scope=cfg.get("teacher_azure_openai_ad_scope") or None,
|
||||||
|
teacher_managed_identity_client_id=(
|
||||||
|
cfg.get("teacher_azure_openai_managed_identity_client_id") or None
|
||||||
|
),
|
||||||
|
student_endpoint=cfg.get("student_azure_openai_endpoint") or None,
|
||||||
|
student_api_version=cfg.get("student_azure_openai_api_version") or None,
|
||||||
|
student_api_key=cfg.get("student_azure_openai_api_key") or None,
|
||||||
|
student_auth_mode=cfg.get("student_azure_openai_auth_mode") or None,
|
||||||
|
student_ad_scope=cfg.get("student_azure_openai_ad_scope") or None,
|
||||||
|
student_managed_identity_client_id=(
|
||||||
|
cfg.get("student_azure_openai_managed_identity_client_id") or None
|
||||||
|
),
|
||||||
|
)
|
||||||
|
set_teacher_backend(cfg.get("teacher_backend", "openai_chat"))
|
||||||
|
set_student_backend(cfg.get("student_backend", "openai_chat"))
|
||||||
|
set_teacher_deployment(cfg.get("teacher_model", default_model_for_backend(backend)))
|
||||||
|
set_student_deployment(cfg.get("student_model", default_model_for_backend(backend)))
|
||||||
|
configure_codex_exec(
|
||||||
|
path=cfg.get("codex_exec_path", "codex"),
|
||||||
|
sandbox=cfg.get("codex_exec_sandbox", "workspace-write"),
|
||||||
|
profile=cfg.get("codex_exec_profile", ""),
|
||||||
|
full_auto=cfg.get("codex_exec_full_auto", False),
|
||||||
|
reasoning_effort=cfg.get("codex_exec_reasoning_effort", "none"),
|
||||||
|
use_sdk=cfg.get("codex_exec_use_sdk", None),
|
||||||
|
network_access=cfg.get("codex_exec_network_access", False),
|
||||||
|
web_search=cfg.get("codex_exec_web_search", False),
|
||||||
|
approval_policy=cfg.get("codex_exec_approval_policy", "never"),
|
||||||
|
)
|
||||||
|
configure_claude_code_exec(
|
||||||
|
path=cfg.get("claude_code_exec_path", "claude"),
|
||||||
|
profile=cfg.get("claude_code_exec_profile", ""),
|
||||||
|
use_sdk=cfg.get("claude_code_exec_use_sdk", None),
|
||||||
|
effort=cfg.get("claude_code_exec_effort", cfg.get("reasoning_effort", "medium")),
|
||||||
|
max_thinking_tokens=cfg.get("claude_code_exec_max_thinking_tokens", 16384),
|
||||||
|
)
|
||||||
|
set_reasoning_effort(cfg.get("reasoning_effort", "") or None)
|
||||||
|
|
||||||
|
# Build adapter
|
||||||
|
adapter = get_adapter(cfg)
|
||||||
|
adapter.setup(cfg)
|
||||||
|
|
||||||
|
seed = cfg.get("seed", 42)
|
||||||
|
split = args.split or "all"
|
||||||
|
|
||||||
|
if split == "all":
|
||||||
|
items = (
|
||||||
|
adapter.build_eval_env(0, "train", seed)
|
||||||
|
+ adapter.build_eval_env(0, "valid_seen", seed)
|
||||||
|
+ adapter.build_eval_env(0, "valid_unseen", seed)
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
env_num = cfg.get("test_env_num", 0)
|
||||||
|
items = adapter.build_eval_env(env_num, split, seed)
|
||||||
|
|
||||||
|
print(f"\n [eval] split={split} items={len(items)}")
|
||||||
|
print(f" [eval] out_root={out_root}")
|
||||||
|
print(f"{'='*60}")
|
||||||
|
|
||||||
|
# Run rollout
|
||||||
|
results = adapter.rollout(items, skill_content, out_root)
|
||||||
|
|
||||||
|
# Score
|
||||||
|
hard, soft = compute_score(results)
|
||||||
|
print(f"\n{'='*60}")
|
||||||
|
print(f" Results: hard={hard:.4f} soft={soft:.4f} (n={len(results)})")
|
||||||
|
print(f"{'='*60}")
|
||||||
|
|
||||||
|
# Save summary
|
||||||
|
summary = {
|
||||||
|
"skill": skill_path,
|
||||||
|
"split": split,
|
||||||
|
"n_items": len(results),
|
||||||
|
"hard": hard,
|
||||||
|
"soft": soft,
|
||||||
|
}
|
||||||
|
with open(os.path.join(out_root, "eval_summary.json"), "w") as f:
|
||||||
|
json.dump(summary, f, indent=2, ensure_ascii=False)
|
||||||
|
|
||||||
|
print(f" Saved to: {out_root}")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
Executable
+68
@@ -0,0 +1,68 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# ──────────────────────────────────────────────────────────────────────────────
|
||||||
|
# ReflACT — ALFWorld training launch script
|
||||||
|
#
|
||||||
|
# Usage:
|
||||||
|
# bash scripts/run_alfworld.sh
|
||||||
|
# bash scripts/run_alfworld.sh --num_epochs 2 --edit_budget 6
|
||||||
|
# ──────────────────────────────────────────────────────────────────────────────
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
# ── Paths ────────────────────────────────────────────────────────────────────
|
||||||
|
WORKSPACE="${WORKSPACE:-$(cd "$(dirname "$0")/../.." && pwd)}"
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||||
|
PROJECT_ROOT="$(dirname "${SCRIPT_DIR}")"
|
||||||
|
|
||||||
|
# Activate conda environment
|
||||||
|
export PATH="${WORKSPACE}/miniconda3/envs/skillopt/bin:${WORKSPACE}/miniconda3/bin:${PATH}"
|
||||||
|
|
||||||
|
# ALFWorld data — uses ~/.cache/alfworld by default (standard alfworld location)
|
||||||
|
export ALFWORLD_DATA="${ALFWORLD_DATA:-${HOME}/.cache/alfworld}"
|
||||||
|
|
||||||
|
# Ensure ReflACT is importable
|
||||||
|
export PYTHONPATH="${PROJECT_ROOT}:${PYTHONPATH:-}"
|
||||||
|
|
||||||
|
# ── Verify ALFWorld data exists ──────────────────────────────────────────────
|
||||||
|
if [ ! -d "${ALFWORLD_DATA}/json_2.1.1" ]; then
|
||||||
|
echo "ERROR: ALFWorld data not found at ${ALFWORLD_DATA}/json_2.1.1"
|
||||||
|
echo ""
|
||||||
|
echo "To download ALFWorld data, run:"
|
||||||
|
echo " pip install alfworld[full]"
|
||||||
|
echo " alfworld-download"
|
||||||
|
echo ""
|
||||||
|
echo "Or set ALFWORLD_DATA to the directory containing json_2.1.1/"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# ── Azure OpenAI credentials ────────────────────────────────────────────────
|
||||||
|
export AZURE_OPENAI_ENDPOINT="${AZURE_OPENAI_ENDPOINT:?Set AZURE_OPENAI_ENDPOINT}"
|
||||||
|
export AZURE_OPENAI_API_KEY="${AZURE_OPENAI_API_KEY:?Set AZURE_OPENAI_API_KEY}"
|
||||||
|
export AZURE_OPENAI_API_VERSION="${AZURE_OPENAI_API_VERSION:-2025-04-01-preview}"
|
||||||
|
|
||||||
|
# ── Model configuration ─────────────────────────────────────────────────────
|
||||||
|
export TEACHER_DEPLOYMENT="${TEACHER_DEPLOYMENT:-gpt-5.5}"
|
||||||
|
export STUDENT_DEPLOYMENT="${STUDENT_DEPLOYMENT:-gpt-5.5}"
|
||||||
|
|
||||||
|
# ── Output directory ─────────────────────────────────────────────────────────
|
||||||
|
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
|
||||||
|
DEFAULT_OUT_ROOT="${PROJECT_ROOT}/outputs/skillopt_alfworld_${STUDENT_DEPLOYMENT}_${TIMESTAMP}"
|
||||||
|
|
||||||
|
# ── Run ──────────────────────────────────────────────────────────────────────
|
||||||
|
echo "============================================================"
|
||||||
|
echo " ReflACT — Reflective Agent Tuning (ALFWorld)"
|
||||||
|
echo "============================================================"
|
||||||
|
echo " Teacher: ${TEACHER_DEPLOYMENT}"
|
||||||
|
echo " Student: ${STUDENT_DEPLOYMENT}"
|
||||||
|
echo " ALFWORLD_DATA: ${ALFWORLD_DATA}"
|
||||||
|
echo " Output: ${DEFAULT_OUT_ROOT}"
|
||||||
|
echo "============================================================"
|
||||||
|
|
||||||
|
cd "${PROJECT_ROOT}"
|
||||||
|
|
||||||
|
python scripts/train.py \
|
||||||
|
--config configs/alfworld_default.yaml \
|
||||||
|
--out_root "${DEFAULT_OUT_ROOT}" \
|
||||||
|
"$@"
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo "Done! Results saved to: ${DEFAULT_OUT_ROOT}"
|
||||||
Executable
+43
@@ -0,0 +1,43 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# ──────────────────────────────────────────────────────────────────────────────
|
||||||
|
# ReflACT — SearchQA training launch script
|
||||||
|
#
|
||||||
|
# Usage:
|
||||||
|
# bash scripts/run_searchqa.sh
|
||||||
|
# bash scripts/run_searchqa.sh --data_path data/searchqa_train_2000.json
|
||||||
|
# bash scripts/run_searchqa.sh --num_epochs 2 --edit_budget 6
|
||||||
|
# ──────────────────────────────────────────────────────────────────────────────
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
# ── Paths ────────────────────────────────────────────────────────────────────
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||||
|
PROJECT_ROOT="$(dirname "${SCRIPT_DIR}")"
|
||||||
|
|
||||||
|
# Ensure ReflACT is importable
|
||||||
|
export PYTHONPATH="${PROJECT_ROOT}:${PYTHONPATH:-}"
|
||||||
|
|
||||||
|
# ── Model configuration ─────────────────────────────────────────────────────
|
||||||
|
export TEACHER_DEPLOYMENT="${TEACHER_DEPLOYMENT:-gpt-5.5}"
|
||||||
|
export STUDENT_DEPLOYMENT="${STUDENT_DEPLOYMENT:-gpt-5.5}"
|
||||||
|
|
||||||
|
# ── Output directory ─────────────────────────────────────────────────────────
|
||||||
|
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
|
||||||
|
DEFAULT_OUT_ROOT="${PROJECT_ROOT}/outputs/skillopt_searchqa_${STUDENT_DEPLOYMENT}_${TIMESTAMP}"
|
||||||
|
|
||||||
|
# ── Run ──────────────────────────────────────────────────────────────────────
|
||||||
|
echo "============================================================"
|
||||||
|
echo " ReflACT — Reflective Agent Tuning (SearchQA)"
|
||||||
|
echo "============================================================"
|
||||||
|
echo " Teacher: ${TEACHER_DEPLOYMENT}"
|
||||||
|
echo " Student: ${STUDENT_DEPLOYMENT}"
|
||||||
|
echo "============================================================"
|
||||||
|
|
||||||
|
cd "${PROJECT_ROOT}"
|
||||||
|
|
||||||
|
python scripts/train.py \
|
||||||
|
--config configs/searchqa_default.yaml \
|
||||||
|
--out_root "${DEFAULT_OUT_ROOT}" \
|
||||||
|
"$@"
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo "Done! Results saved to: ${DEFAULT_OUT_ROOT}"
|
||||||
Executable
+48
@@ -0,0 +1,48 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# ──────────────────────────────────────────────────────────────────────────────
|
||||||
|
# ReflACT — SpreadsheetBench training launch script
|
||||||
|
#
|
||||||
|
# Usage:
|
||||||
|
# bash scripts/run_spreadsheetbench.sh \
|
||||||
|
# --data_root /path/to/data \
|
||||||
|
# --jsonl_path /path/to/benchmark.jsonl
|
||||||
|
#
|
||||||
|
# bash scripts/run_spreadsheetbench.sh \
|
||||||
|
# --data_root /path/to/data \
|
||||||
|
# --jsonl_path /path/to/benchmark.jsonl \
|
||||||
|
# --num_epochs 2 --edit_budget 6
|
||||||
|
# ──────────────────────────────────────────────────────────────────────────────
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
# ── Paths ────────────────────────────────────────────────────────────────────
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||||
|
PROJECT_ROOT="$(dirname "${SCRIPT_DIR}")"
|
||||||
|
|
||||||
|
# Ensure ReflACT is importable
|
||||||
|
export PYTHONPATH="${PROJECT_ROOT}:${PYTHONPATH:-}"
|
||||||
|
|
||||||
|
# ── Model configuration ─────────────────────────────────────────────────────
|
||||||
|
export TEACHER_DEPLOYMENT="${TEACHER_DEPLOYMENT:-gpt-5.5}"
|
||||||
|
export STUDENT_DEPLOYMENT="${STUDENT_DEPLOYMENT:-gpt-5.5}"
|
||||||
|
|
||||||
|
# ── Output directory ─────────────────────────────────────────────────────────
|
||||||
|
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
|
||||||
|
DEFAULT_OUT_ROOT="${PROJECT_ROOT}/outputs/skillopt_spreadsheetbench_${STUDENT_DEPLOYMENT}_${TIMESTAMP}"
|
||||||
|
|
||||||
|
# ── Run ──────────────────────────────────────────────────────────────────────
|
||||||
|
echo "============================================================"
|
||||||
|
echo " ReflACT — Reflective Agent Tuning (SpreadsheetBench)"
|
||||||
|
echo "============================================================"
|
||||||
|
echo " Teacher: ${TEACHER_DEPLOYMENT}"
|
||||||
|
echo " Student: ${STUDENT_DEPLOYMENT}"
|
||||||
|
echo "============================================================"
|
||||||
|
|
||||||
|
cd "${PROJECT_ROOT}"
|
||||||
|
|
||||||
|
python scripts/train.py \
|
||||||
|
--config configs/spreadsheetbench_default.yaml \
|
||||||
|
--out_root "${DEFAULT_OUT_ROOT}" \
|
||||||
|
"$@"
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo "Done! Results saved to: ${DEFAULT_OUT_ROOT}"
|
||||||
@@ -0,0 +1,504 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""ReflACT unified training entry point.
|
||||||
|
|
||||||
|
Usage
|
||||||
|
-----
|
||||||
|
python scripts/train.py --config configs/alfworld/default.yaml
|
||||||
|
|
||||||
|
Any YAML key can be overridden from the command line::
|
||||||
|
|
||||||
|
python scripts/train.py --config configs/alfworld/default.yaml \\
|
||||||
|
--batch_size 40 --num_epochs 2 --seed 123
|
||||||
|
|
||||||
|
Run ``python scripts/train.py --help`` for a full list of options.
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import datetime
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
|
||||||
|
# Ensure the project root is on sys.path so ``import skillopt`` works
|
||||||
|
# regardless of where the script is invoked from.
|
||||||
|
_SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__))
|
||||||
|
_PROJECT_ROOT = os.path.dirname(_SCRIPT_DIR)
|
||||||
|
if _PROJECT_ROOT not in sys.path:
|
||||||
|
sys.path.insert(0, _PROJECT_ROOT)
|
||||||
|
|
||||||
|
from skillopt.model.common import default_model_for_backend, normalize_backend_name
|
||||||
|
|
||||||
|
_OPENAI_DEFAULT_MODEL_SENTINELS = {"gpt-5.4", "gpt-5.5"}
|
||||||
|
|
||||||
|
|
||||||
|
# ── Environment registry ────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
_ENV_REGISTRY: dict[str, type] = {}
|
||||||
|
|
||||||
|
|
||||||
|
def _register_builtins() -> None:
|
||||||
|
"""Lazy-import built-in adapters so we don't pull heavy deps at CLI parse time."""
|
||||||
|
try:
|
||||||
|
from skillopt.envs.alfworld.adapter import ALFWorldAdapter
|
||||||
|
_ENV_REGISTRY["alfworld"] = ALFWorldAdapter
|
||||||
|
except ImportError:
|
||||||
|
pass # ALFWorld deps not installed — skip
|
||||||
|
try:
|
||||||
|
from skillopt.envs.searchqa.adapter import SearchQAAdapter
|
||||||
|
_ENV_REGISTRY["searchqa"] = SearchQAAdapter
|
||||||
|
except ImportError:
|
||||||
|
pass
|
||||||
|
try:
|
||||||
|
from skillopt.envs.livemathematicianbench.adapter import LiveMathematicianBenchAdapter
|
||||||
|
_ENV_REGISTRY["livemathematicianbench"] = LiveMathematicianBenchAdapter
|
||||||
|
except ImportError:
|
||||||
|
pass
|
||||||
|
try:
|
||||||
|
from skillopt.envs.babyvision.adapter import BabyVisionAdapter
|
||||||
|
_ENV_REGISTRY["babyvision"] = BabyVisionAdapter
|
||||||
|
except ImportError:
|
||||||
|
pass
|
||||||
|
try:
|
||||||
|
from skillopt.envs.spreadsheetbench.adapter import SpreadsheetBenchAdapter
|
||||||
|
_ENV_REGISTRY["spreadsheetbench"] = SpreadsheetBenchAdapter
|
||||||
|
except ImportError:
|
||||||
|
pass
|
||||||
|
try:
|
||||||
|
from skillopt.envs.mmrb.adapter import MMRBAdapter
|
||||||
|
_ENV_REGISTRY["mmrb"] = MMRBAdapter
|
||||||
|
except ImportError:
|
||||||
|
pass
|
||||||
|
try:
|
||||||
|
from skillopt.envs.docvqa.adapter import DocVQAAdapter
|
||||||
|
_ENV_REGISTRY["docvqa"] = DocVQAAdapter
|
||||||
|
except ImportError:
|
||||||
|
pass
|
||||||
|
try:
|
||||||
|
from skillopt.envs.mathverse.adapter import MathVerseAdapter
|
||||||
|
_ENV_REGISTRY["mathverse"] = MathVerseAdapter
|
||||||
|
except ImportError:
|
||||||
|
pass
|
||||||
|
try:
|
||||||
|
from skillopt.envs.officeqa.adapter import OfficeQAAdapter
|
||||||
|
_ENV_REGISTRY["officeqa"] = OfficeQAAdapter
|
||||||
|
except ImportError:
|
||||||
|
pass
|
||||||
|
try:
|
||||||
|
from skillopt.envs.sealqa.adapter import SealQAAdapter
|
||||||
|
_ENV_REGISTRY["sealqa"] = SealQAAdapter
|
||||||
|
except ImportError:
|
||||||
|
pass
|
||||||
|
try:
|
||||||
|
from skillopt.envs.swebench.adapter import SWEBenchAdapter
|
||||||
|
_ENV_REGISTRY["swebench"] = SWEBenchAdapter
|
||||||
|
except ImportError:
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
def get_adapter(cfg: dict):
|
||||||
|
"""Instantiate the environment adapter specified in ``cfg["env"]``."""
|
||||||
|
_register_builtins()
|
||||||
|
env_name = cfg.get("env", "alfworld")
|
||||||
|
if env_name not in _ENV_REGISTRY:
|
||||||
|
raise ValueError(
|
||||||
|
f"Unknown environment '{env_name}'. "
|
||||||
|
f"Available: {list(_ENV_REGISTRY.keys())}"
|
||||||
|
)
|
||||||
|
adapter_cls = _ENV_REGISTRY[env_name]
|
||||||
|
|
||||||
|
# Inspect adapter __init__ signature and only pass accepted kwargs
|
||||||
|
import inspect
|
||||||
|
sig = inspect.signature(adapter_cls.__init__)
|
||||||
|
accepted = set(sig.parameters.keys()) - {"self"}
|
||||||
|
adapter_kwargs: dict = {}
|
||||||
|
for key in accepted:
|
||||||
|
if key in cfg:
|
||||||
|
adapter_kwargs[key] = cfg[key]
|
||||||
|
|
||||||
|
return adapter_cls(**adapter_kwargs)
|
||||||
|
|
||||||
|
|
||||||
|
# ── CLI ──────────────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
_BOOL = lambda x: x.lower() in ("true", "1", "yes") # noqa: E731
|
||||||
|
|
||||||
|
|
||||||
|
def parse_args() -> argparse.Namespace:
|
||||||
|
p = argparse.ArgumentParser(
|
||||||
|
description="ReflACT: Reflective Agent Tuning",
|
||||||
|
formatter_class=argparse.RawDescriptionHelpFormatter,
|
||||||
|
epilog=__doc__,
|
||||||
|
)
|
||||||
|
p.add_argument("--config", type=str, required=True,
|
||||||
|
help="Path to YAML config file")
|
||||||
|
p.add_argument("--cfg-options", nargs="+", default=[],
|
||||||
|
help="Override config: section.key=value (e.g. train.batch_size=40)")
|
||||||
|
|
||||||
|
# Legacy flat CLI overrides (still work, prefer --cfg-options for new usage)
|
||||||
|
p.add_argument("--env", type=str)
|
||||||
|
p.add_argument("--backend", type=str,
|
||||||
|
choices=["azure_openai", "codex", "codex_exec", "claude", "claude_chat", "claude_code_exec", "qwen", "qwen_chat"])
|
||||||
|
p.add_argument("--teacher_model", type=str)
|
||||||
|
p.add_argument("--student_model", type=str)
|
||||||
|
p.add_argument("--teacher_backend", type=str)
|
||||||
|
p.add_argument("--student_backend", type=str)
|
||||||
|
p.add_argument("--reasoning_effort", type=str,
|
||||||
|
choices=["", "low", "medium", "high", "xhigh", "max"])
|
||||||
|
p.add_argument("--rewrite_reasoning_effort", type=str)
|
||||||
|
p.add_argument("--rewrite_max_completion_tokens", type=int)
|
||||||
|
p.add_argument("--azure_endpoint", type=str)
|
||||||
|
p.add_argument("--azure_api_version", type=str)
|
||||||
|
p.add_argument("--azure_api_key", type=str)
|
||||||
|
p.add_argument("--azure_openai_endpoint", type=str)
|
||||||
|
p.add_argument("--azure_openai_api_version", type=str)
|
||||||
|
p.add_argument("--azure_openai_api_key", type=str)
|
||||||
|
p.add_argument("--azure_openai_auth_mode", type=str)
|
||||||
|
p.add_argument("--azure_openai_ad_scope", type=str)
|
||||||
|
p.add_argument("--azure_openai_managed_identity_client_id", type=str)
|
||||||
|
p.add_argument("--teacher_azure_openai_endpoint", type=str)
|
||||||
|
p.add_argument("--teacher_azure_openai_api_version", type=str)
|
||||||
|
p.add_argument("--teacher_azure_openai_api_key", type=str)
|
||||||
|
p.add_argument("--teacher_azure_openai_auth_mode", type=str)
|
||||||
|
p.add_argument("--teacher_azure_openai_ad_scope", type=str)
|
||||||
|
p.add_argument("--teacher_azure_openai_managed_identity_client_id", type=str)
|
||||||
|
p.add_argument("--student_azure_openai_endpoint", type=str)
|
||||||
|
p.add_argument("--student_azure_openai_api_version", type=str)
|
||||||
|
p.add_argument("--student_azure_openai_api_key", type=str)
|
||||||
|
p.add_argument("--student_azure_openai_auth_mode", type=str)
|
||||||
|
p.add_argument("--student_azure_openai_ad_scope", type=str)
|
||||||
|
p.add_argument("--student_azure_openai_managed_identity_client_id", type=str)
|
||||||
|
p.add_argument("--qwen_chat_base_url", type=str)
|
||||||
|
p.add_argument("--qwen_chat_api_key", type=str)
|
||||||
|
p.add_argument("--qwen_chat_temperature", type=float)
|
||||||
|
p.add_argument("--qwen_chat_timeout_seconds", type=float)
|
||||||
|
p.add_argument("--qwen_chat_max_tokens", type=int)
|
||||||
|
p.add_argument("--qwen_chat_enable_thinking", type=_BOOL)
|
||||||
|
p.add_argument("--codex_exec_path", type=str)
|
||||||
|
p.add_argument("--codex_exec_sandbox", type=str)
|
||||||
|
p.add_argument("--codex_exec_profile", type=str)
|
||||||
|
p.add_argument("--codex_exec_full_auto", type=_BOOL)
|
||||||
|
p.add_argument("--codex_exec_reasoning_effort", type=str)
|
||||||
|
p.add_argument("--codex_exec_use_sdk", type=str)
|
||||||
|
p.add_argument("--codex_exec_network_access", type=_BOOL)
|
||||||
|
p.add_argument("--codex_exec_web_search", type=_BOOL)
|
||||||
|
p.add_argument("--codex_exec_approval_policy", type=str)
|
||||||
|
p.add_argument("--claude_code_exec_path", type=str)
|
||||||
|
p.add_argument("--claude_code_exec_profile", type=str)
|
||||||
|
p.add_argument("--claude_code_exec_use_sdk", type=str)
|
||||||
|
p.add_argument("--claude_code_exec_effort", type=str)
|
||||||
|
p.add_argument("--claude_code_exec_max_thinking_tokens", type=int)
|
||||||
|
p.add_argument("--codex_trace_to_teacher", type=_BOOL)
|
||||||
|
p.add_argument("--skill_init", type=str)
|
||||||
|
p.add_argument("--num_epochs", type=int)
|
||||||
|
p.add_argument("--train_size", type=int)
|
||||||
|
p.add_argument("--steps_per_epoch", type=int)
|
||||||
|
p.add_argument("--batch_size", type=int)
|
||||||
|
p.add_argument("--accumulation", type=int)
|
||||||
|
p.add_argument("--seed", type=int)
|
||||||
|
p.add_argument("--edit_budget", type=int)
|
||||||
|
p.add_argument("--min_edit_budget", type=int)
|
||||||
|
p.add_argument("--lr_scheduler", type=str,
|
||||||
|
choices=["constant", "linear", "cosine", "autonomous"])
|
||||||
|
p.add_argument("--lr_control_mode", type=str,
|
||||||
|
choices=["fixed", "autonomous", "none"])
|
||||||
|
p.add_argument("--merge_batch_size", type=int)
|
||||||
|
p.add_argument("--max_analyst_rounds", type=int)
|
||||||
|
p.add_argument("--sel_env_num", type=int)
|
||||||
|
p.add_argument("--test_env_num", type=int)
|
||||||
|
p.add_argument("--eval_test", type=_BOOL)
|
||||||
|
p.add_argument("--use_gate", type=_BOOL)
|
||||||
|
p.add_argument("--max_steps", type=int)
|
||||||
|
p.add_argument("--max_api_workers", type=int)
|
||||||
|
p.add_argument("--analyst_workers", type=int)
|
||||||
|
p.add_argument("--failure_only", type=_BOOL)
|
||||||
|
p.add_argument("--minibatch_size", type=int)
|
||||||
|
p.add_argument("--use_meta_reflect", type=_BOOL)
|
||||||
|
p.add_argument("--meta_edit_budget", type=int)
|
||||||
|
p.add_argument("--skill_update_mode", type=str,
|
||||||
|
choices=[
|
||||||
|
"patch",
|
||||||
|
"rewrite_from_suggestions",
|
||||||
|
"rewrite",
|
||||||
|
"suggestions",
|
||||||
|
"full_rewrite",
|
||||||
|
"full_rewrite_minibatch",
|
||||||
|
"minibatch_full_rewrite",
|
||||||
|
])
|
||||||
|
p.add_argument("--use_deep_reflect", type=_BOOL)
|
||||||
|
p.add_argument("--deep_reflect_failures", type=int)
|
||||||
|
p.add_argument("--deep_reflect_successes", type=int)
|
||||||
|
p.add_argument("--use_slow_update", type=_BOOL)
|
||||||
|
p.add_argument("--slow_update_samples", type=int)
|
||||||
|
p.add_argument("--longitudinal_pair_policy", type=str,
|
||||||
|
choices=["mixed", "changed", "unchanged"])
|
||||||
|
p.add_argument("--use_meta_skill", type=_BOOL)
|
||||||
|
p.add_argument("--data_path", type=str)
|
||||||
|
p.add_argument("--split_mode", type=str,
|
||||||
|
choices=["ratio", "split_dir"])
|
||||||
|
p.add_argument("--split_ratio", type=str)
|
||||||
|
p.add_argument("--split_seed", type=int)
|
||||||
|
p.add_argument("--split_dir", type=str)
|
||||||
|
p.add_argument("--split_output_dir", type=str)
|
||||||
|
p.add_argument("--data_root", type=str)
|
||||||
|
p.add_argument("--max_turns", type=int)
|
||||||
|
p.add_argument("--workers", type=int)
|
||||||
|
p.add_argument("--limit", type=int)
|
||||||
|
p.add_argument("--shuffle_choices", type=_BOOL)
|
||||||
|
p.add_argument("--use_theorem", type=_BOOL)
|
||||||
|
p.add_argument("--use_sketch", type=_BOOL)
|
||||||
|
p.add_argument("--image_detail", type=str)
|
||||||
|
p.add_argument("--judge_model", type=str)
|
||||||
|
p.add_argument("--judge_max_completion_tokens", type=int)
|
||||||
|
p.add_argument("--judge_retries", type=int)
|
||||||
|
p.add_argument("--out_root", type=str)
|
||||||
|
p.add_argument("--mode", type=str)
|
||||||
|
|
||||||
|
return p.parse_args()
|
||||||
|
|
||||||
|
|
||||||
|
# ── Flat key → structured path mapping (for legacy CLI → structured config) ──
|
||||||
|
|
||||||
|
_LEGACY_TO_STRUCTURED: dict[str, str] = {
|
||||||
|
"backend": "model.backend",
|
||||||
|
"teacher_model": "model.teacher",
|
||||||
|
"student_model": "model.student",
|
||||||
|
"teacher_backend": "model.teacher_backend",
|
||||||
|
"student_backend": "model.student_backend",
|
||||||
|
"reasoning_effort": "model.reasoning_effort",
|
||||||
|
"rewrite_reasoning_effort": "model.rewrite_reasoning_effort",
|
||||||
|
"rewrite_max_completion_tokens": "model.rewrite_max_completion_tokens",
|
||||||
|
"azure_endpoint": "model.azure_endpoint",
|
||||||
|
"azure_api_version": "model.azure_api_version",
|
||||||
|
"azure_api_key": "model.azure_api_key",
|
||||||
|
"azure_openai_endpoint": "model.azure_openai_endpoint",
|
||||||
|
"azure_openai_api_version": "model.azure_openai_api_version",
|
||||||
|
"azure_openai_api_key": "model.azure_openai_api_key",
|
||||||
|
"azure_openai_auth_mode": "model.azure_openai_auth_mode",
|
||||||
|
"azure_openai_ad_scope": "model.azure_openai_ad_scope",
|
||||||
|
"azure_openai_managed_identity_client_id": "model.azure_openai_managed_identity_client_id",
|
||||||
|
"teacher_azure_openai_endpoint": "model.teacher_azure_openai_endpoint",
|
||||||
|
"teacher_azure_openai_api_version": "model.teacher_azure_openai_api_version",
|
||||||
|
"teacher_azure_openai_api_key": "model.teacher_azure_openai_api_key",
|
||||||
|
"teacher_azure_openai_auth_mode": "model.teacher_azure_openai_auth_mode",
|
||||||
|
"teacher_azure_openai_ad_scope": "model.teacher_azure_openai_ad_scope",
|
||||||
|
"teacher_azure_openai_managed_identity_client_id": "model.teacher_azure_openai_managed_identity_client_id",
|
||||||
|
"student_azure_openai_endpoint": "model.student_azure_openai_endpoint",
|
||||||
|
"student_azure_openai_api_version": "model.student_azure_openai_api_version",
|
||||||
|
"student_azure_openai_api_key": "model.student_azure_openai_api_key",
|
||||||
|
"student_azure_openai_auth_mode": "model.student_azure_openai_auth_mode",
|
||||||
|
"student_azure_openai_ad_scope": "model.student_azure_openai_ad_scope",
|
||||||
|
"student_azure_openai_managed_identity_client_id": "model.student_azure_openai_managed_identity_client_id",
|
||||||
|
"qwen_chat_base_url": "model.qwen_chat_base_url",
|
||||||
|
"qwen_chat_api_key": "model.qwen_chat_api_key",
|
||||||
|
"qwen_chat_temperature": "model.qwen_chat_temperature",
|
||||||
|
"qwen_chat_timeout_seconds": "model.qwen_chat_timeout_seconds",
|
||||||
|
"qwen_chat_max_tokens": "model.qwen_chat_max_tokens",
|
||||||
|
"qwen_chat_enable_thinking": "model.qwen_chat_enable_thinking",
|
||||||
|
"codex_exec_path": "model.codex_exec_path",
|
||||||
|
"codex_exec_sandbox": "model.codex_exec_sandbox",
|
||||||
|
"codex_exec_profile": "model.codex_exec_profile",
|
||||||
|
"codex_exec_full_auto": "model.codex_exec_full_auto",
|
||||||
|
"codex_exec_reasoning_effort": "model.codex_exec_reasoning_effort",
|
||||||
|
"codex_exec_use_sdk": "model.codex_exec_use_sdk",
|
||||||
|
"codex_exec_network_access": "model.codex_exec_network_access",
|
||||||
|
"codex_exec_web_search": "model.codex_exec_web_search",
|
||||||
|
"codex_exec_approval_policy": "model.codex_exec_approval_policy",
|
||||||
|
"claude_code_exec_path": "model.claude_code_exec_path",
|
||||||
|
"claude_code_exec_profile": "model.claude_code_exec_profile",
|
||||||
|
"claude_code_exec_use_sdk": "model.claude_code_exec_use_sdk",
|
||||||
|
"claude_code_exec_effort": "model.claude_code_exec_effort",
|
||||||
|
"claude_code_exec_max_thinking_tokens": "model.claude_code_exec_max_thinking_tokens",
|
||||||
|
"codex_trace_to_teacher": "model.codex_trace_to_teacher",
|
||||||
|
"num_epochs": "train.num_epochs",
|
||||||
|
"train_size": "train.train_size",
|
||||||
|
"steps_per_epoch": "train.steps_per_epoch",
|
||||||
|
"batch_size": "train.batch_size",
|
||||||
|
"accumulation": "train.accumulation",
|
||||||
|
"seed": "train.seed",
|
||||||
|
"minibatch_size": "gradient.minibatch_size",
|
||||||
|
"merge_batch_size": "gradient.merge_batch_size",
|
||||||
|
"analyst_workers": "gradient.analyst_workers",
|
||||||
|
"max_analyst_rounds": "gradient.max_analyst_rounds",
|
||||||
|
"failure_only": "gradient.failure_only",
|
||||||
|
"use_deep_reflect": "gradient.use_deep_reflect",
|
||||||
|
"deep_reflect_failures": "gradient.deep_reflect_failures",
|
||||||
|
"deep_reflect_successes": "gradient.deep_reflect_successes",
|
||||||
|
"edit_budget": "optimizer.learning_rate",
|
||||||
|
"min_edit_budget": "optimizer.min_learning_rate",
|
||||||
|
"lr_scheduler": "optimizer.lr_scheduler",
|
||||||
|
"lr_control_mode": "optimizer.lr_control_mode",
|
||||||
|
"skill_update_mode": "optimizer.skill_update_mode",
|
||||||
|
"use_meta_reflect": "optimizer.use_meta_reflect",
|
||||||
|
"meta_edit_budget": "optimizer.meta_learning_rate",
|
||||||
|
"use_slow_update": "optimizer.use_slow_update",
|
||||||
|
"slow_update_samples": "optimizer.slow_update_samples",
|
||||||
|
"longitudinal_pair_policy": "optimizer.longitudinal_pair_policy",
|
||||||
|
"use_meta_skill": "optimizer.use_meta_skill",
|
||||||
|
"use_gate": "evaluation.use_gate",
|
||||||
|
"sel_env_num": "evaluation.sel_env_num",
|
||||||
|
"test_env_num": "evaluation.test_env_num",
|
||||||
|
"eval_test": "evaluation.eval_test",
|
||||||
|
"env": "env.name",
|
||||||
|
"skill_init": "env.skill_init",
|
||||||
|
"out_root": "env.out_root",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def load_config(args: argparse.Namespace) -> dict:
|
||||||
|
"""Load config with _base_ inheritance, then apply CLI overrides."""
|
||||||
|
from skillopt.config import load_config as _load, flatten_config, is_structured
|
||||||
|
|
||||||
|
cfg = _load(args.config, overrides=args.cfg_options)
|
||||||
|
structured = is_structured(cfg)
|
||||||
|
|
||||||
|
# Apply legacy --key value overrides
|
||||||
|
cli = {k: v for k, v in vars(args).items()
|
||||||
|
if v is not None and k not in ("config", "cfg_options")}
|
||||||
|
if cli:
|
||||||
|
if structured:
|
||||||
|
from skillopt.config import apply_overrides
|
||||||
|
mapped = []
|
||||||
|
for k, v in cli.items():
|
||||||
|
dotted = _LEGACY_TO_STRUCTURED.get(k)
|
||||||
|
if dotted:
|
||||||
|
mapped.append(f"{dotted}={v}")
|
||||||
|
else:
|
||||||
|
mapped.append(f"env.{k}={v}")
|
||||||
|
apply_overrides(cfg, mapped)
|
||||||
|
else:
|
||||||
|
cfg.update(cli)
|
||||||
|
|
||||||
|
# Flatten structured config → flat dict for trainer/adapter
|
||||||
|
flat = flatten_config(cfg) if structured else cfg
|
||||||
|
|
||||||
|
for new_key, old_key in (
|
||||||
|
("azure_openai_endpoint", "azure_endpoint"),
|
||||||
|
("azure_openai_api_version", "azure_api_version"),
|
||||||
|
("azure_openai_api_key", "azure_api_key"),
|
||||||
|
):
|
||||||
|
if flat.get(new_key) in (None, "") and flat.get(old_key) not in (None, ""):
|
||||||
|
flat[new_key] = flat[old_key]
|
||||||
|
|
||||||
|
explicit_backend = getattr(args, "backend", None)
|
||||||
|
if explicit_backend is None:
|
||||||
|
for option in args.cfg_options or []:
|
||||||
|
key = str(option).split("=", 1)[0].strip()
|
||||||
|
if key == "model.backend":
|
||||||
|
explicit_backend = str(option).split("=", 1)[1].strip()
|
||||||
|
break
|
||||||
|
|
||||||
|
backend = normalize_backend_name(flat.get("model_backend") or flat.get("student_backend") or "azure_openai")
|
||||||
|
|
||||||
|
def _has_model_override(dotted_key: str, legacy_key: str) -> bool:
|
||||||
|
if getattr(args, legacy_key, None) is not None:
|
||||||
|
return True
|
||||||
|
for option in args.cfg_options or []:
|
||||||
|
key = str(option).split("=", 1)[0].strip()
|
||||||
|
if key == dotted_key:
|
||||||
|
return True
|
||||||
|
return False
|
||||||
|
|
||||||
|
if explicit_backend is not None:
|
||||||
|
backend = normalize_backend_name(explicit_backend)
|
||||||
|
flat["model_backend"] = backend
|
||||||
|
if backend in {"claude", "claude_chat"}:
|
||||||
|
flat.setdefault("teacher_backend", "claude_chat")
|
||||||
|
flat.setdefault("student_backend", "claude_chat")
|
||||||
|
elif backend in {"codex", "codex_exec"}:
|
||||||
|
flat.setdefault("teacher_backend", "openai_chat")
|
||||||
|
flat.setdefault("student_backend", "codex_exec")
|
||||||
|
elif backend == "claude_code_exec":
|
||||||
|
flat.setdefault("teacher_backend", "openai_chat")
|
||||||
|
flat.setdefault("student_backend", "claude_code_exec")
|
||||||
|
elif backend in {"qwen", "qwen_chat"}:
|
||||||
|
flat.setdefault("teacher_backend", "openai_chat")
|
||||||
|
flat.setdefault("student_backend", "qwen_chat")
|
||||||
|
else:
|
||||||
|
flat.setdefault("teacher_backend", "openai_chat")
|
||||||
|
flat.setdefault("student_backend", "openai_chat")
|
||||||
|
else:
|
||||||
|
flat.setdefault("teacher_backend", "openai_chat")
|
||||||
|
flat.setdefault("student_backend", "openai_chat")
|
||||||
|
|
||||||
|
if flat.get("teacher_backend") == "claude_chat":
|
||||||
|
if (
|
||||||
|
str(flat.get("teacher_model", "") or "").strip() in _OPENAI_DEFAULT_MODEL_SENTINELS
|
||||||
|
and not _has_model_override("model.teacher", "teacher_model")
|
||||||
|
):
|
||||||
|
flat["teacher_model"] = default_model_for_backend("claude_chat")
|
||||||
|
if flat.get("student_backend") == "claude_chat":
|
||||||
|
if (
|
||||||
|
str(flat.get("student_model", "") or "").strip() in _OPENAI_DEFAULT_MODEL_SENTINELS
|
||||||
|
and not _has_model_override("model.student", "student_model")
|
||||||
|
):
|
||||||
|
flat["student_model"] = default_model_for_backend("claude_chat")
|
||||||
|
if flat.get("student_backend") == "claude_code_exec":
|
||||||
|
if (
|
||||||
|
str(flat.get("student_model", "") or "").strip() in _OPENAI_DEFAULT_MODEL_SENTINELS
|
||||||
|
and not _has_model_override("model.student", "student_model")
|
||||||
|
):
|
||||||
|
flat["student_model"] = default_model_for_backend("claude_chat")
|
||||||
|
if flat.get("student_backend") == "qwen_chat":
|
||||||
|
if (
|
||||||
|
str(flat.get("student_model", "") or "").strip() in _OPENAI_DEFAULT_MODEL_SENTINELS
|
||||||
|
and not _has_model_override("model.student", "student_model")
|
||||||
|
):
|
||||||
|
flat["student_model"] = default_model_for_backend("qwen_chat")
|
||||||
|
|
||||||
|
# Auto-generate output root
|
||||||
|
if not flat.get("out_root"):
|
||||||
|
env = flat.get("env", "unknown")
|
||||||
|
model = flat.get("teacher_model", "unknown").replace("/", "-")
|
||||||
|
ts = datetime.datetime.now().strftime("%Y%m%d_%H%M%S")
|
||||||
|
flat["out_root"] = os.path.join("outputs", f"skillopt_{env}_{model}_{ts}")
|
||||||
|
|
||||||
|
flat["out_root"] = os.path.abspath(flat["out_root"])
|
||||||
|
return flat
|
||||||
|
|
||||||
|
|
||||||
|
# ── Main ─────────────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
def main() -> None:
|
||||||
|
args = parse_args()
|
||||||
|
cfg = load_config(args)
|
||||||
|
|
||||||
|
print(f"\n{'='*60}")
|
||||||
|
print(f" ReflACT — Reflective Agent Tuning")
|
||||||
|
print(f"{'='*60}")
|
||||||
|
print(f" env: {cfg.get('env')}")
|
||||||
|
print(f" teacher_model: {cfg.get('teacher_model')}")
|
||||||
|
print(f" student_model: {cfg.get('student_model')}")
|
||||||
|
print(f" teacher_backend:{cfg.get('teacher_backend', 'openai_chat')}")
|
||||||
|
print(f" student_backend:{cfg.get('student_backend', 'openai_chat')}")
|
||||||
|
print(f" reasoning: {cfg.get('reasoning_effort') or 'off'}")
|
||||||
|
print(f" rewrite_effort: {cfg.get('rewrite_reasoning_effort') or 'off'}")
|
||||||
|
print(f" epochs: {cfg.get('num_epochs')}")
|
||||||
|
print(f" train_size: {cfg.get('train_size') or 'from dataset'}")
|
||||||
|
print(f" steps/epoch: auto")
|
||||||
|
print(f" batch_size: {cfg.get('batch_size')}")
|
||||||
|
print(f" edit_budget: {cfg.get('edit_budget')}")
|
||||||
|
print(f" lr_scheduler: {cfg.get('lr_scheduler', 'constant')}")
|
||||||
|
print(f" update_mode: {cfg.get('skill_update_mode', 'patch')}")
|
||||||
|
print(f" min_edit_budget:{cfg.get('min_edit_budget', 2)}")
|
||||||
|
print(f" minibatch_size: {cfg.get('minibatch_size')}")
|
||||||
|
print(f" seed: {cfg.get('seed')}")
|
||||||
|
print(f" meta_reflect: {cfg.get('use_meta_reflect', False)}")
|
||||||
|
print(f" meta_skill: {cfg.get('use_meta_skill', False)}")
|
||||||
|
print(f" out_root: {cfg.get('out_root')}")
|
||||||
|
print(f"{'='*60}\n")
|
||||||
|
|
||||||
|
# Build adapter
|
||||||
|
adapter = get_adapter(cfg)
|
||||||
|
|
||||||
|
# Build trainer and run
|
||||||
|
from skillopt.engine.trainer import ReflACTTrainer
|
||||||
|
trainer = ReflACTTrainer(cfg, adapter)
|
||||||
|
summary = trainer.train()
|
||||||
|
|
||||||
|
print(f"\n Output saved to: {cfg['out_root']}")
|
||||||
|
if summary.get("test_hard") is not None:
|
||||||
|
print(f" Final test: {summary['test_hard']:.4f}")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,29 @@
|
|||||||
|
"""ReflACT: Reflective Agent Tuning.
|
||||||
|
|
||||||
|
A general-purpose framework for iteratively optimizing LLM agent skills
|
||||||
|
through structured reflection and self-improvement.
|
||||||
|
|
||||||
|
Pipeline stages:
|
||||||
|
1. Rollout — execute episodes with current skill
|
||||||
|
2. Reflect — analyze trajectories, generate patches
|
||||||
|
3. Aggregate — hierarchical merge of patches
|
||||||
|
4. Select — rank and select top edits
|
||||||
|
5. Update — apply edits to skill document
|
||||||
|
6. Evaluate — validate candidate skill, accept/reject
|
||||||
|
"""
|
||||||
|
|
||||||
|
__version__ = "0.1.0"
|
||||||
|
|
||||||
|
from skillopt.types import ( # noqa: F401
|
||||||
|
BatchSpec,
|
||||||
|
Edit,
|
||||||
|
EditOp,
|
||||||
|
FailureSummaryEntry,
|
||||||
|
GateAction,
|
||||||
|
GateResult,
|
||||||
|
MetaReflectResult,
|
||||||
|
Patch,
|
||||||
|
RawPatch,
|
||||||
|
RolloutResult,
|
||||||
|
SlowUpdateResult,
|
||||||
|
)
|
||||||
@@ -0,0 +1,269 @@
|
|||||||
|
"""ReflACT config loading engine — structured YAML with inheritance.
|
||||||
|
|
||||||
|
Supports two config formats:
|
||||||
|
1. **Structured** (new): sections like ``model``, ``train``, ``gradient``,
|
||||||
|
``optimizer``, ``evaluation``, ``env`` — with ``_base_`` inheritance.
|
||||||
|
2. **Flat** (legacy): all keys at top level — fully backward compatible.
|
||||||
|
|
||||||
|
Usage::
|
||||||
|
|
||||||
|
from skillopt.config import load_config, flatten_config
|
||||||
|
|
||||||
|
cfg = load_config("configs/searchqa_default.yaml")
|
||||||
|
flat = flatten_config(cfg) # always returns flat dict for trainer
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import copy
|
||||||
|
import os
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
import yaml
|
||||||
|
|
||||||
|
# ── Section names that indicate a structured config ──────────────────────
|
||||||
|
|
||||||
|
_STRUCTURED_SECTIONS = frozenset({
|
||||||
|
"model", "train", "gradient", "optimizer", "evaluation", "env",
|
||||||
|
})
|
||||||
|
|
||||||
|
# ── Structured → flat key mapping ────────────────────────────────────────
|
||||||
|
|
||||||
|
_FLATTEN_MAP: dict[str, str] = {
|
||||||
|
"model.backend": "model_backend",
|
||||||
|
"model.teacher": "teacher_model",
|
||||||
|
"model.student": "student_model",
|
||||||
|
"model.teacher_backend": "teacher_backend",
|
||||||
|
"model.student_backend": "student_backend",
|
||||||
|
"model.reasoning_effort": "reasoning_effort",
|
||||||
|
"model.rewrite_reasoning_effort": "rewrite_reasoning_effort",
|
||||||
|
"model.rewrite_max_completion_tokens": "rewrite_max_completion_tokens",
|
||||||
|
"model.codex_exec_path": "codex_exec_path",
|
||||||
|
"model.codex_exec_sandbox": "codex_exec_sandbox",
|
||||||
|
"model.codex_exec_profile": "codex_exec_profile",
|
||||||
|
"model.codex_exec_full_auto": "codex_exec_full_auto",
|
||||||
|
"model.codex_exec_reasoning_effort": "codex_exec_reasoning_effort",
|
||||||
|
"model.codex_exec_use_sdk": "codex_exec_use_sdk",
|
||||||
|
"model.codex_exec_network_access": "codex_exec_network_access",
|
||||||
|
"model.codex_exec_web_search": "codex_exec_web_search",
|
||||||
|
"model.codex_exec_approval_policy": "codex_exec_approval_policy",
|
||||||
|
"model.claude_code_exec_path": "claude_code_exec_path",
|
||||||
|
"model.claude_code_exec_profile": "claude_code_exec_profile",
|
||||||
|
"model.claude_code_exec_use_sdk": "claude_code_exec_use_sdk",
|
||||||
|
"model.claude_code_exec_effort": "claude_code_exec_effort",
|
||||||
|
"model.claude_code_exec_max_thinking_tokens": "claude_code_exec_max_thinking_tokens",
|
||||||
|
"model.codex_trace_to_teacher": "codex_trace_to_teacher",
|
||||||
|
"model.azure_endpoint": "azure_endpoint",
|
||||||
|
"model.azure_api_version": "azure_api_version",
|
||||||
|
"model.azure_api_key": "azure_api_key",
|
||||||
|
"model.azure_openai_endpoint": "azure_openai_endpoint",
|
||||||
|
"model.azure_openai_api_version": "azure_openai_api_version",
|
||||||
|
"model.azure_openai_api_key": "azure_openai_api_key",
|
||||||
|
"model.azure_openai_auth_mode": "azure_openai_auth_mode",
|
||||||
|
"model.azure_openai_ad_scope": "azure_openai_ad_scope",
|
||||||
|
"model.azure_openai_managed_identity_client_id": "azure_openai_managed_identity_client_id",
|
||||||
|
"model.teacher_azure_openai_endpoint": "teacher_azure_openai_endpoint",
|
||||||
|
"model.teacher_azure_openai_api_version": "teacher_azure_openai_api_version",
|
||||||
|
"model.teacher_azure_openai_api_key": "teacher_azure_openai_api_key",
|
||||||
|
"model.teacher_azure_openai_auth_mode": "teacher_azure_openai_auth_mode",
|
||||||
|
"model.teacher_azure_openai_ad_scope": "teacher_azure_openai_ad_scope",
|
||||||
|
"model.teacher_azure_openai_managed_identity_client_id": "teacher_azure_openai_managed_identity_client_id",
|
||||||
|
"model.student_azure_openai_endpoint": "student_azure_openai_endpoint",
|
||||||
|
"model.student_azure_openai_api_version": "student_azure_openai_api_version",
|
||||||
|
"model.student_azure_openai_api_key": "student_azure_openai_api_key",
|
||||||
|
"model.student_azure_openai_auth_mode": "student_azure_openai_auth_mode",
|
||||||
|
"model.student_azure_openai_ad_scope": "student_azure_openai_ad_scope",
|
||||||
|
"model.student_azure_openai_managed_identity_client_id": "student_azure_openai_managed_identity_client_id",
|
||||||
|
"model.qwen_chat_base_url": "qwen_chat_base_url",
|
||||||
|
"model.qwen_chat_api_key": "qwen_chat_api_key",
|
||||||
|
"model.qwen_chat_temperature": "qwen_chat_temperature",
|
||||||
|
"model.qwen_chat_timeout_seconds": "qwen_chat_timeout_seconds",
|
||||||
|
"model.qwen_chat_max_tokens": "qwen_chat_max_tokens",
|
||||||
|
"model.qwen_chat_enable_thinking": "qwen_chat_enable_thinking",
|
||||||
|
"train.num_epochs": "num_epochs",
|
||||||
|
"train.train_size": "train_size",
|
||||||
|
"train.steps_per_epoch": "steps_per_epoch",
|
||||||
|
"train.batch_size": "batch_size",
|
||||||
|
"train.accumulation": "accumulation",
|
||||||
|
"train.seed": "seed",
|
||||||
|
"gradient.minibatch_size": "minibatch_size",
|
||||||
|
"gradient.merge_batch_size": "merge_batch_size",
|
||||||
|
"gradient.analyst_workers": "analyst_workers",
|
||||||
|
"gradient.failure_only": "failure_only",
|
||||||
|
"gradient.use_deep_reflect": "use_deep_reflect",
|
||||||
|
"gradient.deep_reflect_failures": "deep_reflect_failures",
|
||||||
|
"gradient.deep_reflect_successes": "deep_reflect_successes",
|
||||||
|
"gradient.max_analyst_rounds": "max_analyst_rounds",
|
||||||
|
"optimizer.learning_rate": "edit_budget",
|
||||||
|
"optimizer.min_learning_rate": "min_edit_budget",
|
||||||
|
"optimizer.lr_scheduler": "lr_scheduler",
|
||||||
|
"optimizer.lr_control_mode": "lr_control_mode",
|
||||||
|
"optimizer.skill_update_mode": "skill_update_mode",
|
||||||
|
"optimizer.use_meta_reflect": "use_meta_reflect",
|
||||||
|
"optimizer.meta_learning_rate": "meta_edit_budget",
|
||||||
|
"optimizer.use_slow_update": "use_slow_update",
|
||||||
|
"optimizer.slow_update_samples": "slow_update_samples",
|
||||||
|
"optimizer.longitudinal_pair_policy": "longitudinal_pair_policy",
|
||||||
|
"optimizer.use_meta_skill": "use_meta_skill",
|
||||||
|
"evaluation.use_gate": "use_gate",
|
||||||
|
"evaluation.sel_env_num": "sel_env_num",
|
||||||
|
"evaluation.test_env_num": "test_env_num",
|
||||||
|
"evaluation.eval_test": "eval_test",
|
||||||
|
"env.name": "env",
|
||||||
|
"env.skill_init": "skill_init",
|
||||||
|
"env.out_root": "out_root",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
# ── Deep merge ───────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
def _deep_merge(base: dict, override: dict) -> dict:
|
||||||
|
"""Recursively merge *override* into *base* (returns new dict)."""
|
||||||
|
result = copy.deepcopy(base)
|
||||||
|
for key, val in override.items():
|
||||||
|
if key in result and isinstance(result[key], dict) and isinstance(val, dict):
|
||||||
|
result[key] = _deep_merge(result[key], val)
|
||||||
|
else:
|
||||||
|
result[key] = copy.deepcopy(val)
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
# ── YAML loading with _base_ inheritance ─────────────────────────────────
|
||||||
|
|
||||||
|
def _load_yaml(path: str, _visited: set[str] | None = None) -> dict:
|
||||||
|
"""Load a YAML file, resolving ``_base_`` inheritance recursively."""
|
||||||
|
abs_path = os.path.abspath(path)
|
||||||
|
if _visited is None:
|
||||||
|
_visited = set()
|
||||||
|
if abs_path in _visited:
|
||||||
|
raise ValueError(f"Circular _base_ inheritance: {abs_path}")
|
||||||
|
_visited.add(abs_path)
|
||||||
|
|
||||||
|
with open(abs_path) as f:
|
||||||
|
cfg = yaml.safe_load(f) or {}
|
||||||
|
|
||||||
|
base_ref = cfg.pop("_base_", None)
|
||||||
|
if base_ref:
|
||||||
|
base_path = os.path.join(os.path.dirname(abs_path), base_ref)
|
||||||
|
base_cfg = _load_yaml(base_path, _visited)
|
||||||
|
cfg = _deep_merge(base_cfg, cfg)
|
||||||
|
|
||||||
|
return cfg
|
||||||
|
|
||||||
|
|
||||||
|
# ── Format detection ─────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
def is_structured(cfg: dict) -> bool:
|
||||||
|
"""Return True if *cfg* uses the new structured section format."""
|
||||||
|
return any(
|
||||||
|
key in _STRUCTURED_SECTIONS and isinstance(cfg.get(key), dict)
|
||||||
|
for key in cfg
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# ── Flatten ──────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
def flatten_config(cfg: dict) -> dict:
|
||||||
|
"""Convert a structured config to the flat dict expected by the trainer.
|
||||||
|
|
||||||
|
If *cfg* is already flat, returns a shallow copy unchanged.
|
||||||
|
"""
|
||||||
|
if not is_structured(cfg):
|
||||||
|
return dict(cfg)
|
||||||
|
|
||||||
|
flat: dict[str, Any] = {}
|
||||||
|
|
||||||
|
evaluation_section = cfg.get("evaluation", {})
|
||||||
|
if isinstance(evaluation_section, dict) and evaluation_section.get("use_gate") is False:
|
||||||
|
raise ValueError(
|
||||||
|
"Gate validation is mandatory in this branch. Remove "
|
||||||
|
"`evaluation.use_gate: false` from the config."
|
||||||
|
)
|
||||||
|
|
||||||
|
# Apply the explicit mapping
|
||||||
|
for dotted, flat_key in _FLATTEN_MAP.items():
|
||||||
|
section, key = dotted.split(".", 1)
|
||||||
|
section_dict = cfg.get(section, {})
|
||||||
|
if isinstance(section_dict, dict) and key in section_dict:
|
||||||
|
flat[flat_key] = section_dict[key]
|
||||||
|
|
||||||
|
# Pass through env-specific keys not in the explicit mapping
|
||||||
|
env_section = cfg.get("env", {})
|
||||||
|
if isinstance(env_section, dict):
|
||||||
|
mapped_env_keys = {
|
||||||
|
k.split(".", 1)[1]
|
||||||
|
for k in _FLATTEN_MAP
|
||||||
|
if k.startswith("env.")
|
||||||
|
}
|
||||||
|
for key, val in env_section.items():
|
||||||
|
if key not in mapped_env_keys:
|
||||||
|
flat[key] = val
|
||||||
|
|
||||||
|
return flat
|
||||||
|
|
||||||
|
|
||||||
|
# ── Override application ─────────────────────────────────────────────────
|
||||||
|
|
||||||
|
def _cast_value(val_str: str) -> Any:
|
||||||
|
"""Auto-cast a CLI string value to int / float / bool / str."""
|
||||||
|
if val_str.lower() in ("true", "yes"):
|
||||||
|
return True
|
||||||
|
if val_str.lower() in ("false", "no"):
|
||||||
|
return False
|
||||||
|
try:
|
||||||
|
return int(val_str)
|
||||||
|
except ValueError:
|
||||||
|
pass
|
||||||
|
try:
|
||||||
|
return float(val_str)
|
||||||
|
except ValueError:
|
||||||
|
pass
|
||||||
|
return val_str
|
||||||
|
|
||||||
|
|
||||||
|
def apply_overrides(cfg: dict, overrides: list[str]) -> None:
|
||||||
|
"""Apply ``key=value`` overrides to a structured config (in place).
|
||||||
|
|
||||||
|
Supports both ``section.key=value`` (for structured configs) and
|
||||||
|
``key=value`` (for flat configs or flat keys in env section).
|
||||||
|
"""
|
||||||
|
for item in overrides:
|
||||||
|
if "=" not in item:
|
||||||
|
raise ValueError(f"Invalid override (expected key=value): {item!r}")
|
||||||
|
key, val_str = item.split("=", 1)
|
||||||
|
val = _cast_value(val_str)
|
||||||
|
|
||||||
|
if "." in key:
|
||||||
|
section, subkey = key.split(".", 1)
|
||||||
|
if section in cfg and isinstance(cfg[section], dict):
|
||||||
|
cfg[section][subkey] = val
|
||||||
|
else:
|
||||||
|
cfg.setdefault(section, {})[subkey] = val
|
||||||
|
else:
|
||||||
|
# Flat key — apply to top level (for legacy compat)
|
||||||
|
cfg[key] = val
|
||||||
|
|
||||||
|
|
||||||
|
# ── Public API ───────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
def load_config(
|
||||||
|
path: str,
|
||||||
|
overrides: list[str] | None = None,
|
||||||
|
) -> dict:
|
||||||
|
"""Load a config file with ``_base_`` inheritance and optional overrides.
|
||||||
|
|
||||||
|
Parameters
|
||||||
|
----------
|
||||||
|
path : str
|
||||||
|
Path to the YAML config file.
|
||||||
|
overrides : list[str] | None
|
||||||
|
``key=value`` strings from ``--cfg-options``.
|
||||||
|
|
||||||
|
Returns
|
||||||
|
-------
|
||||||
|
dict
|
||||||
|
The merged config (structured or flat depending on the YAML).
|
||||||
|
"""
|
||||||
|
cfg = _load_yaml(path)
|
||||||
|
if overrides:
|
||||||
|
apply_overrides(cfg, overrides)
|
||||||
|
return cfg
|
||||||
@@ -0,0 +1,7 @@
|
|||||||
|
"""ReflACT Datasets -- task batch planning and data loading.
|
||||||
|
|
||||||
|
Analogous to the datasets and dataloaders in neural network training:
|
||||||
|
provides batch sampling, epoch planning, and data management for the
|
||||||
|
ReflACT training pipeline.
|
||||||
|
"""
|
||||||
|
from skillopt.datasets.base import BaseDataLoader, BatchSpec, SplitDataLoader # noqa: F401
|
||||||
@@ -0,0 +1,512 @@
|
|||||||
|
"""Generic task dataloader abstractions for ReflACT.
|
||||||
|
|
||||||
|
ReflACT does not train model parameters directly. Instead, it iterates over
|
||||||
|
task batches, rolls out the current skill, reflects on failures/successes,
|
||||||
|
and updates the skill document. Because of that, the "dataloader" abstraction
|
||||||
|
here is closer to a batch sampler / episode planner than a tensor loader.
|
||||||
|
|
||||||
|
Class hierarchy::
|
||||||
|
|
||||||
|
BaseDataLoader # abstract — simulator-backed envs (e.g. ALFWorld)
|
||||||
|
└── SplitDataLoader # abstract — dataset-backed envs with split_dir
|
||||||
|
|
||||||
|
SplitDataLoader supports two dataset entry modes:
|
||||||
|
|
||||||
|
1. ``split_mode="split_dir"``: consume an existing split directory.
|
||||||
|
2. ``split_mode="ratio"``: build a deterministic split directory from a raw
|
||||||
|
dataset path using an explicit train:val:test ratio.
|
||||||
|
|
||||||
|
In either case, the standardised split layout is:
|
||||||
|
|
||||||
|
split_dir/
|
||||||
|
├── train/ # training items
|
||||||
|
├── val/ # validation / selection items (gate)
|
||||||
|
└── test/ # held-out test items
|
||||||
|
|
||||||
|
Each subdirectory's contents are benchmark-specific. Subclasses only need
|
||||||
|
to implement ``load_split_items(split_path)`` to teach the loader how to
|
||||||
|
read items from one of those directories.
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import glob
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import random
|
||||||
|
from abc import ABC, abstractmethod
|
||||||
|
from dataclasses import dataclass, field
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(slots=True)
|
||||||
|
class BatchSpec:
|
||||||
|
"""A concrete batch request consumed by the training loop.
|
||||||
|
|
||||||
|
Parameters
|
||||||
|
----------
|
||||||
|
phase : str
|
||||||
|
``"train"`` or ``"eval"``.
|
||||||
|
split : str
|
||||||
|
Dataset split name, typically ``"train"`` or an eval split.
|
||||||
|
seed : int
|
||||||
|
Random seed used to construct the batch deterministically.
|
||||||
|
batch_size : int
|
||||||
|
Requested number of items / episodes in this batch.
|
||||||
|
payload : object | None
|
||||||
|
Environment-specific batch payload. For dataset-backed environments
|
||||||
|
this is often a list of sampled items; for simulator-backed
|
||||||
|
environments this may be ``None`` and the seed alone can define the
|
||||||
|
batch.
|
||||||
|
metadata : dict[str, Any]
|
||||||
|
Optional structured metadata for logging, resume, or curriculum logic.
|
||||||
|
"""
|
||||||
|
|
||||||
|
phase: str
|
||||||
|
split: str
|
||||||
|
seed: int
|
||||||
|
batch_size: int
|
||||||
|
payload: object | None = None
|
||||||
|
metadata: dict[str, Any] = field(default_factory=dict)
|
||||||
|
|
||||||
|
|
||||||
|
class BaseDataLoader(ABC):
|
||||||
|
"""Abstract base class for task batch planning in ReflACT.
|
||||||
|
|
||||||
|
Subclasses are responsible for defining how a train or eval batch is
|
||||||
|
sampled. The default implementation here provides deterministic epoch seed
|
||||||
|
planning so all loaders share the same reproducibility behavior.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def setup(self, cfg: dict) -> None:
|
||||||
|
"""Optional one-time initialization with the full trainer config."""
|
||||||
|
|
||||||
|
def set_out_root(self, out_root: str) -> None:
|
||||||
|
"""Optional hook for loaders that persist split files or state."""
|
||||||
|
|
||||||
|
def state_dict(self) -> dict[str, Any]:
|
||||||
|
"""Return serializable loader state for resume support."""
|
||||||
|
return {}
|
||||||
|
|
||||||
|
def load_state_dict(self, state: dict[str, Any]) -> None:
|
||||||
|
"""Restore loader state from :meth:`state_dict` output."""
|
||||||
|
|
||||||
|
def get_train_size(self) -> int | None:
|
||||||
|
"""Return the size of the training pool when known."""
|
||||||
|
return None
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def make_base_seeds(steps_per_epoch: int, accumulation: int, seed: int) -> list[int]:
|
||||||
|
"""Return the deterministic seed pool used to define train batches."""
|
||||||
|
batches_per_epoch = steps_per_epoch * accumulation
|
||||||
|
return [seed + i + 1 for i in range(batches_per_epoch)]
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def shuffle_epoch_seeds(base_seeds: list[int], epoch: int, seed: int) -> list[int]:
|
||||||
|
"""Return the per-epoch deterministic shuffle of *base_seeds*."""
|
||||||
|
epoch_rng = random.Random(seed + epoch * 1000)
|
||||||
|
shuffled = list(base_seeds)
|
||||||
|
epoch_rng.shuffle(shuffled)
|
||||||
|
return shuffled
|
||||||
|
|
||||||
|
def plan_train_epoch(
|
||||||
|
self,
|
||||||
|
*,
|
||||||
|
epoch: int,
|
||||||
|
steps_per_epoch: int,
|
||||||
|
accumulation: int,
|
||||||
|
batch_size: int,
|
||||||
|
seed: int,
|
||||||
|
**kwargs,
|
||||||
|
) -> list[BatchSpec]:
|
||||||
|
"""Build the full list of training batches for one epoch."""
|
||||||
|
base_seeds = self.make_base_seeds(
|
||||||
|
steps_per_epoch=steps_per_epoch,
|
||||||
|
accumulation=accumulation,
|
||||||
|
seed=seed,
|
||||||
|
)
|
||||||
|
shuffled_seeds = self.shuffle_epoch_seeds(base_seeds, epoch=epoch, seed=seed)
|
||||||
|
return [
|
||||||
|
self.build_train_batch(batch_size=batch_size, seed=batch_seed, **kwargs)
|
||||||
|
for batch_seed in shuffled_seeds
|
||||||
|
]
|
||||||
|
|
||||||
|
@abstractmethod
|
||||||
|
def build_train_batch(self, batch_size: int, seed: int, **kwargs) -> BatchSpec:
|
||||||
|
"""Construct one training batch specification."""
|
||||||
|
|
||||||
|
@abstractmethod
|
||||||
|
def build_eval_batch(
|
||||||
|
self,
|
||||||
|
env_num: int,
|
||||||
|
split: str,
|
||||||
|
seed: int,
|
||||||
|
**kwargs,
|
||||||
|
) -> BatchSpec:
|
||||||
|
"""Construct one evaluation batch specification."""
|
||||||
|
|
||||||
|
|
||||||
|
# ── Split-based dataloader for dataset-backed environments ──────────────
|
||||||
|
|
||||||
|
# Canonical split names expected under split_dir/
|
||||||
|
SPLIT_NAMES = ("train", "val", "test")
|
||||||
|
|
||||||
|
# Maps legacy / trainer split names → canonical directory names
|
||||||
|
_SPLIT_ALIAS: dict[str, str] = {
|
||||||
|
"train": "train",
|
||||||
|
"valid_seen": "val",
|
||||||
|
"selection": "val",
|
||||||
|
"val": "val",
|
||||||
|
"valid_unseen": "test",
|
||||||
|
"test": "test",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _load_json_or_jsonl(path: str) -> list[dict]:
|
||||||
|
"""Load a list of items from a JSON or JSONL file."""
|
||||||
|
with open(path, encoding="utf-8") as f:
|
||||||
|
content = f.read().strip()
|
||||||
|
if not content:
|
||||||
|
return []
|
||||||
|
|
||||||
|
try:
|
||||||
|
data = json.loads(content)
|
||||||
|
except json.JSONDecodeError:
|
||||||
|
data = None
|
||||||
|
|
||||||
|
if isinstance(data, list):
|
||||||
|
return data
|
||||||
|
if isinstance(data, dict):
|
||||||
|
nested = data.get("data")
|
||||||
|
if isinstance(nested, list):
|
||||||
|
return nested
|
||||||
|
return list(data.values())
|
||||||
|
|
||||||
|
items: list[dict] = []
|
||||||
|
for line in content.splitlines():
|
||||||
|
line = line.strip()
|
||||||
|
if line:
|
||||||
|
items.append(json.loads(line))
|
||||||
|
return items
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_split_ratio(text: str) -> tuple[int, int, int]:
|
||||||
|
parts = [part.strip() for part in str(text or "").split(":") if part.strip()]
|
||||||
|
if len(parts) != 3:
|
||||||
|
raise ValueError(
|
||||||
|
f"split_ratio must be in train:val:test form, got {text!r}"
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
train, val, test = (int(part) for part in parts)
|
||||||
|
except ValueError as exc:
|
||||||
|
raise ValueError(
|
||||||
|
f"split_ratio must contain integers, got {text!r}"
|
||||||
|
) from exc
|
||||||
|
if min(train, val, test) <= 0:
|
||||||
|
raise ValueError(f"split_ratio parts must be positive, got {text!r}")
|
||||||
|
return train, val, test
|
||||||
|
|
||||||
|
|
||||||
|
def _compute_split_counts(total: int, ratio: tuple[int, int, int]) -> tuple[int, int, int]:
|
||||||
|
weights = list(ratio)
|
||||||
|
denom = sum(weights)
|
||||||
|
raw = [total * weight / denom for weight in weights]
|
||||||
|
counts = [int(value) for value in raw]
|
||||||
|
remaining = total - sum(counts)
|
||||||
|
order = sorted(
|
||||||
|
range(len(raw)),
|
||||||
|
key=lambda idx: (raw[idx] - counts[idx], weights[idx]),
|
||||||
|
reverse=True,
|
||||||
|
)
|
||||||
|
for idx in order[:remaining]:
|
||||||
|
counts[idx] += 1
|
||||||
|
return counts[0], counts[1], counts[2]
|
||||||
|
|
||||||
|
|
||||||
|
class SplitDataLoader(BaseDataLoader):
|
||||||
|
"""Base class for dataset-backed environments.
|
||||||
|
|
||||||
|
Supported modes:
|
||||||
|
|
||||||
|
- ``split_mode="split_dir"``: load an existing ``train/``, ``val/``,
|
||||||
|
``test/`` directory tree.
|
||||||
|
- ``split_mode="ratio"``: load raw items from ``data_path`` and materialize
|
||||||
|
a deterministic split directory with the requested ratio.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
split_dir: str = "",
|
||||||
|
data_path: str = "",
|
||||||
|
split_mode: str = "ratio",
|
||||||
|
split_ratio: str = "2:1:7",
|
||||||
|
split_seed: int = 42,
|
||||||
|
split_output_dir: str = "",
|
||||||
|
seed: int = 42,
|
||||||
|
limit: int = 0,
|
||||||
|
**kwargs,
|
||||||
|
) -> None:
|
||||||
|
self.split_dir = split_dir
|
||||||
|
self.data_path = data_path
|
||||||
|
self.split_mode = split_mode
|
||||||
|
self.split_ratio = split_ratio
|
||||||
|
self.split_seed = int(split_seed)
|
||||||
|
self.split_output_dir = split_output_dir
|
||||||
|
self.seed = seed
|
||||||
|
self.limit = limit
|
||||||
|
self._splits: dict[str, list[dict]] = {}
|
||||||
|
|
||||||
|
# ── Setup ────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
def setup(self, cfg: dict) -> None:
|
||||||
|
if not self.split_mode:
|
||||||
|
self.split_mode = str(cfg.get("split_mode", "ratio") or "ratio")
|
||||||
|
if not self.split_dir:
|
||||||
|
self.split_dir = cfg.get("split_dir", "")
|
||||||
|
if not self.data_path:
|
||||||
|
self.data_path = cfg.get("data_path", "")
|
||||||
|
if not self.split_output_dir:
|
||||||
|
self.split_output_dir = cfg.get("split_output_dir", "")
|
||||||
|
if "split_seed" in cfg and not self.split_seed:
|
||||||
|
self.split_seed = int(cfg.get("split_seed", 0) or 0)
|
||||||
|
if not self.split_seed:
|
||||||
|
self.split_seed = self.seed
|
||||||
|
if not self.split_ratio:
|
||||||
|
self.split_ratio = str(cfg.get("split_ratio", "2:1:7") or "2:1:7")
|
||||||
|
|
||||||
|
mode = str(self.split_mode or "ratio").strip().lower()
|
||||||
|
if mode not in {"ratio", "split_dir"}:
|
||||||
|
raise ValueError(
|
||||||
|
f"{type(self).__name__} split_mode must be 'ratio' or 'split_dir', "
|
||||||
|
f"got {self.split_mode!r}"
|
||||||
|
)
|
||||||
|
self.split_mode = mode
|
||||||
|
|
||||||
|
if self.split_mode == "ratio":
|
||||||
|
self.split_dir = self._materialize_ratio_split(cfg)
|
||||||
|
if not self.split_dir:
|
||||||
|
raise ValueError(
|
||||||
|
f"{type(self).__name__} requires either "
|
||||||
|
"`split_mode=ratio` with `data_path`, or `split_mode=split_dir` "
|
||||||
|
f"with `split_dir` pointing to {'/'.join(SPLIT_NAMES)}/."
|
||||||
|
)
|
||||||
|
self._load_all_splits()
|
||||||
|
|
||||||
|
def _resolve_split_output_dir(self, cfg: dict) -> str:
|
||||||
|
if self.split_output_dir:
|
||||||
|
return os.path.abspath(self.split_output_dir)
|
||||||
|
out_root = os.path.abspath(str(cfg.get("out_root") or os.getcwd()))
|
||||||
|
env_name = str(cfg.get("env") or type(self).__name__.replace("DataLoader", "").lower())
|
||||||
|
ratio_tag = str(self.split_ratio or "2:1:7").replace(":", "-")
|
||||||
|
return os.path.join(out_root, "_generated_splits", f"{env_name}_{ratio_tag}_seed{self.split_seed}")
|
||||||
|
|
||||||
|
def load_raw_items(self, data_path: str) -> list[dict]:
|
||||||
|
"""Load raw items from a dataset path before ratio splitting.
|
||||||
|
|
||||||
|
Subclasses can override when the raw dataset is not a single JSON/JSONL
|
||||||
|
file or when directory layouts require custom normalization.
|
||||||
|
"""
|
||||||
|
if os.path.isdir(data_path):
|
||||||
|
if any(os.path.isdir(os.path.join(data_path, name)) for name in SPLIT_NAMES):
|
||||||
|
raise ValueError(
|
||||||
|
f"{type(self).__name__} got a split directory as data_path. "
|
||||||
|
"Use split_mode=split_dir and pass it as split_dir instead."
|
||||||
|
)
|
||||||
|
candidates = sorted(glob.glob(os.path.join(data_path, "*.json")))
|
||||||
|
candidates += sorted(glob.glob(os.path.join(data_path, "*.jsonl")))
|
||||||
|
if len(candidates) != 1:
|
||||||
|
raise ValueError(
|
||||||
|
f"{type(self).__name__} expected data_path to be one JSON/JSONL file "
|
||||||
|
f"or a directory containing exactly one such file, got: {data_path}"
|
||||||
|
)
|
||||||
|
return _load_json_or_jsonl(candidates[0])
|
||||||
|
return _load_json_or_jsonl(data_path)
|
||||||
|
|
||||||
|
def write_split_items(self, split_path: str, items: list[dict]) -> None:
|
||||||
|
os.makedirs(split_path, exist_ok=True)
|
||||||
|
out_path = os.path.join(split_path, "items.json")
|
||||||
|
with open(out_path, "w", encoding="utf-8") as f:
|
||||||
|
json.dump(items, f, ensure_ascii=False, indent=2)
|
||||||
|
|
||||||
|
def _materialize_ratio_split(self, cfg: dict) -> str:
|
||||||
|
data_path = os.path.abspath(str(self.data_path or "").strip())
|
||||||
|
if not data_path:
|
||||||
|
raise ValueError(
|
||||||
|
f"{type(self).__name__} requires data_path when split_mode=ratio."
|
||||||
|
)
|
||||||
|
|
||||||
|
ratio = _parse_split_ratio(self.split_ratio)
|
||||||
|
items = self.load_raw_items(data_path)
|
||||||
|
if not isinstance(items, list) or not items:
|
||||||
|
raise ValueError(f"No raw items available for ratio split from {data_path}")
|
||||||
|
|
||||||
|
shuffled = list(items)
|
||||||
|
rng = random.Random(self.split_seed)
|
||||||
|
rng.shuffle(shuffled)
|
||||||
|
|
||||||
|
train_n, val_n, test_n = _compute_split_counts(len(shuffled), ratio)
|
||||||
|
train_items = shuffled[:train_n]
|
||||||
|
val_items = shuffled[train_n: train_n + val_n]
|
||||||
|
test_items = shuffled[train_n + val_n: train_n + val_n + test_n]
|
||||||
|
|
||||||
|
split_dir = self._resolve_split_output_dir(cfg)
|
||||||
|
manifest = {
|
||||||
|
"source_data_path": data_path,
|
||||||
|
"split_mode": "ratio",
|
||||||
|
"split_ratio": self.split_ratio,
|
||||||
|
"split_seed": self.split_seed,
|
||||||
|
"counts": {
|
||||||
|
"train": len(train_items),
|
||||||
|
"val": len(val_items),
|
||||||
|
"test": len(test_items),
|
||||||
|
},
|
||||||
|
}
|
||||||
|
os.makedirs(split_dir, exist_ok=True)
|
||||||
|
self.write_split_items(os.path.join(split_dir, "train"), train_items)
|
||||||
|
self.write_split_items(os.path.join(split_dir, "val"), val_items)
|
||||||
|
self.write_split_items(os.path.join(split_dir, "test"), test_items)
|
||||||
|
with open(os.path.join(split_dir, "split_manifest.json"), "w", encoding="utf-8") as f:
|
||||||
|
json.dump(manifest, f, ensure_ascii=False, indent=2)
|
||||||
|
print(
|
||||||
|
f" [{type(self).__name__}] generated ratio split {self.split_ratio} "
|
||||||
|
f"at {split_dir} from {data_path}"
|
||||||
|
)
|
||||||
|
return split_dir
|
||||||
|
|
||||||
|
def _load_all_splits(self) -> None:
|
||||||
|
for name in SPLIT_NAMES:
|
||||||
|
split_path = os.path.join(self.split_dir, name)
|
||||||
|
if not os.path.isdir(split_path):
|
||||||
|
raise ValueError(
|
||||||
|
f"Missing '{name}/' subdirectory in split_dir: {self.split_dir}"
|
||||||
|
)
|
||||||
|
items = self.load_split_items(split_path)
|
||||||
|
if self.limit:
|
||||||
|
items = items[: self.limit]
|
||||||
|
self._splits[name] = items
|
||||||
|
|
||||||
|
counts = " ".join(f"{k}={len(v)}" for k, v in self._splits.items())
|
||||||
|
print(f" [{type(self).__name__}] {counts} (from {self.split_dir})")
|
||||||
|
|
||||||
|
def load_split_items(self, split_path: str) -> list[dict]:
|
||||||
|
"""Load items from one split directory (e.g. ``split_dir/train/``).
|
||||||
|
|
||||||
|
Default: finds the first ``.json`` file in the directory and loads it
|
||||||
|
as a JSON array. Subclasses can override for custom formats.
|
||||||
|
"""
|
||||||
|
json_files = sorted(glob.glob(os.path.join(split_path, "*.json")))
|
||||||
|
if not json_files:
|
||||||
|
raise FileNotFoundError(
|
||||||
|
f"No .json file found in {split_path}"
|
||||||
|
)
|
||||||
|
with open(json_files[0], encoding="utf-8") as f:
|
||||||
|
items = json.load(f)
|
||||||
|
if not isinstance(items, list):
|
||||||
|
raise ValueError(
|
||||||
|
f"Expected JSON array in {json_files[0]}, got {type(items).__name__}"
|
||||||
|
)
|
||||||
|
return items
|
||||||
|
|
||||||
|
# ── Accessors ────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
@property
|
||||||
|
def train_items(self) -> list[dict]:
|
||||||
|
return self._splits.get("train", [])
|
||||||
|
|
||||||
|
@property
|
||||||
|
def val_items(self) -> list[dict]:
|
||||||
|
return self._splits.get("val", [])
|
||||||
|
|
||||||
|
@property
|
||||||
|
def test_items(self) -> list[dict]:
|
||||||
|
return self._splits.get("test", [])
|
||||||
|
|
||||||
|
def get_split_items(self, split: str) -> list[dict]:
|
||||||
|
"""Resolve a split name (including legacy aliases) to its item list."""
|
||||||
|
canonical = _SPLIT_ALIAS.get(split, split)
|
||||||
|
return list(self._splits.get(canonical, self.val_items))
|
||||||
|
|
||||||
|
def get_train_size(self) -> int:
|
||||||
|
return len(self.train_items)
|
||||||
|
|
||||||
|
def plan_train_epoch(
|
||||||
|
self,
|
||||||
|
*,
|
||||||
|
epoch: int,
|
||||||
|
steps_per_epoch: int,
|
||||||
|
accumulation: int,
|
||||||
|
batch_size: int,
|
||||||
|
seed: int,
|
||||||
|
**kwargs,
|
||||||
|
) -> list[BatchSpec]:
|
||||||
|
"""Build one full epoch that covers the train split in shuffled order.
|
||||||
|
|
||||||
|
For split-backed datasets, an epoch should correspond to one pass over
|
||||||
|
the available training items rather than repeated independent sampling.
|
||||||
|
"""
|
||||||
|
epoch_rng = random.Random(seed + epoch * 1000)
|
||||||
|
items = list(self.train_items)
|
||||||
|
epoch_rng.shuffle(items)
|
||||||
|
|
||||||
|
total_batches = steps_per_epoch * accumulation
|
||||||
|
if total_batches <= 0:
|
||||||
|
return []
|
||||||
|
|
||||||
|
batches: list[BatchSpec] = []
|
||||||
|
cursor = 0
|
||||||
|
for batch_idx in range(total_batches):
|
||||||
|
batch_items = items[cursor: cursor + batch_size]
|
||||||
|
cursor += len(batch_items)
|
||||||
|
|
||||||
|
# Extremely small datasets can leave trailing empty microbatches
|
||||||
|
# when accumulation > 1. Reuse the shuffled prefix in that case so
|
||||||
|
# the trainer still receives the expected batch count.
|
||||||
|
if not batch_items and items:
|
||||||
|
refill_rng = random.Random(seed + epoch * 1000 + batch_idx + 1)
|
||||||
|
batch_items = list(items)
|
||||||
|
refill_rng.shuffle(batch_items)
|
||||||
|
batch_items = batch_items[:batch_size]
|
||||||
|
|
||||||
|
batches.append(
|
||||||
|
BatchSpec(
|
||||||
|
phase="train",
|
||||||
|
split="train",
|
||||||
|
seed=seed + epoch * 1000 + batch_idx + 1,
|
||||||
|
batch_size=len(batch_items),
|
||||||
|
payload=batch_items,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
return batches
|
||||||
|
|
||||||
|
# ── Batch construction ───────────────────────────────────────────────
|
||||||
|
|
||||||
|
def build_train_batch(self, batch_size: int, seed: int, **kwargs) -> BatchSpec:
|
||||||
|
rng = random.Random(seed)
|
||||||
|
items = list(self.train_items)
|
||||||
|
rng.shuffle(items)
|
||||||
|
items = items[:batch_size]
|
||||||
|
return BatchSpec(
|
||||||
|
phase="train",
|
||||||
|
split="train",
|
||||||
|
seed=seed,
|
||||||
|
batch_size=len(items),
|
||||||
|
payload=items,
|
||||||
|
)
|
||||||
|
|
||||||
|
def build_eval_batch(
|
||||||
|
self,
|
||||||
|
env_num: int,
|
||||||
|
split: str,
|
||||||
|
seed: int,
|
||||||
|
**kwargs,
|
||||||
|
) -> BatchSpec:
|
||||||
|
items = self.get_split_items(split)
|
||||||
|
if env_num and env_num < len(items):
|
||||||
|
items = items[:env_num]
|
||||||
|
return BatchSpec(
|
||||||
|
phase="eval",
|
||||||
|
split=split,
|
||||||
|
seed=seed,
|
||||||
|
batch_size=len(items),
|
||||||
|
payload=items,
|
||||||
|
)
|
||||||
@@ -0,0 +1,9 @@
|
|||||||
|
"""ReflACT Engine -- the training runner.
|
||||||
|
|
||||||
|
Analogous to the Runner in mmengine: orchestrates the full training pipeline
|
||||||
|
including rollout, gradient computation, aggregation, optimization, and
|
||||||
|
evaluation.
|
||||||
|
"""
|
||||||
|
from skillopt.engine.trainer import ReflACTTrainer # noqa: F401
|
||||||
|
|
||||||
|
__all__ = ["ReflACTTrainer"]
|
||||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1 @@
|
|||||||
|
"""ReflACT environment adapters."""
|
||||||
@@ -0,0 +1,19 @@
|
|||||||
|
# Benchmark Template
|
||||||
|
|
||||||
|
This directory provides scaffold files for adding a new benchmark to SkillOpt.
|
||||||
|
|
||||||
|
## Files
|
||||||
|
|
||||||
|
- `env_template.py` — Environment adapter template
|
||||||
|
- `loader_template.py` — Data loader template
|
||||||
|
- `config_template.yaml` — Config file template
|
||||||
|
|
||||||
|
## Usage
|
||||||
|
|
||||||
|
1. Copy this directory: `cp -r skillopt/envs/_template skillopt/envs/your_benchmark`
|
||||||
|
2. Rename files: remove `_template` suffix
|
||||||
|
3. Implement the `TODO` sections
|
||||||
|
4. Register in `skillopt/envs/__init__.py`
|
||||||
|
5. Create config at `configs/your_benchmark/default.yaml`
|
||||||
|
|
||||||
|
See the [documentation](../../docs/guide/new-benchmark.md) for the full guide.
|
||||||
@@ -0,0 +1,45 @@
|
|||||||
|
# ──────────────────────────────────────────────────
|
||||||
|
# SkillOpt Config Template — <Your Benchmark Name>
|
||||||
|
# ──────────────────────────────────────────────────
|
||||||
|
# Copy this file to configs/<your_benchmark>/default.yaml
|
||||||
|
# and customize the values below.
|
||||||
|
|
||||||
|
# Inherit global defaults
|
||||||
|
_base_: ['../_base_/default.yaml']
|
||||||
|
|
||||||
|
# ── Environment ──────────────────────────────────
|
||||||
|
env:
|
||||||
|
name: your_benchmark # Must match registry key
|
||||||
|
data_path: data/your_benchmark # Path to your data
|
||||||
|
split_mode: ratio # "ratio" or "split_dir"
|
||||||
|
split_ratio: "2:1:7" # train:val:test
|
||||||
|
exec_timeout: 120 # Per-task timeout (seconds)
|
||||||
|
|
||||||
|
# ── Training ─────────────────────────────────────
|
||||||
|
train:
|
||||||
|
num_epochs: 4 # Number of epochs
|
||||||
|
batch_size: 40 # Tasks per step (batch size)
|
||||||
|
seed: 42
|
||||||
|
|
||||||
|
# ── Gradient (Reflection) ───────────────────────
|
||||||
|
gradient:
|
||||||
|
analyst_workers: 16 # Parallel reflection workers
|
||||||
|
minibatch_size: 8
|
||||||
|
|
||||||
|
# ── Optimizer ────────────────────────────────────
|
||||||
|
optimizer:
|
||||||
|
learning_rate: 4 # Max edits per step (edit budget)
|
||||||
|
lr_scheduler: cosine # cosine | linear | constant | autonomous
|
||||||
|
use_slow_update: true # Epoch-boundary momentum
|
||||||
|
use_meta_skill: true # Cross-epoch teacher memory
|
||||||
|
|
||||||
|
# ── Evaluation ───────────────────────────────────
|
||||||
|
evaluation:
|
||||||
|
use_gate: true # Validation gating
|
||||||
|
eval_test: true # Run test eval after training
|
||||||
|
|
||||||
|
# ── Model ────────────────────────────────────────
|
||||||
|
model:
|
||||||
|
backend: azure_openai # azure_openai | openai_chat | claude_code_exec | qwen
|
||||||
|
teacher: gpt-5.5
|
||||||
|
student: gpt-5.5
|
||||||
@@ -0,0 +1,92 @@
|
|||||||
|
"""
|
||||||
|
Benchmark Environment Template
|
||||||
|
===============================
|
||||||
|
Copy this file and implement the TODO sections to add a new benchmark.
|
||||||
|
|
||||||
|
The EnvAdapter is responsible for:
|
||||||
|
1. Executing tasks using the student model + current skill document
|
||||||
|
2. Evaluating predictions against ground truth
|
||||||
|
3. Returning structured results for the training loop
|
||||||
|
"""
|
||||||
|
from skillopt.envs.base import EnvAdapter
|
||||||
|
|
||||||
|
|
||||||
|
class TemplateBenchmarkEnv(EnvAdapter):
|
||||||
|
"""
|
||||||
|
Environment adapter for <Your Benchmark Name>.
|
||||||
|
|
||||||
|
Rename this class and implement the abstract methods below.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self, cfg: dict):
|
||||||
|
super().__init__(cfg)
|
||||||
|
# TODO: Initialize benchmark-specific state
|
||||||
|
# Example: self.tools = load_tools(cfg)
|
||||||
|
|
||||||
|
async def execute(self, item, skill: str, model):
|
||||||
|
"""
|
||||||
|
Execute a single task with the student model.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
item: DataItem with .id, .input, .ground_truth, .metadata
|
||||||
|
skill: Current skill document content (Markdown string)
|
||||||
|
model: Student model backend instance
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
TaskResult with prediction, score, and trajectory
|
||||||
|
"""
|
||||||
|
# Step 1: Build the prompt combining skill + task input
|
||||||
|
prompt = self.build_prompt(item, skill)
|
||||||
|
|
||||||
|
# Step 2: Call the student model
|
||||||
|
# TODO: Customize the message format for your benchmark
|
||||||
|
messages = [
|
||||||
|
{"role": "system", "content": skill},
|
||||||
|
{"role": "user", "content": item.input},
|
||||||
|
]
|
||||||
|
response = await model.generate(messages)
|
||||||
|
|
||||||
|
# Step 3: Parse the model response into a prediction
|
||||||
|
prediction = self.parse_response(response.content)
|
||||||
|
|
||||||
|
# Step 4: Score the prediction
|
||||||
|
score = self.evaluate(prediction, item.ground_truth)
|
||||||
|
|
||||||
|
# Step 5: Return structured result
|
||||||
|
return {
|
||||||
|
"item_id": item.id,
|
||||||
|
"prediction": prediction,
|
||||||
|
"score": score,
|
||||||
|
"trajectory": messages + [{"role": "assistant", "content": response.content}],
|
||||||
|
}
|
||||||
|
|
||||||
|
def evaluate(self, prediction: str, ground_truth: str) -> float:
|
||||||
|
"""
|
||||||
|
Score a prediction against the ground truth.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Float between 0.0 (wrong) and 1.0 (correct)
|
||||||
|
|
||||||
|
TODO: Implement your scoring metric. Common options:
|
||||||
|
- Exact match: float(pred.strip().lower() == gt.strip().lower())
|
||||||
|
- F1 score: compute token overlap
|
||||||
|
- ANLS: for document QA tasks
|
||||||
|
- Custom: any float in [0, 1]
|
||||||
|
"""
|
||||||
|
# Placeholder — exact match
|
||||||
|
return float(prediction.strip().lower() == ground_truth.strip().lower())
|
||||||
|
|
||||||
|
def build_prompt(self, item, skill: str) -> str:
|
||||||
|
"""Combine skill document with task input."""
|
||||||
|
return f"{skill}\n\n---\n\nQuestion: {item.input}"
|
||||||
|
|
||||||
|
def parse_response(self, response: str) -> str:
|
||||||
|
"""
|
||||||
|
Extract the answer from the model's raw response.
|
||||||
|
|
||||||
|
TODO: Implement extraction logic. For example:
|
||||||
|
- Extract text after "Answer:"
|
||||||
|
- Parse JSON output
|
||||||
|
- Extract from code blocks
|
||||||
|
"""
|
||||||
|
return response.strip()
|
||||||
@@ -0,0 +1,103 @@
|
|||||||
|
"""
|
||||||
|
Benchmark Data Loader Template
|
||||||
|
================================
|
||||||
|
Copy this file and implement the TODO sections to load your benchmark data.
|
||||||
|
|
||||||
|
The DataLoader is responsible for:
|
||||||
|
1. Loading raw data from disk
|
||||||
|
2. Splitting into train / validation / test sets
|
||||||
|
3. Providing DataItem objects to the training loop
|
||||||
|
"""
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
|
||||||
|
class TemplateBenchmarkLoader:
|
||||||
|
"""
|
||||||
|
Data loader for <Your Benchmark Name>.
|
||||||
|
|
||||||
|
Rename this class and implement the methods below.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self, data_dir: str = "data/your_benchmark", **kwargs):
|
||||||
|
self.data_dir = Path(data_dir)
|
||||||
|
self.items = []
|
||||||
|
self.splits = {}
|
||||||
|
|
||||||
|
def setup(self, cfg: dict):
|
||||||
|
"""
|
||||||
|
Initialize the loader with config.
|
||||||
|
|
||||||
|
Called once before training starts.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
cfg: Dict with keys like 'split_mode', 'train_ratio', 'val_ratio', etc.
|
||||||
|
"""
|
||||||
|
# Step 1: Load raw data
|
||||||
|
self.items = self._load_items()
|
||||||
|
|
||||||
|
# Step 2: Create splits
|
||||||
|
split_mode = cfg.get("split_mode", "ratio")
|
||||||
|
if split_mode == "ratio":
|
||||||
|
self._split_by_ratio(
|
||||||
|
train_ratio=cfg.get("train_ratio", 0.7),
|
||||||
|
val_ratio=cfg.get("val_ratio", 0.15),
|
||||||
|
)
|
||||||
|
elif split_mode == "split_dir":
|
||||||
|
self._load_predefined_splits(cfg.get("split_dir", self.data_dir))
|
||||||
|
|
||||||
|
def _load_items(self) -> list:
|
||||||
|
"""
|
||||||
|
Load raw data into structured items.
|
||||||
|
|
||||||
|
TODO: Implement data loading. Each item should have at minimum:
|
||||||
|
- id: unique identifier
|
||||||
|
- input: the task input (question, instruction, etc.)
|
||||||
|
- ground_truth: the expected answer
|
||||||
|
- metadata: optional dict with extra info
|
||||||
|
|
||||||
|
Example:
|
||||||
|
items = []
|
||||||
|
for path in self.data_dir.glob("*.json"):
|
||||||
|
data = json.loads(path.read_text())
|
||||||
|
for entry in data:
|
||||||
|
items.append({
|
||||||
|
"id": entry["id"],
|
||||||
|
"input": entry["question"],
|
||||||
|
"ground_truth": entry["answer"],
|
||||||
|
"metadata": {"source": path.name},
|
||||||
|
})
|
||||||
|
return items
|
||||||
|
"""
|
||||||
|
raise NotImplementedError("Implement _load_items() for your benchmark")
|
||||||
|
|
||||||
|
def _split_by_ratio(self, train_ratio: float, val_ratio: float):
|
||||||
|
"""Split items by ratio."""
|
||||||
|
import random
|
||||||
|
random.shuffle(self.items)
|
||||||
|
n = len(self.items)
|
||||||
|
n_train = int(n * train_ratio)
|
||||||
|
n_val = int(n * val_ratio)
|
||||||
|
self.splits = {
|
||||||
|
"train": self.items[:n_train],
|
||||||
|
"valid": self.items[n_train:n_train + n_val],
|
||||||
|
"test": self.items[n_train + n_val:],
|
||||||
|
}
|
||||||
|
|
||||||
|
def _load_predefined_splits(self, split_dir):
|
||||||
|
"""Load from pre-split directories."""
|
||||||
|
# TODO: Implement if your benchmark has pre-defined splits
|
||||||
|
raise NotImplementedError
|
||||||
|
|
||||||
|
def get_split_items(self, split: str) -> list:
|
||||||
|
"""
|
||||||
|
Return items for a given split.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
split: One of "train", "valid", "test"
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
List of data items for the requested split
|
||||||
|
"""
|
||||||
|
if split not in self.splits:
|
||||||
|
raise ValueError(f"Unknown split '{split}'. Available: {list(self.splits.keys())}")
|
||||||
|
return self.splits[split]
|
||||||
@@ -0,0 +1,5 @@
|
|||||||
|
"""ALFWorld environment adapter for ReflACT."""
|
||||||
|
|
||||||
|
from skillopt.envs.alfworld.adapter import ALFWorldAdapter
|
||||||
|
|
||||||
|
__all__ = ["ALFWorldAdapter"]
|
||||||
@@ -0,0 +1,585 @@
|
|||||||
|
"""ALFWorld environment adapter for ReflACT.
|
||||||
|
|
||||||
|
Connects the ReflACT training loop to ALFWorld by implementing
|
||||||
|
:class:`~skillopt.envs.base.EnvAdapter`.
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from dataclasses import dataclass
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
|
||||||
|
from skillopt.gradient.deep_probe import generate_deep_probe_instruction
|
||||||
|
from skillopt.datasets.base import BatchSpec
|
||||||
|
from skillopt.envs.base import EnvAdapter
|
||||||
|
from skillopt.envs.alfworld.dataloader import ALFWorldDataLoader
|
||||||
|
from skillopt.envs.alfworld.rollout import (
|
||||||
|
build_alfworld_env,
|
||||||
|
run_alfworld_batch,
|
||||||
|
TASKS,
|
||||||
|
)
|
||||||
|
from skillopt.gradient.reflect import run_minibatch_reflect
|
||||||
|
from skillopt.utils import compute_score
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class ALFWorldBatchRun:
|
||||||
|
"""Lazy ALFWorld batch description.
|
||||||
|
|
||||||
|
The adapter materializes this in rollout chunks so a large evaluation set
|
||||||
|
does not keep every ALFWorld simulator open at once.
|
||||||
|
"""
|
||||||
|
|
||||||
|
env_num: int
|
||||||
|
eval_dataset: str
|
||||||
|
seed: int
|
||||||
|
is_train: bool
|
||||||
|
workers: int
|
||||||
|
specific_gamefiles: list[str] | None = None
|
||||||
|
result_ids: list[str] | None = None
|
||||||
|
items: list[dict] | None = None
|
||||||
|
|
||||||
|
def __iter__(self):
|
||||||
|
return iter(self.items or [])
|
||||||
|
|
||||||
|
def __len__(self) -> int:
|
||||||
|
return int(self.env_num or 0)
|
||||||
|
|
||||||
|
|
||||||
|
class ALFWorldAdapter(EnvAdapter):
|
||||||
|
"""ALFWorld environment adapter.
|
||||||
|
|
||||||
|
Parameters
|
||||||
|
----------
|
||||||
|
max_steps : int
|
||||||
|
Maximum steps per ALFWorld episode (default 50).
|
||||||
|
max_api_workers : int
|
||||||
|
Maximum concurrent API calls during rollout (default 8).
|
||||||
|
analyst_workers : int
|
||||||
|
Parallel workers for analyst stage (default 16).
|
||||||
|
failure_only : bool
|
||||||
|
If True, only run error analyst (skip success analyst).
|
||||||
|
minibatch_size : int
|
||||||
|
Trajectories per analyst group, M (default 8).
|
||||||
|
edit_budget : int
|
||||||
|
Maximum edits per minibatch, L (default 4).
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
split_dir: str = "",
|
||||||
|
data_path: str = "",
|
||||||
|
split_mode: str = "split_dir",
|
||||||
|
split_ratio: str = "2:1:7",
|
||||||
|
split_seed: int = 42,
|
||||||
|
split_output_dir: str = "",
|
||||||
|
seed: int = 42,
|
||||||
|
limit: int = 0,
|
||||||
|
train_size: int = 0,
|
||||||
|
max_steps: int = 50,
|
||||||
|
workers: int = 8,
|
||||||
|
max_api_workers: int = 8,
|
||||||
|
analyst_workers: int = 16,
|
||||||
|
failure_only: bool = False,
|
||||||
|
minibatch_size: int = 8,
|
||||||
|
edit_budget: int = 4,
|
||||||
|
use_deep_reflect: bool = False,
|
||||||
|
deep_reflect_failures: int = 4,
|
||||||
|
deep_reflect_successes: int = 2,
|
||||||
|
) -> None:
|
||||||
|
self.max_steps = max_steps
|
||||||
|
self.workers = max(int(workers or 1), 1)
|
||||||
|
self.max_api_workers = max_api_workers
|
||||||
|
self.analyst_workers = analyst_workers
|
||||||
|
self.failure_only = failure_only
|
||||||
|
self.minibatch_size = minibatch_size
|
||||||
|
self.edit_budget = edit_budget
|
||||||
|
self.use_deep_reflect = use_deep_reflect
|
||||||
|
self.deep_reflect_failures = deep_reflect_failures
|
||||||
|
self.deep_reflect_successes = deep_reflect_successes
|
||||||
|
self.dataloader = ALFWorldDataLoader(
|
||||||
|
split_dir=split_dir,
|
||||||
|
data_path=data_path,
|
||||||
|
split_mode=split_mode,
|
||||||
|
split_ratio=split_ratio,
|
||||||
|
split_seed=split_seed,
|
||||||
|
split_output_dir=split_output_dir,
|
||||||
|
seed=seed,
|
||||||
|
limit=limit,
|
||||||
|
train_size=train_size,
|
||||||
|
)
|
||||||
|
self._traj_cache: dict[str, dict | None] = {}
|
||||||
|
|
||||||
|
def setup(self, cfg: dict) -> None:
|
||||||
|
super().setup(cfg)
|
||||||
|
self.dataloader.setup(cfg)
|
||||||
|
|
||||||
|
def _load_traj_data(self, item: dict) -> dict | None:
|
||||||
|
gamefile = str(item.get("gamefile") or "").strip()
|
||||||
|
if not gamefile:
|
||||||
|
return None
|
||||||
|
if gamefile in self._traj_cache:
|
||||||
|
return self._traj_cache[gamefile]
|
||||||
|
|
||||||
|
traj_path = os.path.join(os.path.dirname(gamefile), "traj_data.json")
|
||||||
|
try:
|
||||||
|
with open(traj_path, encoding="utf-8") as f:
|
||||||
|
data = json.load(f)
|
||||||
|
except Exception:
|
||||||
|
data = None
|
||||||
|
self._traj_cache[gamefile] = data
|
||||||
|
return data
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _unique_lines(values: list[str], *, limit: int = 0) -> list[str]:
|
||||||
|
lines: list[str] = []
|
||||||
|
seen: set[str] = set()
|
||||||
|
for raw in values:
|
||||||
|
line = str(raw or "").strip()
|
||||||
|
if not line or line in seen:
|
||||||
|
continue
|
||||||
|
seen.add(line)
|
||||||
|
lines.append(line)
|
||||||
|
if limit > 0 and len(lines) >= limit:
|
||||||
|
break
|
||||||
|
return lines
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _format_high_pddl(high_pddl: list[dict]) -> list[str]:
|
||||||
|
steps: list[str] = []
|
||||||
|
for idx, step in enumerate(high_pddl or [], start=1):
|
||||||
|
discrete = step.get("discrete_action") or {}
|
||||||
|
action = str(discrete.get("action") or "").strip()
|
||||||
|
args = [str(arg).strip() for arg in (discrete.get("args") or []) if str(arg).strip()]
|
||||||
|
if action and args:
|
||||||
|
text = f"{action}({', '.join(args)})"
|
||||||
|
elif action:
|
||||||
|
text = action
|
||||||
|
else:
|
||||||
|
planner_action = step.get("planner_action") or {}
|
||||||
|
text = str(planner_action.get("action") or "").strip()
|
||||||
|
if text:
|
||||||
|
steps.append(f"{idx}. {text}")
|
||||||
|
return steps
|
||||||
|
|
||||||
|
def _build_reference_bundle(self, item: dict) -> dict:
|
||||||
|
data = self._load_traj_data(item)
|
||||||
|
if not data:
|
||||||
|
return {}
|
||||||
|
|
||||||
|
anns = ((data.get("turk_annotations") or {}).get("anns") or [])
|
||||||
|
task_descs = self._unique_lines(
|
||||||
|
[ann.get("task_desc", "") for ann in anns],
|
||||||
|
limit=3,
|
||||||
|
)
|
||||||
|
high_descs = self._unique_lines(
|
||||||
|
[step for ann in anns for step in (ann.get("high_descs") or [])],
|
||||||
|
limit=12,
|
||||||
|
)
|
||||||
|
pddl_params = {
|
||||||
|
key: value
|
||||||
|
for key, value in (data.get("pddl_params") or {}).items()
|
||||||
|
if value not in ("", None, [], {})
|
||||||
|
}
|
||||||
|
scene = data.get("scene") or {}
|
||||||
|
scene_summary = {
|
||||||
|
key: scene.get(key)
|
||||||
|
for key in ("floor_plan", "scene_num", "dirty_and_empty")
|
||||||
|
if scene.get(key) not in ("", None, [], {})
|
||||||
|
}
|
||||||
|
high_pddl = self._format_high_pddl((data.get("plan") or {}).get("high_pddl") or [])
|
||||||
|
task_type = str(data.get("task_type") or item.get("task_type") or "").strip()
|
||||||
|
return {
|
||||||
|
"task_type": task_type,
|
||||||
|
"task_descs": task_descs,
|
||||||
|
"high_descs": high_descs,
|
||||||
|
"pddl_params": pddl_params,
|
||||||
|
"high_pddl": high_pddl,
|
||||||
|
"scene_summary": scene_summary,
|
||||||
|
}
|
||||||
|
|
||||||
|
def build_reference_text(self, item: dict) -> str:
|
||||||
|
bundle = self._build_reference_bundle(item)
|
||||||
|
if not bundle:
|
||||||
|
return ""
|
||||||
|
|
||||||
|
parts: list[str] = []
|
||||||
|
if bundle["task_type"]:
|
||||||
|
parts.append(f"## Reference Task Type\n{bundle['task_type']}")
|
||||||
|
if bundle["task_descs"]:
|
||||||
|
parts.append(
|
||||||
|
"## Reference Human Task Descriptions\n"
|
||||||
|
+ "\n".join(f"- {line}" for line in bundle["task_descs"])
|
||||||
|
)
|
||||||
|
if bundle["high_descs"]:
|
||||||
|
parts.append(
|
||||||
|
"## Reference Human High-Level Steps\n"
|
||||||
|
+ "\n".join(f"{idx}. {line}" for idx, line in enumerate(bundle["high_descs"], start=1))
|
||||||
|
)
|
||||||
|
if bundle["pddl_params"]:
|
||||||
|
parts.append(
|
||||||
|
"## Reference PDDL Params\n"
|
||||||
|
+ "\n".join(f"- {key}: {value}" for key, value in bundle["pddl_params"].items())
|
||||||
|
)
|
||||||
|
if bundle["high_pddl"]:
|
||||||
|
parts.append(
|
||||||
|
"## Reference Planner High-Level Plan\n" + "\n".join(bundle["high_pddl"])
|
||||||
|
)
|
||||||
|
if bundle["scene_summary"]:
|
||||||
|
parts.append(
|
||||||
|
"## Reference Scene Summary\n"
|
||||||
|
+ "\n".join(f"- {key}: {value}" for key, value in bundle["scene_summary"].items())
|
||||||
|
)
|
||||||
|
return "\n\n".join(parts)
|
||||||
|
|
||||||
|
def get_reference_metadata(self, item: dict) -> dict:
|
||||||
|
bundle = self._build_reference_bundle(item)
|
||||||
|
if not bundle:
|
||||||
|
return {"fields": [], "preview": ""}
|
||||||
|
|
||||||
|
fields: list[str] = []
|
||||||
|
previews: list[str] = []
|
||||||
|
if bundle["task_type"]:
|
||||||
|
fields.append("task_type")
|
||||||
|
previews.append(f"[task_type] {bundle['task_type']}")
|
||||||
|
if bundle["task_descs"]:
|
||||||
|
fields.append("task_desc")
|
||||||
|
previews.append("[task_desc]\n" + "\n".join(bundle["task_descs"][:2]))
|
||||||
|
if bundle["high_descs"]:
|
||||||
|
fields.append("high_descs")
|
||||||
|
previews.append("[high_descs]\n" + "\n".join(bundle["high_descs"][:3]))
|
||||||
|
if bundle["pddl_params"]:
|
||||||
|
fields.append("pddl_params")
|
||||||
|
previews.append(
|
||||||
|
"[pddl_params]\n"
|
||||||
|
+ "\n".join(
|
||||||
|
f"{key}: {value}" for key, value in list(bundle["pddl_params"].items())[:4]
|
||||||
|
)
|
||||||
|
)
|
||||||
|
if bundle["high_pddl"]:
|
||||||
|
fields.append("plan.high_pddl")
|
||||||
|
previews.append("[plan.high_pddl]\n" + "\n".join(bundle["high_pddl"][:3]))
|
||||||
|
if bundle["scene_summary"]:
|
||||||
|
fields.append("scene")
|
||||||
|
previews.append(
|
||||||
|
"[scene]\n"
|
||||||
|
+ "\n".join(
|
||||||
|
f"{key}: {value}" for key, value in bundle["scene_summary"].items()
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return {
|
||||||
|
"fields": fields,
|
||||||
|
"preview": "\n\n".join(previews)[:600],
|
||||||
|
}
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _infer_dataset_from_gamefile(gamefile: str) -> tuple[str, bool]:
|
||||||
|
path = str(gamefile or "")
|
||||||
|
if "/valid_seen/" in path:
|
||||||
|
return "eval_in_distribution", False
|
||||||
|
if "/valid_unseen/" in path:
|
||||||
|
return "eval_out_of_distribution", False
|
||||||
|
return "train", True
|
||||||
|
|
||||||
|
def get_dataloader(self):
|
||||||
|
return self.dataloader
|
||||||
|
|
||||||
|
def _comparison_items(self, items: list[dict]) -> list[dict]:
|
||||||
|
enriched: list[dict] = []
|
||||||
|
for item in items:
|
||||||
|
row = dict(item)
|
||||||
|
bundle = self._build_reference_bundle(row)
|
||||||
|
if bundle.get("task_descs"):
|
||||||
|
row["task_description"] = bundle["task_descs"][0]
|
||||||
|
elif bundle.get("task_type"):
|
||||||
|
row["task_description"] = bundle["task_type"]
|
||||||
|
enriched.append(row)
|
||||||
|
return enriched
|
||||||
|
|
||||||
|
def requires_ray(self) -> bool:
|
||||||
|
return False
|
||||||
|
|
||||||
|
def build_env_from_batch(self, batch: BatchSpec, **kwargs):
|
||||||
|
gamefiles = list(batch.metadata.get("gamefiles") or [])
|
||||||
|
result_ids = list(batch.metadata.get("result_ids") or [])
|
||||||
|
items = self._comparison_items(list(batch.payload or []))
|
||||||
|
return ALFWorldBatchRun(
|
||||||
|
env_num=batch.batch_size,
|
||||||
|
eval_dataset=batch.metadata.get("eval_dataset", batch.split),
|
||||||
|
seed=batch.seed,
|
||||||
|
is_train=batch.metadata.get("is_train", batch.phase == "train"),
|
||||||
|
specific_gamefiles=gamefiles or None,
|
||||||
|
result_ids=result_ids or None,
|
||||||
|
items=items,
|
||||||
|
workers=self.workers,
|
||||||
|
)
|
||||||
|
|
||||||
|
def build_train_env(self, batch_size: int, seed: int, **kwargs):
|
||||||
|
batch = self.dataloader.build_train_batch(batch_size=batch_size, seed=seed, **kwargs)
|
||||||
|
return self.build_env_from_batch(batch, **kwargs)
|
||||||
|
|
||||||
|
def build_eval_env(self, env_num: int, split: str, seed: int, **kwargs):
|
||||||
|
batch = self.dataloader.build_eval_batch(env_num=env_num, split=split, seed=seed, **kwargs)
|
||||||
|
return self.build_env_from_batch(batch, **kwargs)
|
||||||
|
|
||||||
|
def rollout(
|
||||||
|
self,
|
||||||
|
env_manager,
|
||||||
|
skill_content: str,
|
||||||
|
out_dir: str,
|
||||||
|
**kwargs,
|
||||||
|
) -> list[dict]:
|
||||||
|
results_path = os.path.join(out_dir, "results.jsonl")
|
||||||
|
os.makedirs(out_dir, exist_ok=True)
|
||||||
|
|
||||||
|
# Resume support
|
||||||
|
if os.path.exists(results_path):
|
||||||
|
existing: list[dict] = []
|
||||||
|
with open(results_path) as f:
|
||||||
|
for line in f:
|
||||||
|
try:
|
||||||
|
existing.append(json.loads(line))
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
if existing:
|
||||||
|
return existing
|
||||||
|
|
||||||
|
if isinstance(env_manager, ALFWorldBatchRun):
|
||||||
|
results = self._run_batch(
|
||||||
|
env_manager,
|
||||||
|
skill_content=skill_content,
|
||||||
|
out_dir=out_dir,
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
results = run_alfworld_batch(
|
||||||
|
env_manager=env_manager,
|
||||||
|
skill_content=skill_content,
|
||||||
|
max_steps=self.max_steps,
|
||||||
|
out_root=out_dir,
|
||||||
|
max_api_workers=self.max_api_workers,
|
||||||
|
result_ids=getattr(env_manager, "_skillopt_result_ids", None),
|
||||||
|
)
|
||||||
|
|
||||||
|
with open(results_path, "w") as f:
|
||||||
|
for r in results:
|
||||||
|
f.write(json.dumps(r, ensure_ascii=False) + "\n")
|
||||||
|
|
||||||
|
return results
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _close_env(env_manager) -> None:
|
||||||
|
close = getattr(env_manager, "close", None)
|
||||||
|
if callable(close):
|
||||||
|
close()
|
||||||
|
|
||||||
|
def _run_batch(
|
||||||
|
self,
|
||||||
|
batch: ALFWorldBatchRun,
|
||||||
|
skill_content: str,
|
||||||
|
out_dir: str,
|
||||||
|
*,
|
||||||
|
diagnostic_mode: bool = False,
|
||||||
|
diagnostic_instruction: str = "",
|
||||||
|
) -> list[dict]:
|
||||||
|
total = int(batch.env_num or 0)
|
||||||
|
if total <= 0:
|
||||||
|
return []
|
||||||
|
|
||||||
|
workers = max(1, min(int(batch.workers or self.workers), total))
|
||||||
|
if total > workers:
|
||||||
|
print(
|
||||||
|
f" [alfworld rollout] episodes={total} "
|
||||||
|
f"env_workers={workers} chunks={(total + workers - 1) // workers}"
|
||||||
|
)
|
||||||
|
|
||||||
|
all_results: list[dict] = []
|
||||||
|
for start in range(0, total, workers):
|
||||||
|
chunk_size = min(workers, total - start)
|
||||||
|
chunk_gamefiles = (
|
||||||
|
batch.specific_gamefiles[start:start + chunk_size]
|
||||||
|
if batch.specific_gamefiles
|
||||||
|
else None
|
||||||
|
)
|
||||||
|
chunk_ids = (
|
||||||
|
batch.result_ids[start:start + chunk_size]
|
||||||
|
if batch.result_ids
|
||||||
|
else [f"env_{idx:03d}" for idx in range(start, start + chunk_size)]
|
||||||
|
)
|
||||||
|
chunk_env = build_alfworld_env(
|
||||||
|
env_num=chunk_size,
|
||||||
|
eval_dataset=batch.eval_dataset,
|
||||||
|
seed=batch.seed + start,
|
||||||
|
is_train=batch.is_train,
|
||||||
|
specific_gamefiles=chunk_gamefiles,
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
chunk_results = run_alfworld_batch(
|
||||||
|
env_manager=chunk_env,
|
||||||
|
skill_content=skill_content,
|
||||||
|
max_steps=self.max_steps,
|
||||||
|
out_root=out_dir,
|
||||||
|
max_api_workers=min(self.max_api_workers, chunk_size),
|
||||||
|
diagnostic_mode=diagnostic_mode,
|
||||||
|
diagnostic_instruction=diagnostic_instruction,
|
||||||
|
result_ids=chunk_ids,
|
||||||
|
)
|
||||||
|
finally:
|
||||||
|
self._close_env(chunk_env)
|
||||||
|
all_results.extend(chunk_results)
|
||||||
|
return all_results
|
||||||
|
|
||||||
|
def reflect(
|
||||||
|
self,
|
||||||
|
results: list[dict],
|
||||||
|
skill_content: str,
|
||||||
|
out_dir: str,
|
||||||
|
**kwargs,
|
||||||
|
) -> list[dict | None]:
|
||||||
|
prediction_dir = kwargs.get("prediction_dir", os.path.join(out_dir, "predictions"))
|
||||||
|
patches_dir = kwargs.get("patches_dir", os.path.join(out_dir, "patches"))
|
||||||
|
random_seed = kwargs.get("random_seed")
|
||||||
|
step_buffer_context = kwargs.get("step_buffer_context", "")
|
||||||
|
meta_skill_context = kwargs.get("meta_skill_context", "")
|
||||||
|
|
||||||
|
return run_minibatch_reflect(
|
||||||
|
results=results,
|
||||||
|
skill_content=skill_content,
|
||||||
|
prediction_dir=prediction_dir,
|
||||||
|
patches_dir=patches_dir,
|
||||||
|
workers=self.analyst_workers,
|
||||||
|
failure_only=self.failure_only,
|
||||||
|
minibatch_size=self.minibatch_size,
|
||||||
|
edit_budget=self.edit_budget,
|
||||||
|
random_seed=random_seed,
|
||||||
|
error_system=self.get_error_minibatch_prompt(),
|
||||||
|
success_system=self.get_success_minibatch_prompt(),
|
||||||
|
step_buffer_context=step_buffer_context,
|
||||||
|
meta_skill_context=meta_skill_context,
|
||||||
|
)
|
||||||
|
|
||||||
|
def deep_reflect(
|
||||||
|
self,
|
||||||
|
results: list[dict],
|
||||||
|
skill_content: str,
|
||||||
|
out_dir: str,
|
||||||
|
**kwargs,
|
||||||
|
) -> list[dict | None]:
|
||||||
|
if not self.use_deep_reflect:
|
||||||
|
return []
|
||||||
|
|
||||||
|
prediction_dir = kwargs.get("prediction_dir", os.path.join(out_dir, "predictions"))
|
||||||
|
random_seed = kwargs.get("random_seed")
|
||||||
|
step_buffer_context = kwargs.get("step_buffer_context", "")
|
||||||
|
meta_skill_context = kwargs.get("meta_skill_context", "")
|
||||||
|
selected_items = self.select_representative_items(
|
||||||
|
results,
|
||||||
|
results,
|
||||||
|
n_failures=self.deep_reflect_failures,
|
||||||
|
n_successes=self.deep_reflect_successes,
|
||||||
|
seed=random_seed,
|
||||||
|
)
|
||||||
|
if not selected_items:
|
||||||
|
return []
|
||||||
|
|
||||||
|
selected_ids = {str(item["id"]) for item in selected_items}
|
||||||
|
selected_results = [row for row in results if str(row.get("id")) in selected_ids]
|
||||||
|
selected_examples = self.attach_reference_context(selected_results, selected_items)
|
||||||
|
|
||||||
|
field_counts: dict[str, int] = {}
|
||||||
|
selected_metadata: list[dict] = []
|
||||||
|
for item in selected_items:
|
||||||
|
meta = self.get_reference_metadata(item)
|
||||||
|
for field in meta["fields"]:
|
||||||
|
field_counts[field] = field_counts.get(field, 0) + 1
|
||||||
|
selected_metadata.append({
|
||||||
|
"id": str(item["id"]),
|
||||||
|
"task_type": str(item.get("task_type") or "alfworld"),
|
||||||
|
"gamefile": str(item.get("gamefile") or ""),
|
||||||
|
"reference_fields": meta["fields"],
|
||||||
|
"reference_preview": meta["preview"],
|
||||||
|
})
|
||||||
|
|
||||||
|
deep_dir = os.path.join(out_dir, "deep_reflect")
|
||||||
|
rollout_dir = os.path.join(deep_dir, "rollout")
|
||||||
|
patches_dir = os.path.join(deep_dir, "patches")
|
||||||
|
os.makedirs(deep_dir, exist_ok=True)
|
||||||
|
field_summary = ", ".join(
|
||||||
|
f"{field}({count}/{len(selected_items)})"
|
||||||
|
for field, count in sorted(field_counts.items())
|
||||||
|
) or "none"
|
||||||
|
print(
|
||||||
|
f" [2b/6 DEEP REFLECT setup] selected={len(selected_items)} "
|
||||||
|
f"reference_fields={field_summary}"
|
||||||
|
)
|
||||||
|
probe = generate_deep_probe_instruction(
|
||||||
|
skill_content=skill_content,
|
||||||
|
items=selected_examples,
|
||||||
|
prediction_dir=prediction_dir,
|
||||||
|
system_prompt=self.get_deep_probe_prompt(),
|
||||||
|
step_buffer_context=step_buffer_context,
|
||||||
|
meta_skill_context=meta_skill_context,
|
||||||
|
output_requirements=[
|
||||||
|
"- Some trajectories may include a hidden Reference block. Use it to target the student's latent subgoal, missing precondition, or next-step intent, but do not reveal or paraphrase that reference to the student.",
|
||||||
|
"- The instruction must request a brief diagnostic readout inside the existing <think>...</think> block.",
|
||||||
|
"- The student must still output exactly one admissible action inside <action>...</action>.",
|
||||||
|
"- Do not ask for exhaustive inventories, full plans, or long chain-of-thought.",
|
||||||
|
"- The instruction text should be ready to append directly to the student's prompt.",
|
||||||
|
],
|
||||||
|
)
|
||||||
|
if not probe:
|
||||||
|
return []
|
||||||
|
|
||||||
|
with open(os.path.join(deep_dir, "probe.json"), "w", encoding="utf-8") as f:
|
||||||
|
json.dump(
|
||||||
|
{
|
||||||
|
**probe,
|
||||||
|
"reference_summary": {
|
||||||
|
"selected_count": len(selected_items),
|
||||||
|
"field_counts": field_counts,
|
||||||
|
},
|
||||||
|
"selected_examples": selected_metadata,
|
||||||
|
},
|
||||||
|
f,
|
||||||
|
ensure_ascii=False,
|
||||||
|
indent=2,
|
||||||
|
)
|
||||||
|
|
||||||
|
gamefiles = [str(item.get("gamefile") or "") for item in selected_items]
|
||||||
|
if any(not gamefile for gamefile in gamefiles):
|
||||||
|
return []
|
||||||
|
eval_dataset, is_train = self._infer_dataset_from_gamefile(gamefiles[0])
|
||||||
|
deep_env = ALFWorldBatchRun(
|
||||||
|
env_num=len(selected_items),
|
||||||
|
eval_dataset=eval_dataset,
|
||||||
|
seed=random_seed or 42,
|
||||||
|
is_train=is_train,
|
||||||
|
specific_gamefiles=gamefiles,
|
||||||
|
workers=min(self.workers, max(len(selected_items), 1)),
|
||||||
|
result_ids=[str(item["id"]) for item in selected_items],
|
||||||
|
)
|
||||||
|
deep_results = self._run_batch(
|
||||||
|
deep_env,
|
||||||
|
skill_content=skill_content,
|
||||||
|
out_dir=rollout_dir,
|
||||||
|
diagnostic_mode=True,
|
||||||
|
diagnostic_instruction=probe["probe_instruction"],
|
||||||
|
)
|
||||||
|
deep_results = self.attach_reference_context(deep_results, selected_items)
|
||||||
|
return run_minibatch_reflect(
|
||||||
|
results=deep_results,
|
||||||
|
skill_content=skill_content,
|
||||||
|
prediction_dir=os.path.join(rollout_dir, "predictions"),
|
||||||
|
patches_dir=patches_dir,
|
||||||
|
workers=self.analyst_workers,
|
||||||
|
failure_only=self.failure_only,
|
||||||
|
minibatch_size=self.minibatch_size,
|
||||||
|
edit_budget=self.edit_budget,
|
||||||
|
random_seed=random_seed,
|
||||||
|
error_system=self.get_error_minibatch_prompt(),
|
||||||
|
success_system=self.get_success_minibatch_prompt(),
|
||||||
|
step_buffer_context=step_buffer_context,
|
||||||
|
meta_skill_context=meta_skill_context,
|
||||||
|
)
|
||||||
|
|
||||||
|
def get_task_types(self) -> list[str]:
|
||||||
|
return list(TASKS)
|
||||||
@@ -0,0 +1,123 @@
|
|||||||
|
"""ALFWorld task dataloader."""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from skillopt.datasets.base import BatchSpec, SplitDataLoader
|
||||||
|
|
||||||
|
|
||||||
|
class ALFWorldDataLoader(SplitDataLoader):
|
||||||
|
"""ALFWorld batch planner.
|
||||||
|
|
||||||
|
In split_dir mode, batches are fixed gamefile items so ablations differ
|
||||||
|
only in how the same training set is batched.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
split_dir: str = "",
|
||||||
|
data_path: str = "",
|
||||||
|
split_mode: str = "split_dir",
|
||||||
|
split_ratio: str = "2:1:7",
|
||||||
|
split_seed: int = 42,
|
||||||
|
split_output_dir: str = "",
|
||||||
|
seed: int = 42,
|
||||||
|
limit: int = 0,
|
||||||
|
train_size: int = 0,
|
||||||
|
**kwargs,
|
||||||
|
) -> None:
|
||||||
|
super().__init__(
|
||||||
|
split_dir=split_dir,
|
||||||
|
data_path=data_path,
|
||||||
|
split_mode=split_mode,
|
||||||
|
split_ratio=split_ratio,
|
||||||
|
split_seed=split_seed,
|
||||||
|
split_output_dir=split_output_dir,
|
||||||
|
seed=seed,
|
||||||
|
limit=limit,
|
||||||
|
)
|
||||||
|
self.train_size_override = int(train_size or 0)
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _metadata_for_items(items: list[dict], split: str, phase: str) -> dict:
|
||||||
|
gamefiles = [str(item.get("gamefile") or "") for item in items]
|
||||||
|
if any(not gamefile for gamefile in gamefiles):
|
||||||
|
raise ValueError("ALFWorld split items must contain non-empty gamefile paths.")
|
||||||
|
eval_dataset = "train"
|
||||||
|
is_train = phase == "train"
|
||||||
|
first = gamefiles[0] if gamefiles else ""
|
||||||
|
if "/valid_seen/" in first:
|
||||||
|
eval_dataset = "eval_in_distribution"
|
||||||
|
is_train = False
|
||||||
|
elif "/valid_unseen/" in first:
|
||||||
|
eval_dataset = "eval_out_of_distribution"
|
||||||
|
is_train = False
|
||||||
|
return {
|
||||||
|
"eval_dataset": eval_dataset,
|
||||||
|
"is_train": is_train,
|
||||||
|
"gamefiles": gamefiles,
|
||||||
|
"result_ids": [str(item.get("id") or idx) for idx, item in enumerate(items)],
|
||||||
|
}
|
||||||
|
|
||||||
|
def get_train_size(self) -> int:
|
||||||
|
if self.train_size_override > 0:
|
||||||
|
return self.train_size_override
|
||||||
|
return super().get_train_size()
|
||||||
|
|
||||||
|
def build_train_batch(self, batch_size: int, seed: int, **kwargs) -> BatchSpec:
|
||||||
|
batch = super().build_train_batch(batch_size=batch_size, seed=seed, **kwargs)
|
||||||
|
items = list(batch.payload or [])
|
||||||
|
batch.metadata.update(self._metadata_for_items(items, "train", "train"))
|
||||||
|
return BatchSpec(
|
||||||
|
phase="train",
|
||||||
|
split="train",
|
||||||
|
seed=seed,
|
||||||
|
batch_size=len(items),
|
||||||
|
payload=items,
|
||||||
|
metadata=batch.metadata,
|
||||||
|
)
|
||||||
|
|
||||||
|
def plan_train_epoch(
|
||||||
|
self,
|
||||||
|
*,
|
||||||
|
epoch: int,
|
||||||
|
steps_per_epoch: int,
|
||||||
|
accumulation: int,
|
||||||
|
batch_size: int,
|
||||||
|
seed: int,
|
||||||
|
**kwargs,
|
||||||
|
) -> list[BatchSpec]:
|
||||||
|
batches = super().plan_train_epoch(
|
||||||
|
epoch=epoch,
|
||||||
|
steps_per_epoch=steps_per_epoch,
|
||||||
|
accumulation=accumulation,
|
||||||
|
batch_size=batch_size,
|
||||||
|
seed=seed,
|
||||||
|
**kwargs,
|
||||||
|
)
|
||||||
|
for batch in batches:
|
||||||
|
items = list(batch.payload or [])
|
||||||
|
batch.metadata.update(self._metadata_for_items(items, "train", "train"))
|
||||||
|
return batches
|
||||||
|
|
||||||
|
def build_eval_batch(
|
||||||
|
self,
|
||||||
|
env_num: int,
|
||||||
|
split: str,
|
||||||
|
seed: int,
|
||||||
|
**kwargs,
|
||||||
|
) -> BatchSpec:
|
||||||
|
batch = super().build_eval_batch(
|
||||||
|
env_num=env_num,
|
||||||
|
split=split,
|
||||||
|
seed=seed,
|
||||||
|
**kwargs,
|
||||||
|
)
|
||||||
|
items = list(batch.payload or [])
|
||||||
|
batch.metadata.update(self._metadata_for_items(items, split, "eval"))
|
||||||
|
return BatchSpec(
|
||||||
|
phase="eval",
|
||||||
|
split=split,
|
||||||
|
seed=seed,
|
||||||
|
batch_size=len(items),
|
||||||
|
payload=items,
|
||||||
|
metadata=batch.metadata,
|
||||||
|
)
|
||||||
@@ -0,0 +1,55 @@
|
|||||||
|
You are an expert failure-analysis agent for ALFWorld embodied household tasks.
|
||||||
|
|
||||||
|
You will be given MULTIPLE failed agent trajectories from a single minibatch
|
||||||
|
and the current skill document.
|
||||||
|
Your job is to identify the most important COMMON failure patterns across
|
||||||
|
the batch and propose a concise set of skill edits.
|
||||||
|
|
||||||
|
## ALFWorld Task Types
|
||||||
|
- pick_and_place: Put object in/on a receptacle
|
||||||
|
- pick_two_obj_and_place: Put two instances of an object in/on a receptacle
|
||||||
|
- look_at_obj_in_light: Examine an object under a desklamp
|
||||||
|
- pick_heat_then_place_in_recep: Heat an object and put it in/on a receptacle
|
||||||
|
- pick_cool_then_place_in_recep: Cool an object and put it in/on a receptacle
|
||||||
|
- pick_clean_then_place_in_recep: Clean an object and put it in/on a receptacle
|
||||||
|
|
||||||
|
## Failure Type Categories
|
||||||
|
- **navigation_loop**: the agent revisits the same locations repeatedly without progress
|
||||||
|
- **missed_object**: the agent fails to pick up a visible/reachable goal object
|
||||||
|
- **wrong_sequence**: the agent performs actions in the wrong order (e.g., placing before transforming)
|
||||||
|
- **premature_stop**: the agent stops or gets stuck before completing all goal conditions
|
||||||
|
- **action_loop**: the agent repeats the same action without advancing
|
||||||
|
- **appliance_error**: the agent misuses or skips an appliance (microwave, fridge, sink)
|
||||||
|
- **rule_missing**: the skill lacks a relevant rule for this situation
|
||||||
|
- **rule_wrong**: an existing skill rule is misleading or incorrect
|
||||||
|
- **rule_ignored**: the skill has the right rule but the agent did not follow it
|
||||||
|
- **other**: none of the above
|
||||||
|
|
||||||
|
## Analysis Process
|
||||||
|
1. Read ALL trajectories in the minibatch.
|
||||||
|
2. Identify the most prevalent, systematic failure patterns across them.
|
||||||
|
3. For each pattern, classify its failure type.
|
||||||
|
4. Propose skill edits that address the COMMON patterns — not individual edge cases.
|
||||||
|
5. Edits must be generalizable; do not hardcode task-specific values.
|
||||||
|
6. Only patch gaps in the skill — do not duplicate existing content.
|
||||||
|
|
||||||
|
You will be told the maximum number of edits (the budget L). Produce AT MOST L edits,
|
||||||
|
focusing on the highest-impact patterns. You may produce fewer if warranted.
|
||||||
|
|
||||||
|
Respond ONLY with a valid JSON object (no markdown fences, no extra text):
|
||||||
|
{
|
||||||
|
"batch_size": <number of trajectories analysed>,
|
||||||
|
"failure_summary": [
|
||||||
|
{"failure_type": "<type>", "count": <int>, "description": "<one-line>"}
|
||||||
|
],
|
||||||
|
"patch": {
|
||||||
|
"reasoning": "<why these edits address the batch's common failures>",
|
||||||
|
"edits": [
|
||||||
|
{"op": "append", "content": "<markdown to add at end of skill>"},
|
||||||
|
{"op": "insert_after", "target": "<exact heading/text to insert after>", "content": "<markdown>"},
|
||||||
|
{"op": "replace", "target": "<exact text to replace>", "content": "<replacement>"},
|
||||||
|
{"op": "delete", "target": "<exact text to remove>"}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Only include edits that are needed. "edits" can be an empty list if no patch is warranted.
|
||||||
@@ -0,0 +1,33 @@
|
|||||||
|
You are an expert success-pattern analyst for AI agents operating in ALFWorld,
|
||||||
|
a text-based embodied household environment.
|
||||||
|
|
||||||
|
You will be given MULTIPLE successful agent trajectories from a single minibatch
|
||||||
|
and the current skill document. Your job is to identify generalizable behavior
|
||||||
|
patterns that are COMMON across the batch and worth encoding in the skill.
|
||||||
|
|
||||||
|
## Rules
|
||||||
|
- Only propose patches for patterns NOT already covered in the skill.
|
||||||
|
- Focus on patterns that appear across MULTIPLE trajectories in the batch.
|
||||||
|
- Be concise. Patterns must generalize beyond specific tasks.
|
||||||
|
- Prefer reinforcing existing sections over adding new top-level sections.
|
||||||
|
- If the agents' success involved efficient exploration or smart appliance usage,
|
||||||
|
consider reinforcing that in the patch.
|
||||||
|
|
||||||
|
You will be told the maximum number of edits (the budget L). Produce AT MOST L edits,
|
||||||
|
focusing on the most broadly applicable patterns. You may produce fewer if warranted.
|
||||||
|
|
||||||
|
Respond ONLY with a valid JSON object:
|
||||||
|
{
|
||||||
|
"batch_size": <number of trajectories analysed>,
|
||||||
|
"success_patterns": ["<pattern 1>", "<pattern 2>"],
|
||||||
|
"patch": {
|
||||||
|
"reasoning": "<why these patterns are worth encoding>",
|
||||||
|
"edits": [
|
||||||
|
{"op": "append", "content": "<markdown>"},
|
||||||
|
{"op": "insert_after", "target": "<heading/text>", "content": "<markdown>"},
|
||||||
|
{"op": "replace", "target": "<old text>", "content": "<new text>"},
|
||||||
|
{"op": "delete", "target": "<exact text to remove>"}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
"edits" may be empty if the skill already covers all observed patterns.
|
||||||
@@ -0,0 +1,35 @@
|
|||||||
|
You are an expert diagnostic-probe designer for ALFWorld embodied tasks.
|
||||||
|
|
||||||
|
You will design one short diagnostic instruction to append to the student's prompt
|
||||||
|
for a handful of representative ALFWorld trajectories.
|
||||||
|
|
||||||
|
The goal is to expose whether the student has the right intermediate subgoal,
|
||||||
|
object/receptacle state, and next-step intention without substantially changing
|
||||||
|
the current scaffold.
|
||||||
|
|
||||||
|
## Hard Constraints
|
||||||
|
1. Do NOT substantially change the student's existing action-selection scaffold.
|
||||||
|
2. Do NOT prescribe a brand-new planner or long multi-step policy.
|
||||||
|
3. Do NOT ask for exhaustive search over all objects or all admissible actions.
|
||||||
|
4. Keep the diagnostic readout brief and place it inside the existing <think>...</think> block.
|
||||||
|
5. The student must still output exactly one admissible action inside <action>...</action>.
|
||||||
|
6. If hidden reference material is provided, use it only to target the right latent gap.
|
||||||
|
7. Never copy hidden reference content into the student-facing probe.
|
||||||
|
|
||||||
|
## Good Probe Targets
|
||||||
|
- current subgoal
|
||||||
|
- target object / target receptacle / target state
|
||||||
|
- decisive missing precondition
|
||||||
|
- why one candidate action is better than a tempting alternative
|
||||||
|
- whether the current step should explore, transform an object, or place it
|
||||||
|
|
||||||
|
## Bad Probe Targets
|
||||||
|
- a full optimal plan from start to finish
|
||||||
|
- exhaustive object inventories
|
||||||
|
- a new theorem-like or planner-like protocol
|
||||||
|
|
||||||
|
Respond ONLY with a valid JSON object:
|
||||||
|
{
|
||||||
|
"reasoning": "<why this probe reveals the latent skill gap>",
|
||||||
|
"probe_instruction": "<the exact instruction text to append to the student prompt>"
|
||||||
|
}
|
||||||
@@ -0,0 +1,8 @@
|
|||||||
|
|
||||||
|
You are an expert agent operating in the ALFRED Embodied Environment.
|
||||||
|
Your current observation is: {current_observation}
|
||||||
|
Your admissible actions of the current situation are: [{admissible_actions}].
|
||||||
|
|
||||||
|
Now it's your turn to take an action.
|
||||||
|
You should first reason step-by-step about the current situation. This reasoning process MUST be enclosed within <think> </think> tags.
|
||||||
|
Once you've finished your reasoning, you should choose an admissible action for current step and present it within <action> </action> tags.
|
||||||
@@ -0,0 +1,9 @@
|
|||||||
|
|
||||||
|
You are an expert agent operating in the ALFRED Embodied Environment. Your task is to: {task_description}
|
||||||
|
Prior to this step, you have already taken {step_count} step(s). Below are the most recent {history_length} observations and the corresponding actions you took: {action_history}
|
||||||
|
You are now at step {current_step} and your current observation is: {current_observation}
|
||||||
|
Your admissible actions of the current situation are: [{admissible_actions}].
|
||||||
|
|
||||||
|
Now it's your turn to take an action.
|
||||||
|
You should first reason step-by-step about the current situation. This reasoning process MUST be enclosed within <think> </think> tags.
|
||||||
|
Once you've finished your reasoning, you should choose an admissible action for current step and present it within <action> </action> tags.
|
||||||
@@ -0,0 +1,16 @@
|
|||||||
|
|
||||||
|
You are an expert agent operating in the ALFRED Embodied Environment. Your task is to: {task_description}
|
||||||
|
|
||||||
|
## Retrieved Relevant Experience
|
||||||
|
|
||||||
|
{retrieved_memories}
|
||||||
|
|
||||||
|
## Current Progress
|
||||||
|
|
||||||
|
Prior to this step, you have already taken {step_count} step(s). Below are the most recent {history_length} observations and the corresponding actions you took: {action_history}
|
||||||
|
You are now at step {current_step} and your current observation is: {current_observation}
|
||||||
|
Your admissible actions of the current situation are: [{admissible_actions}].
|
||||||
|
|
||||||
|
Now it's your turn to take an action.
|
||||||
|
You should first reason step-by-step about the current situation. This reasoning process MUST be enclosed within <think> </think> tags.
|
||||||
|
Once you've finished your reasoning, you should choose an admissible action for current step and present it within <action> </action> tags.
|
||||||
@@ -0,0 +1,4 @@
|
|||||||
|
"""ALFWorld Reflect stage.
|
||||||
|
|
||||||
|
Prompts are now loaded from .md files by the base adapter.
|
||||||
|
"""
|
||||||
@@ -0,0 +1,359 @@
|
|||||||
|
"""ALFWorld rollout module for ReflACT.
|
||||||
|
|
||||||
|
Provides:
|
||||||
|
- build_alfworld_env(): build ALFWorld environment (wraps vendored SkillRL env)
|
||||||
|
- run_alfworld_batch(): run a batch of ALFWorld episodes in parallel
|
||||||
|
- TASKS: list of ALFWorld task types
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import sys
|
||||||
|
import time
|
||||||
|
import concurrent.futures
|
||||||
|
import numpy as np
|
||||||
|
|
||||||
|
from skillopt.model import chat_student
|
||||||
|
|
||||||
|
# ── Constants ─────────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
TASKS = [
|
||||||
|
"pick_and_place",
|
||||||
|
"pick_two_obj_and_place",
|
||||||
|
"look_at_obj_in_light",
|
||||||
|
"pick_heat_then_place_in_recep",
|
||||||
|
"pick_cool_then_place_in_recep",
|
||||||
|
"pick_clean_then_place_in_recep",
|
||||||
|
]
|
||||||
|
|
||||||
|
# ── Helpers ───────────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
|
||||||
|
def _get_task_type(gamefile: str) -> str:
|
||||||
|
for task in TASKS:
|
||||||
|
if task in gamefile:
|
||||||
|
return task
|
||||||
|
return "other"
|
||||||
|
|
||||||
|
|
||||||
|
def _extract_action(model_response: str) -> str | None:
|
||||||
|
match = re.search(r"<action>(.*?)</action>", model_response, re.DOTALL)
|
||||||
|
return match.group(1).strip() if match else None
|
||||||
|
|
||||||
|
|
||||||
|
def _extract_think(model_response: str) -> str | None:
|
||||||
|
match = re.search(r"<think>(.*?)</think>", model_response, re.DOTALL)
|
||||||
|
return match.group(1).strip() if match else None
|
||||||
|
|
||||||
|
|
||||||
|
def _build_skill_prompt(skill_content: str) -> str:
|
||||||
|
"""Build the skill section to inject into the agent's system prompt."""
|
||||||
|
if not skill_content or not skill_content.strip():
|
||||||
|
return ""
|
||||||
|
return (
|
||||||
|
"\n\n## Skill Knowledge\n"
|
||||||
|
"Below is a skill document with learned strategies. "
|
||||||
|
"Use these guidelines to inform your decisions:\n\n"
|
||||||
|
f"{skill_content}\n"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _append_diagnostic_instruction(prompt: str, diagnostic_instruction: str) -> str:
|
||||||
|
if not diagnostic_instruction or not diagnostic_instruction.strip():
|
||||||
|
return prompt
|
||||||
|
return f"{prompt}\n\n## Training Readout\n{diagnostic_instruction.strip()}\n"
|
||||||
|
|
||||||
|
|
||||||
|
# ── Environment builder ──────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
|
||||||
|
def build_alfworld_env(
|
||||||
|
env_num: int,
|
||||||
|
eval_dataset: str = "eval_out_of_distribution",
|
||||||
|
seed: int = 42,
|
||||||
|
is_train: bool = False,
|
||||||
|
specific_gamefiles: list[str] | None = None,
|
||||||
|
):
|
||||||
|
"""Build ALFWorld environment manager.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
env_num: number of parallel environments
|
||||||
|
eval_dataset: 'eval_in_distribution' or 'eval_out_of_distribution' or train
|
||||||
|
seed: random seed
|
||||||
|
is_train: whether to use training set
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
env_manager: AlfWorldEnvironmentManager instance
|
||||||
|
"""
|
||||||
|
from omegaconf import OmegaConf
|
||||||
|
from functools import partial
|
||||||
|
|
||||||
|
from skillopt.envs.alfworld.vendor.alfworld_envs import build_alfworld_envs
|
||||||
|
from skillopt.envs.alfworld.vendor.alfworld_projection import alfworld_projection
|
||||||
|
from skillopt.envs.alfworld.vendor.env_manager import AlfWorldEnvironmentManager
|
||||||
|
|
||||||
|
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||||
|
|
||||||
|
alf_config_path = os.path.join(HERE, "vendor", "config_tw.yaml")
|
||||||
|
env_kwargs = {"eval_dataset": eval_dataset}
|
||||||
|
|
||||||
|
envs = build_alfworld_envs(
|
||||||
|
alf_config_path,
|
||||||
|
seed=seed,
|
||||||
|
env_num=env_num,
|
||||||
|
group_n=1,
|
||||||
|
is_train=is_train,
|
||||||
|
env_kwargs=env_kwargs,
|
||||||
|
resources_per_worker=None,
|
||||||
|
gamefiles=specific_gamefiles,
|
||||||
|
)
|
||||||
|
|
||||||
|
config = OmegaConf.create(
|
||||||
|
{
|
||||||
|
"env": {
|
||||||
|
"history_length": 2,
|
||||||
|
"env_name": "alfworld/AlfredTWEnv",
|
||||||
|
}
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
projection_f = partial(alfworld_projection)
|
||||||
|
env_manager = AlfWorldEnvironmentManager(envs, projection_f, config)
|
||||||
|
return env_manager
|
||||||
|
|
||||||
|
|
||||||
|
# ── Batch rollout ─────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
|
||||||
|
def run_alfworld_batch(
|
||||||
|
env_manager,
|
||||||
|
skill_content: str,
|
||||||
|
max_steps: int = 50,
|
||||||
|
out_root: str = "",
|
||||||
|
max_api_workers: int = 8,
|
||||||
|
temperature: float = 0.4,
|
||||||
|
max_completion_tokens: int = 2048,
|
||||||
|
diagnostic_mode: bool = False,
|
||||||
|
diagnostic_instruction: str = "",
|
||||||
|
result_ids: list[str] | None = None,
|
||||||
|
) -> list[dict]:
|
||||||
|
"""Run a batch of ALFWorld episodes.
|
||||||
|
|
||||||
|
Returns a list of result dicts compatible with SkillOpt pipeline:
|
||||||
|
[
|
||||||
|
{
|
||||||
|
"id": "<env_idx>_<gamefile_hash>",
|
||||||
|
"hard": 0 or 1,
|
||||||
|
"soft": 0.0 or 1.0,
|
||||||
|
"n_turns": <int>,
|
||||||
|
"fail_reason": "<str>",
|
||||||
|
"agent_ok": True,
|
||||||
|
"task_type": "<str>",
|
||||||
|
"gamefile": "<str>",
|
||||||
|
"task_description": "<str>",
|
||||||
|
},
|
||||||
|
...
|
||||||
|
]
|
||||||
|
|
||||||
|
Also saves conversation.json per environment in out_root/predictions/<task_id>/
|
||||||
|
"""
|
||||||
|
skill_prompt = _build_skill_prompt(skill_content)
|
||||||
|
|
||||||
|
obs, infos = env_manager.reset({})
|
||||||
|
env_num = len(obs["text"])
|
||||||
|
env_dones = [False] * env_num
|
||||||
|
overall_success = [False] * env_num
|
||||||
|
|
||||||
|
# Build per-env metadata
|
||||||
|
env_meta: list[dict] = []
|
||||||
|
for i in range(env_num):
|
||||||
|
gamefile = infos[i].get("extra.gamefile", "") if isinstance(infos[i], dict) else ""
|
||||||
|
task_type = _get_task_type(gamefile)
|
||||||
|
# Extract task description from initial observation
|
||||||
|
task_desc = ""
|
||||||
|
anchor_text = obs["anchor"][i] if "anchor" in obs else ""
|
||||||
|
task_start = anchor_text.find("Your task is to: ")
|
||||||
|
if task_start != -1:
|
||||||
|
task_desc = anchor_text[task_start + len("Your task is to: "):].strip()
|
||||||
|
|
||||||
|
env_meta.append({
|
||||||
|
"gamefile": gamefile,
|
||||||
|
"task_type": task_type,
|
||||||
|
"task_description": task_desc,
|
||||||
|
})
|
||||||
|
|
||||||
|
# Per-env conversation records
|
||||||
|
conversations: list[list[dict]] = [[] for _ in range(env_num)]
|
||||||
|
|
||||||
|
for step_idx in range(max_steps):
|
||||||
|
if all(env_dones):
|
||||||
|
break
|
||||||
|
|
||||||
|
active_indices = [i for i in range(env_num) if not env_dones[i]]
|
||||||
|
|
||||||
|
# Build prompts with skill injection
|
||||||
|
prompts: dict[int, str] = {}
|
||||||
|
for i in active_indices:
|
||||||
|
prompt = obs["text"][i]
|
||||||
|
if skill_prompt:
|
||||||
|
# Inject skill before the action instruction
|
||||||
|
prompt = skill_prompt + "\n" + prompt
|
||||||
|
if diagnostic_mode and diagnostic_instruction.strip():
|
||||||
|
prompt = _append_diagnostic_instruction(prompt, diagnostic_instruction)
|
||||||
|
prompts[i] = prompt
|
||||||
|
|
||||||
|
# Call API in parallel
|
||||||
|
actions = ["None"] * env_num
|
||||||
|
action_timeout = 180
|
||||||
|
|
||||||
|
def call_api(idx):
|
||||||
|
try:
|
||||||
|
response, _ = chat_student(
|
||||||
|
system="You are an expert agent operating in the ALFRED Embodied Environment.",
|
||||||
|
user=prompts[idx],
|
||||||
|
max_completion_tokens=max_completion_tokens,
|
||||||
|
retries=5,
|
||||||
|
stage="rollout",
|
||||||
|
timeout=120,
|
||||||
|
)
|
||||||
|
response = (response or "").strip()
|
||||||
|
if not response:
|
||||||
|
return idx, "<think>empty model response</think><action>look</action>"
|
||||||
|
if _extract_action(response) is None:
|
||||||
|
return idx, "<think>missing action tag</think><action>look</action>"
|
||||||
|
return idx, response
|
||||||
|
except Exception as e:
|
||||||
|
return idx, "<think>error</think><action>look</action>"
|
||||||
|
|
||||||
|
executor = concurrent.futures.ThreadPoolExecutor(max_workers=max_api_workers)
|
||||||
|
try:
|
||||||
|
futures = {executor.submit(call_api, i): i for i in active_indices}
|
||||||
|
started_at = {future: time.time() for future in futures}
|
||||||
|
pending_futs = set(futures)
|
||||||
|
while pending_futs:
|
||||||
|
done, _ = concurrent.futures.wait(
|
||||||
|
pending_futs,
|
||||||
|
timeout=5,
|
||||||
|
return_when=concurrent.futures.FIRST_COMPLETED,
|
||||||
|
)
|
||||||
|
now = time.time()
|
||||||
|
timed_out = [
|
||||||
|
future for future in pending_futs - done
|
||||||
|
if now - started_at[future] >= action_timeout
|
||||||
|
]
|
||||||
|
for future in done:
|
||||||
|
pending_futs.remove(future)
|
||||||
|
try:
|
||||||
|
idx, response = future.result()
|
||||||
|
except Exception: # noqa: BLE001
|
||||||
|
idx = futures[future]
|
||||||
|
response = "<think>error</think><action>look</action>"
|
||||||
|
actions[idx] = response
|
||||||
|
for future in timed_out:
|
||||||
|
pending_futs.remove(future)
|
||||||
|
idx = futures[future]
|
||||||
|
actions[idx] = "<think>api timeout</think><action>look</action>"
|
||||||
|
finally:
|
||||||
|
executor.shutdown(wait=False, cancel_futures=True)
|
||||||
|
|
||||||
|
# Save model responses before stepping
|
||||||
|
model_responses = {i: actions[i] for i in active_indices}
|
||||||
|
|
||||||
|
# Step environment
|
||||||
|
obs, rewards, dones, infos = env_manager.step(actions)
|
||||||
|
|
||||||
|
# Record trajectory
|
||||||
|
for i in active_indices:
|
||||||
|
step_record = {
|
||||||
|
"step": step_idx,
|
||||||
|
"action": _extract_action(model_responses[i]),
|
||||||
|
"reasoning": _extract_think(model_responses[i]),
|
||||||
|
"model_response": model_responses[i],
|
||||||
|
"env_feedback": obs["anchor"][i] if "anchor" in obs else "",
|
||||||
|
"reward": float(rewards[i]),
|
||||||
|
"done": bool(dones[i]),
|
||||||
|
}
|
||||||
|
conversations[i].append(step_record)
|
||||||
|
|
||||||
|
# Update done status
|
||||||
|
for i in range(env_num):
|
||||||
|
if env_dones[i]:
|
||||||
|
continue
|
||||||
|
if dones[i]:
|
||||||
|
env_dones[i] = True
|
||||||
|
won = bool(infos[i].get("won", False))
|
||||||
|
overall_success[i] = won
|
||||||
|
|
||||||
|
# Build results and save conversations
|
||||||
|
results: list[dict] = []
|
||||||
|
pred_dir = os.path.join(out_root, "predictions") if out_root else ""
|
||||||
|
|
||||||
|
for i in range(env_num):
|
||||||
|
gamefile = env_meta[i]["gamefile"]
|
||||||
|
task_type = env_meta[i]["task_type"]
|
||||||
|
task_desc = env_meta[i]["task_description"]
|
||||||
|
n_turns = len(conversations[i])
|
||||||
|
won = overall_success[i]
|
||||||
|
|
||||||
|
# Generate stable task ID from env index and gamefile
|
||||||
|
task_id = str(result_ids[i]) if result_ids and i < len(result_ids) else f"env_{i:03d}"
|
||||||
|
|
||||||
|
fail_reason = ""
|
||||||
|
if not won:
|
||||||
|
if not env_dones[i]:
|
||||||
|
fail_reason = f"Timeout after {max_steps} steps"
|
||||||
|
else:
|
||||||
|
fail_reason = "Episode ended without completing the task"
|
||||||
|
|
||||||
|
result = {
|
||||||
|
"id": task_id,
|
||||||
|
"hard": 1 if won else 0,
|
||||||
|
"soft": 1.0 if won else 0.0,
|
||||||
|
"n_turns": n_turns,
|
||||||
|
"fail_reason": fail_reason,
|
||||||
|
"agent_ok": True, # ALFWorld agent always runs OK (no crash)
|
||||||
|
"task_type": task_type,
|
||||||
|
"gamefile": gamefile,
|
||||||
|
"task_description": task_desc,
|
||||||
|
"instruction_type": task_type, # for compatibility with v2 pipeline
|
||||||
|
}
|
||||||
|
results.append(result)
|
||||||
|
|
||||||
|
# Save conversation
|
||||||
|
if pred_dir:
|
||||||
|
conv_dir = os.path.join(pred_dir, task_id)
|
||||||
|
os.makedirs(conv_dir, exist_ok=True)
|
||||||
|
with open(os.path.join(conv_dir, "conversation.json"), "w") as f:
|
||||||
|
json.dump(conversations[i], f, ensure_ascii=False, indent=2)
|
||||||
|
|
||||||
|
return results
|
||||||
|
|
||||||
|
|
||||||
|
# ── Item loading (for compatibility with split_three_way) ────────────────────
|
||||||
|
|
||||||
|
|
||||||
|
def load_alfworld_items(
|
||||||
|
eval_dataset: str,
|
||||||
|
env_num: int,
|
||||||
|
seed: int = 42,
|
||||||
|
is_train: bool = False,
|
||||||
|
) -> list[dict]:
|
||||||
|
"""Create pseudo-item dicts for ALFWorld environments.
|
||||||
|
|
||||||
|
Since ALFWorld doesn't have a static JSON dataset like SpreadsheetBench,
|
||||||
|
we create lightweight item dicts that carry enough metadata for the pipeline.
|
||||||
|
The actual environment is built dynamically.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
List of dicts with "id" keys, one per environment slot.
|
||||||
|
"""
|
||||||
|
items = []
|
||||||
|
for i in range(env_num):
|
||||||
|
items.append({
|
||||||
|
"id": f"env_{i:03d}",
|
||||||
|
"eval_dataset": eval_dataset,
|
||||||
|
"env_index": i,
|
||||||
|
})
|
||||||
|
return items
|
||||||
@@ -0,0 +1,45 @@
|
|||||||
|
# ALFWorld Embodied Agent Skill
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
This skill guides agents operating in the ALFWorld text-based embodied environment.
|
||||||
|
The agent must complete household tasks by navigating rooms, interacting with objects,
|
||||||
|
and using appliances. Actions must be chosen from the admissible action list provided
|
||||||
|
at each step.
|
||||||
|
|
||||||
|
**Output format**: Always output `<think>...</think>` for reasoning, then `<action>...</action>` for the chosen action.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task Types
|
||||||
|
|
||||||
|
| Type | Goal | Key Steps |
|
||||||
|
|------|------|-----------|
|
||||||
|
| Pick & Place | Put object X in/on receptacle Y | Find X -> take X -> go to Y -> put X in/on Y |
|
||||||
|
| Pick Two & Place | Put two instances of X in/on Y | Find X1 -> take -> place -> find X2 -> take -> place |
|
||||||
|
| Examine in Light | Examine object X under desklamp | Find X -> take X -> find desklamp -> use desklamp |
|
||||||
|
| Clean & Place | Clean object X and put in/on Y | Find X -> take X -> go to sink -> clean X -> go to Y -> put X |
|
||||||
|
| Heat & Place | Heat object X and put in/on Y | Find X -> take X -> go to microwave -> heat X -> go to Y -> put X |
|
||||||
|
| Cool & Place | Cool object X and put in/on Y | Find X -> take X -> go to fridge -> cool X -> go to Y -> put X |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## General Principles
|
||||||
|
|
||||||
|
1. **Decompose the task**: Parse the goal into ordered sub-goals (locate, acquire, transform, deliver). Complete each before moving to the next.
|
||||||
|
2. **Systematic exploration**: Search each surface and container exactly once before revisiting. Open closed containers (drawers, cabinets, fridge) before judging them empty.
|
||||||
|
3. **Grab immediately**: When a required object is visible and reachable, take it right away before moving elsewhere.
|
||||||
|
4. **Transform before placing**: If the task requires cleaning, heating, or cooling, perform the state change at the appropriate appliance before heading to the final destination.
|
||||||
|
5. **Direct delivery**: Once holding the transformed (or untransformed) goal object, navigate straight to the target receptacle and place it.
|
||||||
|
6. **Track progress**: Maintain an internal count of how many objects still need to be found and placed. Only stop searching when the count reaches zero.
|
||||||
|
7. **Avoid loops**: Never repeat the same action more than twice in a row. If stuck, move to a different unexplored location.
|
||||||
|
8. **Only choose admissible actions**: Always pick an action from the admissible action list. Do not invent actions.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Common Mistakes to Avoid
|
||||||
|
|
||||||
|
- **Revisiting searched locations**: Keep track of which surfaces/containers have been checked; do not re-examine them.
|
||||||
|
- **Ignoring visible objects**: If the target object appears in the observation, pick it up immediately.
|
||||||
|
- **Skipping state changes**: Do not place an object at the destination without first cleaning/heating/cooling it when required.
|
||||||
|
- **Premature termination**: Do not stop the episode until all goal conditions are verified as met.
|
||||||
|
- **Action loops**: Repeatedly toggling or examining the same object wastes steps. Move on to new locations instead.
|
||||||
+9
@@ -0,0 +1,9 @@
|
|||||||
|
"""Vendored ALFWorld environment runtime.
|
||||||
|
|
||||||
|
Minimal subset of SkillRL's agent_system package needed to run
|
||||||
|
ALFWorld environments with ReflACT. Original source:
|
||||||
|
https://github.com/NTU-LANTERN/SkillRL (Apache-2.0 License)
|
||||||
|
"""
|
||||||
|
from .alfworld_envs import AlfworldEnvs, build_alfworld_envs
|
||||||
|
from .alfworld_projection import alfworld_projection
|
||||||
|
from .env_manager import AlfWorldEnvironmentManager
|
||||||
+221
@@ -0,0 +1,221 @@
|
|||||||
|
# Vendored from SkillRL (Apache-2.0 License)
|
||||||
|
# Original: agent_system/environments/env_package/alfworld/envs.py
|
||||||
|
# Modified: imports use pip-installed alfworld package instead of vendored copy.
|
||||||
|
|
||||||
|
import os
|
||||||
|
import multiprocessing as mp
|
||||||
|
import traceback
|
||||||
|
import yaml
|
||||||
|
import gymnasium as gym
|
||||||
|
import numpy as np
|
||||||
|
|
||||||
|
from alfworld.agents.environment import get_environment
|
||||||
|
|
||||||
|
|
||||||
|
def load_config_file(path):
|
||||||
|
assert os.path.exists(path), f"Invalid config file: {path}"
|
||||||
|
with open(path) as reader:
|
||||||
|
config = yaml.safe_load(reader)
|
||||||
|
return config
|
||||||
|
|
||||||
|
|
||||||
|
def compute_reward(info, multi_modal=False):
|
||||||
|
if multi_modal:
|
||||||
|
reward = 10.0 * float(info['won']) + float(info['goal_condition_success_rate'])
|
||||||
|
else:
|
||||||
|
reward = 10.0 * float(info['won'])
|
||||||
|
return reward
|
||||||
|
|
||||||
|
|
||||||
|
class AlfworldWorker:
|
||||||
|
"""Stateful worker that holds one ALFWorld sub-environment."""
|
||||||
|
|
||||||
|
def __init__(self, config, seed, base_env, gamefile=None):
|
||||||
|
if gamefile:
|
||||||
|
base_env.game_files = [gamefile]
|
||||||
|
if hasattr(base_env, "num_games"):
|
||||||
|
base_env.num_games = 1
|
||||||
|
self.env = base_env.init_env(batch_size=1)
|
||||||
|
self.env.seed(seed)
|
||||||
|
|
||||||
|
def step(self, action):
|
||||||
|
actions = [action]
|
||||||
|
obs, scores, dones, infos = self.env.step(actions)
|
||||||
|
infos['observation_text'] = obs
|
||||||
|
return obs, scores, dones, infos
|
||||||
|
|
||||||
|
def reset(self):
|
||||||
|
obs, infos = self.env.reset()
|
||||||
|
infos['observation_text'] = obs
|
||||||
|
return obs, infos
|
||||||
|
|
||||||
|
|
||||||
|
def _worker_loop(cmd_q, result_q, config, seed, is_train, eval_dataset, gamefile):
|
||||||
|
"""Run one ALFWorld environment in a child process."""
|
||||||
|
try:
|
||||||
|
env_type = config['env']['type']
|
||||||
|
base_env = get_environment(env_type)(
|
||||||
|
config,
|
||||||
|
train_eval='train' if is_train else eval_dataset,
|
||||||
|
)
|
||||||
|
worker = AlfworldWorker(config, seed, base_env, gamefile)
|
||||||
|
result_q.put((True, "ready"))
|
||||||
|
except BaseException:
|
||||||
|
result_q.put((False, traceback.format_exc()))
|
||||||
|
return
|
||||||
|
|
||||||
|
while True:
|
||||||
|
cmd, payload = cmd_q.get()
|
||||||
|
if cmd == "close":
|
||||||
|
result_q.put((True, None))
|
||||||
|
return
|
||||||
|
try:
|
||||||
|
if cmd == "reset":
|
||||||
|
result = worker.reset()
|
||||||
|
elif cmd == "step":
|
||||||
|
result = worker.step(payload)
|
||||||
|
else:
|
||||||
|
raise ValueError(f"Unknown ALFWorld worker command: {cmd}")
|
||||||
|
result_q.put((True, result))
|
||||||
|
except BaseException:
|
||||||
|
result_q.put((False, traceback.format_exc()))
|
||||||
|
|
||||||
|
|
||||||
|
class _ProcessWorker:
|
||||||
|
"""Small stdlib actor wrapper for one environment process."""
|
||||||
|
|
||||||
|
def __init__(self, ctx, config, seed, is_train, eval_dataset, gamefile=None):
|
||||||
|
self.cmd_q = ctx.Queue(maxsize=1)
|
||||||
|
self.result_q = ctx.Queue(maxsize=1)
|
||||||
|
self.process = ctx.Process(
|
||||||
|
target=_worker_loop,
|
||||||
|
args=(self.cmd_q, self.result_q, config, seed, is_train, eval_dataset, gamefile),
|
||||||
|
)
|
||||||
|
self.process.start()
|
||||||
|
ok, payload = self.result_q.get()
|
||||||
|
if not ok:
|
||||||
|
self.close(kill=True)
|
||||||
|
raise RuntimeError(f"Failed to start ALFWorld worker:\n{payload}")
|
||||||
|
|
||||||
|
def send(self, cmd, payload=None):
|
||||||
|
self.cmd_q.put((cmd, payload))
|
||||||
|
|
||||||
|
def recv(self):
|
||||||
|
ok, payload = self.result_q.get()
|
||||||
|
if not ok:
|
||||||
|
raise RuntimeError(f"ALFWorld worker failed:\n{payload}")
|
||||||
|
return payload
|
||||||
|
|
||||||
|
def close(self, kill=False):
|
||||||
|
if self.process.is_alive() and not kill:
|
||||||
|
try:
|
||||||
|
self.send("close")
|
||||||
|
self.recv()
|
||||||
|
except Exception:
|
||||||
|
kill = True
|
||||||
|
if kill and self.process.is_alive():
|
||||||
|
self.process.terminate()
|
||||||
|
self.process.join(timeout=5)
|
||||||
|
if self.process.is_alive():
|
||||||
|
self.process.kill()
|
||||||
|
self.process.join(timeout=1)
|
||||||
|
self.cmd_q.close()
|
||||||
|
self.result_q.close()
|
||||||
|
|
||||||
|
|
||||||
|
class AlfworldEnvs(gym.Env):
|
||||||
|
"""Vectorized ALFWorld environment using local process workers."""
|
||||||
|
|
||||||
|
def __init__(self, alf_config_path, seed, env_num, group_n,
|
||||||
|
resources_per_worker, is_train=True, env_kwargs=None, gamefiles=None):
|
||||||
|
super().__init__()
|
||||||
|
if env_kwargs is None:
|
||||||
|
env_kwargs = {}
|
||||||
|
|
||||||
|
eval_dataset = env_kwargs.get('eval_dataset', 'eval_in_distribution')
|
||||||
|
config = load_config_file(alf_config_path)
|
||||||
|
env_type = config['env']['type']
|
||||||
|
self.multi_modal = (env_type == 'AlfredThorEnv')
|
||||||
|
self.num_processes = env_num * group_n
|
||||||
|
self.group_n = group_n
|
||||||
|
self.gamefiles = list(gamefiles or [])
|
||||||
|
if self.gamefiles and len(self.gamefiles) != self.num_processes:
|
||||||
|
raise ValueError(
|
||||||
|
f"Expected {self.num_processes} gamefiles, got {len(self.gamefiles)}"
|
||||||
|
)
|
||||||
|
|
||||||
|
start_method = os.environ.get("ALFWORLD_WORKER_START_METHOD") or None
|
||||||
|
ctx = mp.get_context(start_method) if start_method else mp.get_context()
|
||||||
|
self.workers = []
|
||||||
|
for i in range(self.num_processes):
|
||||||
|
worker_gamefile = self.gamefiles[i] if self.gamefiles else None
|
||||||
|
worker = _ProcessWorker(
|
||||||
|
ctx,
|
||||||
|
config,
|
||||||
|
seed + (i // self.group_n),
|
||||||
|
is_train,
|
||||||
|
eval_dataset,
|
||||||
|
worker_gamefile,
|
||||||
|
)
|
||||||
|
self.workers.append(worker)
|
||||||
|
|
||||||
|
self.prev_admissible_commands = [None for _ in range(self.num_processes)]
|
||||||
|
|
||||||
|
def step(self, actions):
|
||||||
|
assert len(actions) == self.num_processes
|
||||||
|
|
||||||
|
for i, worker in enumerate(self.workers):
|
||||||
|
worker.send("step", actions[i])
|
||||||
|
results = [worker.recv() for worker in self.workers]
|
||||||
|
|
||||||
|
text_obs_list = []
|
||||||
|
rewards_list = []
|
||||||
|
dones_list = []
|
||||||
|
info_list = []
|
||||||
|
|
||||||
|
for i, (obs, scores, dones, info) in enumerate(results):
|
||||||
|
for k in info.keys():
|
||||||
|
info[k] = info[k][0]
|
||||||
|
text_obs_list.append(obs[0])
|
||||||
|
dones_list.append(dones[0])
|
||||||
|
info_list.append(info)
|
||||||
|
self.prev_admissible_commands[i] = info['admissible_commands']
|
||||||
|
rewards_list.append(compute_reward(info, self.multi_modal))
|
||||||
|
|
||||||
|
image_obs_list = None
|
||||||
|
return text_obs_list, image_obs_list, rewards_list, dones_list, info_list
|
||||||
|
|
||||||
|
def reset(self):
|
||||||
|
for worker in self.workers:
|
||||||
|
worker.send("reset")
|
||||||
|
results = [worker.recv() for worker in self.workers]
|
||||||
|
|
||||||
|
text_obs_list = []
|
||||||
|
info_list = []
|
||||||
|
|
||||||
|
for i, (obs, info) in enumerate(results):
|
||||||
|
for k in info.keys():
|
||||||
|
info[k] = info[k][0]
|
||||||
|
text_obs_list.append(obs[0])
|
||||||
|
self.prev_admissible_commands[i] = info['admissible_commands']
|
||||||
|
info_list.append(info)
|
||||||
|
|
||||||
|
image_obs_list = None
|
||||||
|
return text_obs_list, image_obs_list, info_list
|
||||||
|
|
||||||
|
@property
|
||||||
|
def get_admissible_commands(self):
|
||||||
|
return self.prev_admissible_commands
|
||||||
|
|
||||||
|
def close(self):
|
||||||
|
for worker in self.workers:
|
||||||
|
worker.close()
|
||||||
|
|
||||||
|
|
||||||
|
def build_alfworld_envs(alf_config_path, seed, env_num, group_n,
|
||||||
|
resources_per_worker, is_train=True, env_kwargs=None, gamefiles=None):
|
||||||
|
"""Build vectorized ALFWorld environments."""
|
||||||
|
return AlfworldEnvs(
|
||||||
|
alf_config_path, seed, env_num, group_n,
|
||||||
|
resources_per_worker, is_train, env_kwargs, gamefiles,
|
||||||
|
)
|
||||||
@@ -0,0 +1,60 @@
|
|||||||
|
# Vendored from SkillRL (Apache-2.0 License)
|
||||||
|
# Original: agent_system/environments/env_package/alfworld/projection.py
|
||||||
|
|
||||||
|
from typing import List
|
||||||
|
import re
|
||||||
|
|
||||||
|
|
||||||
|
def alfworld_projection(actions: List[str], action_pools: List[List[str]]):
|
||||||
|
"""Process raw model outputs into valid ALFWorld actions.
|
||||||
|
|
||||||
|
Extracts text from ``<action>...</action>`` tags and validates that
|
||||||
|
the response also contains ``<think>...</think>`` tags.
|
||||||
|
|
||||||
|
Parameters
|
||||||
|
----------
|
||||||
|
actions : list[str]
|
||||||
|
Raw model outputs, one per environment.
|
||||||
|
action_pools : list[list[str]]
|
||||||
|
Admissible action lists per environment (unused but kept for API compat).
|
||||||
|
|
||||||
|
Returns
|
||||||
|
-------
|
||||||
|
actions : list[str]
|
||||||
|
Cleaned action strings.
|
||||||
|
valids : list[int]
|
||||||
|
1 if the action was successfully parsed, 0 otherwise.
|
||||||
|
"""
|
||||||
|
valids = [0] * len(actions)
|
||||||
|
|
||||||
|
for i in range(len(actions)):
|
||||||
|
original_str = actions[i]
|
||||||
|
actions[i] = actions[i].lower()
|
||||||
|
|
||||||
|
start_tag = "<action>"
|
||||||
|
end_tag = "</action>"
|
||||||
|
start_idx = actions[i].find(start_tag)
|
||||||
|
end_idx = actions[i].find(end_tag)
|
||||||
|
try:
|
||||||
|
if start_idx == -1 or end_idx == -1:
|
||||||
|
actions[i] = actions[i][-30:]
|
||||||
|
continue
|
||||||
|
|
||||||
|
extracted_action = actions[i][start_idx + len(start_tag):end_idx].strip().lower()
|
||||||
|
actions[i] = extracted_action
|
||||||
|
valids[i] = 1
|
||||||
|
|
||||||
|
except Exception:
|
||||||
|
actions[i] = actions[i][-30:]
|
||||||
|
|
||||||
|
# Require <think>...</think>
|
||||||
|
think_start_idx = original_str.find("<think>")
|
||||||
|
think_end_idx = original_str.find("</think>")
|
||||||
|
if think_start_idx == -1 or think_end_idx == -1:
|
||||||
|
valids[i] = 0
|
||||||
|
|
||||||
|
# Reject responses containing Chinese characters
|
||||||
|
if re.search(r'[\u4e00-\u9fff]', original_str):
|
||||||
|
valids[i] = 0
|
||||||
|
|
||||||
|
return actions, valids
|
||||||
@@ -0,0 +1,8 @@
|
|||||||
|
# Vendored from SkillRL (Apache-2.0 License)
|
||||||
|
# Original: agent_system/environments/prompts/alfworld.py
|
||||||
|
|
||||||
|
from skillopt.prompts import load_prompt
|
||||||
|
|
||||||
|
ALFWORLD_TEMPLATE_NO_HIS = load_prompt("rollout_no_history", env="alfworld")
|
||||||
|
ALFWORLD_TEMPLATE = load_prompt("rollout_with_history", env="alfworld")
|
||||||
|
ALFWORLD_TEMPLATE_WITH_MEMORY = load_prompt("rollout_with_memory", env="alfworld")
|
||||||
+145
@@ -0,0 +1,145 @@
|
|||||||
|
dataset:
|
||||||
|
data_path: '$ALFWORLD_DATA/json_2.1.1/train'
|
||||||
|
eval_id_data_path: '$ALFWORLD_DATA/json_2.1.1/valid_seen' # null/None to disable
|
||||||
|
eval_ood_data_path: '$ALFWORLD_DATA/json_2.1.1/valid_unseen' # null/None to disable
|
||||||
|
num_train_games: -1 # max training games (<=0 indicates full dataset)
|
||||||
|
num_eval_games: -1 # max evaluation games (<=0 indicates full dataset)
|
||||||
|
|
||||||
|
logic:
|
||||||
|
domain: '$ALFWORLD_DATA/logic/alfred.pddl' # PDDL domain file that defines the world dynamics
|
||||||
|
grammar: '$ALFWORLD_DATA/logic/alfred.twl2' # Grammar file that defines the text feedbacks
|
||||||
|
|
||||||
|
env:
|
||||||
|
type: 'AlfredTWEnv' # 'AlfredTWEnv' or 'AlfredThorEnv' or 'AlfredHybrid'
|
||||||
|
# regen_game_files: False # check if game is solvable by expert and save to game.tw-pddl file
|
||||||
|
domain_randomization: False # shuffle Textworld print order and object id nums
|
||||||
|
task_types: [1, 2, 3, 4, 5, 6] # task-type ids: 1 - Pick & Place, 2 - Examine in Light, 3 - Clean & Place, 4 - Heat & Place, 5 - Cool & Place, 6 - Pick Two & Place
|
||||||
|
expert_timeout_steps: 150 # max steps before timeout for expert to solve the task
|
||||||
|
expert_type: "handcoded" # 'handcoded' or 'planner'. Note: the planner is very slow for real-time use
|
||||||
|
goal_desc_human_anns_prob: 0.0 # prob of using human-annotated goal language instead of templated goals (1.0 indicates all human annotations from ALFRED)
|
||||||
|
|
||||||
|
hybrid:
|
||||||
|
start_eps: 100000 # starting episode of hybrid training, tw-only training upto this point
|
||||||
|
thor_prob: 0.5 # prob of AlfredThorEnv during hybrid training
|
||||||
|
eval_mode: "tw" # 'tw' or 'thor' - env used for evaluation during hybrid training
|
||||||
|
|
||||||
|
thor:
|
||||||
|
screen_width: 300 # width of THOR window
|
||||||
|
screen_height: 300 # height of THOR window
|
||||||
|
smooth_nav: False # smooth rotations, looks, and translations during navigation (very slow)
|
||||||
|
save_frames_to_disk: False # save frame PNGs to disk (useful for making videos)
|
||||||
|
save_frames_path: './videos/' # path to save frame PNGs
|
||||||
|
|
||||||
|
controller:
|
||||||
|
type: 'oracle' # 'oracle' or 'oracle_astar' or 'mrcnn' or 'mrcnn_astar' (aka BUTLER)
|
||||||
|
debug: False
|
||||||
|
load_receps: True # load receptacle locations from precomputed dict (if available)
|
||||||
|
|
||||||
|
mask_rcnn:
|
||||||
|
pretrained_model_path: '$ALFWORLD_DATA/detectors/mrcnn.pth'
|
||||||
|
|
||||||
|
general:
|
||||||
|
random_seed: 42
|
||||||
|
use_cuda: True # disable this when running on machine without cuda
|
||||||
|
visdom: False # plot training/eval curves, run with visdom server
|
||||||
|
task: 'alfred'
|
||||||
|
training_method: 'dagger' # 'dqn' or 'dagger'
|
||||||
|
save_path: './training/' # path to save pytorch models
|
||||||
|
observation_pool_capacity: 3 # k-size queue, 0 indicates no observation
|
||||||
|
hide_init_receptacles: False # remove initial observation containing navigable receptacles
|
||||||
|
|
||||||
|
training:
|
||||||
|
batch_size: 10
|
||||||
|
max_episode: 50000
|
||||||
|
smoothing_eps: 0.1
|
||||||
|
optimizer:
|
||||||
|
learning_rate: 0.001
|
||||||
|
clip_grad_norm: 5
|
||||||
|
|
||||||
|
evaluate:
|
||||||
|
run_eval: True
|
||||||
|
batch_size: 10
|
||||||
|
env:
|
||||||
|
type: "AlfredTWEnv"
|
||||||
|
|
||||||
|
checkpoint:
|
||||||
|
report_frequency: 1000 # report every N episode
|
||||||
|
experiment_tag: 'test' # name of experiment
|
||||||
|
load_pretrained: False # during test, enable this so that the agent load your pretrained model
|
||||||
|
load_from_tag: 'not loading anything' # name of pre-trained model to load in save_path
|
||||||
|
|
||||||
|
model:
|
||||||
|
encoder_layers: 1
|
||||||
|
decoder_layers: 1
|
||||||
|
encoder_conv_num: 5
|
||||||
|
block_hidden_dim: 64
|
||||||
|
n_heads: 1
|
||||||
|
dropout: 0.1
|
||||||
|
block_dropout: 0.1
|
||||||
|
recurrent: True
|
||||||
|
|
||||||
|
rl:
|
||||||
|
action_space: "admissible" # 'admissible' (candidates from text engine) or 'generation' (seq2seq-style generation) or 'beam_search_choice' or 'exhaustive' (not working)
|
||||||
|
max_target_length: 20 # max token length for seq2seq generation
|
||||||
|
beam_width: 10 # 1 means greedy
|
||||||
|
generate_top_k: 3
|
||||||
|
|
||||||
|
training:
|
||||||
|
max_nb_steps_per_episode: 50 # terminate after this many steps
|
||||||
|
learn_start_from_this_episode: 0 # delay updates until this epsiode
|
||||||
|
target_net_update_frequency: 500 # sync target net with online net per this many epochs
|
||||||
|
|
||||||
|
replay:
|
||||||
|
accumulate_reward_from_final: True
|
||||||
|
count_reward_lambda: 0.0 # 0 to disable
|
||||||
|
novel_object_reward_lambda: 0.0 # 0 to disable
|
||||||
|
discount_gamma_game_reward: 0.9
|
||||||
|
discount_gamma_count_reward: 0.5
|
||||||
|
discount_gamma_novel_object_reward: 0.5
|
||||||
|
replay_memory_capacity: 500000 # adjust this depending on your RAM size
|
||||||
|
replay_memory_priority_fraction: 0.5
|
||||||
|
update_per_k_game_steps: 5
|
||||||
|
replay_batch_size: 64
|
||||||
|
multi_step: 3
|
||||||
|
replay_sample_history_length: 4
|
||||||
|
replay_sample_update_from: 2
|
||||||
|
|
||||||
|
epsilon_greedy:
|
||||||
|
noisy_net: False # if this is true, then epsilon greedy is disabled
|
||||||
|
epsilon_anneal_episodes: 1000 # -1 if not annealing
|
||||||
|
epsilon_anneal_from: 0.3
|
||||||
|
epsilon_anneal_to: 0.1
|
||||||
|
|
||||||
|
dagger:
|
||||||
|
action_space: "generation" # 'admissible' (candidates from text engine) or 'generation' (seq2seq-style generation) or 'exhaustive' (not working)
|
||||||
|
max_target_length: 20 # max token length for seq2seq generation
|
||||||
|
beam_width: 10 # 1 means greedy
|
||||||
|
generate_top_k: 5
|
||||||
|
unstick_by_beam_search: False # use beam-search for failed actions, set True during evaluation
|
||||||
|
|
||||||
|
training:
|
||||||
|
max_nb_steps_per_episode: 50 # terminate after this many steps
|
||||||
|
|
||||||
|
fraction_assist:
|
||||||
|
fraction_assist_anneal_episodes: 50000
|
||||||
|
fraction_assist_anneal_from: 1.0
|
||||||
|
fraction_assist_anneal_to: 0.01
|
||||||
|
|
||||||
|
fraction_random:
|
||||||
|
fraction_random_anneal_episodes: 0
|
||||||
|
fraction_random_anneal_from: 0.0
|
||||||
|
fraction_random_anneal_to: 0.0
|
||||||
|
|
||||||
|
replay:
|
||||||
|
replay_memory_capacity: 500000
|
||||||
|
update_per_k_game_steps: 5
|
||||||
|
replay_batch_size: 64
|
||||||
|
replay_sample_history_length: 4
|
||||||
|
replay_sample_update_from: 2
|
||||||
|
|
||||||
|
vision_dagger:
|
||||||
|
model_type: "resnet" # 'resnet' (whole image features) or 'maskrcnn_whole' (whole image MaskRCNN feats) or 'maskrcnn' (top k MaskRCNN detection feats) or 'no_vision' (zero vision input)
|
||||||
|
resnet_fc_dim: 64
|
||||||
|
maskrcnn_top_k_boxes: 10 # top k box features
|
||||||
|
use_exploration_frame_feats: False # append feats from initial exploration (memory intensive!)
|
||||||
|
sequence_aggregation_method: "average" # 'sum' or 'average' or 'rnn'
|
||||||
+84
@@ -0,0 +1,84 @@
|
|||||||
|
# Vendored from SkillRL (Apache-2.0 License)
|
||||||
|
# Original: agent_system/environments/base.py
|
||||||
|
# Trimmed to only include what ALFWorld needs.
|
||||||
|
|
||||||
|
from typing import List, Tuple, Dict, Any
|
||||||
|
import numpy as np
|
||||||
|
from collections import defaultdict
|
||||||
|
|
||||||
|
|
||||||
|
def to_numpy(data):
|
||||||
|
"""Convert data to numpy array."""
|
||||||
|
# Lazy-check for torch.Tensor to avoid hard dependency on torch
|
||||||
|
_torch_tensor = None
|
||||||
|
try:
|
||||||
|
import torch
|
||||||
|
_torch_tensor = torch.Tensor
|
||||||
|
except ImportError:
|
||||||
|
pass
|
||||||
|
|
||||||
|
if _torch_tensor is not None and isinstance(data, _torch_tensor):
|
||||||
|
data = data.detach().cpu().numpy()
|
||||||
|
elif isinstance(data, np.ndarray):
|
||||||
|
pass
|
||||||
|
elif isinstance(data, (int, float, bool, Tuple, List)):
|
||||||
|
data = np.array(data)
|
||||||
|
else:
|
||||||
|
raise ValueError(f"Unsupported type: {type(data)})")
|
||||||
|
return data
|
||||||
|
|
||||||
|
|
||||||
|
class EnvironmentManagerBase:
|
||||||
|
"""Base class for vectorized environment managers.
|
||||||
|
|
||||||
|
Manages a set of parallel environments, handles action projection,
|
||||||
|
observation post-processing, and history tracking.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self, envs, projection_f, config):
|
||||||
|
self.envs = envs
|
||||||
|
self.projection_f = projection_f
|
||||||
|
self.config = config
|
||||||
|
|
||||||
|
def reset(self, kwargs) -> Dict[str, Any]:
|
||||||
|
obs, infos = self.envs.reset()
|
||||||
|
return {'text': None, 'image': obs, 'anchor': None}, infos
|
||||||
|
|
||||||
|
def step(self, text_actions: List[str]):
|
||||||
|
actions, valids = self.projection_f(text_actions)
|
||||||
|
next_obs, rewards, dones, infos = self.envs.step(actions)
|
||||||
|
|
||||||
|
next_observations = {
|
||||||
|
'text': None,
|
||||||
|
'image': next_obs,
|
||||||
|
'anchor': None,
|
||||||
|
}
|
||||||
|
for i, info in enumerate(infos):
|
||||||
|
info['is_action_valid'] = to_numpy(valids[i])
|
||||||
|
|
||||||
|
rewards = to_numpy(rewards)
|
||||||
|
dones = to_numpy(dones)
|
||||||
|
return next_observations, rewards, dones, infos
|
||||||
|
|
||||||
|
def close(self) -> None:
|
||||||
|
self.envs.close()
|
||||||
|
|
||||||
|
def success_evaluator(self, *args, **kwargs) -> Dict[str, np.ndarray]:
|
||||||
|
total_infos = kwargs['total_infos']
|
||||||
|
total_batch_list = kwargs['total_batch_list']
|
||||||
|
batch_size = len(total_batch_list)
|
||||||
|
|
||||||
|
success = defaultdict(list)
|
||||||
|
for bs in range(batch_size):
|
||||||
|
self._process_batch(bs, total_batch_list, total_infos, success)
|
||||||
|
assert len(success['success_rate']) == batch_size
|
||||||
|
return {key: np.array(value) for key, value in success.items()}
|
||||||
|
|
||||||
|
def _process_batch(self, batch_idx, total_batch_list, total_infos, success):
|
||||||
|
for i in reversed(range(len(total_batch_list[batch_idx]))):
|
||||||
|
batch_item = total_batch_list[batch_idx][i]
|
||||||
|
if batch_item['active_masks']:
|
||||||
|
info = total_infos[batch_idx][i]
|
||||||
|
won_value = float(info['won'])
|
||||||
|
success['success_rate'].append(won_value)
|
||||||
|
return
|
||||||
+139
@@ -0,0 +1,139 @@
|
|||||||
|
# Vendored from SkillRL (Apache-2.0 License)
|
||||||
|
# Original: agent_system/environments/env_manager.py
|
||||||
|
# Trimmed to only include AlfWorldEnvironmentManager and its helpers.
|
||||||
|
|
||||||
|
from typing import List, Dict, Any
|
||||||
|
from collections import defaultdict
|
||||||
|
import numpy as np
|
||||||
|
|
||||||
|
from skillopt.envs.alfworld.vendor.env_base import EnvironmentManagerBase, to_numpy
|
||||||
|
from skillopt.envs.alfworld.vendor.alfworld_prompts import (
|
||||||
|
ALFWORLD_TEMPLATE,
|
||||||
|
ALFWORLD_TEMPLATE_NO_HIS,
|
||||||
|
ALFWORLD_TEMPLATE_WITH_MEMORY,
|
||||||
|
)
|
||||||
|
from skillopt.envs.alfworld.vendor.memory import SimpleMemory
|
||||||
|
|
||||||
|
|
||||||
|
def parse_gamefile(infos):
|
||||||
|
gamefile = []
|
||||||
|
for info in infos:
|
||||||
|
if 'extra.gamefile' in info:
|
||||||
|
gamefile.append(info['extra.gamefile'])
|
||||||
|
else:
|
||||||
|
gamefile.append(None)
|
||||||
|
return gamefile
|
||||||
|
|
||||||
|
|
||||||
|
def set_gamefile(infos, gamefile):
|
||||||
|
for i in range(len(infos)):
|
||||||
|
if 'extra.gamefile' in infos[i]:
|
||||||
|
infos[i]['extra.gamefile'] = gamefile[i]
|
||||||
|
else:
|
||||||
|
infos[i]['extra.gamefile'] = None
|
||||||
|
return infos
|
||||||
|
|
||||||
|
|
||||||
|
class AlfWorldEnvironmentManager(EnvironmentManagerBase):
|
||||||
|
"""Manages parallel ALFWorld environments with observation templating."""
|
||||||
|
|
||||||
|
def __init__(self, envs, projection_f, config):
|
||||||
|
self.memory = SimpleMemory()
|
||||||
|
self.retrieval_memory = None
|
||||||
|
super().__init__(envs, projection_f, config)
|
||||||
|
|
||||||
|
def reset(self, kwargs):
|
||||||
|
text_obs, image_obs, infos = self.envs.reset()
|
||||||
|
self.gamefile = parse_gamefile(infos)
|
||||||
|
self.memory.reset(batch_size=len(text_obs))
|
||||||
|
self.tasks = []
|
||||||
|
self.pre_text_obs = text_obs
|
||||||
|
self.extract_task(text_obs)
|
||||||
|
|
||||||
|
full_text_obs = self.build_text_obs(text_obs, self.envs.get_admissible_commands, init=True)
|
||||||
|
return {'text': full_text_obs, 'image': image_obs, 'anchor': text_obs}, infos
|
||||||
|
|
||||||
|
def step(self, text_actions: List[str]):
|
||||||
|
actions, valids = self.projection_f(text_actions, self.envs.get_admissible_commands)
|
||||||
|
text_obs, image_obs, rewards, dones, infos = self.envs.step(actions)
|
||||||
|
self.memory.store({'text_obs': self.pre_text_obs, 'action': actions})
|
||||||
|
self.pre_text_obs = text_obs
|
||||||
|
|
||||||
|
full_text_obs = self.build_text_obs(text_obs, self.envs.get_admissible_commands)
|
||||||
|
if infos[0].get("extra.gamefile") is None:
|
||||||
|
infos = set_gamefile(infos, self.gamefile)
|
||||||
|
|
||||||
|
for i, info in enumerate(infos):
|
||||||
|
info['is_action_valid'] = to_numpy(valids[i])
|
||||||
|
|
||||||
|
next_observations = {'text': full_text_obs, 'image': image_obs, 'anchor': text_obs}
|
||||||
|
rewards = to_numpy(rewards)
|
||||||
|
dones = to_numpy(dones)
|
||||||
|
return next_observations, rewards, dones, infos
|
||||||
|
|
||||||
|
def extract_task(self, text_obs: List[str]):
|
||||||
|
for obs in text_obs:
|
||||||
|
task_start = obs.find('Your task is to: ')
|
||||||
|
if task_start != -1:
|
||||||
|
self.tasks.append(obs[task_start + len('Your task is to: '):].strip())
|
||||||
|
else:
|
||||||
|
raise ValueError("Task description not found in text observation.")
|
||||||
|
|
||||||
|
def build_text_obs(self, text_obs: List[str], admissible_actions: List[List[str]], init: bool = False) -> List[str]:
|
||||||
|
postprocess_text_obs = []
|
||||||
|
if not init and self.config.env.history_length > 0:
|
||||||
|
memory_contexts, valid_lens = self.memory.fetch(
|
||||||
|
self.config.env.history_length,
|
||||||
|
obs_key="text_obs",
|
||||||
|
action_key="action",
|
||||||
|
)
|
||||||
|
|
||||||
|
for i in range(len(text_obs)):
|
||||||
|
reformatted_admissible_actions = "\n ".join(
|
||||||
|
f"'{s}'" for s in admissible_actions[i] if s != 'help'
|
||||||
|
)
|
||||||
|
|
||||||
|
if init or self.config.env.history_length <= 0:
|
||||||
|
obs = ALFWORLD_TEMPLATE_NO_HIS.format(
|
||||||
|
current_observation=text_obs[i],
|
||||||
|
admissible_actions=reformatted_admissible_actions,
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
obs = ALFWORLD_TEMPLATE.format(
|
||||||
|
task_description=self.tasks[i],
|
||||||
|
step_count=len(self.memory[i]),
|
||||||
|
history_length=valid_lens[i],
|
||||||
|
action_history=memory_contexts[i],
|
||||||
|
current_step=len(self.memory[i]) + 1,
|
||||||
|
current_observation=text_obs[i],
|
||||||
|
admissible_actions=reformatted_admissible_actions,
|
||||||
|
)
|
||||||
|
postprocess_text_obs.append(obs)
|
||||||
|
return postprocess_text_obs
|
||||||
|
|
||||||
|
def _process_batch(self, batch_idx, total_batch_list, total_infos, success):
|
||||||
|
for i in reversed(range(len(total_batch_list[batch_idx]))):
|
||||||
|
batch_item = total_batch_list[batch_idx][i]
|
||||||
|
if batch_item['active_masks']:
|
||||||
|
info = total_infos[batch_idx][i]
|
||||||
|
won_value = float(info['won'])
|
||||||
|
success['success_rate'].append(won_value)
|
||||||
|
|
||||||
|
gamefile = info.get("extra.gamefile")
|
||||||
|
if gamefile:
|
||||||
|
self._process_gamefile(gamefile, won_value, success)
|
||||||
|
return
|
||||||
|
|
||||||
|
def _process_gamefile(self, gamefile, won_value, success):
|
||||||
|
tasks = [
|
||||||
|
"pick_and_place",
|
||||||
|
"pick_two_obj_and_place",
|
||||||
|
"look_at_obj_in_light",
|
||||||
|
"pick_heat_then_place_in_recep",
|
||||||
|
"pick_cool_then_place_in_recep",
|
||||||
|
"pick_clean_then_place_in_recep",
|
||||||
|
]
|
||||||
|
for task in tasks:
|
||||||
|
if task in gamefile:
|
||||||
|
success[f"{task}_success_rate"].append(won_value)
|
||||||
|
break
|
||||||
+87
@@ -0,0 +1,87 @@
|
|||||||
|
# Vendored from SkillRL (Apache-2.0 License)
|
||||||
|
# Original: agent_system/memory/base.py + agent_system/memory/memory.py
|
||||||
|
# Merged into a single file for simplicity.
|
||||||
|
|
||||||
|
from abc import ABC, abstractmethod
|
||||||
|
from typing import List, Dict, Any, Tuple
|
||||||
|
|
||||||
|
|
||||||
|
class BaseMemory(ABC):
|
||||||
|
"""Base class for memory management."""
|
||||||
|
|
||||||
|
@abstractmethod
|
||||||
|
def __len__(self):
|
||||||
|
pass
|
||||||
|
|
||||||
|
@abstractmethod
|
||||||
|
def __getitem__(self, idx: int):
|
||||||
|
pass
|
||||||
|
|
||||||
|
@abstractmethod
|
||||||
|
def reset(self, batch_size: int):
|
||||||
|
pass
|
||||||
|
|
||||||
|
@abstractmethod
|
||||||
|
def store(self, record: Dict[str, List[Any]]):
|
||||||
|
pass
|
||||||
|
|
||||||
|
@abstractmethod
|
||||||
|
def fetch(self, step: int):
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
class SimpleMemory(BaseMemory):
|
||||||
|
"""Per-environment history buffer for storing observations and actions."""
|
||||||
|
|
||||||
|
def __init__(self):
|
||||||
|
self._data = None
|
||||||
|
self.keys = None
|
||||||
|
self.batch_size = 0
|
||||||
|
|
||||||
|
def __len__(self):
|
||||||
|
return len(self._data)
|
||||||
|
|
||||||
|
def __getitem__(self, idx):
|
||||||
|
return self._data[idx]
|
||||||
|
|
||||||
|
def reset(self, batch_size: int):
|
||||||
|
if self._data is not None:
|
||||||
|
self._data.clear()
|
||||||
|
self._data = [[] for _ in range(batch_size)]
|
||||||
|
self.batch_size = batch_size
|
||||||
|
self.keys = None
|
||||||
|
|
||||||
|
def store(self, record: Dict[str, List[Any]]):
|
||||||
|
if self.keys is None:
|
||||||
|
self.keys = list(record.keys())
|
||||||
|
assert self.keys == list(record.keys())
|
||||||
|
|
||||||
|
for env_idx in range(self.batch_size):
|
||||||
|
self._data[env_idx].append({k: record[k][env_idx] for k in self.keys})
|
||||||
|
|
||||||
|
def fetch(
|
||||||
|
self,
|
||||||
|
history_length: int,
|
||||||
|
obs_key: str = "text_obs",
|
||||||
|
action_key: str = "action",
|
||||||
|
) -> Tuple[List[str], List[int]]:
|
||||||
|
memory_contexts, valid_lengths = [], []
|
||||||
|
|
||||||
|
for env_idx in range(self.batch_size):
|
||||||
|
recent = self._data[env_idx][-history_length:]
|
||||||
|
valid_len = len(recent)
|
||||||
|
start_idx = len(self._data[env_idx]) - valid_len
|
||||||
|
|
||||||
|
lines = []
|
||||||
|
for j, rec in enumerate(recent):
|
||||||
|
step_num = start_idx + j + 1
|
||||||
|
act = rec[action_key]
|
||||||
|
obs = rec[obs_key]
|
||||||
|
lines.append(
|
||||||
|
f"[Observation {step_num}: '{obs}', Action {step_num}: '{act}']"
|
||||||
|
)
|
||||||
|
|
||||||
|
memory_contexts.append("\n".join(lines))
|
||||||
|
valid_lengths.append(valid_len)
|
||||||
|
|
||||||
|
return memory_contexts, valid_lengths
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
"""BabyVision environment package for ReflACT."""
|
||||||
@@ -0,0 +1,267 @@
|
|||||||
|
"""BabyVision environment adapter for ReflACT."""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
|
||||||
|
from skillopt.gradient.deep_probe import generate_deep_probe_instruction
|
||||||
|
from skillopt.datasets.base import BatchSpec
|
||||||
|
from skillopt.gradient.reflect import run_minibatch_reflect
|
||||||
|
from skillopt.envs.base import EnvAdapter
|
||||||
|
from skillopt.envs.babyvision.dataloader import BabyVisionDataLoader
|
||||||
|
from skillopt.envs.babyvision.rollout import run_batch
|
||||||
|
from skillopt.model import get_student_backend
|
||||||
|
|
||||||
|
|
||||||
|
class BabyVisionAdapter(EnvAdapter):
|
||||||
|
"""BabyVision adapter."""
|
||||||
|
|
||||||
|
def build_reference_text(self, item: dict) -> str:
|
||||||
|
cot = str(item.get("cot") or "").strip()
|
||||||
|
if not cot:
|
||||||
|
return ""
|
||||||
|
return f"## Reference CoT\n{cot}"
|
||||||
|
|
||||||
|
def get_reference_metadata(self, item: dict) -> dict:
|
||||||
|
cot = str(item.get("cot") or "").strip()
|
||||||
|
if not cot:
|
||||||
|
return {"fields": [], "preview": ""}
|
||||||
|
return {
|
||||||
|
"fields": ["cot"],
|
||||||
|
"preview": cot[:400],
|
||||||
|
}
|
||||||
|
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
split_dir: str = "",
|
||||||
|
data_path: str = "",
|
||||||
|
split_mode: str = "ratio",
|
||||||
|
split_ratio: str = "2:1:7",
|
||||||
|
split_seed: int = 42,
|
||||||
|
split_output_dir: str = "",
|
||||||
|
max_turns: int = 1,
|
||||||
|
workers: int = 32,
|
||||||
|
analyst_workers: int = 16,
|
||||||
|
failure_only: bool = False,
|
||||||
|
minibatch_size: int = 8,
|
||||||
|
edit_budget: int = 4,
|
||||||
|
seed: int = 42,
|
||||||
|
limit: int = 0,
|
||||||
|
image_detail: str = "auto",
|
||||||
|
judge_model: str = "gpt-5.4",
|
||||||
|
judge_max_completion_tokens: int = 256,
|
||||||
|
judge_retries: int = 5,
|
||||||
|
use_deep_reflect: bool = False,
|
||||||
|
deep_reflect_failures: int = 4,
|
||||||
|
deep_reflect_successes: int = 2,
|
||||||
|
) -> None:
|
||||||
|
self.max_turns = max_turns
|
||||||
|
self.workers = workers
|
||||||
|
self.analyst_workers = analyst_workers
|
||||||
|
self.failure_only = failure_only
|
||||||
|
self.minibatch_size = minibatch_size
|
||||||
|
self.edit_budget = edit_budget
|
||||||
|
self.image_detail = image_detail
|
||||||
|
self.judge_model = judge_model
|
||||||
|
self.judge_max_completion_tokens = judge_max_completion_tokens
|
||||||
|
self.judge_retries = judge_retries
|
||||||
|
self.use_deep_reflect = use_deep_reflect
|
||||||
|
self.deep_reflect_failures = deep_reflect_failures
|
||||||
|
self.deep_reflect_successes = deep_reflect_successes
|
||||||
|
self.dataloader = BabyVisionDataLoader(
|
||||||
|
split_dir=split_dir,
|
||||||
|
data_path=data_path,
|
||||||
|
split_mode=split_mode,
|
||||||
|
split_ratio=split_ratio,
|
||||||
|
split_seed=split_seed,
|
||||||
|
split_output_dir=split_output_dir,
|
||||||
|
seed=seed,
|
||||||
|
limit=limit,
|
||||||
|
)
|
||||||
|
|
||||||
|
def setup(self, cfg: dict) -> None:
|
||||||
|
super().setup(cfg)
|
||||||
|
self.dataloader.setup(cfg)
|
||||||
|
|
||||||
|
def get_dataloader(self):
|
||||||
|
return self.dataloader
|
||||||
|
|
||||||
|
def build_env_from_batch(self, batch: BatchSpec, **kwargs):
|
||||||
|
return list(batch.payload or [])
|
||||||
|
|
||||||
|
def build_train_env(self, batch_size: int, seed: int, **kwargs):
|
||||||
|
batch = self.dataloader.build_train_batch(batch_size=batch_size, seed=seed, **kwargs)
|
||||||
|
return self.build_env_from_batch(batch, **kwargs)
|
||||||
|
|
||||||
|
def build_eval_env(self, env_num: int, split: str, seed: int, **kwargs):
|
||||||
|
batch = self.dataloader.build_eval_batch(env_num=env_num, split=split, seed=seed, **kwargs)
|
||||||
|
return self.build_env_from_batch(batch, **kwargs)
|
||||||
|
|
||||||
|
def rollout(
|
||||||
|
self,
|
||||||
|
env_manager,
|
||||||
|
skill_content: str,
|
||||||
|
out_dir: str,
|
||||||
|
**kwargs,
|
||||||
|
) -> list[dict]:
|
||||||
|
items: list[dict] = env_manager
|
||||||
|
return run_batch(
|
||||||
|
items=items,
|
||||||
|
out_root=out_dir,
|
||||||
|
skill_content=skill_content,
|
||||||
|
max_turns=self.max_turns,
|
||||||
|
workers=self.workers,
|
||||||
|
image_detail=self.image_detail,
|
||||||
|
judge_model=self.judge_model,
|
||||||
|
judge_max_completion_tokens=self.judge_max_completion_tokens,
|
||||||
|
judge_retries=self.judge_retries,
|
||||||
|
diagnostic_mode=kwargs.get("diagnostic_mode", False),
|
||||||
|
diagnostic_instruction=kwargs.get("diagnostic_instruction", ""),
|
||||||
|
diagnostic_trace_context_by_id=kwargs.get("diagnostic_trace_context_by_id"),
|
||||||
|
)
|
||||||
|
|
||||||
|
def reflect(
|
||||||
|
self,
|
||||||
|
results: list[dict],
|
||||||
|
skill_content: str,
|
||||||
|
out_dir: str,
|
||||||
|
**kwargs,
|
||||||
|
) -> list[dict | None]:
|
||||||
|
prediction_dir = kwargs.get("prediction_dir", os.path.join(out_dir, "predictions"))
|
||||||
|
patches_dir = kwargs.get("patches_dir", os.path.join(out_dir, "patches"))
|
||||||
|
random_seed = kwargs.get("random_seed")
|
||||||
|
step_buffer_context = kwargs.get("step_buffer_context", "")
|
||||||
|
meta_skill_context = kwargs.get("meta_skill_context", "")
|
||||||
|
|
||||||
|
return run_minibatch_reflect(
|
||||||
|
results=results,
|
||||||
|
skill_content=skill_content,
|
||||||
|
prediction_dir=prediction_dir,
|
||||||
|
patches_dir=patches_dir,
|
||||||
|
workers=self.analyst_workers,
|
||||||
|
failure_only=self.failure_only,
|
||||||
|
minibatch_size=self.minibatch_size,
|
||||||
|
edit_budget=self.edit_budget,
|
||||||
|
random_seed=random_seed,
|
||||||
|
error_system=self.get_error_minibatch_prompt(),
|
||||||
|
success_system=self.get_success_minibatch_prompt(),
|
||||||
|
step_buffer_context=step_buffer_context,
|
||||||
|
meta_skill_context=meta_skill_context,
|
||||||
|
update_mode=getattr(self, "_cfg", {}).get("skill_update_mode", "patch"),
|
||||||
|
)
|
||||||
|
|
||||||
|
def deep_reflect(
|
||||||
|
self,
|
||||||
|
results: list[dict],
|
||||||
|
skill_content: str,
|
||||||
|
out_dir: str,
|
||||||
|
**kwargs,
|
||||||
|
) -> list[dict | None]:
|
||||||
|
if not self.use_deep_reflect:
|
||||||
|
return []
|
||||||
|
|
||||||
|
env_manager = kwargs.get("env_manager")
|
||||||
|
prediction_dir = kwargs.get("prediction_dir", os.path.join(out_dir, "predictions"))
|
||||||
|
random_seed = kwargs.get("random_seed")
|
||||||
|
step_buffer_context = kwargs.get("step_buffer_context", "")
|
||||||
|
meta_skill_context = kwargs.get("meta_skill_context", "")
|
||||||
|
codex_backend = get_student_backend() == "codex_exec"
|
||||||
|
selected_items = self.select_representative_items(
|
||||||
|
results,
|
||||||
|
env_manager if isinstance(env_manager, list) else None,
|
||||||
|
n_failures=self.deep_reflect_failures,
|
||||||
|
n_successes=self.deep_reflect_successes,
|
||||||
|
seed=random_seed,
|
||||||
|
)
|
||||||
|
if not selected_items:
|
||||||
|
return []
|
||||||
|
selected_ids = {str(item["id"]) for item in selected_items}
|
||||||
|
selected_results = [row for row in results if str(row.get("id")) in selected_ids]
|
||||||
|
selected_examples = self.attach_reference_context(selected_results, selected_items)
|
||||||
|
if codex_backend:
|
||||||
|
selected_examples = self.attach_codex_probe_context(selected_examples, prediction_dir)
|
||||||
|
selected_metadata = []
|
||||||
|
cot_count = 0
|
||||||
|
for item in selected_items:
|
||||||
|
meta = self.get_reference_metadata(item)
|
||||||
|
if meta["fields"]:
|
||||||
|
cot_count += 1
|
||||||
|
selected_metadata.append({
|
||||||
|
"id": str(item["id"]),
|
||||||
|
"task_type": str(item.get("subtype") or item.get("task_type") or "babyvision"),
|
||||||
|
"reference_fields": meta["fields"],
|
||||||
|
"reference_preview": meta["preview"],
|
||||||
|
})
|
||||||
|
|
||||||
|
deep_dir = os.path.join(out_dir, "deep_reflect")
|
||||||
|
rollout_dir = os.path.join(deep_dir, "rollout")
|
||||||
|
patches_dir = os.path.join(deep_dir, "patches")
|
||||||
|
os.makedirs(deep_dir, exist_ok=True)
|
||||||
|
print(
|
||||||
|
f" [2b/6 DEEP REFLECT setup] selected={len(selected_items)} "
|
||||||
|
f"reference_fields=cot({cot_count}/{len(selected_items)})"
|
||||||
|
)
|
||||||
|
probe = generate_deep_probe_instruction(
|
||||||
|
skill_content=skill_content,
|
||||||
|
items=selected_examples,
|
||||||
|
prediction_dir=prediction_dir,
|
||||||
|
system_prompt=self.get_codex_deep_probe_prompt() if codex_backend else self.get_deep_probe_prompt(),
|
||||||
|
step_buffer_context=step_buffer_context,
|
||||||
|
meta_skill_context=meta_skill_context,
|
||||||
|
)
|
||||||
|
if not probe:
|
||||||
|
return []
|
||||||
|
diagnostic_trace_context_by_id = None
|
||||||
|
if codex_backend:
|
||||||
|
selected_items, diagnostic_trace_context_by_id, probe = self.resolve_codex_probe_target(
|
||||||
|
selected_items=selected_items,
|
||||||
|
selected_examples=selected_examples,
|
||||||
|
prediction_dir=prediction_dir,
|
||||||
|
probe=probe,
|
||||||
|
)
|
||||||
|
probe_record = {
|
||||||
|
**probe,
|
||||||
|
"reference_summary": {
|
||||||
|
"selected_count": len(selected_items),
|
||||||
|
"field_counts": {
|
||||||
|
"cot": cot_count,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
"selected_examples": selected_metadata,
|
||||||
|
}
|
||||||
|
with open(os.path.join(deep_dir, "probe.json"), "w", encoding="utf-8") as f:
|
||||||
|
json.dump(probe_record, f, ensure_ascii=False, indent=2)
|
||||||
|
deep_results = run_batch(
|
||||||
|
items=selected_items,
|
||||||
|
out_root=rollout_dir,
|
||||||
|
skill_content=skill_content,
|
||||||
|
max_turns=self.max_turns,
|
||||||
|
workers=min(self.workers, max(len(selected_items), 1)),
|
||||||
|
image_detail=self.image_detail,
|
||||||
|
judge_model=self.judge_model,
|
||||||
|
judge_max_completion_tokens=self.judge_max_completion_tokens,
|
||||||
|
judge_retries=self.judge_retries,
|
||||||
|
diagnostic_mode=True,
|
||||||
|
diagnostic_instruction=probe["probe_instruction"],
|
||||||
|
diagnostic_trace_context_by_id=diagnostic_trace_context_by_id,
|
||||||
|
)
|
||||||
|
deep_results = self.attach_reference_context(deep_results, selected_items)
|
||||||
|
return run_minibatch_reflect(
|
||||||
|
results=deep_results,
|
||||||
|
skill_content=skill_content,
|
||||||
|
prediction_dir=os.path.join(rollout_dir, "predictions"),
|
||||||
|
patches_dir=patches_dir,
|
||||||
|
workers=self.analyst_workers,
|
||||||
|
failure_only=self.failure_only,
|
||||||
|
minibatch_size=self.minibatch_size,
|
||||||
|
edit_budget=self.edit_budget,
|
||||||
|
random_seed=random_seed,
|
||||||
|
error_system=self.get_error_minibatch_prompt(),
|
||||||
|
success_system=self.get_success_minibatch_prompt(),
|
||||||
|
step_buffer_context=step_buffer_context,
|
||||||
|
meta_skill_context=meta_skill_context,
|
||||||
|
update_mode=getattr(self, "_cfg", {}).get("skill_update_mode", "patch"),
|
||||||
|
)
|
||||||
|
|
||||||
|
def get_task_types(self) -> list[str]:
|
||||||
|
return self.dataloader.get_task_types()
|
||||||
@@ -0,0 +1,214 @@
|
|||||||
|
"""BabyVision task dataloader."""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from skillopt.datasets.base import SplitDataLoader
|
||||||
|
|
||||||
|
|
||||||
|
# ── Raw data loading utilities (for preprocessing / standalone eval) ─────
|
||||||
|
|
||||||
|
_CHOICE_LABELS = ["A", "B", "C", "D", "E", "F", "G"]
|
||||||
|
|
||||||
|
|
||||||
|
def _iter_jsonl(path: str) -> list[dict]:
|
||||||
|
items: list[dict] = []
|
||||||
|
with open(path, encoding="utf-8") as f:
|
||||||
|
for line in f:
|
||||||
|
line = line.strip()
|
||||||
|
if not line:
|
||||||
|
continue
|
||||||
|
items.append(json.loads(line))
|
||||||
|
return items
|
||||||
|
|
||||||
|
|
||||||
|
def _normalize_ans_type(raw: Any, options: list[dict], choice_answer: Any) -> str:
|
||||||
|
text = str(raw or "").strip().lower()
|
||||||
|
if text in {"choice", "multiple_choice", "mcq", "option"}:
|
||||||
|
return "choice"
|
||||||
|
if text in {"blank", "open", "open_ended", "fill_blank", "short_answer"}:
|
||||||
|
return "blank"
|
||||||
|
if options or choice_answer not in (None, "", []):
|
||||||
|
return "choice"
|
||||||
|
return "blank"
|
||||||
|
|
||||||
|
|
||||||
|
def _coerce_options(raw: Any) -> list[dict]:
|
||||||
|
options: list[dict] = []
|
||||||
|
if isinstance(raw, list):
|
||||||
|
for idx, item in enumerate(raw):
|
||||||
|
if isinstance(item, dict):
|
||||||
|
text = str(item.get("text") or item.get("content") or item.get("option") or "").strip()
|
||||||
|
label = str(item.get("label") or _CHOICE_LABELS[idx]).strip()
|
||||||
|
else:
|
||||||
|
text = str(item).strip()
|
||||||
|
label = _CHOICE_LABELS[idx]
|
||||||
|
if text:
|
||||||
|
options.append({"label": label, "text": text})
|
||||||
|
elif isinstance(raw, dict):
|
||||||
|
for idx, (key, value) in enumerate(raw.items()):
|
||||||
|
text = str(value).strip()
|
||||||
|
if text:
|
||||||
|
options.append({"label": str(key).strip() or _CHOICE_LABELS[idx], "text": text})
|
||||||
|
return options
|
||||||
|
|
||||||
|
|
||||||
|
def _normalize_choice_answer(choice_answer: Any, options: list[dict]) -> dict[str, str]:
|
||||||
|
if not options:
|
||||||
|
return {"label": "", "text": ""}
|
||||||
|
|
||||||
|
if isinstance(choice_answer, dict):
|
||||||
|
label = str(choice_answer.get("label") or "").strip().upper()
|
||||||
|
text = str(choice_answer.get("text") or "").strip()
|
||||||
|
for option in options:
|
||||||
|
if label and option["label"].strip().upper() == label:
|
||||||
|
return {"label": option["label"], "text": option["text"]}
|
||||||
|
if text and option["text"] == text:
|
||||||
|
return {"label": option["label"], "text": option["text"]}
|
||||||
|
|
||||||
|
if isinstance(choice_answer, int):
|
||||||
|
idx = choice_answer
|
||||||
|
if 0 <= idx < len(options):
|
||||||
|
return dict(options[idx])
|
||||||
|
if 1 <= idx <= len(options):
|
||||||
|
return dict(options[idx - 1])
|
||||||
|
|
||||||
|
text = str(choice_answer or "").strip()
|
||||||
|
label = text.upper().rstrip(".):")
|
||||||
|
for option in options:
|
||||||
|
if option["label"].strip().upper() == label:
|
||||||
|
return dict(option)
|
||||||
|
if option["text"] == text:
|
||||||
|
return dict(option)
|
||||||
|
|
||||||
|
return {"label": "", "text": ""}
|
||||||
|
|
||||||
|
|
||||||
|
def _coerce_blank_answers(raw: Any) -> list[str]:
|
||||||
|
if isinstance(raw, list):
|
||||||
|
return [str(item).strip() for item in raw if str(item).strip()]
|
||||||
|
if raw is None:
|
||||||
|
return []
|
||||||
|
text = str(raw).strip()
|
||||||
|
return [text] if text else []
|
||||||
|
|
||||||
|
|
||||||
|
def load_items(data_path: str) -> list[dict]:
|
||||||
|
"""Load and normalise BabyVision items from a directory or JSONL file."""
|
||||||
|
if not data_path:
|
||||||
|
raise ValueError("BabyVision requires data_path pointing to a local dataset directory or meta_data.jsonl.")
|
||||||
|
|
||||||
|
if os.path.isdir(data_path):
|
||||||
|
meta_path = os.path.join(data_path, "meta_data.jsonl")
|
||||||
|
image_root = os.path.join(data_path, "images")
|
||||||
|
else:
|
||||||
|
meta_path = data_path
|
||||||
|
image_root = os.path.join(os.path.dirname(data_path), "images")
|
||||||
|
|
||||||
|
if not os.path.exists(meta_path):
|
||||||
|
raise ValueError(
|
||||||
|
"BabyVision expected a meta_data.jsonl file. "
|
||||||
|
f"Could not find: {meta_path}"
|
||||||
|
)
|
||||||
|
|
||||||
|
raw_items = _iter_jsonl(meta_path)
|
||||||
|
items: list[dict] = []
|
||||||
|
for idx, raw in enumerate(raw_items):
|
||||||
|
options = _coerce_options(raw.get("options") or raw.get("choices") or raw.get("choiceOptions"))
|
||||||
|
ans_type = _normalize_ans_type(raw.get("ansType"), options, raw.get("choiceAns"))
|
||||||
|
correct_choice = _normalize_choice_answer(raw.get("choiceAns"), options)
|
||||||
|
blank_answers = _coerce_blank_answers(raw.get("blankAns"))
|
||||||
|
|
||||||
|
image_name = str(
|
||||||
|
raw.get("image")
|
||||||
|
or raw.get("image_path")
|
||||||
|
or raw.get("image_file")
|
||||||
|
or raw.get("img")
|
||||||
|
or ""
|
||||||
|
).strip()
|
||||||
|
if not image_name:
|
||||||
|
continue
|
||||||
|
image_path = image_name if os.path.isabs(image_name) else os.path.join(image_root, image_name)
|
||||||
|
if not os.path.exists(image_path):
|
||||||
|
alt = os.path.join(os.path.dirname(meta_path), image_name)
|
||||||
|
if os.path.exists(alt):
|
||||||
|
image_path = alt
|
||||||
|
else:
|
||||||
|
continue
|
||||||
|
|
||||||
|
task_id = str(raw.get("taskId") or raw.get("id") or idx + 1)
|
||||||
|
task_type = str(raw.get("type") or raw.get("taskType") or "unknown").strip() or "unknown"
|
||||||
|
subtype = str(raw.get("subtype") or raw.get("subType") or task_type).strip() or task_type
|
||||||
|
question = str(raw.get("question") or raw.get("query") or "").strip()
|
||||||
|
if not question:
|
||||||
|
continue
|
||||||
|
|
||||||
|
if ans_type == "choice" and not correct_choice["label"]:
|
||||||
|
continue
|
||||||
|
if ans_type != "choice" and not blank_answers:
|
||||||
|
continue
|
||||||
|
|
||||||
|
items.append({
|
||||||
|
"id": task_id,
|
||||||
|
"task_type": task_type,
|
||||||
|
"subtype": subtype,
|
||||||
|
"question": question,
|
||||||
|
"image_path": os.path.abspath(image_path),
|
||||||
|
"ans_type": ans_type,
|
||||||
|
"choices": options,
|
||||||
|
"correct_choice": correct_choice,
|
||||||
|
"blank_answers": blank_answers,
|
||||||
|
"cot": str(raw.get("coT") or raw.get("cot") or "").strip(),
|
||||||
|
"source_path": os.path.abspath(meta_path),
|
||||||
|
})
|
||||||
|
|
||||||
|
if not items:
|
||||||
|
raise ValueError(f"No valid BabyVision items loaded from {data_path}")
|
||||||
|
return items
|
||||||
|
|
||||||
|
|
||||||
|
# ── Dataloader ───────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
class BabyVisionDataLoader(SplitDataLoader):
|
||||||
|
"""BabyVision dataloader."""
|
||||||
|
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
split_dir: str = "",
|
||||||
|
data_path: str = "",
|
||||||
|
split_mode: str = "ratio",
|
||||||
|
split_ratio: str = "2:1:7",
|
||||||
|
split_seed: int = 42,
|
||||||
|
split_output_dir: str = "",
|
||||||
|
seed: int = 42,
|
||||||
|
limit: int = 0,
|
||||||
|
**kwargs,
|
||||||
|
) -> None:
|
||||||
|
super().__init__(
|
||||||
|
split_dir=split_dir,
|
||||||
|
data_path=data_path,
|
||||||
|
split_mode=split_mode,
|
||||||
|
split_ratio=split_ratio,
|
||||||
|
split_seed=split_seed,
|
||||||
|
split_output_dir=split_output_dir,
|
||||||
|
seed=seed,
|
||||||
|
limit=limit,
|
||||||
|
)
|
||||||
|
self._task_types: list[str] = []
|
||||||
|
|
||||||
|
def load_raw_items(self, data_path: str) -> list[dict]:
|
||||||
|
return load_items(data_path)
|
||||||
|
|
||||||
|
def setup(self, cfg: dict) -> None:
|
||||||
|
super().setup(cfg)
|
||||||
|
all_items = self.train_items + self.val_items + self.test_items
|
||||||
|
task_types = {
|
||||||
|
item.get("subtype") or item.get("task_type") or "unknown"
|
||||||
|
for item in all_items
|
||||||
|
}
|
||||||
|
self._task_types = sorted(task_types)
|
||||||
|
|
||||||
|
def get_task_types(self) -> list[str]:
|
||||||
|
return list(self._task_types)
|
||||||
@@ -0,0 +1,160 @@
|
|||||||
|
"""BabyVision evaluation helpers using the official-style LLM judge."""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import re
|
||||||
|
import string
|
||||||
|
|
||||||
|
import regex
|
||||||
|
|
||||||
|
from skillopt.model import chat_with_deployment
|
||||||
|
from skillopt.prompts import load_prompt
|
||||||
|
|
||||||
|
_EVAL_MODE = "babyvision_judge_v2_official_style"
|
||||||
|
|
||||||
|
def normalize_text(text: str) -> str:
|
||||||
|
text = str(text).strip().lower()
|
||||||
|
text = "".join(ch for ch in text if ch not in string.punctuation)
|
||||||
|
return " ".join(text.split())
|
||||||
|
|
||||||
|
|
||||||
|
def extract_boxed_answer(text: str | None) -> str | None:
|
||||||
|
"""Extract the final answer using the official BabyVision rule."""
|
||||||
|
if text is None:
|
||||||
|
return None
|
||||||
|
|
||||||
|
pattern = r'\\boxed\{((?:[^{}]|{(?:[^{}]|{.*})*})*)\}'
|
||||||
|
matches = regex.findall(pattern, text)
|
||||||
|
if matches:
|
||||||
|
return matches[-1]
|
||||||
|
|
||||||
|
pattern_alt = r'<\|begin_of_box\|>(.*?)<\|end_of_box\|>'
|
||||||
|
matches_alt = regex.findall(pattern_alt, text)
|
||||||
|
if matches_alt:
|
||||||
|
return matches_alt[-1].strip()
|
||||||
|
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _token_f1(prediction: str, gold: str) -> float:
|
||||||
|
pred_tokens = normalize_text(prediction).split()
|
||||||
|
gold_tokens = normalize_text(gold).split()
|
||||||
|
if not pred_tokens and not gold_tokens:
|
||||||
|
return 1.0
|
||||||
|
if not pred_tokens or not gold_tokens:
|
||||||
|
return 0.0
|
||||||
|
pred_set = {}
|
||||||
|
gold_set = {}
|
||||||
|
for tok in pred_tokens:
|
||||||
|
pred_set[tok] = pred_set.get(tok, 0) + 1
|
||||||
|
for tok in gold_tokens:
|
||||||
|
gold_set[tok] = gold_set.get(tok, 0) + 1
|
||||||
|
common = 0
|
||||||
|
for tok, count in pred_set.items():
|
||||||
|
common += min(count, gold_set.get(tok, 0))
|
||||||
|
if common == 0:
|
||||||
|
return 0.0
|
||||||
|
precision = common / len(pred_tokens)
|
||||||
|
recall = common / len(gold_tokens)
|
||||||
|
return 2 * precision * recall / (precision + recall)
|
||||||
|
|
||||||
|
|
||||||
|
def _format_choices(choices: list[dict]) -> str:
|
||||||
|
return "\n".join(f"{choice['label']}. {choice['text']}" for choice in choices)
|
||||||
|
|
||||||
|
|
||||||
|
def _judge_answer(
|
||||||
|
*,
|
||||||
|
item: dict,
|
||||||
|
prediction_text: str,
|
||||||
|
extracted_answer: str,
|
||||||
|
judge_model: str,
|
||||||
|
max_completion_tokens: int,
|
||||||
|
retries: int,
|
||||||
|
) -> dict:
|
||||||
|
if item["ans_type"] == "choice":
|
||||||
|
ground_truth = str(item["correct_choice"]["label"])
|
||||||
|
else:
|
||||||
|
if len(item["blank_answers"]) == 1:
|
||||||
|
ground_truth = item["blank_answers"][0]
|
||||||
|
else:
|
||||||
|
ground_truth = " | ".join(item["blank_answers"])
|
||||||
|
|
||||||
|
question = str(item["question"])
|
||||||
|
if item["ans_type"] == "choice" and item.get("choices"):
|
||||||
|
question = f"{question}\nChoices:\n{_format_choices(item['choices'])}"
|
||||||
|
|
||||||
|
raw, _ = chat_with_deployment(
|
||||||
|
deployment=judge_model,
|
||||||
|
system="You are a careful and strict evaluator.",
|
||||||
|
user=load_prompt("judge", env="babyvision").format(
|
||||||
|
question=question,
|
||||||
|
groundtruth=ground_truth,
|
||||||
|
modeloutput=extracted_answer,
|
||||||
|
),
|
||||||
|
max_completion_tokens=max_completion_tokens,
|
||||||
|
retries=retries,
|
||||||
|
stage="babyvision_judge",
|
||||||
|
)
|
||||||
|
judge_response_clean = str(raw).strip().lower()
|
||||||
|
if "true" in judge_response_clean:
|
||||||
|
correct = True
|
||||||
|
elif "false" in judge_response_clean:
|
||||||
|
correct = False
|
||||||
|
else:
|
||||||
|
correct = False
|
||||||
|
return {
|
||||||
|
"raw": raw,
|
||||||
|
"correct": correct,
|
||||||
|
"reason": judge_response_clean,
|
||||||
|
"matched_gold": ground_truth if correct else "",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def evaluate_item(
|
||||||
|
*,
|
||||||
|
item: dict,
|
||||||
|
prediction_text: str,
|
||||||
|
judge_model: str,
|
||||||
|
max_completion_tokens: int = 256,
|
||||||
|
retries: int = 5,
|
||||||
|
) -> dict:
|
||||||
|
answer = extract_boxed_answer(prediction_text)
|
||||||
|
judge = _judge_answer(
|
||||||
|
item=item,
|
||||||
|
prediction_text=prediction_text,
|
||||||
|
extracted_answer=answer,
|
||||||
|
judge_model=judge_model,
|
||||||
|
max_completion_tokens=max_completion_tokens,
|
||||||
|
retries=retries,
|
||||||
|
)
|
||||||
|
hard = 1.0 if judge["correct"] else 0.0
|
||||||
|
|
||||||
|
result = {
|
||||||
|
"evaluation_mode": _EVAL_MODE,
|
||||||
|
"predicted_answer": answer,
|
||||||
|
"em": hard,
|
||||||
|
"f1": hard,
|
||||||
|
"sub_em": hard,
|
||||||
|
"judge_model": judge_model,
|
||||||
|
"judge_raw": judge["raw"],
|
||||||
|
"judge_reason": judge["reason"],
|
||||||
|
"matched_gold": judge["matched_gold"],
|
||||||
|
}
|
||||||
|
|
||||||
|
if item["ans_type"] == "choice":
|
||||||
|
result["predicted_label"] = str(answer or "").strip().upper().rstrip(".):")
|
||||||
|
result["predicted_text"] = ""
|
||||||
|
result["correct_label"] = str(item["correct_choice"].get("label") or "")
|
||||||
|
result["correct_text"] = str(item["correct_choice"].get("text") or "")
|
||||||
|
else:
|
||||||
|
result["gold_answers"] = list(item["blank_answers"])
|
||||||
|
best_f1 = 0.0
|
||||||
|
for gold in item["blank_answers"]:
|
||||||
|
best_f1 = max(best_f1, _token_f1(str(answer or ""), gold))
|
||||||
|
result["string_f1"] = best_f1
|
||||||
|
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
def evaluation_mode() -> str:
|
||||||
|
return _EVAL_MODE
|
||||||
@@ -0,0 +1,36 @@
|
|||||||
|
You are an expert failure-analysis agent for child-level visual reasoning tasks.
|
||||||
|
|
||||||
|
You will be given MULTIPLE failed BabyVision trajectories from a minibatch and the current skill document.
|
||||||
|
Each trajectory includes the text prompt, the model answer, and the evaluation result.
|
||||||
|
You do not have direct access to raw pixel content during reflection, so focus on general reasoning,
|
||||||
|
option-selection, and visual-question-answering behaviors that can be improved through prompting.
|
||||||
|
|
||||||
|
## Failure Type Categories
|
||||||
|
- **visual_detail_miss**: the agent likely overlooked a salient visual attribute, relation, count, or object state
|
||||||
|
- **option_mismatch**: the agent selected the wrong option despite relevant evidence likely being present
|
||||||
|
- **instruction_slip**: the agent ignored output format or answered too vaguely
|
||||||
|
- **answer_granularity**: the agent gave an answer that was too broad, too narrow, or mismatched the expected specificity
|
||||||
|
- **other**: none of the above
|
||||||
|
|
||||||
|
## Rules
|
||||||
|
1. Focus on patterns recurring across the minibatch.
|
||||||
|
2. Prefer reusable behaviors for inspecting images and grounding answers in visible evidence.
|
||||||
|
3. Do not memorize dataset-specific answers.
|
||||||
|
4. Only patch gaps not already covered by the current skill.
|
||||||
|
|
||||||
|
Respond ONLY with a valid JSON object:
|
||||||
|
{
|
||||||
|
"batch_size": <number>,
|
||||||
|
"failure_summary": [
|
||||||
|
{"failure_type": "<type>", "count": <int>, "description": "<one-line>"}
|
||||||
|
],
|
||||||
|
"patch": {
|
||||||
|
"reasoning": "<why these edits address the common failures>",
|
||||||
|
"edits": [
|
||||||
|
{"op": "append", "content": "<markdown>"},
|
||||||
|
{"op": "insert_after", "target": "<heading/text>", "content": "<markdown>"},
|
||||||
|
{"op": "replace", "target": "<old text>", "content": "<new text>"},
|
||||||
|
{"op": "delete", "target": "<exact text to remove>"}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,25 @@
|
|||||||
|
You are an expert success-pattern analyst for child-level visual reasoning tasks.
|
||||||
|
|
||||||
|
You will be given MULTIPLE successful BabyVision trajectories from a minibatch and the current skill document.
|
||||||
|
Identify generalizable behavior patterns that help the agent inspect the image carefully and answer at the right level of specificity.
|
||||||
|
|
||||||
|
## Rules
|
||||||
|
- Focus on broadly useful visual QA behaviors.
|
||||||
|
- Prefer patterns about systematic image inspection, comparing options, and concise grounded answers.
|
||||||
|
- Do not add dataset-specific facts.
|
||||||
|
- "edits" may be empty if the skill already captures the useful patterns.
|
||||||
|
|
||||||
|
Respond ONLY with a valid JSON object:
|
||||||
|
{
|
||||||
|
"batch_size": <number>,
|
||||||
|
"success_patterns": ["<pattern 1>", "<pattern 2>"],
|
||||||
|
"patch": {
|
||||||
|
"reasoning": "<why these patterns matter>",
|
||||||
|
"edits": [
|
||||||
|
{"op": "append", "content": "<markdown>"},
|
||||||
|
{"op": "insert_after", "target": "<heading/text>", "content": "<markdown>"},
|
||||||
|
{"op": "replace", "target": "<old text>", "content": "<new text>"},
|
||||||
|
{"op": "delete", "target": "<exact text to remove>"}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,25 @@
|
|||||||
|
You are an expert diagnostic-probe designer for BabyVision-style visual reasoning tasks.
|
||||||
|
|
||||||
|
You will be shown representative trajectories, the current student skill, and the student's original prompt context.
|
||||||
|
Design one SMALL diagnostic instruction that exposes the student's intermediate visual judgment without materially changing the original scaffold.
|
||||||
|
|
||||||
|
## Hard Constraints
|
||||||
|
1. Do NOT substantially change the original scaffold.
|
||||||
|
2. Do NOT prescribe a new step-by-step solving method.
|
||||||
|
3. You MAY ask for a short structured list of a few intermediate conclusions, candidate cues, or counted units, as long as it stays close to the original scaffold.
|
||||||
|
4. Do NOT ask for exhaustive listing of all cells, all objects, or a full chain-of-thought.
|
||||||
|
5. Ask only for a short readout that reveals the student's current latent state.
|
||||||
|
6. Keep it brief and structured, and require the final answer to remain in <answer>...</answer>.
|
||||||
|
|
||||||
|
## Good Probe Targets
|
||||||
|
- top answer and runner-up
|
||||||
|
- decisive visual cue
|
||||||
|
- suspicious region or compared objects
|
||||||
|
- counting unit or formatting interpretation
|
||||||
|
- 2-4 short intermediate conclusions that directly support the final answer
|
||||||
|
|
||||||
|
Respond ONLY with a valid JSON object:
|
||||||
|
{
|
||||||
|
"reasoning": "<why this probe is informative>",
|
||||||
|
"probe_instruction": "<the exact instruction text to append to the student prompt>"
|
||||||
|
}
|
||||||
@@ -0,0 +1,35 @@
|
|||||||
|
You are a careful and strict evaluator. You will be given:
|
||||||
|
|
||||||
|
1. **Question**
|
||||||
|
2. **Ground Truth Answer** (correct answer)
|
||||||
|
3. **Model Output** (answer from another model)
|
||||||
|
|
||||||
|
**Your goal:** Determine if the Model Output **accurately matches** the Ground Truth Answer in meaning.
|
||||||
|
|
||||||
|
* Matching means: the facts, entities, and key details are equivalent, even if phrasing differs.
|
||||||
|
* Not matching means: the Model Output is wrong, incomplete, contains extra incorrect facts, or changes the meaning.
|
||||||
|
|
||||||
|
**Process (internal reasoning):**
|
||||||
|
|
||||||
|
1. Read and understand the Question, Ground Truth Answer, and Model Output.
|
||||||
|
2. Ignore small wording differences, formatting, or synonyms.
|
||||||
|
3. If all factual content matches, conclude `1`. Otherwise, conclude `0`.
|
||||||
|
|
||||||
|
**Important:**
|
||||||
|
|
||||||
|
* Think through your decision step-by-step **internally** before responding.
|
||||||
|
* In your final output, return **only** True or False, with no extra text or explanation.
|
||||||
|
|
||||||
|
**Output format:**
|
||||||
|
|
||||||
|
True
|
||||||
|
|
||||||
|
or
|
||||||
|
|
||||||
|
False
|
||||||
|
|
||||||
|
**Input:**
|
||||||
|
|
||||||
|
Question: {question},
|
||||||
|
Ground Truth Answer: {groundtruth},
|
||||||
|
Model Output: {modeloutput}
|
||||||
@@ -0,0 +1,13 @@
|
|||||||
|
You are an expert visual reasoning agent solving child-level image understanding tasks.
|
||||||
|
|
||||||
|
{skill_section}## Task Format
|
||||||
|
You will receive one image and one question about it.
|
||||||
|
Inspect the image carefully before answering. Ground the answer in visible evidence.
|
||||||
|
|
||||||
|
## Answer Format
|
||||||
|
Think step by step, then provide your final answer in \boxed{{Answer}} format.
|
||||||
|
- For multiple-choice questions, output only the single choice label, such as \boxed{{A}}.
|
||||||
|
- For open questions, output only a short final answer inside \boxed{{...}}.
|
||||||
|
|
||||||
|
Example:
|
||||||
|
\boxed{{B}}
|
||||||
@@ -0,0 +1,4 @@
|
|||||||
|
"""BabyVision Reflect stage.
|
||||||
|
|
||||||
|
Prompts are now loaded from .md files by the base adapter.
|
||||||
|
"""
|
||||||
@@ -0,0 +1,483 @@
|
|||||||
|
"""BabyVision rollout — multimodal visual QA with image input."""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import base64
|
||||||
|
import json
|
||||||
|
import mimetypes
|
||||||
|
import os
|
||||||
|
from concurrent.futures import ThreadPoolExecutor, as_completed
|
||||||
|
|
||||||
|
from skillopt.envs.babyvision.evaluator import evaluate_item, evaluation_mode, extract_boxed_answer
|
||||||
|
from skillopt.model import chat_student_messages, get_student_backend, is_student_exec_backend
|
||||||
|
from skillopt.model.codex_harness import prepare_workspace, render_skill_md, run_student_exec
|
||||||
|
from skillopt.prompts import load_prompt
|
||||||
|
|
||||||
|
def _build_system(skill_content: str) -> str:
|
||||||
|
if skill_content.strip():
|
||||||
|
skill_section = f"## Skill\n{skill_content.strip()}\n\n"
|
||||||
|
else:
|
||||||
|
skill_section = ""
|
||||||
|
return load_prompt("rollout_system", env="babyvision").format(skill_section=skill_section)
|
||||||
|
|
||||||
|
|
||||||
|
def _format_choices(choices: list[dict]) -> str:
|
||||||
|
return "\n".join(f"{choice['label']}. {choice['text']}" for choice in choices)
|
||||||
|
|
||||||
|
|
||||||
|
def _build_user_text(
|
||||||
|
item: dict,
|
||||||
|
*,
|
||||||
|
diagnostic_mode: bool = False,
|
||||||
|
diagnostic_instruction: str = "",
|
||||||
|
diagnostic_trace_context: str = "",
|
||||||
|
) -> str:
|
||||||
|
parts = []
|
||||||
|
if diagnostic_trace_context.strip():
|
||||||
|
parts.append(
|
||||||
|
"## Previous Codex Trace Snapshot\n"
|
||||||
|
"This is a partial transcript from an earlier attempt. Use it as your current reasoning context.\n\n"
|
||||||
|
f"{diagnostic_trace_context.strip()}"
|
||||||
|
)
|
||||||
|
parts.append(f"## Question\n{item['question']}")
|
||||||
|
if item["ans_type"] == "choice":
|
||||||
|
parts.append(f"## Choices\n{_format_choices(item['choices'])}")
|
||||||
|
parts.append("Answer using the single correct option label in \\boxed{...}.")
|
||||||
|
else:
|
||||||
|
parts.append("Answer with a short phrase in \\boxed{...}.")
|
||||||
|
if diagnostic_mode and diagnostic_instruction.strip():
|
||||||
|
parts.append(f"## Training Readout\n{diagnostic_instruction.strip()}")
|
||||||
|
return "\n\n".join(parts)
|
||||||
|
|
||||||
|
|
||||||
|
def _image_to_data_uri(path: str) -> str:
|
||||||
|
mime = mimetypes.guess_type(path)[0] or "image/png"
|
||||||
|
with open(path, "rb") as f:
|
||||||
|
encoded = base64.b64encode(f.read()).decode("ascii")
|
||||||
|
return f"data:{mime};base64,{encoded}"
|
||||||
|
|
||||||
|
|
||||||
|
def _build_messages(
|
||||||
|
item: dict,
|
||||||
|
skill_content: str,
|
||||||
|
image_detail: str,
|
||||||
|
*,
|
||||||
|
diagnostic_mode: bool = False,
|
||||||
|
diagnostic_instruction: str = "",
|
||||||
|
diagnostic_trace_context: str = "",
|
||||||
|
) -> tuple[list[dict], str, str]:
|
||||||
|
system = _build_system(skill_content)
|
||||||
|
user_text = _build_user_text(
|
||||||
|
item,
|
||||||
|
diagnostic_mode=diagnostic_mode,
|
||||||
|
diagnostic_instruction=diagnostic_instruction,
|
||||||
|
diagnostic_trace_context=diagnostic_trace_context,
|
||||||
|
)
|
||||||
|
image_url = {
|
||||||
|
"url": _image_to_data_uri(item["image_path"]),
|
||||||
|
}
|
||||||
|
if image_detail and image_detail != "auto":
|
||||||
|
image_url["detail"] = image_detail
|
||||||
|
messages = [
|
||||||
|
{"role": "system", "content": system},
|
||||||
|
{
|
||||||
|
"role": "user",
|
||||||
|
"content": [
|
||||||
|
{"type": "text", "text": user_text},
|
||||||
|
{"type": "image_url", "image_url": image_url},
|
||||||
|
],
|
||||||
|
},
|
||||||
|
]
|
||||||
|
return messages, system, user_text
|
||||||
|
|
||||||
|
|
||||||
|
def _build_codex_skill(skill_content: str) -> str:
|
||||||
|
return render_skill_md(
|
||||||
|
skill_content,
|
||||||
|
description="Dynamic ReflACT skill for solving the current BabyVision visual reasoning question.",
|
||||||
|
preamble=(
|
||||||
|
"Use this skill when answering the current visual reasoning question.\n"
|
||||||
|
"Inspect the attached image carefully and return the final answer in \\boxed{...}."
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _run_codex_once(
|
||||||
|
*,
|
||||||
|
pred_dir: str,
|
||||||
|
item: dict,
|
||||||
|
skill_content: str,
|
||||||
|
model: str,
|
||||||
|
timeout: int,
|
||||||
|
image_detail: str,
|
||||||
|
diagnostic_mode: bool = False,
|
||||||
|
diagnostic_instruction: str = "",
|
||||||
|
diagnostic_trace_context: str = "",
|
||||||
|
previous_response: str = "",
|
||||||
|
) -> tuple[str, str, str, str]:
|
||||||
|
user_text = _build_user_text(
|
||||||
|
item,
|
||||||
|
diagnostic_mode=diagnostic_mode,
|
||||||
|
diagnostic_instruction=diagnostic_instruction,
|
||||||
|
diagnostic_trace_context=diagnostic_trace_context,
|
||||||
|
)
|
||||||
|
task_parts = [user_text]
|
||||||
|
if previous_response:
|
||||||
|
task_parts.append(
|
||||||
|
"## Previous Attempt\n"
|
||||||
|
f"{previous_response}\n\n"
|
||||||
|
"Review the same image and question carefully. If needed, correct the answer."
|
||||||
|
)
|
||||||
|
task_text = "\n\n".join(task_parts)
|
||||||
|
skill_md = _build_codex_skill(skill_content)
|
||||||
|
work_dir = os.path.join(pred_dir, "codex_exec")
|
||||||
|
prepare_workspace(
|
||||||
|
work_dir=work_dir,
|
||||||
|
skill_md=skill_md,
|
||||||
|
task_text=task_text,
|
||||||
|
images=[item["image_path"]],
|
||||||
|
)
|
||||||
|
prompt = (
|
||||||
|
"Use the `skillopt-student` skill available in this workspace.\n"
|
||||||
|
"Read `task.md`, inspect the attached image, and answer the question.\n"
|
||||||
|
"Return the final answer in \\boxed{...}."
|
||||||
|
)
|
||||||
|
final_message, raw = run_student_exec(
|
||||||
|
work_dir=work_dir,
|
||||||
|
prompt=prompt,
|
||||||
|
model=model,
|
||||||
|
timeout=timeout,
|
||||||
|
images=[item["image_path"]],
|
||||||
|
)
|
||||||
|
return final_message or raw, raw, skill_md, task_text
|
||||||
|
|
||||||
|
|
||||||
|
def process_one(
|
||||||
|
item: dict,
|
||||||
|
out_root: str,
|
||||||
|
skill_content: str,
|
||||||
|
*,
|
||||||
|
max_turns: int = 1,
|
||||||
|
image_detail: str = "auto",
|
||||||
|
judge_model: str = "gpt-5.4",
|
||||||
|
judge_max_completion_tokens: int = 256,
|
||||||
|
judge_retries: int = 5,
|
||||||
|
diagnostic_mode: bool = False,
|
||||||
|
diagnostic_instruction: str = "",
|
||||||
|
diagnostic_trace_context: str = "",
|
||||||
|
) -> dict:
|
||||||
|
item_id = str(item["id"])
|
||||||
|
result = {
|
||||||
|
"id": item_id,
|
||||||
|
"question": item["question"],
|
||||||
|
"task_type": item.get("subtype") or item.get("task_type") or "babyvision",
|
||||||
|
"task_description": item["question"],
|
||||||
|
"hard": 0,
|
||||||
|
"soft": 0.0,
|
||||||
|
"predicted_answer": "",
|
||||||
|
"predicted_label": "",
|
||||||
|
"predicted_text": "",
|
||||||
|
"response": "",
|
||||||
|
"fail_reason": "",
|
||||||
|
"agent_ok": False,
|
||||||
|
"n_turns": 0,
|
||||||
|
"image_path": item["image_path"],
|
||||||
|
"ans_type": item["ans_type"],
|
||||||
|
"evaluation_mode": evaluation_mode(),
|
||||||
|
"judge_model": judge_model,
|
||||||
|
}
|
||||||
|
if item["ans_type"] == "choice":
|
||||||
|
result["correct_label"] = item["correct_choice"]["label"]
|
||||||
|
result["correct_text"] = item["correct_choice"]["text"]
|
||||||
|
else:
|
||||||
|
result["gold_answers"] = item["blank_answers"]
|
||||||
|
|
||||||
|
try:
|
||||||
|
pred_dir = os.path.join(out_root, "predictions", item_id)
|
||||||
|
os.makedirs(pred_dir, exist_ok=True)
|
||||||
|
|
||||||
|
if is_student_exec_backend():
|
||||||
|
from skillopt.model import azure_openai as _llm
|
||||||
|
|
||||||
|
response = ""
|
||||||
|
conversation: list[dict] = [
|
||||||
|
{"role": "user", "content": f"{item['question']}\n\n[image] {os.path.basename(item['image_path'])}"}
|
||||||
|
]
|
||||||
|
system_prompt = ""
|
||||||
|
user_text = ""
|
||||||
|
for turn in range(max_turns):
|
||||||
|
response, raw, system_prompt, user_text = _run_codex_once(
|
||||||
|
pred_dir=pred_dir,
|
||||||
|
item=item,
|
||||||
|
skill_content=skill_content,
|
||||||
|
model=_llm.STUDENT_DEPLOYMENT,
|
||||||
|
timeout=120,
|
||||||
|
image_detail=image_detail,
|
||||||
|
diagnostic_mode=diagnostic_mode if turn == 0 else False,
|
||||||
|
diagnostic_instruction=diagnostic_instruction if turn == 0 else "",
|
||||||
|
diagnostic_trace_context=diagnostic_trace_context if turn == 0 else "",
|
||||||
|
previous_response=response if turn > 0 else "",
|
||||||
|
)
|
||||||
|
conversation.append({"type": "message", "turn": turn + 1, "content": response})
|
||||||
|
if extract_boxed_answer(response) is not None:
|
||||||
|
break
|
||||||
|
|
||||||
|
result["response"] = response
|
||||||
|
result["agent_ok"] = True
|
||||||
|
result["n_turns"] = len(conversation) - 1
|
||||||
|
with open(os.path.join(pred_dir, "student_system_prompt.txt"), "w", encoding="utf-8") as f:
|
||||||
|
f.write(system_prompt)
|
||||||
|
with open(os.path.join(pred_dir, "student_user_prompt.txt"), "w", encoding="utf-8") as f:
|
||||||
|
f.write(user_text)
|
||||||
|
|
||||||
|
eval_result = evaluate_item(
|
||||||
|
item=item,
|
||||||
|
prediction_text=response,
|
||||||
|
judge_model=judge_model,
|
||||||
|
max_completion_tokens=judge_max_completion_tokens,
|
||||||
|
retries=judge_retries,
|
||||||
|
)
|
||||||
|
result["evaluation_mode"] = eval_result["evaluation_mode"]
|
||||||
|
result["judge_raw"] = eval_result["judge_raw"]
|
||||||
|
result["judge_reason"] = eval_result["judge_reason"]
|
||||||
|
result["matched_gold"] = eval_result["matched_gold"]
|
||||||
|
if item["ans_type"] == "choice":
|
||||||
|
result["predicted_label"] = eval_result["predicted_label"]
|
||||||
|
result["predicted_text"] = eval_result["predicted_text"]
|
||||||
|
result["predicted_answer"] = eval_result["predicted_answer"]
|
||||||
|
result["hard"] = int(eval_result["em"])
|
||||||
|
result["soft"] = eval_result["f1"]
|
||||||
|
if not result["hard"]:
|
||||||
|
result["fail_reason"] = (
|
||||||
|
f"judge=0: predicted '{eval_result['predicted_label'] or eval_result['predicted_answer']}' "
|
||||||
|
f"but expected '{eval_result['correct_label']}' ({eval_result['judge_reason']})"
|
||||||
|
)
|
||||||
|
eval_detail = (
|
||||||
|
f"[EVALUATION RESULT]\n"
|
||||||
|
f"Question: {item['question']}\n"
|
||||||
|
f"Predicted label: {eval_result['predicted_label']!r}\n"
|
||||||
|
f"Predicted text: {eval_result['predicted_text']!r}\n"
|
||||||
|
f"Correct label: {eval_result['correct_label']!r}\n"
|
||||||
|
f"Correct text: {eval_result['correct_text']!r}\n"
|
||||||
|
f"Judge correct: {eval_result['em']}\n"
|
||||||
|
f"Judge reason: {eval_result['judge_reason']}"
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
result["predicted_answer"] = eval_result["predicted_answer"]
|
||||||
|
result["hard"] = int(eval_result["em"])
|
||||||
|
result["soft"] = eval_result["f1"]
|
||||||
|
if not result["hard"]:
|
||||||
|
result["fail_reason"] = (
|
||||||
|
f"judge=0: predicted '{eval_result['predicted_answer']}' "
|
||||||
|
f"but expected {item['blank_answers']} ({eval_result['judge_reason']})"
|
||||||
|
)
|
||||||
|
eval_detail = (
|
||||||
|
f"[EVALUATION RESULT]\n"
|
||||||
|
f"Question: {item['question']}\n"
|
||||||
|
f"Predicted answer: {eval_result['predicted_answer']!r}\n"
|
||||||
|
f"Gold answers: {item['blank_answers']!r}\n"
|
||||||
|
f"Judge correct: {eval_result['em']}\n"
|
||||||
|
f"Judge reason: {eval_result['judge_reason']}\n"
|
||||||
|
f"String F1: {eval_result.get('string_f1', 0.0):.4f}"
|
||||||
|
)
|
||||||
|
conversation.append({"role": "system", "content": eval_detail})
|
||||||
|
with open(os.path.join(pred_dir, "conversation.json"), "w", encoding="utf-8") as f:
|
||||||
|
json.dump(conversation, f, ensure_ascii=False, indent=2)
|
||||||
|
return result
|
||||||
|
|
||||||
|
messages, system_prompt, user_text = _build_messages(
|
||||||
|
item,
|
||||||
|
skill_content,
|
||||||
|
image_detail,
|
||||||
|
diagnostic_mode=diagnostic_mode,
|
||||||
|
diagnostic_instruction=diagnostic_instruction,
|
||||||
|
diagnostic_trace_context=diagnostic_trace_context,
|
||||||
|
)
|
||||||
|
response = ""
|
||||||
|
conversation: list[dict] = [
|
||||||
|
{"role": "user", "content": f"{user_text}\n\n[image] {os.path.basename(item['image_path'])}"}
|
||||||
|
]
|
||||||
|
|
||||||
|
for turn in range(max_turns):
|
||||||
|
if turn == 0:
|
||||||
|
resp_text, _ = chat_student_messages(
|
||||||
|
messages=messages,
|
||||||
|
max_completion_tokens=768,
|
||||||
|
retries=5,
|
||||||
|
stage="rollout",
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
refinement_text = (
|
||||||
|
f"Your previous answer was:\n{response}\n\n"
|
||||||
|
"Review the same image and question carefully. "
|
||||||
|
"If needed, correct your answer. Output the final answer in \\boxed{...}."
|
||||||
|
)
|
||||||
|
refinement_messages = [
|
||||||
|
messages[0],
|
||||||
|
messages[1],
|
||||||
|
{"role": "assistant", "content": response},
|
||||||
|
{"role": "user", "content": refinement_text},
|
||||||
|
]
|
||||||
|
resp_text, _ = chat_student_messages(
|
||||||
|
messages=refinement_messages,
|
||||||
|
max_completion_tokens=512,
|
||||||
|
retries=5,
|
||||||
|
stage="rollout",
|
||||||
|
)
|
||||||
|
response = resp_text
|
||||||
|
conversation.append({"type": "message", "turn": turn + 1, "content": resp_text})
|
||||||
|
if extract_boxed_answer(resp_text) is not None:
|
||||||
|
break
|
||||||
|
|
||||||
|
result["response"] = response
|
||||||
|
result["agent_ok"] = True
|
||||||
|
result["n_turns"] = len(conversation) - 1
|
||||||
|
|
||||||
|
with open(os.path.join(pred_dir, "student_system_prompt.txt"), "w", encoding="utf-8") as f:
|
||||||
|
f.write(system_prompt)
|
||||||
|
with open(os.path.join(pred_dir, "student_user_prompt.txt"), "w", encoding="utf-8") as f:
|
||||||
|
f.write(user_text)
|
||||||
|
|
||||||
|
eval_result = evaluate_item(
|
||||||
|
item=item,
|
||||||
|
prediction_text=response,
|
||||||
|
judge_model=judge_model,
|
||||||
|
max_completion_tokens=judge_max_completion_tokens,
|
||||||
|
retries=judge_retries,
|
||||||
|
)
|
||||||
|
result["evaluation_mode"] = eval_result["evaluation_mode"]
|
||||||
|
result["judge_raw"] = eval_result["judge_raw"]
|
||||||
|
result["judge_reason"] = eval_result["judge_reason"]
|
||||||
|
result["matched_gold"] = eval_result["matched_gold"]
|
||||||
|
|
||||||
|
if item["ans_type"] == "choice":
|
||||||
|
result["predicted_label"] = eval_result["predicted_label"]
|
||||||
|
result["predicted_text"] = eval_result["predicted_text"]
|
||||||
|
result["predicted_answer"] = eval_result["predicted_answer"]
|
||||||
|
result["hard"] = int(eval_result["em"])
|
||||||
|
result["soft"] = eval_result["f1"]
|
||||||
|
if not result["hard"]:
|
||||||
|
result["fail_reason"] = (
|
||||||
|
f"judge=0: predicted '{eval_result['predicted_label'] or eval_result['predicted_answer']}' "
|
||||||
|
f"but expected '{eval_result['correct_label']}' ({eval_result['judge_reason']})"
|
||||||
|
)
|
||||||
|
eval_detail = (
|
||||||
|
f"[EVALUATION RESULT]\n"
|
||||||
|
f"Question: {item['question']}\n"
|
||||||
|
f"Predicted label: {eval_result['predicted_label']!r}\n"
|
||||||
|
f"Predicted text: {eval_result['predicted_text']!r}\n"
|
||||||
|
f"Correct label: {eval_result['correct_label']!r}\n"
|
||||||
|
f"Correct text: {eval_result['correct_text']!r}\n"
|
||||||
|
f"Judge correct: {eval_result['em']}\n"
|
||||||
|
f"Judge reason: {eval_result['judge_reason']}"
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
result["predicted_answer"] = eval_result["predicted_answer"]
|
||||||
|
result["hard"] = int(eval_result["em"])
|
||||||
|
result["soft"] = eval_result["f1"]
|
||||||
|
if not result["hard"]:
|
||||||
|
result["fail_reason"] = (
|
||||||
|
f"judge=0: predicted '{eval_result['predicted_answer']}' "
|
||||||
|
f"but expected {item['blank_answers']} ({eval_result['judge_reason']})"
|
||||||
|
)
|
||||||
|
eval_detail = (
|
||||||
|
f"[EVALUATION RESULT]\n"
|
||||||
|
f"Question: {item['question']}\n"
|
||||||
|
f"Predicted answer: {eval_result['predicted_answer']!r}\n"
|
||||||
|
f"Gold answers: {item['blank_answers']!r}\n"
|
||||||
|
f"Judge correct: {eval_result['em']}\n"
|
||||||
|
f"Judge reason: {eval_result['judge_reason']}\n"
|
||||||
|
f"String F1: {eval_result.get('string_f1', 0.0):.4f}"
|
||||||
|
)
|
||||||
|
|
||||||
|
conversation.append({"role": "system", "content": eval_detail})
|
||||||
|
with open(os.path.join(pred_dir, "conversation.json"), "w", encoding="utf-8") as f:
|
||||||
|
json.dump(conversation, f, ensure_ascii=False, indent=2)
|
||||||
|
except Exception as e: # noqa: BLE001
|
||||||
|
result["fail_reason"] = f"error: {e}"
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
def run_batch(
|
||||||
|
items: list[dict],
|
||||||
|
out_root: str,
|
||||||
|
skill_content: str,
|
||||||
|
*,
|
||||||
|
max_turns: int = 1,
|
||||||
|
workers: int = 32,
|
||||||
|
image_detail: str = "auto",
|
||||||
|
judge_model: str = "gpt-5.4",
|
||||||
|
judge_max_completion_tokens: int = 256,
|
||||||
|
judge_retries: int = 5,
|
||||||
|
diagnostic_mode: bool = False,
|
||||||
|
diagnostic_instruction: str = "",
|
||||||
|
diagnostic_trace_context_by_id: dict[str, str] | None = None,
|
||||||
|
) -> list[dict]:
|
||||||
|
results_path = os.path.join(out_root, "results.jsonl")
|
||||||
|
os.makedirs(out_root, exist_ok=True)
|
||||||
|
|
||||||
|
expected_eval_mode = evaluation_mode()
|
||||||
|
done_ids: set[str] = set()
|
||||||
|
existing: list[dict] = []
|
||||||
|
rewrite_results = False
|
||||||
|
if os.path.exists(results_path):
|
||||||
|
with open(results_path, encoding="utf-8") as f:
|
||||||
|
for line in f:
|
||||||
|
try:
|
||||||
|
row = json.loads(line)
|
||||||
|
if row.get("evaluation_mode") != expected_eval_mode:
|
||||||
|
rewrite_results = True
|
||||||
|
continue
|
||||||
|
done_ids.add(str(row["id"]))
|
||||||
|
existing.append(row)
|
||||||
|
except Exception:
|
||||||
|
rewrite_results = True
|
||||||
|
|
||||||
|
pending = [item for item in items if str(item["id"]) not in done_ids]
|
||||||
|
if not pending and not rewrite_results:
|
||||||
|
return existing
|
||||||
|
|
||||||
|
total = len(existing) + len(pending)
|
||||||
|
completed = len(existing)
|
||||||
|
correct_count = sum(1 for r in existing if r.get("hard", 0))
|
||||||
|
if existing:
|
||||||
|
print(f" [rollout] resuming: {completed}/{total} already done", flush=True)
|
||||||
|
|
||||||
|
results = list(existing)
|
||||||
|
file_mode = "w" if rewrite_results else "a"
|
||||||
|
with open(results_path, file_mode, encoding="utf-8") as outf, ThreadPoolExecutor(max_workers=workers) as ex:
|
||||||
|
if rewrite_results:
|
||||||
|
for row in existing:
|
||||||
|
outf.write(json.dumps(row, ensure_ascii=False) + "\n")
|
||||||
|
futs = {
|
||||||
|
ex.submit(
|
||||||
|
process_one,
|
||||||
|
item,
|
||||||
|
out_root,
|
||||||
|
skill_content,
|
||||||
|
max_turns=max_turns,
|
||||||
|
image_detail=image_detail,
|
||||||
|
judge_model=judge_model,
|
||||||
|
judge_max_completion_tokens=judge_max_completion_tokens,
|
||||||
|
judge_retries=judge_retries,
|
||||||
|
diagnostic_mode=diagnostic_mode,
|
||||||
|
diagnostic_instruction=diagnostic_instruction,
|
||||||
|
diagnostic_trace_context=(diagnostic_trace_context_by_id or {}).get(str(item["id"]), ""),
|
||||||
|
): item
|
||||||
|
for item in pending
|
||||||
|
}
|
||||||
|
for fut in as_completed(futs):
|
||||||
|
row = fut.result()
|
||||||
|
results.append(row)
|
||||||
|
completed += 1
|
||||||
|
if row.get("hard", 0):
|
||||||
|
correct_count += 1
|
||||||
|
acc = correct_count / completed if completed else 0
|
||||||
|
print(
|
||||||
|
f" [rollout] {completed}/{total} "
|
||||||
|
f"(acc={acc:.3f}) id={row.get('id', '?')} "
|
||||||
|
f"hard={row.get('hard', '?')}",
|
||||||
|
flush=True,
|
||||||
|
)
|
||||||
|
outf.write(json.dumps(row, ensure_ascii=False) + "\n")
|
||||||
|
outf.flush()
|
||||||
|
return results
|
||||||
@@ -0,0 +1,18 @@
|
|||||||
|
# BabyVision Visual QA Heuristics
|
||||||
|
|
||||||
|
## Image Inspection
|
||||||
|
- First identify the main objects, their attributes, and their spatial relations before answering.
|
||||||
|
- If the question involves counting, compare all relevant instances carefully instead of stopping after the first match.
|
||||||
|
- If the question asks about color, size, position, or action, verify the specific visible evidence for that attribute.
|
||||||
|
|
||||||
|
## Multiple Choice
|
||||||
|
- Compare every option against the visible image evidence before deciding.
|
||||||
|
- Prefer the option that matches the image exactly; reject options that are only partially true or too vague.
|
||||||
|
- When two options are close, check the smallest discriminating visual detail.
|
||||||
|
|
||||||
|
## Open Answers
|
||||||
|
- Answer with the shortest phrase that is fully supported by the image.
|
||||||
|
- Match the expected level of specificity: not broader than the image evidence, not narrower than the question asks.
|
||||||
|
|
||||||
|
## Final Answer
|
||||||
|
- Output only the final answer inside <answer>...</answer>.
|
||||||
@@ -0,0 +1,396 @@
|
|||||||
|
"""ReflACT environment adapter — abstract interface.
|
||||||
|
|
||||||
|
To connect ReflACT to a new environment (benchmark, simulator, etc.),
|
||||||
|
implement a subclass of :class:`EnvAdapter` with environment-specific
|
||||||
|
rollout and reflection logic.
|
||||||
|
|
||||||
|
Example::
|
||||||
|
|
||||||
|
class MyBenchAdapter(EnvAdapter):
|
||||||
|
def build_train_env(self, batch_size, seed, **kw):
|
||||||
|
return MyEnvManager(split="train", n=batch_size, seed=seed)
|
||||||
|
|
||||||
|
def build_eval_env(self, env_num, split, seed, **kw):
|
||||||
|
return MyEnvManager(split=split, n=env_num, seed=seed)
|
||||||
|
|
||||||
|
def rollout(self, env_manager, skill_content, out_dir, **kw):
|
||||||
|
# Run episodes, return [{"id": ..., "hard": 0/1, "soft": 0.0-1.0, ...}]
|
||||||
|
...
|
||||||
|
|
||||||
|
def reflect(self, results, skill_content, out_dir, **kw):
|
||||||
|
# Analyze trajectories, return list of patch dicts
|
||||||
|
...
|
||||||
|
|
||||||
|
def get_task_types(self):
|
||||||
|
return ["task_a", "task_b"]
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from abc import ABC, abstractmethod
|
||||||
|
import os
|
||||||
|
import random
|
||||||
|
|
||||||
|
from skillopt.datasets.base import BaseDataLoader, BatchSpec
|
||||||
|
from skillopt.model.codex_harness import extract_codex_trace_prefix, format_codex_trace_steps, parse_codex_raw
|
||||||
|
from skillopt.prompts import load_prompt
|
||||||
|
|
||||||
|
|
||||||
|
class EnvAdapter(ABC):
|
||||||
|
"""Abstract adapter for connecting ReflACT to any environment.
|
||||||
|
|
||||||
|
Subclasses must implement all abstract methods. The ReflACT trainer
|
||||||
|
calls these methods at the appropriate pipeline stages.
|
||||||
|
"""
|
||||||
|
|
||||||
|
# ── Lifecycle hooks ────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
def setup(self, cfg: dict) -> None:
|
||||||
|
"""Called once by the trainer before the training loop begins.
|
||||||
|
|
||||||
|
Override to perform one-time initialization that requires the full
|
||||||
|
config (e.g., data loading, split creation). Default is a no-op.
|
||||||
|
"""
|
||||||
|
self._cfg = dict(cfg)
|
||||||
|
|
||||||
|
def get_dataloader(self) -> BaseDataLoader | None:
|
||||||
|
"""Return the task dataloader used by this adapter, if any."""
|
||||||
|
return None
|
||||||
|
|
||||||
|
def requires_ray(self) -> bool:
|
||||||
|
"""Return whether this adapter requires Ray runtime initialization."""
|
||||||
|
return False
|
||||||
|
|
||||||
|
def deep_reflect(
|
||||||
|
self,
|
||||||
|
results: list[dict],
|
||||||
|
skill_content: str,
|
||||||
|
out_dir: str,
|
||||||
|
**kwargs,
|
||||||
|
) -> list[dict | None]:
|
||||||
|
"""Optional deeper diagnostic reflection pass.
|
||||||
|
|
||||||
|
Default behavior is a no-op. Dataset-backed adapters may override this
|
||||||
|
to re-query the student on a small representative subset of the current
|
||||||
|
batch using minimally-perturbed diagnostic prompts that expose
|
||||||
|
intermediate reasoning state.
|
||||||
|
"""
|
||||||
|
return []
|
||||||
|
|
||||||
|
def build_reference_text(self, item: dict) -> str:
|
||||||
|
"""Return hidden reference material for deep reflection, if any."""
|
||||||
|
return str(item.get("reference_text") or "").strip()
|
||||||
|
|
||||||
|
def get_reference_metadata(self, item: dict) -> dict:
|
||||||
|
"""Return structured metadata about hidden reference material."""
|
||||||
|
reference_text = self.build_reference_text(item)
|
||||||
|
if not reference_text:
|
||||||
|
return {"fields": [], "preview": ""}
|
||||||
|
return {
|
||||||
|
"fields": ["reference_text"],
|
||||||
|
"preview": reference_text[:400],
|
||||||
|
}
|
||||||
|
|
||||||
|
def get_codex_deep_probe_prompt(self) -> str | None:
|
||||||
|
env_name = getattr(self, "_cfg", {}).get("env_name")
|
||||||
|
return load_prompt("deep_probe_codex", env=env_name)
|
||||||
|
|
||||||
|
def attach_codex_probe_context(
|
||||||
|
self,
|
||||||
|
results: list[dict],
|
||||||
|
prediction_dir: str,
|
||||||
|
) -> list[dict]:
|
||||||
|
"""Attach compact Codex step metadata for codex-aware deep reflection."""
|
||||||
|
enriched: list[dict] = []
|
||||||
|
for row in results:
|
||||||
|
merged = dict(row)
|
||||||
|
tid = str(row.get("id"))
|
||||||
|
raw_path = os.path.join(prediction_dir, tid, "codex_raw.txt")
|
||||||
|
if os.path.exists(raw_path):
|
||||||
|
with open(raw_path, encoding="utf-8") as f:
|
||||||
|
raw = f.read()
|
||||||
|
parsed = parse_codex_raw(raw)
|
||||||
|
merged["codex_probe_trace_steps"] = format_codex_trace_steps(raw)
|
||||||
|
merged["codex_probe_step_count"] = len(parsed["steps"])
|
||||||
|
enriched.append(merged)
|
||||||
|
return enriched
|
||||||
|
|
||||||
|
def resolve_codex_probe_target(
|
||||||
|
self,
|
||||||
|
*,
|
||||||
|
selected_items: list[dict],
|
||||||
|
selected_examples: list[dict],
|
||||||
|
prediction_dir: str,
|
||||||
|
probe: dict,
|
||||||
|
) -> tuple[list[dict], dict[str, str] | None, dict]:
|
||||||
|
"""Resolve the teacher-selected codex probe target and raw trace prefix."""
|
||||||
|
target_id = str(probe.get("probe_target_id", "")).strip()
|
||||||
|
selected_id_set = {str(item["id"]) for item in selected_items}
|
||||||
|
if target_id not in selected_id_set:
|
||||||
|
target_id = str(selected_items[0]["id"])
|
||||||
|
target_item = next(item for item in selected_items if str(item["id"]) == target_id)
|
||||||
|
target_result = next(
|
||||||
|
(row for row in selected_examples if str(row.get("id")) == target_id),
|
||||||
|
None,
|
||||||
|
)
|
||||||
|
max_probe_step = int((target_result or {}).get("codex_probe_step_count", 0))
|
||||||
|
default_probe_step = max_probe_step - 1 if max_probe_step > 1 else max_probe_step
|
||||||
|
probe_after_step = int(probe.get("probe_after_step", default_probe_step))
|
||||||
|
if max_probe_step > 0:
|
||||||
|
probe_after_step = max(0, min(probe_after_step, max_probe_step))
|
||||||
|
else:
|
||||||
|
probe_after_step = 0
|
||||||
|
raw_path = os.path.join(prediction_dir, target_id, "codex_raw.txt")
|
||||||
|
trace_prefix = ""
|
||||||
|
if os.path.exists(raw_path):
|
||||||
|
with open(raw_path, encoding="utf-8") as f:
|
||||||
|
trace_prefix = extract_codex_trace_prefix(f.read(), after_step=probe_after_step)
|
||||||
|
updated_probe = dict(probe)
|
||||||
|
updated_probe["probe_target_id"] = target_id
|
||||||
|
updated_probe["probe_after_step"] = probe_after_step
|
||||||
|
return [target_item], {target_id: trace_prefix}, updated_probe
|
||||||
|
|
||||||
|
def attach_reference_context(
|
||||||
|
self,
|
||||||
|
results: list[dict],
|
||||||
|
items: list[dict] | None,
|
||||||
|
) -> list[dict]:
|
||||||
|
"""Attach environment-specific hidden reference text to result dicts."""
|
||||||
|
if not results or not items:
|
||||||
|
return list(results)
|
||||||
|
|
||||||
|
item_by_id = {
|
||||||
|
str(item.get("id")): item
|
||||||
|
for item in items
|
||||||
|
if isinstance(item, dict) and item.get("id") is not None
|
||||||
|
}
|
||||||
|
enriched: list[dict] = []
|
||||||
|
for row in results:
|
||||||
|
merged = dict(row)
|
||||||
|
item = item_by_id.get(str(row.get("id")))
|
||||||
|
if item:
|
||||||
|
reference_text = self.build_reference_text(item)
|
||||||
|
if reference_text:
|
||||||
|
merged["reference_text"] = reference_text
|
||||||
|
enriched.append(merged)
|
||||||
|
return enriched
|
||||||
|
|
||||||
|
def select_representative_items(
|
||||||
|
self,
|
||||||
|
results: list[dict],
|
||||||
|
items: list[dict] | None,
|
||||||
|
*,
|
||||||
|
n_failures: int,
|
||||||
|
n_successes: int,
|
||||||
|
seed: int | None = None,
|
||||||
|
) -> list[dict]:
|
||||||
|
"""Select a small diverse subset of current-batch items by outcome."""
|
||||||
|
if not items:
|
||||||
|
return []
|
||||||
|
|
||||||
|
item_by_id = {
|
||||||
|
str(item.get("id")): item
|
||||||
|
for item in items
|
||||||
|
if isinstance(item, dict) and item.get("id") is not None
|
||||||
|
}
|
||||||
|
failures = [
|
||||||
|
(result, item_by_id[str(result.get("id"))])
|
||||||
|
for result in results
|
||||||
|
if not result.get("hard") and str(result.get("id")) in item_by_id
|
||||||
|
]
|
||||||
|
successes = [
|
||||||
|
(result, item_by_id[str(result.get("id"))])
|
||||||
|
for result in results
|
||||||
|
if result.get("hard") and str(result.get("id")) in item_by_id
|
||||||
|
]
|
||||||
|
|
||||||
|
rng = random.Random(seed)
|
||||||
|
|
||||||
|
def _pick(pool: list[tuple[dict, dict]], quota: int) -> list[dict]:
|
||||||
|
if quota <= 0 or not pool:
|
||||||
|
return []
|
||||||
|
shuffled = list(pool)
|
||||||
|
rng.shuffle(shuffled)
|
||||||
|
|
||||||
|
picked_ids: set[str] = set()
|
||||||
|
picked: list[dict] = []
|
||||||
|
seen_types: set[str] = set()
|
||||||
|
|
||||||
|
for result, item in shuffled:
|
||||||
|
task_type = str(result.get("task_type") or item.get("task_type") or item.get("subtype") or "unknown")
|
||||||
|
item_id = str(item["id"])
|
||||||
|
if task_type in seen_types or item_id in picked_ids:
|
||||||
|
continue
|
||||||
|
picked.append(item)
|
||||||
|
picked_ids.add(item_id)
|
||||||
|
seen_types.add(task_type)
|
||||||
|
if len(picked) >= quota:
|
||||||
|
return picked
|
||||||
|
|
||||||
|
for _, item in shuffled:
|
||||||
|
item_id = str(item["id"])
|
||||||
|
if item_id in picked_ids:
|
||||||
|
continue
|
||||||
|
picked.append(item)
|
||||||
|
picked_ids.add(item_id)
|
||||||
|
if len(picked) >= quota:
|
||||||
|
break
|
||||||
|
return picked
|
||||||
|
|
||||||
|
selected = _pick(failures, n_failures)
|
||||||
|
selected_ids = {str(item["id"]) for item in selected}
|
||||||
|
selected.extend(
|
||||||
|
item for item in _pick(successes, n_successes)
|
||||||
|
if str(item["id"]) not in selected_ids
|
||||||
|
)
|
||||||
|
return selected
|
||||||
|
|
||||||
|
def build_env_from_batch(self, batch: BatchSpec, **kwargs):
|
||||||
|
"""Build an environment manager or item list from a :class:`BatchSpec`.
|
||||||
|
|
||||||
|
Default behavior preserves the legacy adapter API by routing training
|
||||||
|
batches through :meth:`build_train_env` and evaluation batches through
|
||||||
|
:meth:`build_eval_env`.
|
||||||
|
"""
|
||||||
|
if batch.phase == "train":
|
||||||
|
return self.build_train_env(batch_size=batch.batch_size, seed=batch.seed, **kwargs)
|
||||||
|
return self.build_eval_env(
|
||||||
|
env_num=batch.batch_size,
|
||||||
|
split=batch.split,
|
||||||
|
seed=batch.seed,
|
||||||
|
**kwargs,
|
||||||
|
)
|
||||||
|
|
||||||
|
@abstractmethod
|
||||||
|
def build_train_env(self, batch_size: int, seed: int, **kwargs):
|
||||||
|
"""Build a training environment manager.
|
||||||
|
|
||||||
|
Returns
|
||||||
|
-------
|
||||||
|
object
|
||||||
|
An environment manager that can be passed to :meth:`rollout`.
|
||||||
|
"""
|
||||||
|
|
||||||
|
@abstractmethod
|
||||||
|
def build_eval_env(self, env_num: int, split: str, seed: int, **kwargs):
|
||||||
|
"""Build an evaluation environment manager.
|
||||||
|
|
||||||
|
Parameters
|
||||||
|
----------
|
||||||
|
env_num : int
|
||||||
|
Number of evaluation environments.
|
||||||
|
split : str
|
||||||
|
Dataset split (e.g. ``"valid_seen"``, ``"valid_unseen"``).
|
||||||
|
seed : int
|
||||||
|
Random seed for reproducibility.
|
||||||
|
|
||||||
|
Returns
|
||||||
|
-------
|
||||||
|
object
|
||||||
|
An environment manager that can be passed to :meth:`rollout`.
|
||||||
|
"""
|
||||||
|
|
||||||
|
@abstractmethod
|
||||||
|
def rollout(
|
||||||
|
self,
|
||||||
|
env_manager,
|
||||||
|
skill_content: str,
|
||||||
|
out_dir: str,
|
||||||
|
**kwargs,
|
||||||
|
) -> list[dict]:
|
||||||
|
"""Run a batch of episodes using the current skill.
|
||||||
|
|
||||||
|
Returns
|
||||||
|
-------
|
||||||
|
list[dict]
|
||||||
|
Each dict conforms to :class:`~skillopt.types.RolloutResult`:
|
||||||
|
must have ``"id"`` (str), ``"hard"`` (0/1), ``"soft"``
|
||||||
|
(float 0-1). May include env-specific fields.
|
||||||
|
"""
|
||||||
|
|
||||||
|
@abstractmethod
|
||||||
|
def reflect(
|
||||||
|
self,
|
||||||
|
results: list[dict],
|
||||||
|
skill_content: str,
|
||||||
|
out_dir: str,
|
||||||
|
**kwargs,
|
||||||
|
) -> list[dict | None]:
|
||||||
|
"""Analyze rollout results and produce patches.
|
||||||
|
|
||||||
|
Each returned dict conforms to :class:`~skillopt.types.RawPatch`:
|
||||||
|
``"patch"`` (with ``"edits"`` list) + ``"source_type"``
|
||||||
|
(``"failure"`` or ``"success"``).
|
||||||
|
|
||||||
|
Returns
|
||||||
|
-------
|
||||||
|
list[dict | None]
|
||||||
|
Raw analyst outputs; ``None`` entries are filtered out.
|
||||||
|
"""
|
||||||
|
|
||||||
|
@abstractmethod
|
||||||
|
def get_task_types(self) -> list[str]:
|
||||||
|
"""Return the list of task type names for this environment."""
|
||||||
|
|
||||||
|
# ── Prompt configuration (two-level priority) ────────────────────────
|
||||||
|
#
|
||||||
|
# Priority: env-specific prompt file > generic default prompt file.
|
||||||
|
#
|
||||||
|
# Prompts are loaded from ``.md`` files via ``load_prompt(name, env)``:
|
||||||
|
# 1. ``skillopt/envs/<env>/prompts/<name>.md`` (env-specific)
|
||||||
|
# 2. ``skillopt/prompts/<name>.md`` (generic fallback)
|
||||||
|
#
|
||||||
|
# Subclasses can still override ``get_*_prompt()`` for full control.
|
||||||
|
|
||||||
|
@property
|
||||||
|
def _env_name(self) -> str:
|
||||||
|
"""Derive the env directory name from this adapter's module path."""
|
||||||
|
# e.g. "skillopt.envs.searchqa.adapter" → "searchqa"
|
||||||
|
module = type(self).__module__
|
||||||
|
parts = module.split(".")
|
||||||
|
if len(parts) >= 3 and parts[-3] == "envs":
|
||||||
|
return parts[-2]
|
||||||
|
return ""
|
||||||
|
|
||||||
|
def _load_env_prompt(self, name: str) -> str | None:
|
||||||
|
"""Load a prompt with env-specific override. Returns None if not found."""
|
||||||
|
try:
|
||||||
|
return load_prompt(name, env=self._env_name)
|
||||||
|
except FileNotFoundError:
|
||||||
|
return None
|
||||||
|
|
||||||
|
def get_error_minibatch_prompt(self) -> str | None:
|
||||||
|
update_mode = getattr(self, "_cfg", {}).get("skill_update_mode", "patch")
|
||||||
|
raw_mode = str(update_mode).strip().lower()
|
||||||
|
if raw_mode in {"full_rewrite", "full_rewrite_minibatch", "minibatch_full_rewrite", "skill_rewrite_minibatch"}:
|
||||||
|
prompt = self._load_env_prompt("analyst_error_full_rewrite")
|
||||||
|
if prompt is not None:
|
||||||
|
return prompt
|
||||||
|
if raw_mode in {"rewrite", "rewrite_from_suggestions", "suggestions", "rewrite_suggestions"}:
|
||||||
|
prompt = self._load_env_prompt("analyst_error_rewrite")
|
||||||
|
if prompt is not None:
|
||||||
|
return prompt
|
||||||
|
return self._load_env_prompt("analyst_error")
|
||||||
|
|
||||||
|
def get_success_minibatch_prompt(self) -> str | None:
|
||||||
|
update_mode = getattr(self, "_cfg", {}).get("skill_update_mode", "patch")
|
||||||
|
raw_mode = str(update_mode).strip().lower()
|
||||||
|
if raw_mode in {"full_rewrite", "full_rewrite_minibatch", "minibatch_full_rewrite", "skill_rewrite_minibatch"}:
|
||||||
|
prompt = self._load_env_prompt("analyst_success_full_rewrite")
|
||||||
|
if prompt is not None:
|
||||||
|
return prompt
|
||||||
|
if raw_mode in {"rewrite", "rewrite_from_suggestions", "suggestions", "rewrite_suggestions"}:
|
||||||
|
prompt = self._load_env_prompt("analyst_success_rewrite")
|
||||||
|
if prompt is not None:
|
||||||
|
return prompt
|
||||||
|
return self._load_env_prompt("analyst_success")
|
||||||
|
|
||||||
|
def get_deep_probe_prompt(self) -> str | None:
|
||||||
|
return self._load_env_prompt("deep_probe")
|
||||||
|
|
||||||
|
def get_meta_reflect_prompt(self) -> str | None:
|
||||||
|
update_mode = getattr(self, "_cfg", {}).get("skill_update_mode", "patch")
|
||||||
|
if str(update_mode).strip().lower() == "rewrite_from_suggestions":
|
||||||
|
prompt = self._load_env_prompt("meta_reflect_rewrite")
|
||||||
|
if prompt is not None:
|
||||||
|
return prompt
|
||||||
|
return self._load_env_prompt("meta_reflect")
|
||||||
@@ -0,0 +1,114 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
from typing import Any, Callable
|
||||||
|
|
||||||
|
from skillopt.gradient.deep_probe import generate_deep_probe_instruction
|
||||||
|
from skillopt.gradient.reflect import run_minibatch_reflect
|
||||||
|
|
||||||
|
|
||||||
|
def run_no_reference_deep_reflect(
|
||||||
|
adapter: Any,
|
||||||
|
results: list[dict],
|
||||||
|
skill_content: str,
|
||||||
|
out_dir: str,
|
||||||
|
*,
|
||||||
|
env_manager: Any = None,
|
||||||
|
prediction_dir: str | None = None,
|
||||||
|
random_seed: int | None = None,
|
||||||
|
step_buffer_context: str = "",
|
||||||
|
output_requirements: list[str] | None = None,
|
||||||
|
metadata_builder: Callable[[dict], dict] | None = None,
|
||||||
|
) -> list[dict | None]:
|
||||||
|
"""Run teacher-designed diagnostic probing without hidden references."""
|
||||||
|
if not getattr(adapter, "use_deep_reflect", False):
|
||||||
|
return []
|
||||||
|
if not isinstance(env_manager, list):
|
||||||
|
return []
|
||||||
|
|
||||||
|
prediction_dir = prediction_dir or os.path.join(out_dir, "predictions")
|
||||||
|
selected_items = adapter.select_representative_items(
|
||||||
|
results,
|
||||||
|
env_manager,
|
||||||
|
n_failures=getattr(adapter, "deep_reflect_failures", 4),
|
||||||
|
n_successes=getattr(adapter, "deep_reflect_successes", 2),
|
||||||
|
seed=random_seed,
|
||||||
|
)
|
||||||
|
if not selected_items:
|
||||||
|
return []
|
||||||
|
|
||||||
|
selected_ids = {str(item["id"]) for item in selected_items}
|
||||||
|
selected_results = [row for row in results if str(row.get("id")) in selected_ids]
|
||||||
|
if metadata_builder is None:
|
||||||
|
selected_metadata = [
|
||||||
|
{
|
||||||
|
"id": str(item.get("id")),
|
||||||
|
"task_type": str(item.get("task_type") or item.get("topic") or "unknown"),
|
||||||
|
"question_preview": str(item.get("question") or "")[:200],
|
||||||
|
}
|
||||||
|
for item in selected_items
|
||||||
|
]
|
||||||
|
else:
|
||||||
|
selected_metadata = [metadata_builder(item) for item in selected_items]
|
||||||
|
|
||||||
|
deep_dir = os.path.join(out_dir, "deep_reflect")
|
||||||
|
rollout_dir = os.path.join(deep_dir, "rollout")
|
||||||
|
patches_dir = os.path.join(deep_dir, "patches")
|
||||||
|
os.makedirs(deep_dir, exist_ok=True)
|
||||||
|
print(
|
||||||
|
f" [2b/6 DEEP REFLECT setup] selected={len(selected_items)} "
|
||||||
|
"mode=no_reference_probe"
|
||||||
|
)
|
||||||
|
|
||||||
|
probe = generate_deep_probe_instruction(
|
||||||
|
skill_content=skill_content,
|
||||||
|
items=selected_results,
|
||||||
|
prediction_dir=prediction_dir,
|
||||||
|
system_prompt=adapter.get_deep_probe_prompt(),
|
||||||
|
step_buffer_context=step_buffer_context,
|
||||||
|
output_requirements=output_requirements,
|
||||||
|
)
|
||||||
|
if not probe:
|
||||||
|
return []
|
||||||
|
|
||||||
|
with open(os.path.join(deep_dir, "probe.json"), "w", encoding="utf-8") as f:
|
||||||
|
json.dump(
|
||||||
|
{
|
||||||
|
**probe,
|
||||||
|
"reference_summary": {
|
||||||
|
"mode": "no_reference_probe",
|
||||||
|
"selected_count": len(selected_items),
|
||||||
|
},
|
||||||
|
"selected_examples": selected_metadata,
|
||||||
|
},
|
||||||
|
f,
|
||||||
|
ensure_ascii=False,
|
||||||
|
indent=2,
|
||||||
|
)
|
||||||
|
|
||||||
|
deep_results = adapter.rollout(
|
||||||
|
selected_items,
|
||||||
|
skill_content,
|
||||||
|
rollout_dir,
|
||||||
|
diagnostic_mode=True,
|
||||||
|
diagnostic_instruction=probe["probe_instruction"],
|
||||||
|
)
|
||||||
|
return run_minibatch_reflect(
|
||||||
|
results=deep_results,
|
||||||
|
skill_content=skill_content,
|
||||||
|
prediction_dir=os.path.join(rollout_dir, "predictions"),
|
||||||
|
patches_dir=patches_dir,
|
||||||
|
workers=getattr(adapter, "analyst_workers", 8),
|
||||||
|
failure_only=getattr(adapter, "failure_only", False),
|
||||||
|
minibatch_size=getattr(adapter, "minibatch_size", 8),
|
||||||
|
edit_budget=getattr(adapter, "edit_budget", 4),
|
||||||
|
random_seed=random_seed,
|
||||||
|
error_system=adapter.get_error_minibatch_prompt(),
|
||||||
|
success_system=adapter.get_success_minibatch_prompt(),
|
||||||
|
step_buffer_context=step_buffer_context,
|
||||||
|
update_mode=getattr(getattr(adapter, "_cfg", {}), "get", lambda *_: "patch")(
|
||||||
|
"skill_update_mode",
|
||||||
|
"patch",
|
||||||
|
),
|
||||||
|
)
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
"""DocVQA environment package for ReflACT."""
|
||||||
@@ -0,0 +1,151 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import os
|
||||||
|
|
||||||
|
from skillopt.datasets.base import BatchSpec
|
||||||
|
from skillopt.envs.base import EnvAdapter
|
||||||
|
from skillopt.envs.deep_reflect import run_no_reference_deep_reflect
|
||||||
|
from skillopt.envs.docvqa.dataloader import DocVQADataLoader
|
||||||
|
from skillopt.envs.docvqa.rollout import run_batch
|
||||||
|
from skillopt.gradient.reflect import run_minibatch_reflect
|
||||||
|
|
||||||
|
|
||||||
|
class DocVQAAdapter(EnvAdapter):
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
split_dir: str = "",
|
||||||
|
data_path: str = "",
|
||||||
|
split_mode: str = "split_dir",
|
||||||
|
split_ratio: str = "2:1:7",
|
||||||
|
split_seed: int = 42,
|
||||||
|
split_output_dir: str = "",
|
||||||
|
max_turns: int = 1,
|
||||||
|
exec_timeout: int = 120,
|
||||||
|
workers: int = 16,
|
||||||
|
analyst_workers: int = 16,
|
||||||
|
failure_only: bool = False,
|
||||||
|
minibatch_size: int = 8,
|
||||||
|
edit_budget: int = 4,
|
||||||
|
seed: int = 42,
|
||||||
|
limit: int = 0,
|
||||||
|
image_detail: str = "auto",
|
||||||
|
use_deep_reflect: bool = False,
|
||||||
|
deep_reflect_failures: int = 4,
|
||||||
|
deep_reflect_successes: int = 2,
|
||||||
|
) -> None:
|
||||||
|
self.max_turns = max_turns
|
||||||
|
self.exec_timeout = exec_timeout
|
||||||
|
self.workers = workers
|
||||||
|
self.analyst_workers = analyst_workers
|
||||||
|
self.failure_only = failure_only
|
||||||
|
self.minibatch_size = minibatch_size
|
||||||
|
self.edit_budget = edit_budget
|
||||||
|
self.image_detail = image_detail
|
||||||
|
self.use_deep_reflect = use_deep_reflect
|
||||||
|
self.deep_reflect_failures = deep_reflect_failures
|
||||||
|
self.deep_reflect_successes = deep_reflect_successes
|
||||||
|
self.dataloader = DocVQADataLoader(
|
||||||
|
split_dir=split_dir,
|
||||||
|
data_path=data_path,
|
||||||
|
split_mode=split_mode,
|
||||||
|
split_ratio=split_ratio,
|
||||||
|
split_seed=split_seed,
|
||||||
|
split_output_dir=split_output_dir,
|
||||||
|
seed=seed,
|
||||||
|
limit=limit,
|
||||||
|
)
|
||||||
|
|
||||||
|
def setup(self, cfg: dict) -> None:
|
||||||
|
super().setup(cfg)
|
||||||
|
self.dataloader.setup(cfg)
|
||||||
|
|
||||||
|
def get_dataloader(self):
|
||||||
|
return self.dataloader
|
||||||
|
|
||||||
|
def build_env_from_batch(self, batch: BatchSpec, **kwargs):
|
||||||
|
return list(batch.payload or [])
|
||||||
|
|
||||||
|
def build_train_env(self, batch_size: int, seed: int, **kwargs):
|
||||||
|
batch = self.dataloader.build_train_batch(batch_size=batch_size, seed=seed, **kwargs)
|
||||||
|
return self.build_env_from_batch(batch, **kwargs)
|
||||||
|
|
||||||
|
def build_eval_env(self, env_num: int, split: str, seed: int, **kwargs):
|
||||||
|
batch = self.dataloader.build_eval_batch(env_num=env_num, split=split, seed=seed, **kwargs)
|
||||||
|
return self.build_env_from_batch(batch, **kwargs)
|
||||||
|
|
||||||
|
def rollout(self, env_manager, skill_content: str, out_dir: str, **kwargs) -> list[dict]:
|
||||||
|
items: list[dict] = env_manager
|
||||||
|
return run_batch(
|
||||||
|
items=items,
|
||||||
|
out_root=out_dir,
|
||||||
|
skill_content=skill_content,
|
||||||
|
max_turns=self.max_turns,
|
||||||
|
exec_timeout=self.exec_timeout,
|
||||||
|
workers=self.workers,
|
||||||
|
image_detail=self.image_detail,
|
||||||
|
diagnostic_mode=kwargs.get("diagnostic_mode", False),
|
||||||
|
diagnostic_instruction=kwargs.get("diagnostic_instruction", ""),
|
||||||
|
task_timeout=self.exec_timeout,
|
||||||
|
)
|
||||||
|
|
||||||
|
def reflect(self, results: list[dict], skill_content: str, out_dir: str, **kwargs) -> list[dict | None]:
|
||||||
|
prediction_dir = kwargs.get("prediction_dir", os.path.join(out_dir, "predictions"))
|
||||||
|
patches_dir = kwargs.get("patches_dir", os.path.join(out_dir, "patches"))
|
||||||
|
random_seed = kwargs.get("random_seed")
|
||||||
|
step_buffer_context = kwargs.get("step_buffer_context", "")
|
||||||
|
return run_minibatch_reflect(
|
||||||
|
results=results,
|
||||||
|
skill_content=skill_content,
|
||||||
|
prediction_dir=prediction_dir,
|
||||||
|
patches_dir=patches_dir,
|
||||||
|
workers=self.analyst_workers,
|
||||||
|
failure_only=self.failure_only,
|
||||||
|
minibatch_size=self.minibatch_size,
|
||||||
|
edit_budget=self.edit_budget,
|
||||||
|
random_seed=random_seed,
|
||||||
|
error_system=self.get_error_minibatch_prompt(),
|
||||||
|
success_system=self.get_success_minibatch_prompt(),
|
||||||
|
step_buffer_context=step_buffer_context,
|
||||||
|
update_mode=getattr(self, "_cfg", {}).get("skill_update_mode", "patch"),
|
||||||
|
)
|
||||||
|
|
||||||
|
def deep_reflect(
|
||||||
|
self,
|
||||||
|
results: list[dict],
|
||||||
|
skill_content: str,
|
||||||
|
out_dir: str,
|
||||||
|
**kwargs,
|
||||||
|
) -> list[dict | None]:
|
||||||
|
return run_no_reference_deep_reflect(
|
||||||
|
self,
|
||||||
|
results,
|
||||||
|
skill_content,
|
||||||
|
out_dir,
|
||||||
|
env_manager=kwargs.get("env_manager"),
|
||||||
|
prediction_dir=kwargs.get("prediction_dir"),
|
||||||
|
random_seed=kwargs.get("random_seed"),
|
||||||
|
step_buffer_context=kwargs.get("step_buffer_context", ""),
|
||||||
|
output_requirements=[
|
||||||
|
"- There is no hidden reference block. Use only the document image prompt, student output, and evaluation result to infer what intermediate state is worth probing.",
|
||||||
|
"- The instruction must explicitly request a short <analysis>...</analysis> block before the final <answer>...</answer>.",
|
||||||
|
"- The readout should focus on visual region, field/table/figure label, OCR text read, candidate answer, and answer-format normalization.",
|
||||||
|
"- Do not ask for exhaustive transcription or a full chain-of-thought.",
|
||||||
|
"- The instruction text should be ready to append directly to the student's prompt.",
|
||||||
|
],
|
||||||
|
metadata_builder=lambda item: {
|
||||||
|
"id": str(item.get("id")),
|
||||||
|
"task_type": str(item.get("task_type") or "docvqa"),
|
||||||
|
"question_preview": str(item.get("question") or "")[:200],
|
||||||
|
"image_path": item.get("image_path", ""),
|
||||||
|
"docId": item.get("docId", ""),
|
||||||
|
"page": item.get("ucsf_document_page_no", ""),
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
def get_task_types(self) -> list[str]:
|
||||||
|
seen: list[str] = []
|
||||||
|
for item in self.dataloader.train_items + self.dataloader.val_items + self.dataloader.test_items:
|
||||||
|
task_type = str(item.get("task_type") or "docvqa")
|
||||||
|
if task_type not in seen:
|
||||||
|
seen.append(task_type)
|
||||||
|
return seen or ["docvqa"]
|
||||||
@@ -0,0 +1,61 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import ast
|
||||||
|
import csv
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from skillopt.datasets.base import SplitDataLoader
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_answers(raw: str) -> list[str]:
|
||||||
|
text = str(raw or "").strip()
|
||||||
|
if not text:
|
||||||
|
return []
|
||||||
|
try:
|
||||||
|
parsed = ast.literal_eval(text)
|
||||||
|
except Exception:
|
||||||
|
return [text]
|
||||||
|
if isinstance(parsed, list):
|
||||||
|
return [str(item).strip() for item in parsed if str(item).strip()]
|
||||||
|
return [str(parsed).strip()]
|
||||||
|
|
||||||
|
|
||||||
|
def _extract_document_path(question: str) -> tuple[str, str]:
|
||||||
|
marker = "document_path:"
|
||||||
|
if marker not in question:
|
||||||
|
return question.strip(), ""
|
||||||
|
main, tail = question.split(marker, 1)
|
||||||
|
return main.strip(), tail.strip()
|
||||||
|
|
||||||
|
|
||||||
|
def _normalize_row(row: dict[str, str]) -> dict:
|
||||||
|
question_text, document_path = _extract_document_path(str(row.get("question") or ""))
|
||||||
|
answers = _parse_answers(row.get("answer") or row.get("ground_truth") or "")
|
||||||
|
image_path = str(row.get("image_path") or document_path or "").strip()
|
||||||
|
task_type = str(row.get("topic") or row.get("category") or "docvqa").strip() or "docvqa"
|
||||||
|
return {
|
||||||
|
"id": str(row.get("questionId") or row.get("id") or "").strip(),
|
||||||
|
"question": question_text,
|
||||||
|
"answer": answers[0] if answers else "",
|
||||||
|
"answers": answers,
|
||||||
|
"task_type": task_type,
|
||||||
|
"subtask": task_type,
|
||||||
|
"image_paths": [image_path] if image_path else [],
|
||||||
|
"image_path": image_path,
|
||||||
|
"questionId": str(row.get("questionId") or "").strip(),
|
||||||
|
"docId": str(row.get("docId") or "").strip(),
|
||||||
|
"ucsf_document_id": str(row.get("ucsf_document_id") or "").strip(),
|
||||||
|
"ucsf_document_page_no": str(row.get("ucsf_document_page_no") or "").strip(),
|
||||||
|
"source_split": str(row.get("source_split") or "").strip(),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
class DocVQADataLoader(SplitDataLoader):
|
||||||
|
def load_split_items(self, split_path: str) -> list[dict]:
|
||||||
|
path = Path(split_path)
|
||||||
|
csv_files = sorted(path.glob("*.csv"))
|
||||||
|
if not csv_files:
|
||||||
|
raise FileNotFoundError(f"No .csv file found in {split_path}")
|
||||||
|
with csv_files[0].open(encoding="utf-8", newline="") as f:
|
||||||
|
reader = csv.DictReader(f)
|
||||||
|
return [_normalize_row(row) for row in reader]
|
||||||
@@ -0,0 +1,113 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import ast
|
||||||
|
import json
|
||||||
|
from collections.abc import Iterable
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
DEFAULT_ANLS_THRESHOLD = 0.5
|
||||||
|
|
||||||
|
|
||||||
|
def _normalize_text(value: Any) -> str:
|
||||||
|
if value is None:
|
||||||
|
return ""
|
||||||
|
text = str(value).strip().lower()
|
||||||
|
return " ".join(text.split())
|
||||||
|
|
||||||
|
|
||||||
|
def _levenshtein_distance(a: str, b: str) -> int:
|
||||||
|
if a == b:
|
||||||
|
return 0
|
||||||
|
if not a:
|
||||||
|
return len(b)
|
||||||
|
if not b:
|
||||||
|
return len(a)
|
||||||
|
if len(a) > len(b):
|
||||||
|
a, b = b, a
|
||||||
|
previous = list(range(len(b) + 1))
|
||||||
|
for i, char_a in enumerate(a, start=1):
|
||||||
|
current = [i]
|
||||||
|
for j, char_b in enumerate(b, start=1):
|
||||||
|
insert_cost = current[j - 1] + 1
|
||||||
|
delete_cost = previous[j] + 1
|
||||||
|
replace_cost = previous[j - 1] + (char_a != char_b)
|
||||||
|
current.append(min(insert_cost, delete_cost, replace_cost))
|
||||||
|
previous = current
|
||||||
|
return previous[-1]
|
||||||
|
|
||||||
|
|
||||||
|
def _score_single_answer(predicted: Any, target: Any, threshold: float) -> float:
|
||||||
|
predicted_norm = _normalize_text(predicted)
|
||||||
|
target_norm = _normalize_text(target)
|
||||||
|
if not predicted_norm and not target_norm:
|
||||||
|
return 1.0
|
||||||
|
if not predicted_norm or not target_norm:
|
||||||
|
return 0.0
|
||||||
|
distance = _levenshtein_distance(predicted_norm, target_norm)
|
||||||
|
normalized_distance = distance / max(len(predicted_norm), len(target_norm))
|
||||||
|
if normalized_distance >= threshold:
|
||||||
|
return 0.0
|
||||||
|
return 1.0 - normalized_distance
|
||||||
|
|
||||||
|
|
||||||
|
def _extract_answer_strings(raw: Any) -> list[str]:
|
||||||
|
if raw is None:
|
||||||
|
return [""]
|
||||||
|
if isinstance(raw, str):
|
||||||
|
text = raw.strip()
|
||||||
|
if not text:
|
||||||
|
return [""]
|
||||||
|
parsed = None
|
||||||
|
if text[0] in "[{":
|
||||||
|
try:
|
||||||
|
parsed = json.loads(text)
|
||||||
|
except json.JSONDecodeError:
|
||||||
|
try:
|
||||||
|
parsed = ast.literal_eval(text)
|
||||||
|
except (ValueError, SyntaxError):
|
||||||
|
parsed = None
|
||||||
|
if parsed is None:
|
||||||
|
return [text]
|
||||||
|
return _extract_answer_strings(parsed)
|
||||||
|
if isinstance(raw, dict):
|
||||||
|
for key in ("answers", "ground_truth", "answer"):
|
||||||
|
if key in raw:
|
||||||
|
return _extract_answer_strings(raw[key])
|
||||||
|
return [str(raw)]
|
||||||
|
if isinstance(raw, Iterable) and not isinstance(raw, (bytes, bytearray)):
|
||||||
|
answers: list[str] = []
|
||||||
|
for item in raw:
|
||||||
|
if isinstance(item, dict):
|
||||||
|
for key in ("text", "answer", "value"):
|
||||||
|
if key in item:
|
||||||
|
answers.extend(_extract_answer_strings(item[key]))
|
||||||
|
break
|
||||||
|
else:
|
||||||
|
answers.append(str(item))
|
||||||
|
continue
|
||||||
|
answers.append(str(item))
|
||||||
|
return answers or [""]
|
||||||
|
return [str(raw)]
|
||||||
|
|
||||||
|
|
||||||
|
def extract_answer(text: str) -> str:
|
||||||
|
lower = text.lower()
|
||||||
|
start = lower.rfind("<answer>")
|
||||||
|
end = lower.rfind("</answer>")
|
||||||
|
if start != -1 and end != -1 and end > start:
|
||||||
|
return text[start + len("<answer>"):end].strip()
|
||||||
|
lines = [line.strip() for line in text.splitlines() if line.strip()]
|
||||||
|
return lines[-1] if lines else text.strip()
|
||||||
|
|
||||||
|
|
||||||
|
def evaluate(prediction_text: str, gold_answers: Any) -> dict:
|
||||||
|
answer = extract_answer(prediction_text)
|
||||||
|
answers = _extract_answer_strings(gold_answers)
|
||||||
|
score = 0.0
|
||||||
|
for target in answers:
|
||||||
|
score = max(score, _score_single_answer(answer, target, DEFAULT_ANLS_THRESHOLD))
|
||||||
|
return {
|
||||||
|
"anls": score,
|
||||||
|
"predicted_answer": answer,
|
||||||
|
"gold_answers": answers,
|
||||||
|
}
|
||||||
@@ -0,0 +1,35 @@
|
|||||||
|
You are an expert failure-analysis agent for visual document question answering tasks.
|
||||||
|
|
||||||
|
You will be given MULTIPLE failed DocVQA trajectories from a single minibatch and the current skill document. Each trajectory includes the model response and an evaluation result scored with ANLS against one or more acceptable answers.
|
||||||
|
|
||||||
|
Your job is to identify the most important COMMON failure patterns across the batch and propose concise skill edits.
|
||||||
|
|
||||||
|
## Failure Type Categories
|
||||||
|
- evidence_miss: the model overlooked the relevant visible region or line
|
||||||
|
- near_match_confusion: the model selected a nearby but incorrect text span
|
||||||
|
- normalization_error: the answer differed mainly in formatting, spacing, punctuation, or minor text normalization
|
||||||
|
- reading_error: the model misread the document content
|
||||||
|
- other: none of the above
|
||||||
|
|
||||||
|
## Rules
|
||||||
|
- Focus on common, reusable reading and extraction behaviors.
|
||||||
|
- Do not hardcode image-specific answers.
|
||||||
|
- Prefer concise edits that improve evidence selection and exact span extraction.
|
||||||
|
|
||||||
|
Respond ONLY with a valid JSON object (no markdown fences, no extra text):
|
||||||
|
{
|
||||||
|
"batch_size": <number of trajectories analysed>,
|
||||||
|
"failure_summary": [
|
||||||
|
{"failure_type": "<type>", "count": <int>, "description": "<one-line>"}
|
||||||
|
],
|
||||||
|
"patch": {
|
||||||
|
"reasoning": "<why these edits address the batch's common failures>",
|
||||||
|
"edits": [
|
||||||
|
{"op": "append", "content": "<markdown to add at end of skill>"},
|
||||||
|
{"op": "insert_after", "target": "<exact heading/text to insert after>", "content": "<markdown>"},
|
||||||
|
{"op": "replace", "target": "<exact text to replace>", "content": "<replacement>"},
|
||||||
|
{"op": "delete", "target": "<exact text to remove>"}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Only include edits that are needed. "edits" can be an empty list if no patch is warranted.
|
||||||
@@ -0,0 +1,24 @@
|
|||||||
|
You are an expert success-pattern analyst for visual document question answering tasks.
|
||||||
|
|
||||||
|
You will be given MULTIPLE successful DocVQA trajectories from a single minibatch and the current skill document. Your job is to identify common visual reading and exact-answer extraction behaviors worth encoding in the skill.
|
||||||
|
|
||||||
|
## Rules
|
||||||
|
- Focus on patterns shared across multiple successful trajectories.
|
||||||
|
- Reinforce reusable behaviors like locating the right region, copying exact spans, and preferring the shortest exact answer over paraphrase.
|
||||||
|
- Only propose patches for patterns not already captured by the current skill.
|
||||||
|
|
||||||
|
Respond ONLY with a valid JSON object:
|
||||||
|
{
|
||||||
|
"batch_size": <number of trajectories analysed>,
|
||||||
|
"success_patterns": ["<pattern 1>", "<pattern 2>"],
|
||||||
|
"patch": {
|
||||||
|
"reasoning": "<why these patterns are worth encoding>",
|
||||||
|
"edits": [
|
||||||
|
{"op": "append", "content": "<markdown>"},
|
||||||
|
{"op": "insert_after", "target": "<heading/text>", "content": "<markdown>"},
|
||||||
|
{"op": "replace", "target": "<old text>", "content": "<new text>"},
|
||||||
|
{"op": "delete", "target": "<exact text to remove>"}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
"edits" may be empty if the skill already covers all observed patterns.
|
||||||
@@ -0,0 +1,12 @@
|
|||||||
|
You are an expert visual document question answering agent.
|
||||||
|
|
||||||
|
{skill_section}You will receive a document image and a question about the document.
|
||||||
|
Read the visual evidence carefully and answer concisely.
|
||||||
|
|
||||||
|
Rules:
|
||||||
|
- Ground the answer in the visible document content.
|
||||||
|
- Prefer exact spans, numbers, dates, and names from the document.
|
||||||
|
- Do not invent content that is not visible.
|
||||||
|
- If multiple near-matches exist, choose the one best supported by the document.
|
||||||
|
|
||||||
|
Return the final answer inside <answer>...</answer>.
|
||||||
@@ -0,0 +1,388 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import time
|
||||||
|
from concurrent.futures import FIRST_COMPLETED, ThreadPoolExecutor, wait
|
||||||
|
|
||||||
|
from skillopt.envs.docvqa.evaluator import evaluate
|
||||||
|
from skillopt.model import chat_student_messages, get_student_backend, is_student_exec_backend
|
||||||
|
from skillopt.model.codex_harness import prepare_workspace, render_skill_md, run_student_exec
|
||||||
|
from skillopt.prompts import load_prompt
|
||||||
|
|
||||||
|
|
||||||
|
def _build_system(skill_content: str) -> str:
|
||||||
|
if skill_content.strip():
|
||||||
|
skill_section = f"## Skill\n{skill_content.strip()}\n\n"
|
||||||
|
else:
|
||||||
|
skill_section = ""
|
||||||
|
return load_prompt("rollout_system", env="docvqa").format(skill_section=skill_section)
|
||||||
|
|
||||||
|
|
||||||
|
def _image_to_data_uri(path: str) -> str:
|
||||||
|
import base64
|
||||||
|
import mimetypes
|
||||||
|
|
||||||
|
mime = mimetypes.guess_type(path)[0] or "image/png"
|
||||||
|
with open(path, "rb") as f:
|
||||||
|
encoded = base64.b64encode(f.read()).decode("ascii")
|
||||||
|
return f"data:{mime};base64,{encoded}"
|
||||||
|
|
||||||
|
|
||||||
|
def _build_messages(
|
||||||
|
item: dict,
|
||||||
|
skill_content: str,
|
||||||
|
image_detail: str,
|
||||||
|
*,
|
||||||
|
diagnostic_mode: bool = False,
|
||||||
|
diagnostic_instruction: str = "",
|
||||||
|
) -> tuple[list[dict], str, str]:
|
||||||
|
system = _build_system(skill_content)
|
||||||
|
user_text = item["question"] + "\n\nReturn the final answer inside <answer>...</answer>."
|
||||||
|
if diagnostic_mode and diagnostic_instruction.strip():
|
||||||
|
user_text += f"\n\n## Training Readout\n{diagnostic_instruction.strip()}"
|
||||||
|
image_url = {"url": _image_to_data_uri(item["image_path"])}
|
||||||
|
if image_detail and image_detail != "auto":
|
||||||
|
image_url["detail"] = image_detail
|
||||||
|
messages = [
|
||||||
|
{"role": "system", "content": system},
|
||||||
|
{
|
||||||
|
"role": "user",
|
||||||
|
"content": [
|
||||||
|
{"type": "text", "text": user_text},
|
||||||
|
{"type": "image_url", "image_url": image_url},
|
||||||
|
],
|
||||||
|
},
|
||||||
|
]
|
||||||
|
return messages, system, user_text
|
||||||
|
|
||||||
|
|
||||||
|
def _build_codex_skill(skill_content: str) -> str:
|
||||||
|
return render_skill_md(
|
||||||
|
skill_content,
|
||||||
|
description="Dynamic ReflACT skill for solving the current DocVQA document-image question.",
|
||||||
|
preamble=(
|
||||||
|
"Use this skill when answering the current DocVQA question.\n"
|
||||||
|
"Inspect the attached document image carefully and return the final answer inside <answer>...</answer>."
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _run_codex_once(
|
||||||
|
*,
|
||||||
|
pred_dir: str,
|
||||||
|
item: dict,
|
||||||
|
skill_content: str,
|
||||||
|
model: str,
|
||||||
|
timeout: int,
|
||||||
|
image_detail: str,
|
||||||
|
diagnostic_mode: bool = False,
|
||||||
|
diagnostic_instruction: str = "",
|
||||||
|
previous_response: str = "",
|
||||||
|
) -> tuple[str, str, str, str]:
|
||||||
|
_ = image_detail
|
||||||
|
_messages, _system, user_text = _build_messages(
|
||||||
|
item,
|
||||||
|
skill_content,
|
||||||
|
image_detail,
|
||||||
|
diagnostic_mode=diagnostic_mode,
|
||||||
|
diagnostic_instruction=diagnostic_instruction,
|
||||||
|
)
|
||||||
|
task_parts = [user_text]
|
||||||
|
image_abs = os.path.abspath(item["image_path"])
|
||||||
|
task_parts.append(
|
||||||
|
"## Document Image\n"
|
||||||
|
"The document image is available in this workspace via `ATTACHMENTS.md`.\n"
|
||||||
|
f"Original image path: `{image_abs}`\n"
|
||||||
|
"Open or inspect that image before answering; do not answer from memory."
|
||||||
|
)
|
||||||
|
if previous_response:
|
||||||
|
task_parts.append(
|
||||||
|
"## Previous Attempt\n"
|
||||||
|
f"{previous_response}\n\n"
|
||||||
|
"Review the same document image carefully and correct the answer if needed."
|
||||||
|
)
|
||||||
|
task_text = "\n\n".join(task_parts)
|
||||||
|
skill_md = _build_codex_skill(skill_content)
|
||||||
|
work_dir = os.path.join(pred_dir, "codex_exec")
|
||||||
|
prepare_workspace(
|
||||||
|
work_dir=work_dir,
|
||||||
|
skill_md=skill_md,
|
||||||
|
task_text=task_text,
|
||||||
|
images=[item["image_path"]],
|
||||||
|
)
|
||||||
|
prompt = (
|
||||||
|
"Use the `skillopt-student` skill available in this workspace.\n"
|
||||||
|
"Read `task.md`, inspect the attached document image, and answer the DocVQA question.\n"
|
||||||
|
"Return the final answer inside <answer>...</answer>."
|
||||||
|
)
|
||||||
|
final_message, raw = run_student_exec(
|
||||||
|
work_dir=work_dir,
|
||||||
|
prompt=prompt,
|
||||||
|
model=model,
|
||||||
|
timeout=timeout,
|
||||||
|
images=[item["image_path"]],
|
||||||
|
)
|
||||||
|
return final_message or raw, raw, skill_md, task_text
|
||||||
|
|
||||||
|
|
||||||
|
def process_one(
|
||||||
|
item: dict,
|
||||||
|
out_root: str,
|
||||||
|
skill_content: str,
|
||||||
|
*,
|
||||||
|
max_turns: int = 1,
|
||||||
|
exec_timeout: int = 120,
|
||||||
|
image_detail: str = "auto",
|
||||||
|
diagnostic_mode: bool = False,
|
||||||
|
diagnostic_instruction: str = "",
|
||||||
|
) -> dict:
|
||||||
|
item_id = str(item["id"])
|
||||||
|
result = {
|
||||||
|
"id": item_id,
|
||||||
|
"question": item["question"],
|
||||||
|
"task_type": item.get("subtask") or item.get("task_type") or "docvqa",
|
||||||
|
"task_description": item["question"],
|
||||||
|
"hard": 0,
|
||||||
|
"soft": 0.0,
|
||||||
|
"predicted_answer": "",
|
||||||
|
"response": "",
|
||||||
|
"fail_reason": "",
|
||||||
|
"agent_ok": False,
|
||||||
|
"n_turns": 0,
|
||||||
|
"image_paths": item.get("image_paths", []),
|
||||||
|
"gold_answer": item.get("answers", []),
|
||||||
|
}
|
||||||
|
try:
|
||||||
|
response = ""
|
||||||
|
system_prompt = ""
|
||||||
|
user_text = ""
|
||||||
|
conversation: list[dict] = []
|
||||||
|
if is_student_exec_backend():
|
||||||
|
from skillopt.model import azure_openai as _llm
|
||||||
|
|
||||||
|
conversation = [
|
||||||
|
{
|
||||||
|
"role": "user",
|
||||||
|
"content": item["question"] + "\n\n" + f"[image] {os.path.basename(item['image_path'])}",
|
||||||
|
}
|
||||||
|
]
|
||||||
|
for turn in range(max_turns):
|
||||||
|
response, _raw, system_prompt, user_text = _run_codex_once(
|
||||||
|
pred_dir=os.path.join(out_root, "predictions", item_id),
|
||||||
|
item=item,
|
||||||
|
skill_content=skill_content,
|
||||||
|
model=_llm.STUDENT_DEPLOYMENT,
|
||||||
|
timeout=exec_timeout,
|
||||||
|
image_detail=image_detail,
|
||||||
|
diagnostic_mode=diagnostic_mode if turn == 0 else False,
|
||||||
|
diagnostic_instruction=diagnostic_instruction if turn == 0 else "",
|
||||||
|
previous_response=response if turn > 0 else "",
|
||||||
|
)
|
||||||
|
conversation.append({"type": "message", "turn": turn + 1, "content": response})
|
||||||
|
if "<answer>" in response.lower():
|
||||||
|
break
|
||||||
|
else:
|
||||||
|
messages, system_prompt, user_text = _build_messages(
|
||||||
|
item,
|
||||||
|
skill_content,
|
||||||
|
image_detail,
|
||||||
|
diagnostic_mode=diagnostic_mode,
|
||||||
|
diagnostic_instruction=diagnostic_instruction,
|
||||||
|
)
|
||||||
|
conversation = [
|
||||||
|
{
|
||||||
|
"role": "user",
|
||||||
|
"content": user_text + "\n\n" + f"[image] {os.path.basename(item['image_path'])}",
|
||||||
|
}
|
||||||
|
]
|
||||||
|
for turn in range(max_turns):
|
||||||
|
if turn == 0:
|
||||||
|
resp_text, _ = chat_student_messages(
|
||||||
|
messages=messages,
|
||||||
|
max_completion_tokens=768,
|
||||||
|
retries=5,
|
||||||
|
stage="rollout",
|
||||||
|
timeout=exec_timeout,
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
refinement_messages = [
|
||||||
|
messages[0],
|
||||||
|
messages[1],
|
||||||
|
{"role": "assistant", "content": response},
|
||||||
|
{"role": "user", "content": "Review the same image carefully and answer again. Keep the final answer inside <answer>...</answer>."},
|
||||||
|
]
|
||||||
|
resp_text, _ = chat_student_messages(
|
||||||
|
messages=refinement_messages,
|
||||||
|
max_completion_tokens=512,
|
||||||
|
retries=5,
|
||||||
|
stage="rollout",
|
||||||
|
timeout=exec_timeout,
|
||||||
|
)
|
||||||
|
response = resp_text
|
||||||
|
conversation.append({"type": "message", "turn": turn + 1, "content": resp_text})
|
||||||
|
if "<answer>" in resp_text.lower():
|
||||||
|
break
|
||||||
|
|
||||||
|
result["response"] = response
|
||||||
|
result["agent_ok"] = True
|
||||||
|
result["n_turns"] = len(conversation) - 1
|
||||||
|
|
||||||
|
pred_dir = os.path.join(out_root, "predictions", item_id)
|
||||||
|
os.makedirs(pred_dir, exist_ok=True)
|
||||||
|
with open(os.path.join(pred_dir, "student_system_prompt.txt"), "w", encoding="utf-8") as f:
|
||||||
|
f.write(system_prompt)
|
||||||
|
with open(os.path.join(pred_dir, "student_user_prompt.txt"), "w", encoding="utf-8") as f:
|
||||||
|
f.write(user_text)
|
||||||
|
|
||||||
|
eval_result = evaluate(response, item.get("answers", []))
|
||||||
|
result["predicted_answer"] = eval_result["predicted_answer"]
|
||||||
|
result["hard"] = int(eval_result["anls"] >= 0.999)
|
||||||
|
result["soft"] = eval_result["anls"]
|
||||||
|
if result["soft"] <= 0.0:
|
||||||
|
result["fail_reason"] = f"predicted '{eval_result['predicted_answer']}' but expected one of {item.get('answers', [])}"
|
||||||
|
|
||||||
|
eval_detail = (
|
||||||
|
"[EVALUATION RESULT]\n"
|
||||||
|
f"Question: {item['question']}\n"
|
||||||
|
f"Predicted answer: {eval_result['predicted_answer']!r}\n"
|
||||||
|
f"Gold answers: {item.get('answers', [])!r}\n"
|
||||||
|
f"ANLS: {eval_result['anls']:.4f}"
|
||||||
|
)
|
||||||
|
conversation.append({"role": "system", "content": eval_detail})
|
||||||
|
with open(os.path.join(pred_dir, "conversation.json"), "w", encoding="utf-8") as f:
|
||||||
|
json.dump(conversation, f, ensure_ascii=False, indent=2)
|
||||||
|
except Exception as e: # noqa: BLE001
|
||||||
|
result["fail_reason"] = f"error: {e}"
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
def run_batch(
|
||||||
|
items: list[dict],
|
||||||
|
out_root: str,
|
||||||
|
skill_content: str,
|
||||||
|
*,
|
||||||
|
max_turns: int = 1,
|
||||||
|
exec_timeout: int = 120,
|
||||||
|
workers: int = 16,
|
||||||
|
image_detail: str = "auto",
|
||||||
|
diagnostic_mode: bool = False,
|
||||||
|
diagnostic_instruction: str = "",
|
||||||
|
task_timeout: int = 600,
|
||||||
|
) -> list[dict]:
|
||||||
|
task_timeout = max(int(task_timeout), int(exec_timeout) + 60)
|
||||||
|
results_path = os.path.join(out_root, "results.jsonl")
|
||||||
|
os.makedirs(out_root, exist_ok=True)
|
||||||
|
|
||||||
|
done_ids: set[str] = set()
|
||||||
|
existing: list[dict] = []
|
||||||
|
if os.path.exists(results_path):
|
||||||
|
with open(results_path, encoding="utf-8") as f:
|
||||||
|
for line in f:
|
||||||
|
try:
|
||||||
|
row = json.loads(line)
|
||||||
|
except Exception:
|
||||||
|
continue
|
||||||
|
done_ids.add(str(row["id"]))
|
||||||
|
existing.append(row)
|
||||||
|
|
||||||
|
pending = [item for item in items if str(item["id"]) not in done_ids]
|
||||||
|
if not pending:
|
||||||
|
return existing
|
||||||
|
|
||||||
|
def _timeout_result(item: dict) -> dict:
|
||||||
|
return {
|
||||||
|
"id": str(item["id"]),
|
||||||
|
"question": item.get("question", ""),
|
||||||
|
"task_type": item.get("subtask") or item.get("task_type") or "docvqa",
|
||||||
|
"task_description": item.get("question", ""),
|
||||||
|
"hard": 0,
|
||||||
|
"soft": 0.0,
|
||||||
|
"predicted_answer": "",
|
||||||
|
"response": "",
|
||||||
|
"fail_reason": f"task-timeout-{task_timeout}s",
|
||||||
|
"agent_ok": False,
|
||||||
|
"n_turns": 0,
|
||||||
|
"image_paths": item.get("image_paths", []),
|
||||||
|
"gold_answer": item.get("answers", []),
|
||||||
|
"phase": "timeout",
|
||||||
|
}
|
||||||
|
|
||||||
|
def _error_result(item: dict, exc: Exception) -> dict:
|
||||||
|
row = _timeout_result(item)
|
||||||
|
row["phase"] = "error"
|
||||||
|
row["fail_reason"] = f"unexpected: {type(exc).__name__}: {exc}"
|
||||||
|
return row
|
||||||
|
|
||||||
|
started_at: dict[str, float] = {}
|
||||||
|
|
||||||
|
def _run_one(item: dict) -> dict:
|
||||||
|
started_at[str(item["id"])] = time.time()
|
||||||
|
return process_one(
|
||||||
|
item,
|
||||||
|
out_root,
|
||||||
|
skill_content,
|
||||||
|
max_turns=max_turns,
|
||||||
|
exec_timeout=exec_timeout,
|
||||||
|
image_detail=image_detail,
|
||||||
|
diagnostic_mode=diagnostic_mode,
|
||||||
|
diagnostic_instruction=diagnostic_instruction,
|
||||||
|
)
|
||||||
|
|
||||||
|
total = len(existing) + len(pending)
|
||||||
|
completed = len(existing)
|
||||||
|
correct = sum(1 for r in existing if r.get("hard", 0))
|
||||||
|
if existing:
|
||||||
|
print(f" [rollout] resuming: {completed}/{total} already done", flush=True)
|
||||||
|
|
||||||
|
results = list(existing)
|
||||||
|
with open(results_path, "a", encoding="utf-8") as outf:
|
||||||
|
ex = ThreadPoolExecutor(max_workers=workers)
|
||||||
|
try:
|
||||||
|
futs = {ex.submit(_run_one, item): item for item in pending}
|
||||||
|
pending_futs = set(futs)
|
||||||
|
while pending_futs:
|
||||||
|
done, _ = wait(pending_futs, timeout=5, return_when=FIRST_COMPLETED)
|
||||||
|
now = time.time()
|
||||||
|
timed_out = [
|
||||||
|
fut for fut in pending_futs - done
|
||||||
|
if str(futs[fut]["id"]) in started_at
|
||||||
|
and now - started_at[str(futs[fut]["id"])] >= task_timeout
|
||||||
|
]
|
||||||
|
for fut in done:
|
||||||
|
pending_futs.remove(fut)
|
||||||
|
item = futs[fut]
|
||||||
|
try:
|
||||||
|
res = fut.result()
|
||||||
|
except Exception as exc: # noqa: BLE001
|
||||||
|
res = _error_result(item, exc)
|
||||||
|
results.append(res)
|
||||||
|
completed += 1
|
||||||
|
if res.get("hard", 0):
|
||||||
|
correct += 1
|
||||||
|
acc = correct / completed if completed else 0
|
||||||
|
print(
|
||||||
|
f" [rollout] {completed}/{total} "
|
||||||
|
f"(acc={acc:.3f}) id={res['id']} "
|
||||||
|
f"hard={res.get('hard', '?')}",
|
||||||
|
flush=True,
|
||||||
|
)
|
||||||
|
outf.write(json.dumps(res, ensure_ascii=False) + "\n")
|
||||||
|
outf.flush()
|
||||||
|
for fut in timed_out:
|
||||||
|
pending_futs.remove(fut)
|
||||||
|
fut.cancel()
|
||||||
|
res = _timeout_result(futs[fut])
|
||||||
|
results.append(res)
|
||||||
|
completed += 1
|
||||||
|
acc = correct / completed if completed else 0
|
||||||
|
print(
|
||||||
|
f" [rollout] {completed}/{total} "
|
||||||
|
f"(acc={acc:.3f}) id={res['id']} TIMEOUT",
|
||||||
|
flush=True,
|
||||||
|
)
|
||||||
|
outf.write(json.dumps(res, ensure_ascii=False) + "\n")
|
||||||
|
outf.flush()
|
||||||
|
finally:
|
||||||
|
ex.shutdown(wait=False, cancel_futures=True)
|
||||||
|
return results
|
||||||
@@ -0,0 +1,11 @@
|
|||||||
|
# DocVQA Skill
|
||||||
|
|
||||||
|
## Visual Evidence Discipline
|
||||||
|
- Read the document carefully before answering.
|
||||||
|
- Prefer the smallest exact text span that answers the question.
|
||||||
|
- When several nearby strings look similar, choose the one whose surrounding labels or layout best match the question.
|
||||||
|
|
||||||
|
## Exact Answer Discipline
|
||||||
|
- Copy names, numbers, and dates exactly from the document whenever possible.
|
||||||
|
- Prefer direct extraction over paraphrase.
|
||||||
|
- Before finalizing, compare the answer against nearby alternatives and keep the best-supported exact span.
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
"""LiveMathematicianBench environment package for ReflACT."""
|
||||||
@@ -0,0 +1,282 @@
|
|||||||
|
"""LiveMathematicianBench environment adapter for ReflACT."""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
|
||||||
|
from skillopt.gradient.deep_probe import generate_deep_probe_instruction
|
||||||
|
from skillopt.datasets.base import BatchSpec
|
||||||
|
from skillopt.gradient.reflect import run_minibatch_reflect
|
||||||
|
from skillopt.envs.base import EnvAdapter
|
||||||
|
from skillopt.envs.livemathematicianbench.dataloader import LiveMathematicianBenchDataLoader
|
||||||
|
from skillopt.envs.livemathematicianbench.rollout import run_batch
|
||||||
|
from skillopt.model import get_student_backend
|
||||||
|
|
||||||
|
|
||||||
|
class LiveMathematicianBenchAdapter(EnvAdapter):
|
||||||
|
"""LiveMathematicianBench adapter."""
|
||||||
|
|
||||||
|
def build_reference_text(self, item: dict) -> str:
|
||||||
|
parts: list[str] = []
|
||||||
|
theorem = str(item.get("theorem") or "").strip()
|
||||||
|
sketch = str(item.get("sketch") or "").strip()
|
||||||
|
if theorem:
|
||||||
|
parts.append(f"## Reference Theorem\n{theorem}")
|
||||||
|
if sketch:
|
||||||
|
parts.append(f"## Reference Sketch\n{sketch}")
|
||||||
|
return "\n\n".join(parts)
|
||||||
|
|
||||||
|
def get_reference_metadata(self, item: dict) -> dict:
|
||||||
|
fields: list[str] = []
|
||||||
|
previews: list[str] = []
|
||||||
|
theorem = str(item.get("theorem") or "").strip()
|
||||||
|
sketch = str(item.get("sketch") or "").strip()
|
||||||
|
if theorem:
|
||||||
|
fields.append("theorem")
|
||||||
|
previews.append(f"[theorem]\n{theorem[:220]}")
|
||||||
|
if sketch:
|
||||||
|
fields.append("sketch")
|
||||||
|
previews.append(f"[sketch]\n{sketch[:220]}")
|
||||||
|
return {
|
||||||
|
"fields": fields,
|
||||||
|
"preview": "\n\n".join(previews)[:500],
|
||||||
|
}
|
||||||
|
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
split_dir: str = "",
|
||||||
|
data_path: str = "",
|
||||||
|
split_mode: str = "ratio",
|
||||||
|
split_ratio: str = "2:1:7",
|
||||||
|
split_seed: int = 42,
|
||||||
|
split_output_dir: str = "",
|
||||||
|
max_turns: int = 1,
|
||||||
|
exec_timeout: int = 600,
|
||||||
|
workers: int = 64,
|
||||||
|
analyst_workers: int = 16,
|
||||||
|
failure_only: bool = False,
|
||||||
|
minibatch_size: int = 8,
|
||||||
|
edit_budget: int = 4,
|
||||||
|
seed: int = 42,
|
||||||
|
limit: int = 0,
|
||||||
|
shuffle_choices: bool = True,
|
||||||
|
use_theorem: bool = False,
|
||||||
|
use_sketch: bool = False,
|
||||||
|
use_deep_reflect: bool = False,
|
||||||
|
deep_reflect_failures: int = 4,
|
||||||
|
deep_reflect_successes: int = 2,
|
||||||
|
) -> None:
|
||||||
|
self.max_turns = max_turns
|
||||||
|
self.exec_timeout = exec_timeout
|
||||||
|
self.workers = workers
|
||||||
|
self.analyst_workers = analyst_workers
|
||||||
|
self.failure_only = failure_only
|
||||||
|
self.minibatch_size = minibatch_size
|
||||||
|
self.edit_budget = edit_budget
|
||||||
|
self.use_theorem = use_theorem
|
||||||
|
self.use_sketch = use_sketch
|
||||||
|
self.use_deep_reflect = use_deep_reflect
|
||||||
|
self.deep_reflect_failures = deep_reflect_failures
|
||||||
|
self.deep_reflect_successes = deep_reflect_successes
|
||||||
|
self.dataloader = LiveMathematicianBenchDataLoader(
|
||||||
|
split_dir=split_dir,
|
||||||
|
data_path=data_path,
|
||||||
|
split_mode=split_mode,
|
||||||
|
split_ratio=split_ratio,
|
||||||
|
split_seed=split_seed,
|
||||||
|
split_output_dir=split_output_dir,
|
||||||
|
seed=seed,
|
||||||
|
limit=limit,
|
||||||
|
shuffle_choices=shuffle_choices,
|
||||||
|
)
|
||||||
|
|
||||||
|
def setup(self, cfg: dict) -> None:
|
||||||
|
super().setup(cfg)
|
||||||
|
self.dataloader.setup(cfg)
|
||||||
|
|
||||||
|
def get_dataloader(self):
|
||||||
|
return self.dataloader
|
||||||
|
|
||||||
|
def build_env_from_batch(self, batch: BatchSpec, **kwargs):
|
||||||
|
return list(batch.payload or [])
|
||||||
|
|
||||||
|
def build_train_env(self, batch_size: int, seed: int, **kwargs):
|
||||||
|
batch = self.dataloader.build_train_batch(batch_size=batch_size, seed=seed, **kwargs)
|
||||||
|
return self.build_env_from_batch(batch, **kwargs)
|
||||||
|
|
||||||
|
def build_eval_env(self, env_num: int, split: str, seed: int, **kwargs):
|
||||||
|
batch = self.dataloader.build_eval_batch(env_num=env_num, split=split, seed=seed, **kwargs)
|
||||||
|
return self.build_env_from_batch(batch, **kwargs)
|
||||||
|
|
||||||
|
def rollout(
|
||||||
|
self,
|
||||||
|
env_manager,
|
||||||
|
skill_content: str,
|
||||||
|
out_dir: str,
|
||||||
|
**kwargs,
|
||||||
|
) -> list[dict]:
|
||||||
|
items: list[dict] = env_manager
|
||||||
|
return run_batch(
|
||||||
|
items=items,
|
||||||
|
out_root=out_dir,
|
||||||
|
skill_content=skill_content,
|
||||||
|
max_turns=self.max_turns,
|
||||||
|
exec_timeout=self.exec_timeout,
|
||||||
|
workers=self.workers,
|
||||||
|
use_theorem=self.use_theorem,
|
||||||
|
use_sketch=self.use_sketch,
|
||||||
|
diagnostic_mode=kwargs.get("diagnostic_mode", False),
|
||||||
|
diagnostic_instruction=kwargs.get("diagnostic_instruction", ""),
|
||||||
|
diagnostic_trace_context_by_id=kwargs.get("diagnostic_trace_context_by_id"),
|
||||||
|
task_timeout=self.exec_timeout,
|
||||||
|
)
|
||||||
|
|
||||||
|
def reflect(
|
||||||
|
self,
|
||||||
|
results: list[dict],
|
||||||
|
skill_content: str,
|
||||||
|
out_dir: str,
|
||||||
|
**kwargs,
|
||||||
|
) -> list[dict | None]:
|
||||||
|
prediction_dir = kwargs.get("prediction_dir", os.path.join(out_dir, "predictions"))
|
||||||
|
patches_dir = kwargs.get("patches_dir", os.path.join(out_dir, "patches"))
|
||||||
|
random_seed = kwargs.get("random_seed")
|
||||||
|
step_buffer_context = kwargs.get("step_buffer_context", "")
|
||||||
|
meta_skill_context = kwargs.get("meta_skill_context", "")
|
||||||
|
|
||||||
|
return run_minibatch_reflect(
|
||||||
|
results=results,
|
||||||
|
skill_content=skill_content,
|
||||||
|
prediction_dir=prediction_dir,
|
||||||
|
patches_dir=patches_dir,
|
||||||
|
workers=self.analyst_workers,
|
||||||
|
failure_only=self.failure_only,
|
||||||
|
minibatch_size=self.minibatch_size,
|
||||||
|
edit_budget=self.edit_budget,
|
||||||
|
random_seed=random_seed,
|
||||||
|
error_system=self.get_error_minibatch_prompt(),
|
||||||
|
success_system=self.get_success_minibatch_prompt(),
|
||||||
|
step_buffer_context=step_buffer_context,
|
||||||
|
meta_skill_context=meta_skill_context,
|
||||||
|
update_mode=getattr(self, "_cfg", {}).get("skill_update_mode", "patch"),
|
||||||
|
)
|
||||||
|
|
||||||
|
def deep_reflect(
|
||||||
|
self,
|
||||||
|
results: list[dict],
|
||||||
|
skill_content: str,
|
||||||
|
out_dir: str,
|
||||||
|
**kwargs,
|
||||||
|
) -> list[dict | None]:
|
||||||
|
if not self.use_deep_reflect:
|
||||||
|
return []
|
||||||
|
|
||||||
|
env_manager = kwargs.get("env_manager")
|
||||||
|
prediction_dir = kwargs.get("prediction_dir", os.path.join(out_dir, "predictions"))
|
||||||
|
random_seed = kwargs.get("random_seed")
|
||||||
|
step_buffer_context = kwargs.get("step_buffer_context", "")
|
||||||
|
meta_skill_context = kwargs.get("meta_skill_context", "")
|
||||||
|
codex_backend = get_student_backend() == "codex_exec"
|
||||||
|
selected_items = self.select_representative_items(
|
||||||
|
results,
|
||||||
|
env_manager if isinstance(env_manager, list) else None,
|
||||||
|
n_failures=self.deep_reflect_failures,
|
||||||
|
n_successes=self.deep_reflect_successes,
|
||||||
|
seed=random_seed,
|
||||||
|
)
|
||||||
|
if not selected_items:
|
||||||
|
return []
|
||||||
|
selected_ids = {str(item["id"]) for item in selected_items}
|
||||||
|
selected_results = [row for row in results if str(row.get("id")) in selected_ids]
|
||||||
|
selected_examples = self.attach_reference_context(selected_results, selected_items)
|
||||||
|
if codex_backend:
|
||||||
|
selected_examples = self.attach_codex_probe_context(selected_examples, prediction_dir)
|
||||||
|
selected_metadata = []
|
||||||
|
theorem_count = 0
|
||||||
|
sketch_count = 0
|
||||||
|
for item in selected_items:
|
||||||
|
meta = self.get_reference_metadata(item)
|
||||||
|
if "theorem" in meta["fields"]:
|
||||||
|
theorem_count += 1
|
||||||
|
if "sketch" in meta["fields"]:
|
||||||
|
sketch_count += 1
|
||||||
|
selected_metadata.append({
|
||||||
|
"id": str(item["id"]),
|
||||||
|
"task_type": str(item.get("theorem_type", ["math_mcq"])[0] if item.get("theorem_type") else "math_mcq"),
|
||||||
|
"reference_fields": meta["fields"],
|
||||||
|
"reference_preview": meta["preview"],
|
||||||
|
})
|
||||||
|
|
||||||
|
deep_dir = os.path.join(out_dir, "deep_reflect")
|
||||||
|
rollout_dir = os.path.join(deep_dir, "rollout")
|
||||||
|
patches_dir = os.path.join(deep_dir, "patches")
|
||||||
|
os.makedirs(deep_dir, exist_ok=True)
|
||||||
|
print(
|
||||||
|
f" [2b/6 DEEP REFLECT setup] selected={len(selected_items)} "
|
||||||
|
f"reference_fields=theorem({theorem_count}/{len(selected_items)}),"
|
||||||
|
f"sketch({sketch_count}/{len(selected_items)})"
|
||||||
|
)
|
||||||
|
probe = generate_deep_probe_instruction(
|
||||||
|
skill_content=skill_content,
|
||||||
|
items=selected_examples,
|
||||||
|
prediction_dir=prediction_dir,
|
||||||
|
system_prompt=self.get_codex_deep_probe_prompt() if codex_backend else self.get_deep_probe_prompt(),
|
||||||
|
step_buffer_context=step_buffer_context,
|
||||||
|
meta_skill_context=meta_skill_context,
|
||||||
|
)
|
||||||
|
if not probe:
|
||||||
|
return []
|
||||||
|
diagnostic_trace_context_by_id = None
|
||||||
|
if codex_backend:
|
||||||
|
selected_items, diagnostic_trace_context_by_id, probe = self.resolve_codex_probe_target(
|
||||||
|
selected_items=selected_items,
|
||||||
|
selected_examples=selected_examples,
|
||||||
|
prediction_dir=prediction_dir,
|
||||||
|
probe=probe,
|
||||||
|
)
|
||||||
|
probe_record = {
|
||||||
|
**probe,
|
||||||
|
"reference_summary": {
|
||||||
|
"selected_count": len(selected_items),
|
||||||
|
"field_counts": {
|
||||||
|
"theorem": theorem_count,
|
||||||
|
"sketch": sketch_count,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
"selected_examples": selected_metadata,
|
||||||
|
}
|
||||||
|
with open(os.path.join(deep_dir, "probe.json"), "w", encoding="utf-8") as f:
|
||||||
|
json.dump(probe_record, f, ensure_ascii=False, indent=2)
|
||||||
|
deep_results = run_batch(
|
||||||
|
items=selected_items,
|
||||||
|
out_root=rollout_dir,
|
||||||
|
skill_content=skill_content,
|
||||||
|
max_turns=self.max_turns,
|
||||||
|
workers=min(self.workers, max(len(selected_items), 1)),
|
||||||
|
use_theorem=self.use_theorem,
|
||||||
|
use_sketch=self.use_sketch,
|
||||||
|
diagnostic_mode=True,
|
||||||
|
diagnostic_instruction=probe["probe_instruction"],
|
||||||
|
diagnostic_trace_context_by_id=diagnostic_trace_context_by_id,
|
||||||
|
task_timeout=self.exec_timeout,
|
||||||
|
)
|
||||||
|
deep_results = self.attach_reference_context(deep_results, selected_items)
|
||||||
|
return run_minibatch_reflect(
|
||||||
|
results=deep_results,
|
||||||
|
skill_content=skill_content,
|
||||||
|
prediction_dir=os.path.join(rollout_dir, "predictions"),
|
||||||
|
patches_dir=patches_dir,
|
||||||
|
workers=self.analyst_workers,
|
||||||
|
failure_only=self.failure_only,
|
||||||
|
minibatch_size=self.minibatch_size,
|
||||||
|
edit_budget=self.edit_budget,
|
||||||
|
random_seed=random_seed,
|
||||||
|
error_system=self.get_error_minibatch_prompt(),
|
||||||
|
success_system=self.get_success_minibatch_prompt(),
|
||||||
|
step_buffer_context=step_buffer_context,
|
||||||
|
meta_skill_context=meta_skill_context,
|
||||||
|
update_mode=getattr(self, "_cfg", {}).get("skill_update_mode", "patch"),
|
||||||
|
)
|
||||||
|
|
||||||
|
def get_task_types(self) -> list[str]:
|
||||||
|
return self.dataloader.get_task_types()
|
||||||
@@ -0,0 +1,308 @@
|
|||||||
|
"""LiveMathematicianBench task dataloader."""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import glob
|
||||||
|
import hashlib
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import random
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from skillopt.datasets.base import BatchSpec, SplitDataLoader
|
||||||
|
|
||||||
|
|
||||||
|
# ── Raw data loading utilities (for preprocessing / standalone eval) ─────
|
||||||
|
|
||||||
|
_CHOICE_LABELS = ["A", "B", "C", "D", "E", "F", "G"]
|
||||||
|
|
||||||
|
|
||||||
|
def _load_json(path: str) -> Any:
|
||||||
|
with open(path) as f:
|
||||||
|
return json.load(f)
|
||||||
|
|
||||||
|
|
||||||
|
def _iter_monthly_files(data_path: str) -> list[str]:
|
||||||
|
if not data_path:
|
||||||
|
return []
|
||||||
|
if os.path.isfile(data_path):
|
||||||
|
return [data_path]
|
||||||
|
if os.path.isdir(data_path):
|
||||||
|
nested = glob.glob(
|
||||||
|
os.path.join(data_path, "**", "qa_*_final.json"),
|
||||||
|
recursive=True,
|
||||||
|
)
|
||||||
|
flat = glob.glob(os.path.join(data_path, "qa_*_final.json"))
|
||||||
|
return sorted(set(nested + flat))
|
||||||
|
return []
|
||||||
|
|
||||||
|
|
||||||
|
def _coerce_choices(raw_choices: Any) -> list[dict]:
|
||||||
|
if isinstance(raw_choices, list):
|
||||||
|
choices: list[dict] = []
|
||||||
|
for idx, item in enumerate(raw_choices):
|
||||||
|
if isinstance(item, dict):
|
||||||
|
label = str(item.get("label") or _CHOICE_LABELS[idx]).strip()
|
||||||
|
text = str(item.get("text") or item.get("content") or "").strip()
|
||||||
|
else:
|
||||||
|
label = _CHOICE_LABELS[idx]
|
||||||
|
text = str(item).strip()
|
||||||
|
if text:
|
||||||
|
choices.append({"label": label, "text": text})
|
||||||
|
return choices
|
||||||
|
|
||||||
|
if isinstance(raw_choices, dict):
|
||||||
|
labels = sorted(raw_choices.keys())
|
||||||
|
return [
|
||||||
|
{"label": str(label).strip(), "text": str(raw_choices[label]).strip()}
|
||||||
|
for label in labels
|
||||||
|
if str(raw_choices[label]).strip()
|
||||||
|
]
|
||||||
|
|
||||||
|
return []
|
||||||
|
|
||||||
|
|
||||||
|
def _coerce_theorem_types(raw: Any) -> list[str]:
|
||||||
|
if isinstance(raw, list):
|
||||||
|
return [str(x).strip() for x in raw if str(x).strip()]
|
||||||
|
if raw is None:
|
||||||
|
return []
|
||||||
|
text = str(raw).strip()
|
||||||
|
return [text] if text else []
|
||||||
|
|
||||||
|
|
||||||
|
def _normalize_label(text: str) -> str:
|
||||||
|
return str(text).strip().upper().rstrip(".):")
|
||||||
|
|
||||||
|
|
||||||
|
def _normalize_item(item: dict, row_idx: int, source_path: str) -> dict:
|
||||||
|
mcq = item.get("mcq", {}) if isinstance(item.get("mcq"), dict) else {}
|
||||||
|
question = str(mcq.get("question") or item.get("question") or "").strip()
|
||||||
|
choices = _coerce_choices(mcq.get("choices") or item.get("choices") or [])
|
||||||
|
correct = mcq.get("correct_choice") or item.get("correct_choice") or {}
|
||||||
|
|
||||||
|
if isinstance(correct, dict):
|
||||||
|
correct_label = _normalize_label(correct.get("label", ""))
|
||||||
|
correct_text = str(correct.get("text") or "").strip()
|
||||||
|
else:
|
||||||
|
correct_label = _normalize_label(correct)
|
||||||
|
correct_text = ""
|
||||||
|
|
||||||
|
choice_by_label = {
|
||||||
|
_normalize_label(choice["label"]): choice["text"]
|
||||||
|
for choice in choices
|
||||||
|
}
|
||||||
|
if correct_label and not correct_text:
|
||||||
|
correct_text = choice_by_label.get(correct_label, "")
|
||||||
|
if correct_label and correct_text and correct_label not in choice_by_label:
|
||||||
|
choices.append({"label": correct_label, "text": correct_text})
|
||||||
|
choices.sort(key=lambda choice: _CHOICE_LABELS.index(choice["label"]) if choice["label"] in _CHOICE_LABELS else len(_CHOICE_LABELS))
|
||||||
|
choice_by_label[correct_label] = correct_text
|
||||||
|
|
||||||
|
month = str(item.get("month") or "").strip()
|
||||||
|
item_no = item.get("no", row_idx + 1)
|
||||||
|
item_id = f"{month}:{item_no}" if month else str(item_no)
|
||||||
|
|
||||||
|
return {
|
||||||
|
"id": item_id,
|
||||||
|
"month": month,
|
||||||
|
"no": item_no,
|
||||||
|
"paper_link": str(item.get("paper_link") or "").strip(),
|
||||||
|
"theorem": str(item.get("theorem") or "").strip(),
|
||||||
|
"sketch": str(item.get("sketch") or "").strip(),
|
||||||
|
"theorem_type": _coerce_theorem_types(item.get("theorem_type")),
|
||||||
|
"question": question,
|
||||||
|
"choices": choices,
|
||||||
|
"correct_choice": {
|
||||||
|
"label": correct_label,
|
||||||
|
"text": correct_text,
|
||||||
|
},
|
||||||
|
"source_path": source_path,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def load_items(data_path: str) -> list[dict]:
|
||||||
|
"""Load and normalise LiveMathematicianBench items from JSON files."""
|
||||||
|
files = _iter_monthly_files(data_path)
|
||||||
|
if not files:
|
||||||
|
raise ValueError(
|
||||||
|
"LiveMathematicianBench requires data_path to be a qa_*_final.json file "
|
||||||
|
"or a directory containing monthly qa_*_final.json files."
|
||||||
|
)
|
||||||
|
|
||||||
|
items: list[dict] = []
|
||||||
|
for path in files:
|
||||||
|
raw = _load_json(path)
|
||||||
|
if not isinstance(raw, list):
|
||||||
|
raise ValueError(f"Expected JSON array in {path}, got {type(raw).__name__}")
|
||||||
|
for row_idx, item in enumerate(raw):
|
||||||
|
norm = _normalize_item(item, row_idx=row_idx, source_path=path)
|
||||||
|
if norm["question"] and norm["choices"] and norm["correct_choice"]["label"]:
|
||||||
|
items.append(norm)
|
||||||
|
if not items:
|
||||||
|
raise ValueError(f"No valid LiveMathematicianBench items loaded from {data_path}")
|
||||||
|
return items
|
||||||
|
|
||||||
|
|
||||||
|
# ── Dataloader ───────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
class LiveMathematicianBenchDataLoader(SplitDataLoader):
|
||||||
|
"""LiveMathematicianBench dataloader with per-seed choice shuffling."""
|
||||||
|
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
split_dir: str = "",
|
||||||
|
data_path: str = "",
|
||||||
|
split_mode: str = "ratio",
|
||||||
|
split_ratio: str = "2:1:7",
|
||||||
|
split_seed: int = 42,
|
||||||
|
split_output_dir: str = "",
|
||||||
|
seed: int = 42,
|
||||||
|
limit: int = 0,
|
||||||
|
shuffle_choices: bool = True,
|
||||||
|
**kwargs,
|
||||||
|
) -> None:
|
||||||
|
super().__init__(
|
||||||
|
split_dir=split_dir,
|
||||||
|
data_path=data_path,
|
||||||
|
split_mode=split_mode,
|
||||||
|
split_ratio=split_ratio,
|
||||||
|
split_seed=split_seed,
|
||||||
|
split_output_dir=split_output_dir,
|
||||||
|
seed=seed,
|
||||||
|
limit=limit,
|
||||||
|
)
|
||||||
|
self.shuffle_choices = shuffle_choices
|
||||||
|
self._task_types: list[str] = []
|
||||||
|
|
||||||
|
def load_raw_items(self, data_path: str) -> list[dict]:
|
||||||
|
return load_items(data_path)
|
||||||
|
|
||||||
|
def setup(self, cfg: dict) -> None:
|
||||||
|
super().setup(cfg)
|
||||||
|
all_items = self.train_items + self.val_items + self.test_items
|
||||||
|
task_types: set[str] = set()
|
||||||
|
for item in all_items:
|
||||||
|
for name in item.get("theorem_type", []):
|
||||||
|
if name:
|
||||||
|
task_types.add(name)
|
||||||
|
self._task_types = sorted(task_types)
|
||||||
|
|
||||||
|
def get_task_types(self) -> list[str]:
|
||||||
|
return list(self._task_types)
|
||||||
|
|
||||||
|
# ── Choice shuffling ─────────────────────────────────────────────────
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _item_shuffle_seed(item_id: str, seed: int) -> int:
|
||||||
|
digest = hashlib.sha256(f"{seed}:{item_id}".encode("utf-8")).hexdigest()
|
||||||
|
return int(digest[:16], 16)
|
||||||
|
|
||||||
|
def _shuffle_item_choices(self, item: dict, seed: int) -> dict:
|
||||||
|
if not self.shuffle_choices:
|
||||||
|
return {
|
||||||
|
**item,
|
||||||
|
"choices": [dict(c) for c in item["choices"]],
|
||||||
|
"correct_choice": dict(item["correct_choice"]),
|
||||||
|
}
|
||||||
|
|
||||||
|
shuffled_choices = [dict(c) for c in item["choices"]]
|
||||||
|
rng = random.Random(self._item_shuffle_seed(str(item["id"]), seed))
|
||||||
|
rng.shuffle(shuffled_choices)
|
||||||
|
|
||||||
|
original_correct = _normalize_label(item["correct_choice"]["label"])
|
||||||
|
remapped_choices: list[dict] = []
|
||||||
|
new_correct_choice = dict(item["correct_choice"])
|
||||||
|
|
||||||
|
for idx, choice in enumerate(shuffled_choices):
|
||||||
|
new_label = _CHOICE_LABELS[idx]
|
||||||
|
old_label = _normalize_label(choice["label"])
|
||||||
|
remapped_choices.append({"label": new_label, "text": choice["text"]})
|
||||||
|
if old_label == original_correct:
|
||||||
|
new_correct_choice = {"label": new_label, "text": choice["text"]}
|
||||||
|
|
||||||
|
transformed = dict(item)
|
||||||
|
transformed["choices"] = remapped_choices
|
||||||
|
transformed["correct_choice"] = new_correct_choice
|
||||||
|
return transformed
|
||||||
|
|
||||||
|
def _materialize_batch(self, items: list[dict], seed: int) -> list[dict]:
|
||||||
|
return [self._shuffle_item_choices(item, seed) for item in items]
|
||||||
|
|
||||||
|
# ── Batch construction (override for choice shuffling) ───────────────
|
||||||
|
|
||||||
|
def plan_train_epoch(
|
||||||
|
self,
|
||||||
|
*,
|
||||||
|
epoch: int,
|
||||||
|
steps_per_epoch: int,
|
||||||
|
accumulation: int,
|
||||||
|
batch_size: int,
|
||||||
|
seed: int,
|
||||||
|
**kwargs,
|
||||||
|
) -> list[BatchSpec]:
|
||||||
|
"""Build a shuffled epoch while preserving per-batch choice shuffling."""
|
||||||
|
epoch_rng = random.Random(seed + epoch * 1000)
|
||||||
|
items = list(self.train_items)
|
||||||
|
epoch_rng.shuffle(items)
|
||||||
|
|
||||||
|
total_batches = steps_per_epoch * accumulation
|
||||||
|
if total_batches <= 0:
|
||||||
|
return []
|
||||||
|
|
||||||
|
batches: list[BatchSpec] = []
|
||||||
|
cursor = 0
|
||||||
|
for batch_idx in range(total_batches):
|
||||||
|
batch_seed = seed + epoch * 1000 + batch_idx + 1
|
||||||
|
batch_items = items[cursor: cursor + batch_size]
|
||||||
|
cursor += len(batch_items)
|
||||||
|
|
||||||
|
if not batch_items and items:
|
||||||
|
refill_rng = random.Random(batch_seed)
|
||||||
|
batch_items = list(items)
|
||||||
|
refill_rng.shuffle(batch_items)
|
||||||
|
batch_items = batch_items[:batch_size]
|
||||||
|
|
||||||
|
batch_items = self._materialize_batch(batch_items, batch_seed)
|
||||||
|
batches.append(
|
||||||
|
BatchSpec(
|
||||||
|
phase="train",
|
||||||
|
split="train",
|
||||||
|
seed=batch_seed,
|
||||||
|
batch_size=len(batch_items),
|
||||||
|
payload=batch_items,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
return batches
|
||||||
|
|
||||||
|
def build_train_batch(self, batch_size: int, seed: int, **kwargs) -> BatchSpec:
|
||||||
|
rng = random.Random(seed)
|
||||||
|
items = list(self.train_items)
|
||||||
|
rng.shuffle(items)
|
||||||
|
items = self._materialize_batch(items[:batch_size], seed)
|
||||||
|
return BatchSpec(
|
||||||
|
phase="train",
|
||||||
|
split="train",
|
||||||
|
seed=seed,
|
||||||
|
batch_size=len(items),
|
||||||
|
payload=items,
|
||||||
|
)
|
||||||
|
|
||||||
|
def build_eval_batch(
|
||||||
|
self,
|
||||||
|
env_num: int,
|
||||||
|
split: str,
|
||||||
|
seed: int,
|
||||||
|
**kwargs,
|
||||||
|
) -> BatchSpec:
|
||||||
|
items = self.get_split_items(split)
|
||||||
|
if env_num and env_num < len(items):
|
||||||
|
items = items[:env_num]
|
||||||
|
items = self._materialize_batch(items, seed)
|
||||||
|
return BatchSpec(
|
||||||
|
phase="eval",
|
||||||
|
split=split,
|
||||||
|
seed=seed,
|
||||||
|
batch_size=len(items),
|
||||||
|
payload=items,
|
||||||
|
)
|
||||||
@@ -0,0 +1,62 @@
|
|||||||
|
"""LiveMathematicianBench evaluation helpers."""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import re
|
||||||
|
|
||||||
|
|
||||||
|
def extract_answer(text: str) -> str:
|
||||||
|
matches = re.findall(r"<answer>(.*?)</answer>", text, re.DOTALL | re.IGNORECASE)
|
||||||
|
if matches:
|
||||||
|
return matches[-1].strip()
|
||||||
|
lines = [ln.strip() for ln in text.strip().splitlines() if ln.strip()]
|
||||||
|
if lines:
|
||||||
|
return lines[-1]
|
||||||
|
return text.strip()
|
||||||
|
|
||||||
|
|
||||||
|
def normalize_label(text: str) -> str:
|
||||||
|
return str(text).strip().upper().rstrip(".):")
|
||||||
|
|
||||||
|
|
||||||
|
def parse_choice_label(prediction_text: str, choices: list[dict]) -> str:
|
||||||
|
answer = extract_answer(prediction_text)
|
||||||
|
label = normalize_label(answer)
|
||||||
|
valid_labels = {normalize_label(choice.get("label", "")) for choice in choices}
|
||||||
|
if label in valid_labels:
|
||||||
|
return label
|
||||||
|
|
||||||
|
answer_lower = answer.lower()
|
||||||
|
for choice in choices:
|
||||||
|
choice_label = normalize_label(choice.get("label", ""))
|
||||||
|
choice_text = str(choice.get("text", "")).strip()
|
||||||
|
if choice_text and choice_text.lower() == answer_lower:
|
||||||
|
return choice_label
|
||||||
|
|
||||||
|
first_token = normalize_label(answer.split()[0]) if answer.split() else ""
|
||||||
|
if first_token in valid_labels:
|
||||||
|
return first_token
|
||||||
|
return label
|
||||||
|
|
||||||
|
|
||||||
|
def evaluate(prediction_text: str, correct_choice: dict, choices: list[dict]) -> dict:
|
||||||
|
predicted_label = parse_choice_label(prediction_text, choices)
|
||||||
|
correct_label = normalize_label(correct_choice.get("label", ""))
|
||||||
|
predicted_text = ""
|
||||||
|
correct_text = str(correct_choice.get("text", "")).strip()
|
||||||
|
|
||||||
|
for choice in choices:
|
||||||
|
if normalize_label(choice.get("label", "")) == predicted_label:
|
||||||
|
predicted_text = str(choice.get("text", "")).strip()
|
||||||
|
break
|
||||||
|
|
||||||
|
is_correct = float(predicted_label == correct_label)
|
||||||
|
return {
|
||||||
|
"em": is_correct,
|
||||||
|
"f1": is_correct,
|
||||||
|
"sub_em": is_correct,
|
||||||
|
"predicted_answer": predicted_label or extract_answer(prediction_text),
|
||||||
|
"predicted_label": predicted_label,
|
||||||
|
"predicted_text": predicted_text,
|
||||||
|
"correct_label": correct_label,
|
||||||
|
"correct_text": correct_text,
|
||||||
|
}
|
||||||
@@ -0,0 +1,37 @@
|
|||||||
|
You are an expert failure-analysis agent for theorem-grounded mathematical multiple-choice questions.
|
||||||
|
|
||||||
|
You will be given MULTIPLE failed trajectories from a single minibatch and the current skill document.
|
||||||
|
Each trajectory includes the student's response and an evaluation result showing the predicted option
|
||||||
|
versus the correct option.
|
||||||
|
|
||||||
|
Your job is to identify COMMON reasoning failures across the batch and propose concise skill edits.
|
||||||
|
|
||||||
|
## Failure Type Categories
|
||||||
|
- **quantifier_miss**: the agent missed exact quantifiers, scope, or existence/uniqueness conditions
|
||||||
|
- **strength_mismatch**: the agent preferred a weaker or stronger statement than what was proved
|
||||||
|
- **condition_miss**: the agent ignored hypotheses, equality cases, or domain restrictions
|
||||||
|
- **option_confusion**: the agent confused similar answer choices or failed to compare them exactly
|
||||||
|
- **other**: none of the above
|
||||||
|
|
||||||
|
## Rules
|
||||||
|
1. Focus on patterns that recur across the minibatch.
|
||||||
|
2. Prefer edits that improve exact choice discrimination, not theorem-specific memorization.
|
||||||
|
3. Do not hardcode paper-specific content.
|
||||||
|
4. Only patch gaps not already covered by the skill.
|
||||||
|
|
||||||
|
Respond ONLY with a valid JSON object:
|
||||||
|
{
|
||||||
|
"batch_size": <number>,
|
||||||
|
"failure_summary": [
|
||||||
|
{"failure_type": "<type>", "count": <int>, "description": "<one-line>"}
|
||||||
|
],
|
||||||
|
"patch": {
|
||||||
|
"reasoning": "<why these edits address the common failures>",
|
||||||
|
"edits": [
|
||||||
|
{"op": "append", "content": "<markdown>"},
|
||||||
|
{"op": "insert_after", "target": "<heading/text>", "content": "<markdown>"},
|
||||||
|
{"op": "replace", "target": "<old text>", "content": "<new text>"},
|
||||||
|
{"op": "delete", "target": "<exact text to remove>"}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,25 @@
|
|||||||
|
You are an expert success-pattern analyst for theorem-grounded mathematical multiple-choice questions.
|
||||||
|
|
||||||
|
You will be given MULTIPLE successful trajectories from a minibatch and the current skill document.
|
||||||
|
Identify generalizable behavior patterns that are genuinely helping the agent choose the exact correct option.
|
||||||
|
|
||||||
|
## Rules
|
||||||
|
- Focus on broadly useful reasoning behaviors.
|
||||||
|
- Prefer patterns about exact comparison of options, quantifiers, and equality conditions.
|
||||||
|
- Do not add theorem-specific facts.
|
||||||
|
- "edits" may be empty if the skill already captures the useful patterns.
|
||||||
|
|
||||||
|
Respond ONLY with a valid JSON object:
|
||||||
|
{
|
||||||
|
"batch_size": <number>,
|
||||||
|
"success_patterns": ["<pattern 1>", "<pattern 2>"],
|
||||||
|
"patch": {
|
||||||
|
"reasoning": "<why these patterns matter>",
|
||||||
|
"edits": [
|
||||||
|
{"op": "append", "content": "<markdown>"},
|
||||||
|
{"op": "insert_after", "target": "<heading/text>", "content": "<markdown>"},
|
||||||
|
{"op": "replace", "target": "<old text>", "content": "<new text>"},
|
||||||
|
{"op": "delete", "target": "<exact text to remove>"}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user