Compare commits
3 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 microsoft-github-policy-service[bot]
|
3088550b02 | ||
|
carpedkm
|
eb4d1ee8d0 | ||
|
microsoft-github-operations[bot]
|
874404df81 |
@@ -1,29 +0,0 @@
|
||||
# SkillOpt Environment Variables
|
||||
# Copy this file to .env and fill in your values.
|
||||
# Usage: set -a; source .env; set +a
|
||||
|
||||
# ── Azure OpenAI (required for openai_chat backend) ──────────────────
|
||||
export AZURE_OPENAI_ENDPOINT=https://your-resource.openai.azure.com/
|
||||
export AZURE_OPENAI_API_VERSION=2024-12-01-preview
|
||||
# Authentication: choose one method
|
||||
# Option 1: API Key
|
||||
export AZURE_OPENAI_API_KEY=
|
||||
# Option 2: Azure CLI (no API key needed, recommended on Azure VMs)
|
||||
# export AZURE_OPENAI_AUTH_MODE=azure_cli
|
||||
# Option 3: Managed Identity
|
||||
# export AZURE_OPENAI_AUTH_MODE=managed_identity
|
||||
# export AZURE_OPENAI_MANAGED_IDENTITY_CLIENT_ID=your-client-id
|
||||
|
||||
# ── OpenAI-compatible endpoints ──────────────────────────────────────
|
||||
# Set AUTH_MODE to openai_compatible and reuse AZURE_OPENAI_ENDPOINT / _API_KEY.
|
||||
# The plain OpenAI client is used; no Azure auth, no api-version header.
|
||||
# export AZURE_OPENAI_ENDPOINT=https://api.openai.com/v1
|
||||
# export AZURE_OPENAI_API_KEY=sk-...
|
||||
# export AZURE_OPENAI_AUTH_MODE=openai_compatible
|
||||
|
||||
# ── Anthropic / Claude (for claude_chat backend) ─────────────────────
|
||||
# export ANTHROPIC_API_KEY=sk-ant-...
|
||||
|
||||
# ── Qwen Local Model (for qwen_chat backend) ────────────────────────
|
||||
# export QWEN_CHAT_BASE_URL=http://localhost:8000/v1
|
||||
# export QWEN_CHAT_MODEL=Qwen/Qwen3.5-4B
|
||||
+423
-36
@@ -1,42 +1,429 @@
|
||||
__pycache__/
|
||||
## Ignore Visual Studio temporary files, build results, and
|
||||
## files generated by popular Visual Studio add-ons.
|
||||
##
|
||||
## Get latest from https://github.com/github/gitignore/blob/main/VisualStudio.gitignore
|
||||
|
||||
# User-specific files
|
||||
*.rsuser
|
||||
*.suo
|
||||
*.user
|
||||
*.userosscache
|
||||
*.sln.docstates
|
||||
*.env
|
||||
|
||||
# User-specific files (MonoDevelop/Xamarin Studio)
|
||||
*.userprefs
|
||||
|
||||
# Mono auto generated files
|
||||
mono_crash.*
|
||||
|
||||
# Build results
|
||||
[Dd]ebug/
|
||||
[Dd]ebugPublic/
|
||||
[Rr]elease/
|
||||
[Rr]eleases/
|
||||
|
||||
[Dd]ebug/x64/
|
||||
[Dd]ebugPublic/x64/
|
||||
[Rr]elease/x64/
|
||||
[Rr]eleases/x64/
|
||||
bin/x64/
|
||||
obj/x64/
|
||||
|
||||
[Dd]ebug/x86/
|
||||
[Dd]ebugPublic/x86/
|
||||
[Rr]elease/x86/
|
||||
[Rr]eleases/x86/
|
||||
bin/x86/
|
||||
obj/x86/
|
||||
|
||||
[Ww][Ii][Nn]32/
|
||||
[Aa][Rr][Mm]/
|
||||
[Aa][Rr][Mm]64/
|
||||
[Aa][Rr][Mm]64[Ee][Cc]/
|
||||
bld/
|
||||
[Oo]bj/
|
||||
[Oo]ut/
|
||||
[Ll]og/
|
||||
[Ll]ogs/
|
||||
|
||||
# Build results on 'Bin' directories
|
||||
**/[Bb]in/*
|
||||
# Uncomment if you have tasks that rely on *.refresh files to move binaries
|
||||
# (https://github.com/github/gitignore/pull/3736)
|
||||
#!**/[Bb]in/*.refresh
|
||||
|
||||
# Visual Studio 2015/2017 cache/options directory
|
||||
.vs/
|
||||
# Uncomment if you have tasks that create the project's static files in wwwroot
|
||||
#wwwroot/
|
||||
|
||||
# Visual Studio 2017 auto generated files
|
||||
Generated\ Files/
|
||||
|
||||
# MSTest test Results
|
||||
[Tt]est[Rr]esult*/
|
||||
[Bb]uild[Ll]og.*
|
||||
*.trx
|
||||
|
||||
# NUnit
|
||||
*.VisualState.xml
|
||||
TestResult.xml
|
||||
nunit-*.xml
|
||||
|
||||
# Approval Tests result files
|
||||
*.received.*
|
||||
|
||||
# Build Results of an ATL Project
|
||||
[Dd]ebugPS/
|
||||
[Rr]eleasePS/
|
||||
dlldata.c
|
||||
|
||||
# Benchmark Results
|
||||
BenchmarkDotNet.Artifacts/
|
||||
|
||||
# .NET Core
|
||||
project.lock.json
|
||||
project.fragment.lock.json
|
||||
artifacts/
|
||||
.artifacts/
|
||||
|
||||
# ASP.NET Scaffolding
|
||||
ScaffoldingReadMe.txt
|
||||
|
||||
# StyleCop
|
||||
StyleCopReport.xml
|
||||
|
||||
# Files built by Visual Studio
|
||||
*_i.c
|
||||
*_p.c
|
||||
*_h.h
|
||||
*.ilk
|
||||
*.meta
|
||||
*.obj
|
||||
*.idb
|
||||
*.iobj
|
||||
*.pch
|
||||
*.pdb
|
||||
*.ipdb
|
||||
*.pgc
|
||||
*.pgd
|
||||
*.rsp
|
||||
# but not Directory.Build.rsp, as it configures directory-level build defaults
|
||||
!Directory.Build.rsp
|
||||
*.sbr
|
||||
*.tlb
|
||||
*.tli
|
||||
*.tlh
|
||||
*.tmp
|
||||
*.tmp_proj
|
||||
*_wpftmp.csproj
|
||||
*.log
|
||||
*.tlog
|
||||
*.vspscc
|
||||
*.vssscc
|
||||
.builds
|
||||
*.pidb
|
||||
*.svclog
|
||||
*.scc
|
||||
|
||||
# Chutzpah Test files
|
||||
_Chutzpah*
|
||||
|
||||
# Visual C++ cache files
|
||||
ipch/
|
||||
*.aps
|
||||
*.ncb
|
||||
*.opendb
|
||||
*.opensdf
|
||||
*.sdf
|
||||
*.cachefile
|
||||
*.VC.db
|
||||
*.VC.VC.opendb
|
||||
|
||||
# Visual Studio profiler
|
||||
*.psess
|
||||
*.vsp
|
||||
*.vspx
|
||||
*.sap
|
||||
|
||||
# Visual Studio Trace Files
|
||||
*.e2e
|
||||
|
||||
# TFS 2012 Local Workspace
|
||||
$tf/
|
||||
|
||||
# Guidance Automation Toolkit
|
||||
*.gpState
|
||||
|
||||
# ReSharper is a .NET coding add-in
|
||||
_ReSharper*/
|
||||
*.[Rr]e[Ss]harper
|
||||
*.DotSettings.user
|
||||
|
||||
# TeamCity is a build add-in
|
||||
_TeamCity*
|
||||
|
||||
# DotCover is a Code Coverage Tool
|
||||
*.dotCover
|
||||
|
||||
# AxoCover is a Code Coverage Tool
|
||||
.axoCover/*
|
||||
!.axoCover/settings.json
|
||||
|
||||
# Coverlet is a free, cross platform Code Coverage Tool
|
||||
coverage*.json
|
||||
coverage*.xml
|
||||
coverage*.info
|
||||
|
||||
# Visual Studio code coverage results
|
||||
*.coverage
|
||||
*.coveragexml
|
||||
|
||||
# NCrunch
|
||||
_NCrunch_*
|
||||
.NCrunch_*
|
||||
.*crunch*.local.xml
|
||||
nCrunchTemp_*
|
||||
|
||||
# MightyMoose
|
||||
*.mm.*
|
||||
AutoTest.Net/
|
||||
|
||||
# Web workbench (sass)
|
||||
.sass-cache/
|
||||
|
||||
# Installshield output folder
|
||||
[Ee]xpress/
|
||||
|
||||
# DocProject is a documentation generator add-in
|
||||
DocProject/buildhelp/
|
||||
DocProject/Help/*.HxT
|
||||
DocProject/Help/*.HxC
|
||||
DocProject/Help/*.hhc
|
||||
DocProject/Help/*.hhk
|
||||
DocProject/Help/*.hhp
|
||||
DocProject/Help/Html2
|
||||
DocProject/Help/html
|
||||
|
||||
# Click-Once directory
|
||||
publish/
|
||||
|
||||
# Publish Web Output
|
||||
*.[Pp]ublish.xml
|
||||
*.azurePubxml
|
||||
# Note: Comment the next line if you want to checkin your web deploy settings,
|
||||
# but database connection strings (with potential passwords) will be unencrypted
|
||||
*.pubxml
|
||||
*.publishproj
|
||||
|
||||
# Microsoft Azure Web App publish settings. Comment the next line if you want to
|
||||
# checkin your Azure Web App publish settings, but sensitive information contained
|
||||
# in these scripts will be unencrypted
|
||||
PublishScripts/
|
||||
|
||||
# NuGet Packages
|
||||
*.nupkg
|
||||
# NuGet Symbol Packages
|
||||
*.snupkg
|
||||
# The packages folder can be ignored because of Package Restore
|
||||
**/[Pp]ackages/*
|
||||
# except build/, which is used as an MSBuild target.
|
||||
!**/[Pp]ackages/build/
|
||||
# Uncomment if necessary however generally it will be regenerated when needed
|
||||
#!**/[Pp]ackages/repositories.config
|
||||
# NuGet v3's project.json files produces more ignorable files
|
||||
*.nuget.props
|
||||
*.nuget.targets
|
||||
|
||||
# Microsoft Azure Build Output
|
||||
csx/
|
||||
*.build.csdef
|
||||
|
||||
# Microsoft Azure Emulator
|
||||
ecf/
|
||||
rcf/
|
||||
|
||||
# Windows Store app package directories and files
|
||||
AppPackages/
|
||||
BundleArtifacts/
|
||||
Package.StoreAssociation.xml
|
||||
_pkginfo.txt
|
||||
*.appx
|
||||
*.appxbundle
|
||||
*.appxupload
|
||||
|
||||
# Visual Studio cache files
|
||||
# files ending in .cache can be ignored
|
||||
*.[Cc]ache
|
||||
# but keep track of directories ending in .cache
|
||||
!?*.[Cc]ache/
|
||||
|
||||
# Others
|
||||
ClientBin/
|
||||
~$*
|
||||
*~
|
||||
*.dbmdl
|
||||
*.dbproj.schemaview
|
||||
*.jfm
|
||||
*.pfx
|
||||
*.publishsettings
|
||||
orleans.codegen.cs
|
||||
|
||||
# Including strong name files can present a security risk
|
||||
# (https://github.com/github/gitignore/pull/2483#issue-259490424)
|
||||
#*.snk
|
||||
|
||||
# Since there are multiple workflows, uncomment next line to ignore bower_components
|
||||
# (https://github.com/github/gitignore/pull/1529#issuecomment-104372622)
|
||||
#bower_components/
|
||||
|
||||
# RIA/Silverlight projects
|
||||
Generated_Code/
|
||||
|
||||
# Backup & report files from converting an old project file
|
||||
# to a newer Visual Studio version. Backup files are not needed,
|
||||
# because we have git ;-)
|
||||
_UpgradeReport_Files/
|
||||
Backup*/
|
||||
UpgradeLog*.XML
|
||||
UpgradeLog*.htm
|
||||
ServiceFabricBackup/
|
||||
*.rptproj.bak
|
||||
|
||||
# SQL Server files
|
||||
*.mdf
|
||||
*.ldf
|
||||
*.ndf
|
||||
|
||||
# Business Intelligence projects
|
||||
*.rdl.data
|
||||
*.bim.layout
|
||||
*.bim_*.settings
|
||||
*.rptproj.rsuser
|
||||
*- [Bb]ackup.rdl
|
||||
*- [Bb]ackup ([0-9]).rdl
|
||||
*- [Bb]ackup ([0-9][0-9]).rdl
|
||||
|
||||
# Microsoft Fakes
|
||||
FakesAssemblies/
|
||||
|
||||
# GhostDoc plugin setting file
|
||||
*.GhostDoc.xml
|
||||
|
||||
# Node.js Tools for Visual Studio
|
||||
.ntvs_analysis.dat
|
||||
node_modules/
|
||||
|
||||
# Visual Studio 6 build log
|
||||
*.plg
|
||||
|
||||
# Visual Studio 6 workspace options file
|
||||
*.opt
|
||||
|
||||
# Visual Studio 6 auto-generated workspace file (contains which files were open etc.)
|
||||
*.vbw
|
||||
|
||||
# Visual Studio 6 workspace and project file (working project files containing files to include in project)
|
||||
*.dsw
|
||||
*.dsp
|
||||
|
||||
# Visual Studio 6 technical files
|
||||
*.ncb
|
||||
*.aps
|
||||
|
||||
# Visual Studio LightSwitch build output
|
||||
**/*.HTMLClient/GeneratedArtifacts
|
||||
**/*.DesktopClient/GeneratedArtifacts
|
||||
**/*.DesktopClient/ModelManifest.xml
|
||||
**/*.Server/GeneratedArtifacts
|
||||
**/*.Server/ModelManifest.xml
|
||||
_Pvt_Extensions
|
||||
|
||||
# Paket dependency manager
|
||||
**/.paket/paket.exe
|
||||
paket-files/
|
||||
|
||||
# FAKE - F# Make
|
||||
**/.fake/
|
||||
|
||||
# CodeRush personal settings
|
||||
**/.cr/personal
|
||||
|
||||
# Python Tools for Visual Studio (PTVS)
|
||||
**/__pycache__/
|
||||
*.pyc
|
||||
*.egg-info/
|
||||
build/
|
||||
dist/
|
||||
site/
|
||||
|
||||
data/
|
||||
outputs/
|
||||
logs/
|
||||
external/
|
||||
# Cake - Uncomment if you are using it
|
||||
#tools/**
|
||||
#!tools/packages.config
|
||||
|
||||
/BabyVision/
|
||||
/MMRB/
|
||||
/SpreadsheetBench/
|
||||
/dl4ir-searchQA/
|
||||
# Tabs Studio
|
||||
*.tss
|
||||
|
||||
configs/local/
|
||||
configs/**/*.local.yaml
|
||||
*.local.md
|
||||
*.secret.md
|
||||
*.bak
|
||||
# Telerik's JustMock configuration file
|
||||
*.jmconfig
|
||||
|
||||
.env
|
||||
.secrets/
|
||||
.codex_azure*/
|
||||
# BizTalk build output
|
||||
*.btp.cs
|
||||
*.btm.cs
|
||||
*.odx.cs
|
||||
*.xsd.cs
|
||||
|
||||
# Internal docs (not for open-source release)
|
||||
docs/ablation_plan.md
|
||||
docs/ablation_paper_tables.md
|
||||
docs/ablation_paper_tables.html
|
||||
docs/experiment_commands.md
|
||||
docs/slow_update_flowchart.md
|
||||
docs/session_memory.md
|
||||
docs/harness_fresh_machine_handoff.md
|
||||
docs/harness_monitoring_memory.md
|
||||
docs/harness_reproduction_secrets.secret.md
|
||||
docs/reflact_conda_env_export.yml
|
||||
docs/reflact_overview.html
|
||||
docs/render_ablation_paper_tables.py
|
||||
docs/让*
|
||||
.gradio/
|
||||
# OpenCover UI analysis results
|
||||
OpenCover/
|
||||
|
||||
# Azure Stream Analytics local run output
|
||||
ASALocalRun/
|
||||
|
||||
# MSBuild Binary and Structured Log
|
||||
*.binlog
|
||||
MSBuild_Logs/
|
||||
|
||||
# AWS SAM Build and Temporary Artifacts folder
|
||||
.aws-sam
|
||||
|
||||
# NVidia Nsight GPU debugger configuration file
|
||||
*.nvuser
|
||||
|
||||
# MFractors (Xamarin productivity tool) working folder
|
||||
**/.mfractor/
|
||||
|
||||
# Local History for Visual Studio
|
||||
**/.localhistory/
|
||||
|
||||
# Visual Studio History (VSHistory) files
|
||||
.vshistory/
|
||||
|
||||
# BeatPulse healthcheck temp database
|
||||
healthchecksdb
|
||||
|
||||
# Backup folder for Package Reference Convert tool in Visual Studio 2017
|
||||
MigrationBackup/
|
||||
|
||||
# Ionide (cross platform F# VS Code tools) working folder
|
||||
**/.ionide/
|
||||
|
||||
# Fody - auto-generated XML schema
|
||||
FodyWeavers.xsd
|
||||
|
||||
# VS Code files for those working on multiple tools
|
||||
.vscode/*
|
||||
!.vscode/settings.json
|
||||
!.vscode/tasks.json
|
||||
!.vscode/launch.json
|
||||
!.vscode/extensions.json
|
||||
!.vscode/*.code-snippets
|
||||
|
||||
# Local History for Visual Studio Code
|
||||
.history/
|
||||
|
||||
# Built Visual Studio Code Extensions
|
||||
*.vsix
|
||||
|
||||
# Windows Installer files from build outputs
|
||||
*.cab
|
||||
*.msi
|
||||
*.msix
|
||||
*.msm
|
||||
*.msp
|
||||
|
||||
@@ -1,43 +0,0 @@
|
||||
# Contributing to SkillOpt
|
||||
|
||||
Thank you for your interest in contributing! SkillOpt welcomes contributions of all kinds.
|
||||
|
||||
## Getting Started
|
||||
|
||||
```bash
|
||||
git clone https://github.com/microsoft/SkillOpt.git
|
||||
cd SkillOpt
|
||||
pip install -e ".[dev]"
|
||||
```
|
||||
|
||||
## How to Contribute
|
||||
|
||||
### 🐛 Bug Reports
|
||||
Open a GitHub issue with reproduction steps, expected/actual behavior, and your config file (remove API keys).
|
||||
|
||||
### 🔧 Add a Benchmark
|
||||
See the [guide](docs/guide/new-benchmark.md) and use the scaffold at `skillopt/envs/_template/`.
|
||||
|
||||
### 🤖 Add a Model Backend
|
||||
See the [guide](docs/guide/new-backend.md).
|
||||
|
||||
### 📝 Improve Documentation
|
||||
```bash
|
||||
pip install -e ".[docs]"
|
||||
mkdocs serve # Preview at http://localhost:8000
|
||||
```
|
||||
|
||||
## Pull Request Process
|
||||
|
||||
1. Fork the repo and create a feature branch
|
||||
2. Make changes and test with an existing benchmark
|
||||
3. Submit a PR with a clear description
|
||||
4. Ensure CI passes
|
||||
|
||||
## Code Style
|
||||
- Follow existing patterns in the codebase
|
||||
- Use type hints for function signatures
|
||||
- Keep docstrings concise
|
||||
|
||||
## License
|
||||
By contributing, you agree your contributions are licensed under the [MIT License](LICENSE).
|
||||
@@ -1,21 +0,0 @@
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2026 Microsoft Corporation
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
@@ -1,288 +0,0 @@
|
||||
# SkillOpt: Executive Strategy for Self-Evolving Agent Skills
|
||||
|
||||
*Train agent skills like you train neural networks — with epochs, (mini-)batchsize, learning rates, and validation gates — but without touching model weights.*
|
||||
|
||||
[](https://microsoft.github.io/SkillOpt/) [](https://arxiv.org/abs/2605.23904) [](https://youtu.be/JUBMDTCiM0M) [](https://www.python.org/) [](LICENSE)
|
||||
|
||||
## 🎬 SkillOpt Demo Video
|
||||
|
||||
https://github.com/user-attachments/assets/eb12d3bc-371c-467f-904d-91b61f339ed7
|
||||
|
||||
<p align="center">
|
||||
<a href="https://youtu.be/JUBMDTCiM0M"><b>▶ Watch the full demo on YouTube</b></a>
|
||||
</p>
|
||||
|
||||
---
|
||||
|
||||
## Documentation
|
||||
|
||||
A complete, self-contained **Documentation & Reproduction Guide** lives at
|
||||
[`docs/guideline.html`](docs/guideline.html). It covers installation, data
|
||||
preparation, training/eval commands, the full configuration reference, the
|
||||
framework internals (training loop, validation gate, slow update, meta skill),
|
||||
and an API/function reference — all in a single page with a left navigation
|
||||
sidebar.
|
||||
|
||||
Because GitHub shows raw source for `.html` files instead of rendering them,
|
||||
open the guide one of these ways:
|
||||
|
||||
- **Locally** — clone the repo and open `docs/guideline.html` in any browser
|
||||
(no build step required).
|
||||
- **Rendered online (no setup)** — via the htmlpreview proxy:
|
||||
[`htmlpreview.github.io/?…/docs/guideline.html`](https://htmlpreview.github.io/?https://github.com/microsoft/SkillOpt/blob/main/docs/guideline.html)
|
||||
- **GitHub Pages** — the repository's GitHub Pages site already serves the
|
||||
project homepage from the repo root, so the guide is reachable alongside it at
|
||||
`https://microsoft.github.io/SkillOpt/docs/guideline.html` (the homepage at
|
||||
`https://microsoft.github.io/SkillOpt/` is unaffected).
|
||||
|
||||
---
|
||||
|
||||
## Install
|
||||
|
||||
**Requirements:** Python 3.10+
|
||||
|
||||
```bash
|
||||
git clone https://github.com/microsoft/SkillOpt.git
|
||||
cd SkillOpt
|
||||
pip install -e .
|
||||
|
||||
# For ALFWorld benchmark (optional):
|
||||
pip install -e ".[alfworld]"
|
||||
alfworld-download
|
||||
```
|
||||
|
||||
### Configure API Credentials
|
||||
|
||||
```bash
|
||||
cp .env.example .env
|
||||
# Edit .env with your API credentials, then:
|
||||
source .env
|
||||
```
|
||||
|
||||
**Azure OpenAI** (recommended):
|
||||
```bash
|
||||
export AZURE_OPENAI_ENDPOINT="https://your-resource.openai.azure.com/"
|
||||
# Option 1: API key auth
|
||||
export AZURE_OPENAI_API_KEY="your-key"
|
||||
# Option 2: Azure CLI auth (no API key needed)
|
||||
export AZURE_OPENAI_AUTH_MODE="azure_cli"
|
||||
```
|
||||
|
||||
> **Note:** `AZURE_OPENAI_ENDPOINT` is required for all three modes (`api_key`, `azure_cli`,
|
||||
> `openai_compatible`). Without it, all LLM calls will fail.
|
||||
|
||||
**OpenAI-compatible endpoints**:
|
||||
```bash
|
||||
export AZURE_OPENAI_ENDPOINT="https://api.openai.com/v1"
|
||||
export AZURE_OPENAI_API_KEY="sk-..."
|
||||
export AZURE_OPENAI_AUTH_MODE="openai_compatible"
|
||||
```
|
||||
|
||||
This routes all calls through the plain OpenAI Python client (no Azure auth, no `api-version`
|
||||
header).
|
||||
|
||||
> **Note:** SkillOpt reuses the `AZURE_OPENAI_*` env var names even in this mode — there is no
|
||||
> separate `OPENAI_API_KEY` knob.
|
||||
|
||||
**Anthropic Claude**:
|
||||
```bash
|
||||
export ANTHROPIC_API_KEY="sk-ant-..."
|
||||
```
|
||||
|
||||
**Qwen (local vLLM)**:
|
||||
```bash
|
||||
export QWEN_CHAT_BASE_URL="http://localhost:8000/v1"
|
||||
export QWEN_CHAT_MODEL="Qwen/Qwen3.5-4B"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Data Preparation
|
||||
|
||||
SkillOpt expects data in a **split directory** with `train/`, `val/`, `test/` subdirectories, each containing a JSON file (e.g., `items.json`).
|
||||
|
||||
```
|
||||
data/my_split/
|
||||
├── train/items.json
|
||||
├── val/items.json
|
||||
└── test/items.json
|
||||
```
|
||||
|
||||
Each JSON file is an array of task items. The required fields depend on the benchmark. For example, SearchQA items look like:
|
||||
|
||||
```json
|
||||
[
|
||||
{
|
||||
"id": "unique_item_id",
|
||||
"question": "Who wrote the novel ...",
|
||||
"context": "[DOC] relevant passage text ...",
|
||||
"answers": ["expected answer"]
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
See `skillopt/envs/<benchmark>/dataloader.py` for the exact format each benchmark expects.
|
||||
|
||||
> **Note:** Benchmark datasets are not included in this repository. Prepare your own data following the format above.
|
||||
|
||||
### Supported Benchmarks
|
||||
|
||||
| Benchmark | Type | Config |
|
||||
|---|---|---|
|
||||
| SearchQA | QA | `configs/searchqa/default.yaml` |
|
||||
| ALFWorld | Embodied agent | `configs/alfworld/default.yaml` |
|
||||
| DocVQA | Document QA | `configs/docvqa/default.yaml` |
|
||||
| LiveMathematicianBench | Math | `configs/livemathematicianbench/default.yaml` |
|
||||
| SpreadsheetBench | Code generation | `configs/spreadsheetbench/default.yaml` |
|
||||
| OfficeQA | Tool-augmented QA | `configs/officeqa/default.yaml` |
|
||||
|
||||
---
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Training
|
||||
|
||||
```bash
|
||||
# Minimal example — train on SearchQA:
|
||||
python scripts/train.py \
|
||||
--config configs/searchqa/default.yaml \
|
||||
--split_dir /path/to/your/searchqa_split \
|
||||
--azure_openai_endpoint https://your-resource.openai.azure.com/ \
|
||||
--optimizer_model gpt-5.5 \
|
||||
--target_model gpt-5.5
|
||||
|
||||
# Train on LiveMathematicianBench:
|
||||
python scripts/train.py \
|
||||
--config configs/livemathematicianbench/default.yaml \
|
||||
--split_dir /path/to/your/livemath_split \
|
||||
--azure_openai_endpoint https://your-resource.openai.azure.com/ \
|
||||
--optimizer_model gpt-5.5 \
|
||||
--target_model gpt-5.5
|
||||
|
||||
# Train on ALFWorld:
|
||||
python scripts/train.py \
|
||||
--config configs/alfworld/default.yaml \
|
||||
--split_dir /path/to/your/alfworld_split \
|
||||
--azure_openai_endpoint https://your-resource.openai.azure.com/ \
|
||||
--optimizer_model gpt-5.5 \
|
||||
--target_model gpt-5.5
|
||||
```
|
||||
|
||||
Key CLI arguments:
|
||||
|
||||
| Argument | Description | Example |
|
||||
|---|---|---|
|
||||
| `--config` | Benchmark config YAML | `configs/searchqa/default.yaml` |
|
||||
| `--split_dir` | Path to data split directory | `/path/to/split` |
|
||||
| `--azure_openai_endpoint` | Azure OpenAI endpoint URL | `https://your-resource.openai.azure.com/` |
|
||||
| `--optimizer_model` | Optimizer model deployment name | `gpt-5.5` |
|
||||
| `--target_model` | Target model deployment name | `gpt-5.5` |
|
||||
| `--num_epochs` | Number of training epochs | `4` |
|
||||
| `--batch_size` | Batch size per step | `40` |
|
||||
| `--workers` | Parallel rollout workers | `8` |
|
||||
| `--out_root` | Output directory | `outputs/my_run` |
|
||||
|
||||
### Eval Only
|
||||
|
||||
Evaluate a trained skill on specific data splits without training:
|
||||
|
||||
```bash
|
||||
# Evaluate the packaged GPT-5.5 SearchQA skill on the test split:
|
||||
python scripts/eval_only.py \
|
||||
--config configs/searchqa/default.yaml \
|
||||
--skill ckpt/searchqa/gpt5.5_skill.md \
|
||||
--split valid_unseen \
|
||||
--split_dir /path/to/searchqa_split \
|
||||
--azure_openai_endpoint https://your-resource.openai.azure.com/
|
||||
|
||||
# Evaluate on all splits (train + val + test):
|
||||
python scripts/eval_only.py \
|
||||
--config configs/searchqa/default.yaml \
|
||||
--skill ckpt/searchqa/gpt5.5_skill.md \
|
||||
--split all \
|
||||
--split_dir /path/to/searchqa_split \
|
||||
--azure_openai_endpoint https://your-resource.openai.azure.com/
|
||||
```
|
||||
|
||||
To evaluate a skill produced by a training run, replace `--skill` with that
|
||||
run's best-skill path, for example `outputs/my_run/best_skill.md`.
|
||||
|
||||
| Split | Description |
|
||||
|---|---|
|
||||
| `valid_unseen` | Test set |
|
||||
| `valid_seen` | Validation set |
|
||||
| `train` | Training set |
|
||||
| `all` | All splits combined (default) |
|
||||
|
||||
### Output Structure
|
||||
|
||||
Each run writes to a structured output directory:
|
||||
|
||||
```
|
||||
outputs/<run_name>/
|
||||
├── config.json # Flattened runtime config
|
||||
├── history.json # Per-step training history
|
||||
├── runtime_state.json # Resume checkpoint
|
||||
├── best_skill.md # Best validated skill document
|
||||
├── skills/skill_vXXXX.md # Skill snapshot per step
|
||||
├── steps/step_XXXX/ # Per-step artifacts (patches, evals)
|
||||
├── slow_update/epoch_XX/ # Slow update logs
|
||||
└── meta_skill/epoch_XX/ # Meta skill logs
|
||||
```
|
||||
|
||||
Re-running the same command auto-resumes from the last completed step.
|
||||
|
||||
---
|
||||
|
||||
## Community-contributed configs
|
||||
|
||||
These are **not** default SkillOpt settings — they are reference configs
|
||||
contributed by users for specific scenarios. The paper-reported numbers
|
||||
were obtained with the default settings, not these.
|
||||
|
||||
- **`configs/examples/soft_gate.yaml`** *(PR #25, contributed by
|
||||
[@lvbaocheng](https://github.com/lvbaocheng))* — switches the
|
||||
validation gate from exact-match (`hard`) to soft / partial-credit
|
||||
(`soft` or `mixed`). Useful when the held-out **selection split is
|
||||
small** (e.g. ≤ ~10 items) and the **reward is continuous**, where the
|
||||
discrete hard gate often rejects every candidate and training stalls.
|
||||
See the comment at the top of the file for details and when not to use
|
||||
it.
|
||||
|
||||
---
|
||||
|
||||
## WebUI
|
||||
|
||||
Launch the monitoring dashboard (optional):
|
||||
|
||||
```bash
|
||||
pip install -e ".[webui]"
|
||||
python -m skillopt_webui.app
|
||||
```
|
||||
|
||||
| Flag | Default | Description |
|
||||
|---|---|---|
|
||||
| `--port` | 7860 | Server port |
|
||||
| `--host` | `0.0.0.0` | Bind address |
|
||||
| `--share` | off | Create a public Gradio share link |
|
||||
|
||||
```bash
|
||||
# With public share link (useful for remote servers)
|
||||
python -m skillopt_webui.app --share
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Citation
|
||||
|
||||
```bibtex
|
||||
@misc{yang2026skilloptexecutivestrategyselfevolving,
|
||||
title={SkillOpt: Executive Strategy for Self-Evolving Agent Skills},
|
||||
author={Yifan Yang and Ziyang Gong and Weiquan Huang and Qihao Yang and Ziwei Zhou and Zisu Huang and Yan Li and Xuemei Gao and Qi Dai and Bei Liu and Kai Qiu and Yuqing Yang and Dongdong Chen and Xue Yang and Chong Luo},
|
||||
year={2026},
|
||||
eprint={2605.23904},
|
||||
archivePrefix={arXiv},
|
||||
primaryClass={cs.AI},
|
||||
url={https://arxiv.org/abs/2605.23904}
|
||||
}
|
||||
```
|
||||
+1222
File diff suppressed because it is too large
Load Diff
@@ -1,113 +0,0 @@
|
||||
# ALFWorld Embodied Agent Skill
|
||||
|
||||
## Overview
|
||||
This skill guides agents operating in the ALFWorld text-based embodied environment.
|
||||
The agent must complete household tasks by navigating rooms, interacting with objects,
|
||||
and using appliances. Actions must be chosen from the admissible action list provided
|
||||
at each step.
|
||||
|
||||
**Output format**: Always output `<think>...</think>` for reasoning, then `<action>...</action>` for the chosen action.
|
||||
|
||||
---
|
||||
|
||||
## Task Types
|
||||
|
||||
| Type | Goal | Key Steps |
|
||||
|------|------|-----------|
|
||||
| Pick & Place | Put object X in/on receptacle Y | Find X -> take X -> go to Y -> put X in/on Y |
|
||||
| Pick Two & Place | Put two instances of X in/on Y | Find X1 -> take -> place -> find X2 -> take -> place |
|
||||
|
||||
### Pick Two Object Bookkeeping
|
||||
For `pick_two_obj_and_place`, choose one destination receptacle instance once it is opened/usable, and remember it as the target. Both object instances should be placed into that same remembered receptacle. After placing the first object, do not remove it again; if the second object was already seen, return directly to its remembered location rather than searching randomly. If the two objects are accidentally split across different receptacles, consolidate them into the chosen target receptacle.
|
||||
| Examine in Light | Examine object X under desklamp | Find X -> take X -> find desklamp -> use desklamp |
|
||||
|
||||
| Examine in Light detail | Final interaction | While holding X where a desklamp is visible, use the desklamp; do not try to place X on the lamp first. |
|
||||
| Clean & Place | Clean object X and put in/on Y | Find X -> take X -> go to sink -> clean X -> go to Y -> put X |
|
||||
| Heat & Place | Heat object X and put in/on Y | Find X -> take X -> go to microwave -> heat X -> go to Y -> put X |
|
||||
| Cool & Place | Cool object X and put in/on Y | Find X -> take X -> go to fridge -> cool X -> go to Y -> put X |
|
||||
|
||||
---
|
||||
|
||||
## General Principles
|
||||
|
||||
1. **Decompose the task**: Parse the goal into ordered sub-goals (locate, acquire, transform, deliver). Complete each before moving to the next.
|
||||
2. **Systematic exploration**: Search each surface and container exactly once before revisiting. Open closed containers (drawers, cabinets, fridge) before judging them empty.
|
||||
|
||||
- Prioritize semantically likely locations first, then broaden systematically: food in fridges/on countertops or dining tables; dishes/utensils/cookware on countertops, dining tables, stoveburners, cabinets, or drawers; office/bedroom items on desks, shelves, dressers, sidetables, or in drawers; newspapers on coffeetables, sidetables, sofas, or tvstands; toiletries/cleaning items near sinks, bathroom counters, shelves, carts, or cabinets.
|
||||
|
||||
- For portable kitchen targets such as bread, mugs, cups, plates, bowls, and utensils, check broad exposed surfaces early: after one or two empty countertops, try dining tables or other open surfaces before opening many cabinets/drawers. For small office/bedroom targets, alternate drawers with exposed desks, shelves, sidetables, and dressers rather than exhausting drawers first.
|
||||
|
||||
- Keep a persistent **searched set** of receptacle instances, e.g. `drawer 1`, `shelf 3`, `countertop 2`. Once an observation shows no needed target object there, mark it searched and do not call it “unexplored” later.
|
||||
- If all locations in the current preferred class are searched, **broaden to any unvisited admissible `go to ...` location** instead of restarting the same sequence. Search broadly across surfaces, furniture, containers, and appliances when relevant.
|
||||
- If a visible object is itself an openable/container-like object, such as a box, and opening/examining it is admissible, inspect it before leaving the area.
|
||||
3. **Grab immediately**: When a required object is visible and reachable, take it right away before moving elsewhere.
|
||||
|
||||
- Pick up only the exact requested object type. Similar or related objects, such as a cup when the task asks for a mug, a spoon when it asks for a knife, or a pot when it asks for a pan, are distractors; leave them in place and mark that location searched for the target.
|
||||
4. **Transform before placing**: If the task requires cleaning, heating, or cooling, perform the state change at the appropriate appliance before heading to the final destination.
|
||||
|
||||
- Do not repeatedly revisit the sink, microwave, fridge, or final destination before holding the target object. If you find the appliance early, remember its location, then resume searching unvisited object locations until the target object is acquired.
|
||||
|
||||
- Use direct admissible appliance/tool commands immediately when available, such as `clean X with sinkbasin`, `heat X with microwave`, `cool X with fridge`, or `use desklamp`. Do not waste steps opening, closing, toggling, or examining the appliance unless the needed action is unavailable or opening is required for searching/placing.
|
||||
5. **Direct delivery**: Once holding the transformed (or untransformed) goal object, navigate straight to the target receptacle and place it.
|
||||
|
||||
- Remember known destination receptacles and return directly to the same instance after pickup/transformation. If the destination is also a semantically likely source location, check/open it early rather than only after exhaustive search: food may already be in the fridge, utensils may be on the diningtable, newspapers may be on/near the sofa, and a target drawer can be opened early for pick-two tasks. If the object starts at the destination but needs cleaning/heating/cooling, take it out, transform it, then return to that same instance and place it back.
|
||||
6. **Track progress**: Maintain an internal count of how many objects still need to be found and placed. Only stop searching when the count reaches zero.
|
||||
7. **Avoid loops**: Never repeat the same action more than twice in a row. If stuck, move to a different unexplored location.
|
||||
8. **Only choose admissible actions**: Always pick an action from the admissible action list. Do not invent actions.
|
||||
|
||||
---
|
||||
|
||||
## Common Mistakes to Avoid
|
||||
|
||||
- **Revisiting searched locations**: Keep track of which surfaces/containers have been checked; do not re-examine them.
|
||||
- **Ignoring visible objects**: If the target object appears in the observation, pick it up immediately.
|
||||
- **Skipping state changes**: Do not place an object at the destination without first cleaning/heating/cooling it when required.
|
||||
- **Premature termination**: Do not stop the episode until all goal conditions are verified as met.
|
||||
- **Action loops**: Repeatedly toggling or examining the same object wastes steps. Move on to new locations instead.
|
||||
|
||||
### Hard Search-Loop Recovery
|
||||
|
||||
- **Exact-instance lockout before pickup**: once a receptacle/surface instance has been observed and does not contain the target object, do not go back to that exact instance while still searching for the object. A phase change, such as holding the object or needing final delivery, is the only reason to return.
|
||||
- **Fast broadening threshold**: after 3-4 misses in the same receptacle class, switch to a different likely class or any unvisited admissible location instead of continuing or restarting that class, unless the target has already been seen there.
|
||||
- **No search reset by recency**: do not say a location is "unsearched" merely because it was not in the last few observations. The searched set is global for the whole episode.
|
||||
- **Finite-class exhaustion**: if all visible instances of a small class have been checked once, such as all stoveburners, diningtables, countertops, or shelves, mark that class exhausted for object search and do not start a second pass. Remember a usable destination instance, then search different receptacle classes.
|
||||
- **Unvisited beats likely-but-searched**: after several misses, prefer any admissible unvisited `go to`, `open`, or `examine` target over revisiting a semantically likely but already-searched location.
|
||||
- **Destination surfaces before pickup**: if the destination receptacle is also a likely object location, inspect each instance at most once before pickup. If it lacks the object, remember it as the final destination but stop using it as a search target until the object has been transformed and is ready to place.
|
||||
- **Kitchen item fallback**: for cookware and dishware, after checking obvious burners/tables/counters once, broaden to unsearched cabinets, drawers, shelves, sinkbasins, and other kitchen storage/surfaces rather than cycling among the obvious locations.
|
||||
|
||||
### Strict Search Ledger Action Filter
|
||||
|
||||
Before every empty-handed search action, apply this hard filter:
|
||||
|
||||
1. If a required target object is visible, take it immediately.
|
||||
2. Otherwise choose an exact receptacle/surface/container instance whose contents have not yet been observed in the current object-search phase.
|
||||
3. Reject any `go to`, `examine`, or `open` action for an exact instance already observed to lack the target, even if it is semantically likely, nearby, recently mentioned, or the final destination type.
|
||||
4. If all likely instances are rejected by the ledger, broaden to any unvisited admissible location/class instead of restarting from instance 1 of a searched class.
|
||||
|
||||
The searched ledger survives inventory checks, appliance visits, placing the first object in a pick-two task, and putting down an irrelevant inspected object/container. These events are not permission to rescan shelves, drawers, cabinets, tables, counters, or destination receptacles from the beginning.
|
||||
|
||||
### Destination-as-Source Lockout
|
||||
|
||||
When the final receptacle type is also a plausible source location, inspect each visible destination instance at most once before pickup. After it lacks the target, remember a usable destination instance and lock that exact instance out of object search until you are holding the required object ready for delivery. Do not alternate between destination instances and other searched source instances while still empty-handed.
|
||||
|
||||
### Pick-Two Phase Memory
|
||||
|
||||
After placing the first object in a pick-two task, do not begin a fresh room/class search. If another required instance was previously seen, return directly to that remembered source location for the second pickup. If no second instance is remembered, continue from the existing unsearched-location ledger rather than revisiting locations already checked before the first placement.
|
||||
|
||||
<!-- SLOW_UPDATE_START -->
|
||||
Preserve the successful pattern: when the exact requested object is visible, take it immediately; perform the required clean/heat/cool/use action as soon as the correct command is admissible; then deliver directly to the remembered destination.
|
||||
|
||||
Treat tool locations as tools, not repeated search targets. If a sinkbasin, fridge, microwave, desklamp, or destination receptacle has already been checked and does not contain the target while you are empty-handed, remember it for later but do not revisit it until you are holding the required object or ready to place/use it.
|
||||
|
||||
Use a next-unsearched-instance pointer for every numbered class. If you leave cabinets, drawers, shelves, countertops, or stoveburners and later return to that class, resume at the lowest exact instance not yet observed; never restart at instance 1 and never revisit an instance already observed to lack the target.
|
||||
|
||||
For pan-to-stoveburner tasks, search in a step-efficient order: make one quick pass over stoveburners only to find a pan or remember an empty destination, then leave stoveburners until delivery. Next check countertops/islands and sinkbasins. Then prioritize cabinets in numeric order, opening each closed cabinet and observing its contents, before low-yield drawers. Do not abandon cabinet search to revisit searched stoveburners, countertops, or drawers.
|
||||
|
||||
For kettle/teapot clean-and-place tasks, after checking obvious countertops/islands, check stoveburners and sinkbasins once, then cabinets in numeric order. If several cabinets are empty, continue to the next unsearched cabinet or broaden to unvisited shelves/carts/dining tables; do not return to already searched countertops. Remember one open/empty cabinet as the final destination, but do not keep using searched cabinets as search targets.
|
||||
|
||||
For dishsponge clean-and-place tasks, check sinkbasin and nearby countertops once, then search unvisited cabinets, drawers, shelves, carts, and other storage/surfaces. Because the sink is needed for cleaning, remember it after the first visit; do not go back to the sink while empty-handed just because the sponge is likely near it. Because shelf is the destination, remember a usable shelf after inspecting it once; after a shelf lacks the sponge, search only unvisited shelves or other unvisited locations until the sponge is found.
|
||||
|
||||
When the step budget is running and you are still empty-handed, prefer any unvisited admissible location over any searched likely location. A location being semantically likely, useful later, or recently mentioned is never a reason to rescan it before acquisition.
|
||||
|
||||
Do not let the broadening threshold cause class restarts. Broadening means move to a different unvisited class or continue at the next unsearched instance of a promising storage class; it never means cycling back through exact instances already observed.
|
||||
<!-- SLOW_UPDATE_END -->
|
||||
@@ -1,26 +0,0 @@
|
||||
# DocVQA Skill
|
||||
|
||||
## Visual Evidence Discipline
|
||||
- Read the document carefully before answering.
|
||||
- Prefer the smallest exact text span that answers the question.
|
||||
|
||||
- For questions asking for a value, count, page number, date, or graph reading, return only the requested value span; omit nearby labels, category names, units, or explanatory words unless the question explicitly asks for them.
|
||||
- When several nearby strings look similar, choose the one whose surrounding labels or layout best match the question.
|
||||
|
||||
## Exact Answer Discipline
|
||||
- Copy names, numbers, and dates exactly from the document whenever possible.
|
||||
|
||||
- Preserve the document's exact spelling and punctuation for names and quoted phrases; do not substitute similar letters or change straight/curly quotes, spacing, or parentheses when the visible text provides them.
|
||||
- Prefer direct extraction over paraphrase.
|
||||
- Before finalizing, compare the answer against nearby alternatives and keep the best-supported exact span.
|
||||
|
||||
## Structured Layout Lookup
|
||||
- For tables, first find the row or entry named in the question, then read the value under the requested column, header, date, or category; answer with that cell only.
|
||||
- For forms, receipts, or labeled fields, locate the exact role, party, or field label mentioned in the question, then copy the filled-in value from the same line, box, block, or immediately adjacent field.
|
||||
- For table-of-contents, indexed, numbered, or bulleted lists, match the requested title, entry, or point number, then follow the same line or list item to the associated value; do not take a nearby value from another item.
|
||||
|
||||
## Anchored Handwriting / Nearby Text
|
||||
- For handwritten or list/table questions with an anchor term, first locate the anchor, then inspect the immediately adjacent text in the same row, column, or nearby margin. If legible, provide the best-supported nearby span rather than leaving the answer blank.
|
||||
|
||||
<!-- SLOW_UPDATE_START -->
|
||||
<!-- SLOW_UPDATE_END -->
|
||||
@@ -1,35 +0,0 @@
|
||||
# Live Mathematical MCQ Heuristics
|
||||
|
||||
## Option Comparison
|
||||
|
||||
### Meta-Options About Stronger Results
|
||||
- Treat options of the form “one of the remaining options is correct, but a stronger result can be proven” as serious candidates, especially when the question asks for the strongest statement.
|
||||
- If a concrete option is true but your theorem or derivation gives a strictly stronger conclusion not exactly listed, choose the meta-option rather than the weaker concrete statement.
|
||||
- When options are nested by strength, rank them explicitly before answering: e.g. finite-time blowup is stronger than merely “not globally bounded”; positive stable growth is stronger than ordinary unboundedness; sharper constants, rates, exceptional-set bounds, endpoint inclusion, or full equivalences are stronger than weaker asymptotic versions.
|
||||
- Compare all options before committing. The correct choice is often the strongest statement justified by the question, while nearby distractors are weaker, overstrong, or miss an equality case.
|
||||
- Track exact quantifiers such as "there exists", "for every", "if and only if", and "exactly when".
|
||||
|
||||
## Theorem-Level Precision
|
||||
|
||||
- Do not add converse, realization, or classification claims unless the theorem explicitly proves them. Phrases such as “conversely,” “every such parameter occurs,” “if and only if,” or “exactly all” add strength beyond a one-way implication.
|
||||
- Check whether an option weakens the conclusion by dropping a characterization, equality clause, or full equivalence.
|
||||
- Check whether an option overstates the theorem by upgrading regularity, removing scale restrictions, or changing an existential statement into a universal one.
|
||||
|
||||
## Hypotheses
|
||||
|
||||
### Exact Conditions and Thresholds
|
||||
- For biconditional/equivalence questions, reject conditions that are merely necessary or merely sufficient. A broader condition, such as congruence modulo a divisor instead of modulo the full modulus, is usually weaker and not equivalent unless the domain collapses the extra cases.
|
||||
- For threshold conditions, verify the exact sign and endpoint: distinguish \(\mu_0\) from \(-\mu_0\), \(<\) from \(\le\), and whether the equality case belongs to the positive, zero, or negative parameter regime.
|
||||
- When options differ by “for every” vs “for sufficiently large,” local vs global domains, strict vs non-strict inequalities, or dependence of constants, rank them by logical strength and match the sharpest justified version.
|
||||
- Verify the hypotheses and domain carefully. Distractors often keep the theorem shape but alter the required assumptions.
|
||||
- Pay close attention to equality cases, extremal conditions, and whether a result applies to the full family or only a restricted subfamily.
|
||||
|
||||
## Final Answer
|
||||
- Output the final answer as the single option label only.
|
||||
|
||||
## Exact Scope and Quantitative Wording
|
||||
- Distinguish global conclusions from localized or completed ones. Equivalence after localization, completion, or at each prime/scale is usually weaker than an unqualified equivalence.
|
||||
- In estimate-heavy options, compare every quantitative detail: exponent, derivative index range, constants and their parameter dependence, log factors, additive terms, one-sided vs two-sided notation, and pointwise vs uniform convergence.
|
||||
|
||||
<!-- SLOW_UPDATE_START -->
|
||||
<!-- SLOW_UPDATE_END -->
|
||||
@@ -1,50 +0,0 @@
|
||||
# OfficeQA Skill
|
||||
|
||||
## Retrieval Discipline
|
||||
|
||||
- When an external official time-series observation is needed, prefer the source's series/data-download/table page once identified. If exact-date or guessed-value searches return empty results, stop repeating them; broaden to the official series name/code plus `data` or `download` and use the table values.
|
||||
|
||||
- Treat provided/oracle parsed pages as primary evidence: if they contain the relevant table and period, extract directly from them before searching elsewhere; search only for missing continuation pages, missing periods, or an official actual value not present.
|
||||
- Start by narrowing to the most likely candidate file before reading long passages.
|
||||
- Prefer targeted search terms that name the exact entity, period, measure, or table concept from the question.
|
||||
- After a promising match, read only a small surrounding span and verify it matches the requested year, basis, and unit.
|
||||
|
||||
- If the requested date range extends beyond the provided/oracle page, first enumerate the required periods and verify that every period is present in evidence. Do not compute from a partial ledger or fill missing periods from memory; retrieve continuation pages, adjacent issues, or a later issue of the same table that contains the missing dates/revisions.
|
||||
|
||||
## Evidence Discipline
|
||||
- Extract the exact value from the retrieved text before doing any arithmetic.
|
||||
- Keep track of each operand's period, unit, and semantic role so nearby proxy values are not mixed in.
|
||||
|
||||
- For Treasury financing narratives, label each amount by transaction role before calculating: offered amount, tenders/subscriptions received, tenders accepted, competitive/noncompetitive accepted, foreign or Government-account exchange tenders, refunding, and **new cash** are not interchangeable.
|
||||
- When converting currencies or scales, make a direction ledger first: source table unit, source currency, exchange-rate orientation (foreign currency per U.S. dollar means divide by the rate; U.S. dollars per foreign unit means multiply), and requested final unit.
|
||||
|
||||
- For tables, align values by row label and exact column header, not proximity alone; watch for continued or unlabeled columns, footnotes, adjacent amount-versus-percent columns, fiscal-year versus calendar-year sections, and repeated month rows under different year blocks.
|
||||
- If the question asks for a transformed or derived quantity, compute only after confirming every operand.
|
||||
|
||||
- For derived comparisons, preserve the direction and sign implied by the wording: “change from A to B” means B minus A; “former than latter” means former minus latter; “share accounted for by X” means X divided by the stated total; paired “gap” questions require computing each within-row difference before ranking.
|
||||
- For statistical, regression, correlation, and growth-rate questions, write a formula ledger before calculating: confirm the exact series/endpoints, ordered vector, elapsed intervals, and requested convention such as continuously compounded rate, CAGR, Pearson correlation, or OLS index/year choice.
|
||||
- For multi-stage questions where one table determines the period/entity used in another lookup, freeze that derived key with evidence first, then retrieve the second measure only for that exact month/year/reporting date/entity.
|
||||
|
||||
- For inclusive time-series ranges, make a period-by-period ledger covering every requested month/year exactly once, preserving calendar versus fiscal basis, end-of-month or end-of-fiscal-month status, source units, and any specified adjustments.
|
||||
|
||||
- For statistical transforms over time-series windows, confirm endpoint inclusion/exclusion exactly as worded, use consecutive time indices for trend regressions when appropriate, sort values before medians, and for logarithmic growth use ln(final/initial) before converting to the requested percentage format.
|
||||
|
||||
## Final Answer Discipline
|
||||
|
||||
- Before finalizing, enforce the requested unit and format: convert thousands/millions/billions or full nominal dollars as needed, then apply no-comma, fixed-decimal, whole-number, or nearest-tenth/thousandth formatting exactly as asked.
|
||||
- Return the final answer only after one last consistency check against the retrieved evidence.
|
||||
- Copy the final answer from a checked value, not from an unverified intermediate guess.
|
||||
|
||||
## Statistical and Time-Series Calculation Checks
|
||||
|
||||
- Before computing any statistic, write the intended formula and denominator convention. If the prompt explicitly says **population standard deviation**, divide by `n`; if it says **sample**, divide by `n-1`; for a z-score comparing one observation against a small set of comparison months/periods and no population convention is stated, estimate dispersion with the **sample** standard deviation of the comparison set. Do not round intermediate operands, weighted averages, logs, exchange-rate conversions, or standard deviations before the final requested rounding.
|
||||
- For long inclusive ranges, first enumerate the expected count of observations and the first/last period, then verify the ledger has exactly that count. Exclude totals, cumulative-to-date columns, comparable-period columns, estimates, and extra latest-month columns outside the requested calendar or fiscal range.
|
||||
- When a page contains multiple nearby sections with similar labels, use only the section whose title and row label match the requested measure exactly; do not compute from the first visible table if the requested measure/table title is absent or only partially shown.
|
||||
- For Treasury security quotations, obey the table's quote basis. If the table states that price decimals are 32nds, convert quotes such as `99.27` as `99 + 27/32`, not as decimal `99.27`. If a task asks for smoothing, averaging, or forecasting in a target currency using period-specific exchange rates, convert each period's observation to the target currency first unless the prompt explicitly says to compute in the source currency and convert only the final result.
|
||||
|
||||
## Stricter Final Formatting
|
||||
|
||||
- Match any requested output template exactly. Unless the prompt explicitly asks for unit words or explanatory text, return only the numeric value or requested list; do not append words such as `million`, `dollars`, `percent`, or `percentage points`. Include symbols/commas only when the prompt requests currency-formatted output or the answer format clearly requires them.
|
||||
|
||||
<!-- SLOW_UPDATE_START -->
|
||||
<!-- SLOW_UPDATE_END -->
|
||||
@@ -1,71 +0,0 @@
|
||||
# Question Answering Skill
|
||||
|
||||
(No learned rules yet. Rules will be added through the reflection process.)
|
||||
|
||||
## Concise Answer Normalization
|
||||
- Prefer the shortest unambiguous answer that directly satisfies the question. Do not include generic descriptors, legal suffixes, or expanded formal names unless the question specifically asks for the full official name or the descriptor is necessary to identify the entity.
|
||||
|
||||
- If the answer appears inside a longer descriptive phrase, strip words that merely repeat the clue's requested type or modifiers already stated in the clue. For short-answer trivia, return the distinctive core entity or headword rather than role titles, product flavor adjectives, or place/facility designators, even when those words are part of a fuller official phrase, unless the full official name is explicitly requested.
|
||||
- For place/name-etymology questions asking for “the name” or “the word” that means something, answer the distinctive name/word itself rather than a larger phrase with a generic type label.
|
||||
|
||||
- For natural geographic features, preserve conventional feature designators such as “Lake,” “River,” “Bay,” “Gorge,” “Mount,” or “Island” when they are part of the proper name or match the requested feature type. Do not shorten “Lake Okeechobee,” “Tampa Bay,” or “Olduvai Gorge” to an ambiguous base name merely to be concise.
|
||||
- For companies, brands, and organizations, answer the common distinctive name when sufficient; omit additions such as “Company,” “Corporation,” “Inc.,” etc. unless explicitly required.
|
||||
|
||||
- Preserve the answer surface form supported by the strongest evidence when exact variants differ: spelling, capitalization, punctuation, and word order can matter. Do not substitute an equivalent official/common variant such as an alternate spelling or inverted institution name if a direct title/snippet/answer field gives the expected form.
|
||||
|
||||
- When copying titles or quoted names, preserve ordinary ASCII punctuation from the evidence, especially straight apostrophes (`'`). Do not replace them with typographic curly quotes/apostrophes unless that exact stylized form is explicitly shown as the supported answer.
|
||||
|
||||
- For nicknames, epithets, saints, and quoted titles, copy the supported surface form exactly, including spacing, capitalization, and conventional abbreviations such as “St.” Do not normalize a stylized or quoted form into a lowercase dictionary word or an expanded spelling when the clue/evidence points to the stylized answer.
|
||||
|
||||
- For person answers in trivia or crossword-style clues, prefer the conventional supported name. Use just a surname, first name, or saint/regnal name only when the clue/source clearly expects that short form; otherwise use the canonical full personal name from the strongest evidence or answer field, especially when a lone given name would be ambiguous.
|
||||
- Return the grammatical base form expected by the clue. Do not add a plural `s` merely because a crossword source pluralizes a shared name or category; if the clue lists people sharing a first name, answer the singular given name.
|
||||
|
||||
- For common-noun category answers, default to the singular dictionary headword in trivia/crossword-style clues, even if the clue uses plural words like “these,” “those,” “places,” or “items” for grammar. Use a plural only when the term is inherently plural or an answer field/source clearly gives a plural phrase.
|
||||
|
||||
- For common-noun clues about things being replaced, used in place of, or substituted by another system/item, answer the broad headword for the thing replaced unless a narrowing modifier is required by the clue or answer field. Do not add adjectives such as “letter,” “regular,” or “standard” merely because they appear in explanatory context.
|
||||
- For fill-in-the-blank or definitional clues using words like “this” or “that,” provide a standalone noun phrase. Avoid context-dependent pronouns or possessives from the source text; use a natural article such as “the” when needed (e.g., answer “the highest point,” not “its highest point”).
|
||||
|
||||
## Context-Grounded Evidence Matching
|
||||
- Start by identifying the most distinctive terms in the question: proper names, dates, titles, quoted phrases, unusual words, roles, relationships, and category descriptors.
|
||||
- Prioritize passages or document titles where several distinctive clue terms occur together, especially if the wording directly repeats or closely paraphrases the question.
|
||||
- Treat document titles as useful evidence: the answer is often named in a title while the snippet confirms the clue facts.
|
||||
|
||||
- Do not assume the document title itself is the answer. If the requested type differs from the title entity, use the title as context and extract the matching typed entity from the snippet or clue relationship.
|
||||
|
||||
- For “known as,” “called,” “defined as,” or category/type clues, choose the canonical term explicitly used in the strongest matching title/snippet or scraped answer field rather than inventing a related derivative or near-synonym from the clue wording. When multiple plausible candidates appear, prefer the candidate whose evidence directly states the requested relationship and repeats the most distinctive clue facts.
|
||||
- Ignore noisy results that only match generic words; prefer evidence that directly connects the clue facts to one specific entity.
|
||||
|
||||
## Clue Interpretation and Answer Type
|
||||
- For Jeopardy-style wording such as “this man,” “this group,” “this film,” “this country,” “this system,” “he,” or “his wife,” infer the expected answer type before choosing the answer.
|
||||
- Use that expected type to validate candidates: answer with the concise person, place, title, organization, object, term, or phrase requested by the clue.
|
||||
|
||||
- Treat modifiers attached to the requested type as hard filters, not background flavor: constraints like dates, “largest,” “2-letter-named,” “1978 remake,” “hot dog brand,” “dual throne,” or “on this company’s board” must all fit the candidate before you answer.
|
||||
- For clues centered on creative works such as books, films, plays, songs, poems, or other media, first determine whether the clue asks for the work itself, its creator, a performer or cast member, a character, a quotation source, or a setting. Verbs such as “wrote,” “directed,” “stars,” “played,” and “set in,” plus pronouns like “he” or “her,” usually determine the target.
|
||||
|
||||
- For fill-in-style clues with placeholders such as “this,” “these,” or “one of these,” substitute each candidate back into the clue and choose the concise answer that makes the full phrase, title, or fact read correctly.
|
||||
- For terse clues that are just examples or names separated by commas, slashes, or “or,” infer the shared category, class, or synonym that links them, then answer with that concise common term.
|
||||
|
||||
- For crossword-style clues, treat parenthetical numbers or stated letter counts as hard constraints on the answer length, and omit generic labels that would violate them. In dual-definition clues using wording like “X, or what Y does,” choose the single word that satisfies both senses and preserve the required inflected form.
|
||||
- If the clue references an unavailable image or link with wording like “seen here,” “pictured,” or parenthetical visual hints, rely on the textual clues and context to infer the answer; do not treat the missing image as necessary evidence.
|
||||
- If multiple snippets support the same entity, use that corroboration to choose the canonical/common form of the answer.
|
||||
|
||||
## Trivia / Jeopardy Snippet Formats
|
||||
- Retrieved trivia snippets may contain the clue and answer in scraped formats such as `CATEGORY | clue | answer`, `clue. ANSWER`, or labels like `right:`.
|
||||
- When the question text matches the clue in such a snippet, extract the answer field or adjacent answer name, not the category or the whole clue sentence.
|
||||
|
||||
## Common Clue Traps
|
||||
- Watch for inverse relationships: if the clue says “His third wife was Jiang Qing,” the requested answer is the husband, not Jiang Qing.
|
||||
|
||||
- More generally, preserve relation direction in clues: “A is evidence of this B,” “A is related to this language,” or “home to these characters” asks for the target of the relationship, not the entity already named in the clue.
|
||||
|
||||
- When a clue says examples, models, breeds, members, or items “include,” “like,” or “such as” named entities, treat those names as evidence for the requested parent class or entity. Answer the encompassing brand, animal, category, place, or term requested by “this,” not one of the examples already given.
|
||||
- If the question gives the start of a quotation or phrase, answer with the exact missing continuation from the context.
|
||||
|
||||
- For song, poem, nursery-rhyme, or quotation clues, first decide whether the question asks for a missing word or phrase from the quote or for the associated creator, performer, or work; use pronouns and answer-type signals to choose the right target.
|
||||
- When a clue asks for a constrained form such as a first name, abbreviation, acronym, or lyric word, return that exact form rather than the fuller person, title, or explanation; preserve conventional punctuation or spelling when it is part of the requested form.
|
||||
- If the clue contains wordplay, quotation marks, or puns, treat them as hints, but answer with the real entity supported by the evidence.
|
||||
|
||||
- If a clue includes a quoted title, quoted narration or lyric, named event, slogan, or other distinctive phrase but asks for an associated “this” entity, treat the quote or name as evidence to identify the requested person, work, place, group, category, source, or term; do not return the quoted anchor unless the clue explicitly asks for it.
|
||||
|
||||
<!-- SLOW_UPDATE_START -->
|
||||
<!-- SLOW_UPDATE_END -->
|
||||
@@ -1,133 +0,0 @@
|
||||
# Spreadsheet Manipulation Skill (xlsx)
|
||||
|
||||
## Overview
|
||||
This skill guides agents in manipulating Excel (.xlsx) spreadsheets using Python.
|
||||
|
||||
**Primary libraries**: `openpyxl` (structure-preserving read/write), `pandas` (data transformation).
|
||||
Never use any other third-party libraries.
|
||||
|
||||
---
|
||||
|
||||
## Common Workflow
|
||||
|
||||
1. **Explore** the input file: list sheets, inspect headers, check dimensions.
|
||||
|
||||
- Inspect actual workbook data beyond the preview, including nearby rows/columns, sample outputs, formulas, labels, headers, and any reference/example sheets such as `Output`, `Manual Result`, or `Desired...` tabs.
|
||||
|
||||
- Treat existing filled cells in the requested output area or adjacent example tables as semantic examples for edge cases and expected formats, but still recompute and write the complete requested target range.
|
||||
- Scan the used range for complete header groups, not just row 1. Tables may start in later rows/columns, have title rows above them, or have multiple source/result tables on the same sheet; use nearby labels and the requested output range to distinguish sources from destinations.
|
||||
- Locate tables, fields, and target ranges by header text, nearby labels, and surrounding nonblank structure rather than fixed coordinates. Build header maps from actual cells when useful, e.g. `{str(cell.value).strip(): cell.column}`.
|
||||
2. **Write `solution.py`** with `INPUT_PATH` and `OUTPUT_PATH` defined at the top.
|
||||
3. **Execute** `python solution.py` and verify the output file was created.
|
||||
4. **Confirm** the target cells/range contain the expected values.
|
||||
|
||||
---
|
||||
|
||||
## Library Selection
|
||||
|
||||
| Use case | Library |
|
||||
|----------|---------|
|
||||
| Preserve formulas, formatting, named ranges | `openpyxl` |
|
||||
| Bulk data transformation, aggregation, sorting | `pandas` → write back with `openpyxl` |
|
||||
| Simple cell read/write | `openpyxl` |
|
||||
|
||||
**Warning**: `pandas.to_excel()` silently destroys existing formulas and named ranges.
|
||||
When writing back to a spreadsheet that contains formulas, always use `openpyxl.save()`.
|
||||
|
||||
**Formula evaluation caution**: `openpyxl` can write formulas but does **not** calculate them or update cached results. If the requested output will be checked as cell values, compute the result in Python and write literal values unless the user explicitly requires live formulas. When existing formulas are inputs to your logic, load a second workbook with `data_only=True` to read cached values while saving changes through the normal workbook:
|
||||
|
||||
```python
|
||||
wb = openpyxl.load_workbook(INPUT_PATH)
|
||||
wb_values = openpyxl.load_workbook(INPUT_PATH, data_only=True)
|
||||
ws = wb["Sheet1"]
|
||||
ws_values = wb_values["Sheet1"]
|
||||
```
|
||||
|
||||
Treat wording such as “write/fix a formula,” “SUMIFS/COUNTIFS,” “VBA,” or “macro” as a description of the spreadsheet logic unless the deliverable explicitly requires live formula text, an `.xlsm`, or a preserved VBA project. For normal `.xlsx` outputs, implement the equivalent logic in Python/openpyxl and write the computed final values to the requested cells so verification does not depend on Excel recalculation or macros.
|
||||
|
||||
When the user provides an existing or broken formula, use it as a semantic specification: honor its referenced lookup ranges, criteria ranges, return ranges, aggregation intent, and error-handling behavior, then write the resulting values rather than guessing different source columns or leaving unevaluated formulas.
|
||||
|
||||
---
|
||||
|
||||
## solution.py Template
|
||||
|
||||
```python
|
||||
import openpyxl
|
||||
import pandas as pd
|
||||
|
||||
INPUT_PATH = "..." # set to the actual input path
|
||||
OUTPUT_PATH = "..." # set to the actual output path
|
||||
|
||||
wb = openpyxl.load_workbook(INPUT_PATH)
|
||||
ws = wb.active # or wb["SheetName"]
|
||||
|
||||
# --- perform manipulation ---
|
||||
|
||||
wb.save(OUTPUT_PATH)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Output Requirements
|
||||
|
||||
- Save the result to `OUTPUT_PATH`.
|
||||
- Do not hardcode row counts or column letters — iterate over actual rows in the workbook.
|
||||
- Preserve sheets and cells not mentioned in the instruction.
|
||||
|
||||
## Matching and Target Range Hygiene
|
||||
|
||||
- Choose the comparison operator from the instruction and examples: use `startswith` for “begins with”, substring search for “contains/search/occurrence”, and exact normalized equality only when a whole-cell match is implied.
|
||||
- Create small helper functions for comparisons and numeric parsing. Normalize text by trimming, collapsing repeated spaces/NBSPs, and casefolding; when names or labels have punctuation/spacing inconsistencies, consider punctuation-insensitive keys. Parse numeric text after removing commas/currency symbols while preserving signs and decimal points; skip `None`/blank and booleans for numeric tests, and handle placeholders such as `"-"`, `"$"`, `"$0"`, blanks, and numeric zero deliberately.
|
||||
- Normalize date keys deliberately: handle `datetime`/`date` objects, Excel serial numbers, and date-like strings, then compare at the granularity implied by the task, such as exact date, month, month/year, fiscal period, or year. For workday/date-window logic, compute the range in Python and exclude weekends/holidays as specified.
|
||||
|
||||
- For monthly or period summary grids, canonicalize period labels from all sources: sheet names, title text, row/column headers, text months such as `March`, and actual date cells. Match summaries by normalized period plus the other stated criteria rather than by fixed month offsets or existing formulas.
|
||||
- For date ranges and rolling windows, infer endpoint inclusivity from wording and examples. Phrases like `X to Y`, `through`, and `up to`, or examples such as `2 to 5` meaning `4 days`, usually require inclusive boundary handling.
|
||||
- For time extraction or time-threshold logic, parse `datetime`, `time`, Excel serial/fractional times, and time-like strings into real Python `time`/`datetime` values. Write real time values with an Excel `number_format` such as `hh:mm:ss AM/PM`; do not write text substrings when the result should behave as a time.
|
||||
- For joins, deduplication, grouping, interval lookups, lookup grids, and ordered outputs, build explicit normalized keys, including composite keys when the task refers to multiple fields. Preserve original source order within each group unless sorting is explicitly requested.
|
||||
- For outputs that depend on other rows or lookup grids, make a first pass to build normalized dictionaries/groups/range structures, then a second pass to write results. Avoid nested full-sheet scans per row; split delimited tokens and ignore empty tokens, and treat error literals such as `#N/A` as meaningful sentinel values when the task refers to them.
|
||||
|
||||
- For lookups, filters, joins, and label/header matching, normalize comparison keys consistently: trim whitespace, skip blanks explicitly, use case-insensitive text matching when appropriate, and treat numeric-looking IDs consistently (`330`, `330.0`, and `"330"`). Keep numeric outputs numeric; use `number_format` for display formatting instead of converting numbers to strings unless text is explicitly required.
|
||||
- When replacing a generated output area, clear only the instructed target range before writing new results so stale values/formulas do not remain. Preserve formatting, column widths, borders, formulas, and unrelated cells unless the instruction explicitly asks to change them.
|
||||
|
||||
- If the instruction includes formatting changes, apply them exactly after writing values and only to the requested cells/range. Use `openpyxl` styles for fills, alignment, fonts, borders, and number formats; convert hex colors to ARGB when needed, for example `#FFC000` → `FFFFC000`. For “format as text,” set `number_format = '@'` and write string values when the expected cell values are text.
|
||||
|
||||
- When the instruction names a destination range or columns, write derived results directly there. Do not insert rows/columns, relocate the source table, or sort/delete source records unless that structural change is explicitly requested.
|
||||
- For filtered lists, summaries, and aggregations, first collect all source records/results in memory, preserving the required order, then write from the first output row and clear leftover cells below the new results in the target columns. When adding rows, copy style/alignment/number format from an existing template row when appropriate; when deleting rows, delete from bottom to top to avoid row-index shifts.
|
||||
- Preserve intended blanks as empty cells (`None`) rather than placeholder text or `0` unless the task specifies otherwise.
|
||||
|
||||
- For numeric aggregation, crosstab, SUMIFS-like, and INDEX/MATCH-style summary outputs, infer missing-match behavior from table semantics and examples: numeric summary grids usually require literal `0` for no matching records, while filtered lists or “show only once” outputs usually require blanks (`None`).
|
||||
- For blank-sensitive logic such as “if input is blank, output blank,” evaluate the driving input with `data_only=True` when it may itself be a formula, and write `None` for truly blank outputs rather than relying on a new formula returning `""`.
|
||||
|
||||
## Robustness for Simple Fill Tasks
|
||||
|
||||
- Prefer simple, auditable row/column loops over complex workbook XML parsing unless the task truly requires unsupported workbook internals. Before returning, run the script once to catch syntax/indentation errors and verify that representative target rows were actually written.
|
||||
|
||||
<!-- SLOW_UPDATE_START -->
|
||||
When the user asks for a formula, macro, VBA code, or a fix to an Excel formula, still deliver the completed workbook state: compute the intended results in Python and write literal final values into the requested cells. Do not write formula strings unless the task explicitly says the output must contain live formulas.
|
||||
|
||||
After writing, reload or inspect the saved workbook and verify that every requested/evaluated target cell contains a non-formula literal where a value is expected. If a target cell is still `None` unexpectedly, fix the script before finishing.
|
||||
|
||||
Use existing formulas in the workbook as examples/specifications, not as output. If a cell contains a reference formula such as `=A25` or an INDEX/MATCH/SUMIFS pattern, parse what source cells/ranges/criteria it refers to, compute those results yourself, and overwrite the destination with the referenced or calculated value.
|
||||
|
||||
For blank-sensitive formula tasks, compute the branch explicitly: if the driving source cell is truly blank, write `None`; otherwise write the actual result such as `0`, `1`, a category label, or a lookup value. Never rely on `IF(...,"",...)` formulas to be recalculated later.
|
||||
|
||||
For lookup/category tasks, locate both the input rows and the lookup table by headers and nearby labels. Support exact keys, numeric-looking keys, and interval/range tables; then fill every destination row that has a driving input, not just the first visible example.
|
||||
|
||||
For “every nth row” or OFFSET-style tasks, infer the source column, first source row, and step from the provided examples or formulas, then copy the actual source values into the requested output range as literals.
|
||||
|
||||
For schedule/calendar fill tasks, build a cycle-day-to-periods mapping from the schedule/template area first, then fill the daily rows across all requested class columns based on each row’s cycle day. Preserve repeated/double periods exactly as shown by the template; do not leave formulas in the schedule cells.
|
||||
|
||||
For INDEX/MATCH problems where the first row works but subsequent rows fail, treat row labels, column/year headers, region/type criteria, and expense/category labels as a multi-key lookup. Fill the whole result matrix with values from the source data table, using cached `data_only` values when source cells are formulas.
|
||||
|
||||
For multi-step macro/VBA-style requests, implement every stated operation in the workbook, not just the first deletion/filtering step. Re-read the numbered requirements before saving and verify later computed columns, totals, and derived fields as well as the obvious filtered rows.
|
||||
|
||||
When a target range includes special rows such as `Total`, `Grand Total`, `min`, `max`, constraints, headers, or blank separators, do not apply ordinary row logic blindly to those rows. Compute totals as aggregates when indicated, and leave constraint/header/blank cells untouched unless explicitly requested.
|
||||
|
||||
For residual-balancing tasks, identify data rows separately from min/max constraint rows. Add positive residuals from unit 1 toward unit 5 without exceeding max values; subtract negative residuals from unit 5 toward unit 1 without going below min values; update only the unit cells in actual data rows.
|
||||
|
||||
For time-threshold rows, decide per row whether it is a normal data row or a summary row. Normal rows use the before/after threshold rule; summary rows should aggregate the computed normal-row results if the workbook labels or examples indicate a total.
|
||||
|
||||
Keep scripts simple enough to run cleanly. Avoid unnecessary dynamic code generation and fragile f-strings with regex expressions inside them. Always execute the final `solution.py`; fix any syntax, indentation, or runtime error, then verify representative target cells.
|
||||
|
||||
If workbook cells contain arbitrary sample text that could be sensitive or trigger content filters, do not quote large raw cell contents in your response. Process them locally in Python with neutral variable names and output only the completed script/workbook changes.
|
||||
<!-- SLOW_UPDATE_END -->
|
||||
@@ -1,88 +0,0 @@
|
||||
# SkillOpt default configuration — base for all environments.
|
||||
# Environment configs should inherit via: _base_: default.yaml
|
||||
|
||||
model:
|
||||
backend: azure_openai
|
||||
optimizer: gpt-5.5
|
||||
target: gpt-5.5
|
||||
optimizer_backend: openai_chat
|
||||
target_backend: openai_chat
|
||||
reasoning_effort: medium
|
||||
rewrite_reasoning_effort: ""
|
||||
rewrite_max_completion_tokens: 64000
|
||||
codex_exec_path: codex
|
||||
codex_exec_sandbox: workspace-write
|
||||
codex_exec_profile: ""
|
||||
codex_exec_full_auto: false
|
||||
codex_exec_reasoning_effort: none
|
||||
codex_exec_use_sdk: auto
|
||||
codex_exec_network_access: false
|
||||
codex_exec_web_search: false
|
||||
codex_exec_approval_policy: never
|
||||
claude_code_exec_path: claude
|
||||
claude_code_exec_profile: ""
|
||||
claude_code_exec_use_sdk: auto
|
||||
claude_code_exec_effort: medium
|
||||
claude_code_exec_max_thinking_tokens: 16384
|
||||
codex_trace_to_optimizer: true
|
||||
azure_openai_endpoint: "" # e.g. "https://your-resource.openai.azure.com/"
|
||||
azure_openai_api_version: "2024-12-01-preview"
|
||||
azure_openai_api_key: "" # Fill locally if you do not export AZURE_OPENAI_API_KEY
|
||||
azure_openai_auth_mode: "" # empty → fall back to AZURE_OPENAI_AUTH_MODE env (default "azure_cli")
|
||||
azure_openai_ad_scope: "https://cognitiveservices.azure.com/.default"
|
||||
azure_openai_managed_identity_client_id: ""
|
||||
optimizer_azure_openai_endpoint: "" # e.g. "https://your-resource.openai.azure.com/"
|
||||
optimizer_azure_openai_api_version: "2024-12-01-preview"
|
||||
optimizer_azure_openai_api_key: ""
|
||||
optimizer_azure_openai_auth_mode: "" # empty → fall back to OPTIMIZER_AZURE_OPENAI_AUTH_MODE env, then shared
|
||||
optimizer_azure_openai_ad_scope: "https://cognitiveservices.azure.com/.default"
|
||||
optimizer_azure_openai_managed_identity_client_id: ""
|
||||
target_azure_openai_endpoint: "" # e.g. "https://your-resource.openai.azure.com/"
|
||||
target_azure_openai_api_version: "2024-12-01-preview"
|
||||
target_azure_openai_api_key: ""
|
||||
target_azure_openai_auth_mode: "" # empty → fall back to TARGET_AZURE_OPENAI_AUTH_MODE env, then shared
|
||||
target_azure_openai_ad_scope: "https://cognitiveservices.azure.com/.default"
|
||||
target_azure_openai_managed_identity_client_id: ""
|
||||
|
||||
train:
|
||||
num_epochs: 4
|
||||
train_size: 0 # 0 = derive from dataset split when available
|
||||
batch_size: 40
|
||||
accumulation: 1
|
||||
seed: 42
|
||||
|
||||
gradient:
|
||||
minibatch_size: 8
|
||||
merge_batch_size: 8
|
||||
analyst_workers: 16
|
||||
max_analyst_rounds: 3
|
||||
failure_only: false
|
||||
|
||||
optimizer:
|
||||
learning_rate: 4 # max edits per step (edit_budget)
|
||||
min_learning_rate: 2 # min edits for decay schedulers
|
||||
lr_scheduler: cosine # constant / linear / cosine / autonomous
|
||||
lr_control_mode: fixed # fixed / autonomous / none
|
||||
skill_update_mode: patch # patch / rewrite_from_suggestions / full_rewrite_minibatch
|
||||
use_slow_update: true
|
||||
slow_update_samples: 20
|
||||
slow_update_gate_with_selection: false
|
||||
longitudinal_pair_policy: mixed # mixed / changed / unchanged
|
||||
use_meta_skill: true
|
||||
|
||||
evaluation:
|
||||
use_gate: true
|
||||
sel_env_num: 0
|
||||
test_env_num: 0
|
||||
eval_test: true
|
||||
|
||||
env:
|
||||
name: ""
|
||||
skill_init: ""
|
||||
split_mode: ratio # ratio = build deterministic split from data_path; split_dir = use pre-split train/val/test
|
||||
split_seed: 42
|
||||
split_dir: ""
|
||||
data_path: ""
|
||||
split_output_dir: ""
|
||||
exec_timeout: 120 # per target model/code-agent call timeout in seconds
|
||||
out_root: ""
|
||||
@@ -1,29 +0,0 @@
|
||||
_base_: ../_base_/default.yaml
|
||||
|
||||
train:
|
||||
train_size: 0
|
||||
accumulation: 1
|
||||
|
||||
gradient:
|
||||
minibatch_size: 8
|
||||
merge_batch_size: 8
|
||||
|
||||
optimizer:
|
||||
learning_rate: 4
|
||||
|
||||
evaluation:
|
||||
sel_env_num: 0
|
||||
test_env_num: 0
|
||||
|
||||
env:
|
||||
name: alfworld
|
||||
skill_init: skillopt/envs/alfworld/skills/initial.md
|
||||
split_mode: split_dir
|
||||
split_dir: data/ablation_splits/alfworld/2-1-7_seed42
|
||||
data_path: ""
|
||||
split_output_dir: ""
|
||||
max_steps: 50
|
||||
max_completion_tokens: 16384
|
||||
workers: 8
|
||||
max_api_workers: 8
|
||||
limit: 0
|
||||
@@ -1,28 +0,0 @@
|
||||
_base_: ../_base_/default.yaml
|
||||
|
||||
model:
|
||||
reasoning_effort: medium
|
||||
|
||||
train:
|
||||
batch_size: 40
|
||||
accumulation: 1
|
||||
|
||||
gradient:
|
||||
minibatch_size: 8
|
||||
merge_batch_size: 8
|
||||
|
||||
optimizer:
|
||||
learning_rate: 4
|
||||
|
||||
env:
|
||||
name: docvqa
|
||||
skill_init: skillopt/envs/docvqa/skills/initial.md
|
||||
split_mode: split_dir
|
||||
split_dir: data/docvqa/splits
|
||||
data_path: ""
|
||||
split_output_dir: ""
|
||||
max_turns: 1
|
||||
max_completion_tokens: 16384
|
||||
workers: 16
|
||||
image_detail: auto
|
||||
limit: 0
|
||||
@@ -1,47 +0,0 @@
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Example: soft / mixed validation-gate metric (community-contributed, PR #25)
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
#
|
||||
# This is NOT a default SkillOpt setting and was NOT used to produce the
|
||||
# numbers reported in the paper. It is provided as a reference for users
|
||||
# who encounter a specific scenario where the default `hard` gate is too
|
||||
# coarse to drive training.
|
||||
#
|
||||
# When to consider this:
|
||||
# - You are running on a custom environment.
|
||||
# - Your held-out *selection* split has very few items (e.g. ≤ ~10).
|
||||
# - Your reward function is continuous / partial-credit (e.g. F1, BLEU,
|
||||
# soft match) rather than purely binary 0/1.
|
||||
#
|
||||
# Symptom this addresses:
|
||||
# With a small selection split + continuous rewards, candidate skills
|
||||
# often improve per-item soft scores (e.g. 0.06 → 0.26 on one item) but
|
||||
# never flip the discrete hard outcome. The default `hard` gate then
|
||||
# rejects every candidate and training stalls. Switching the gate to
|
||||
# `soft` or `mixed` lets these partial improvements be accepted.
|
||||
#
|
||||
# When NOT to use this:
|
||||
# - When reproducing the paper. The paper-reported numbers were obtained
|
||||
# under the default `hard` gate.
|
||||
# - When your selection split is large (dozens+ items) and / or your
|
||||
# reward is already binary — `hard` is the more conservative choice
|
||||
# and matches the design described in the paper.
|
||||
#
|
||||
# To use: inherit your env config from this file, e.g.
|
||||
# _base_: ../examples/soft_gate.yaml
|
||||
# or copy the `evaluation:` block below into your config.
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
_base_: ../_base_/default.yaml
|
||||
|
||||
evaluation:
|
||||
# Three options:
|
||||
# 'hard' — default; exact-match accuracy. Use this to reproduce the paper.
|
||||
# 'soft' — per-item soft / partial-credit score (recommended for the
|
||||
# small-split + continuous-reward scenario described above).
|
||||
# 'mixed' — weighted average: (1 - w) * hard + w * soft, with `w` set by
|
||||
# `gate_mixed_weight` below.
|
||||
gate_metric: soft
|
||||
|
||||
# Only used when gate_metric == 'mixed'. Ignored otherwise.
|
||||
gate_mixed_weight: 0.5
|
||||
@@ -1,22 +0,0 @@
|
||||
_base_: ../_base_/default.yaml
|
||||
|
||||
train:
|
||||
train_size: 0
|
||||
batch_size: 40
|
||||
accumulation: 1
|
||||
|
||||
env:
|
||||
name: livemathematicianbench
|
||||
skill_init: skillopt/envs/livemathematicianbench/skills/initial.md
|
||||
split_mode: split_dir
|
||||
split_dir: data/ablation_splits/livemathematicianbench/2-1-7_seed42
|
||||
data_path: ""
|
||||
split_output_dir: ""
|
||||
max_turns: 1
|
||||
max_completion_tokens: 16384
|
||||
exec_timeout: 300
|
||||
workers: 64
|
||||
limit: 0
|
||||
shuffle_choices: true
|
||||
use_theorem: false
|
||||
use_sketch: false
|
||||
@@ -1,34 +0,0 @@
|
||||
_base_: ../_base_/default.yaml
|
||||
|
||||
model:
|
||||
reasoning_effort: medium
|
||||
|
||||
train:
|
||||
batch_size: 40
|
||||
accumulation: 1
|
||||
|
||||
gradient:
|
||||
minibatch_size: 8
|
||||
merge_batch_size: 8
|
||||
|
||||
optimizer:
|
||||
learning_rate: 4
|
||||
|
||||
env:
|
||||
name: officeqa
|
||||
skill_init: skillopt/envs/officeqa/skills/initial.md
|
||||
split_mode: split_dir
|
||||
split_dir: data/officeqa_split
|
||||
data_dirs:
|
||||
- data/officeqa_docs_official
|
||||
workers: 4
|
||||
max_tool_turns: 24
|
||||
max_completion_tokens: 16384
|
||||
search_mode: offline
|
||||
max_queries_per_turn: 4
|
||||
search_api_url: http://apisix.westus2.cloudapp.azure.com/search_tool/search
|
||||
search_auth_env: OFFICEQA_CUSTOM_SEARCH_AUTH
|
||||
search_provider: duckduckgo
|
||||
search_max_num_results: 4
|
||||
search_timeout_seconds: 20
|
||||
limit: 0
|
||||
@@ -1,32 +0,0 @@
|
||||
_base_: ../_base_/default.yaml
|
||||
|
||||
model:
|
||||
reasoning_effort: medium
|
||||
|
||||
train:
|
||||
train_size: 400
|
||||
batch_size: 40
|
||||
accumulation: 1
|
||||
|
||||
gradient:
|
||||
minibatch_size: 8
|
||||
merge_batch_size: 8
|
||||
|
||||
optimizer:
|
||||
learning_rate: 4
|
||||
|
||||
evaluation:
|
||||
sel_env_num: 0
|
||||
test_env_num: 0
|
||||
|
||||
env:
|
||||
name: searchqa
|
||||
skill_init: skillopt/envs/searchqa/skills/initial.md
|
||||
split_mode: split_dir
|
||||
split_dir: data/searchqa_split
|
||||
data_path: ""
|
||||
split_output_dir: ""
|
||||
max_turns: 1
|
||||
max_completion_tokens: 16384
|
||||
workers: 24
|
||||
limit: 0
|
||||
@@ -1,34 +0,0 @@
|
||||
_base_: ../_base_/default.yaml
|
||||
|
||||
model:
|
||||
reasoning_effort: medium
|
||||
|
||||
train:
|
||||
train_size: 80
|
||||
batch_size: 40
|
||||
accumulation: 1
|
||||
|
||||
gradient:
|
||||
minibatch_size: 8
|
||||
merge_batch_size: 8
|
||||
|
||||
optimizer:
|
||||
learning_rate: 4
|
||||
|
||||
evaluation:
|
||||
sel_env_num: 0
|
||||
test_env_num: 0
|
||||
|
||||
env:
|
||||
name: spreadsheetbench
|
||||
skill_init: skillopt/envs/spreadsheetbench/skills/initial.md
|
||||
split_mode: split_dir
|
||||
split_dir: data/spreadsheetbench_split
|
||||
data_path: ""
|
||||
split_output_dir: ""
|
||||
data_root: data/spreadsheetbench_verified_400
|
||||
mode: multi
|
||||
max_turns: 30
|
||||
max_completion_tokens: 16384
|
||||
exec_timeout: 600
|
||||
workers: 24
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -1,602 +0,0 @@
|
||||
[
|
||||
{
|
||||
"id": "1758dc50625e46ee814e44a6061f091d"
|
||||
},
|
||||
{
|
||||
"id": "52967b20d9564d5b94756bc0b3d113d6"
|
||||
},
|
||||
{
|
||||
"id": "e77dbfc0d45e476b9ed5b2204b6d9fc8"
|
||||
},
|
||||
{
|
||||
"id": "675f01b5343e46599626104968c0f45a"
|
||||
},
|
||||
{
|
||||
"id": "dccd0f04cd3b41bc8c4182e41a413b3e"
|
||||
},
|
||||
{
|
||||
"id": "efde5820a29949488feaa18ce7e429f9"
|
||||
},
|
||||
{
|
||||
"id": "98773558971e490d8de41088e41a3207"
|
||||
},
|
||||
{
|
||||
"id": "70e3e1e1d41f4795ba45678d5f4b6524"
|
||||
},
|
||||
{
|
||||
"id": "206a9ffb8ed24686a24db1492c9f10dc"
|
||||
},
|
||||
{
|
||||
"id": "e91f1225cdb84ee7a2d773b5177945c1"
|
||||
},
|
||||
{
|
||||
"id": "b68a14a7b43842a0a50a3965d3bbffbb"
|
||||
},
|
||||
{
|
||||
"id": "e2e679989b544e94bf8dbc3466e49003"
|
||||
},
|
||||
{
|
||||
"id": "c51c26c07b384bbe865976b81a86736d"
|
||||
},
|
||||
{
|
||||
"id": "83928d21f3504843aa6294ce9d9c8da1"
|
||||
},
|
||||
{
|
||||
"id": "f822cde36d864c508b426f3e2b5245b2"
|
||||
},
|
||||
{
|
||||
"id": "f03757766e034c539bf2595428c03626"
|
||||
},
|
||||
{
|
||||
"id": "349be67be2e141438287d21cb6a94131"
|
||||
},
|
||||
{
|
||||
"id": "7f3e18272c6d43dd8c6385c4758b685e"
|
||||
},
|
||||
{
|
||||
"id": "66dc8aee931142b1834da63363511491"
|
||||
},
|
||||
{
|
||||
"id": "52ac78645e144b4598958d67920617dc"
|
||||
},
|
||||
{
|
||||
"id": "32a66c67a8d7490b8572387d8f1d762a"
|
||||
},
|
||||
{
|
||||
"id": "b028bea99d2641268532cd2a32d4ad2d"
|
||||
},
|
||||
{
|
||||
"id": "c9f3a971abe64853aca3e0bb1fea6962"
|
||||
},
|
||||
{
|
||||
"id": "5324041c25d04316a800ccba2d79a2c7"
|
||||
},
|
||||
{
|
||||
"id": "9960a3f1ac954282a7ea1d7f7c8a86f9"
|
||||
},
|
||||
{
|
||||
"id": "03bd8fc86a9d465e9370dcdaedb3c422"
|
||||
},
|
||||
{
|
||||
"id": "a1d0d03fcac54483a1e88542b767afd5"
|
||||
},
|
||||
{
|
||||
"id": "c21a41745ab9402da8b1deea67055a8d"
|
||||
},
|
||||
{
|
||||
"id": "d5c48bd600824c8a9328df80a8baf964"
|
||||
},
|
||||
{
|
||||
"id": "746806ac0b1c48cb80d7ab5ad0a63517"
|
||||
},
|
||||
{
|
||||
"id": "e3b96fc1b99247ebada09a84fa5fd205"
|
||||
},
|
||||
{
|
||||
"id": "bbeeec20466a4dab9e4d73d86c964cba"
|
||||
},
|
||||
{
|
||||
"id": "2b85b097d130404daeb26a3349bdc297"
|
||||
},
|
||||
{
|
||||
"id": "fbaae95260474ed58ccbb05ac482089b"
|
||||
},
|
||||
{
|
||||
"id": "28b837ac304845bd961a46cedc9246af"
|
||||
},
|
||||
{
|
||||
"id": "691e3a24e5994c749ff11c6bc43b2d1c"
|
||||
},
|
||||
{
|
||||
"id": "12d5b51c793a43b1aa3b7e1bf5f4bc73"
|
||||
},
|
||||
{
|
||||
"id": "a3a3892296c74fc4b23b2475156901b1"
|
||||
},
|
||||
{
|
||||
"id": "8065f2e8639848f5b2f16a4f2877b7c2"
|
||||
},
|
||||
{
|
||||
"id": "b06e0d2be0a5465ea12ff2bf2b27d387"
|
||||
},
|
||||
{
|
||||
"id": "e50328dc70634863ac39580ef6f8dd5a"
|
||||
},
|
||||
{
|
||||
"id": "17b441db9b444435951d859e6e081f3b"
|
||||
},
|
||||
{
|
||||
"id": "0fc37a4bd8424ecf8eede56e956edb3a"
|
||||
},
|
||||
{
|
||||
"id": "09c769c7262e43eb95a52e874042c02a"
|
||||
},
|
||||
{
|
||||
"id": "2417ce75564b4b8eab1ddd755b10d962"
|
||||
},
|
||||
{
|
||||
"id": "8f4beaa8a7704eba93f753a0e7040165"
|
||||
},
|
||||
{
|
||||
"id": "6d8e99531264435badb67bf6ac209075"
|
||||
},
|
||||
{
|
||||
"id": "940ddc06648f4c9bac6b3bf5df402e5a"
|
||||
},
|
||||
{
|
||||
"id": "fa0fcf36e99644d0b51be95869d87adc"
|
||||
},
|
||||
{
|
||||
"id": "1882f20b777146608474ac0056d54a5e"
|
||||
},
|
||||
{
|
||||
"id": "439e908aa7c7422daf5632af7e0440ec"
|
||||
},
|
||||
{
|
||||
"id": "2d90c36ab1dd4724b1c108b2507316e4"
|
||||
},
|
||||
{
|
||||
"id": "efa5cc2f750c421a807bc4315db0dc1e"
|
||||
},
|
||||
{
|
||||
"id": "3b1c596e1720499ebe790d14f34a67af"
|
||||
},
|
||||
{
|
||||
"id": "c87234a3a1074b5ea5500114ad9aa16d"
|
||||
},
|
||||
{
|
||||
"id": "9d08ada791844519b7a4bb247d50c89c"
|
||||
},
|
||||
{
|
||||
"id": "74c45e35e5a740798fbd528ff08a70e3"
|
||||
},
|
||||
{
|
||||
"id": "650b683d39dd45c99ad7435fc7d33fe3"
|
||||
},
|
||||
{
|
||||
"id": "191ce59c97e143f7b4c33400913d3570"
|
||||
},
|
||||
{
|
||||
"id": "74441a0f7dae42ca9ee19d8be8c9b318"
|
||||
},
|
||||
{
|
||||
"id": "baef3cef84f14ed98655ac73293a56db"
|
||||
},
|
||||
{
|
||||
"id": "099a1ae148184bd5a93c9bc9889306e7"
|
||||
},
|
||||
{
|
||||
"id": "4fb4332a7a0e44e4b5655de1d7c2e5ac"
|
||||
},
|
||||
{
|
||||
"id": "cdf86d1982f64cb49d6595680b1c53b8"
|
||||
},
|
||||
{
|
||||
"id": "fa4cbba4ffed4f4e8ed4c148711cbc78"
|
||||
},
|
||||
{
|
||||
"id": "41c691d3b7e94ffd993647c7e531e94e"
|
||||
},
|
||||
{
|
||||
"id": "df43ea868a6d409ea67eb632fc9f285c"
|
||||
},
|
||||
{
|
||||
"id": "746448dcf441410db997a59f7137a2c8"
|
||||
},
|
||||
{
|
||||
"id": "3d0cad58c3c145f3b98d51c06a65a0c9"
|
||||
},
|
||||
{
|
||||
"id": "9cb4317831024d589622e2a3cf544360"
|
||||
},
|
||||
{
|
||||
"id": "802015dbc9f5406d8df3b468497f6553"
|
||||
},
|
||||
{
|
||||
"id": "5463f1d3fc124cadb2ad016076a699f5"
|
||||
},
|
||||
{
|
||||
"id": "8c77419b4e864778b965753f411bb469"
|
||||
},
|
||||
{
|
||||
"id": "2e112a7c9d97482fa0deb079c6c09172"
|
||||
},
|
||||
{
|
||||
"id": "f0c26f4f2e9f4d1d84b4f13cb201cbeb"
|
||||
},
|
||||
{
|
||||
"id": "231d7d9955d54f60a1f1ff4f0bf1c5b6"
|
||||
},
|
||||
{
|
||||
"id": "9b56a920d67a48e8b33932802d17d374"
|
||||
},
|
||||
{
|
||||
"id": "092a8aab433c4184bde9c8aa08789464"
|
||||
},
|
||||
{
|
||||
"id": "266a4337fa5742c8bbcc5aa79e571e85"
|
||||
},
|
||||
{
|
||||
"id": "d67cecf38f954800a7c651cca70bdc66"
|
||||
},
|
||||
{
|
||||
"id": "e634fa14bd4941ef96e1d21e796a9cd3"
|
||||
},
|
||||
{
|
||||
"id": "dbc29aef324e4c4f9f19082dcb1b5cb1"
|
||||
},
|
||||
{
|
||||
"id": "e688c5081a84488694b111a11a50ef7e"
|
||||
},
|
||||
{
|
||||
"id": "80b114d03dbb4598b5da648bbe174d0f"
|
||||
},
|
||||
{
|
||||
"id": "ce42d55ff5874c6380e018896b16e128"
|
||||
},
|
||||
{
|
||||
"id": "ef2b37212c98453390c9dca445d69de1"
|
||||
},
|
||||
{
|
||||
"id": "a66639dbb5ed4dfb9e65fe359ed8dc03"
|
||||
},
|
||||
{
|
||||
"id": "10eacaf72a4e4ad49cc43d3a710ee24a"
|
||||
},
|
||||
{
|
||||
"id": "b7e3d45f77bd4a4db768505fc4726592"
|
||||
},
|
||||
{
|
||||
"id": "0d1592f640f04ecb9c7add7164f87c5a"
|
||||
},
|
||||
{
|
||||
"id": "296a678ada814b09bdd0a53629f7f3f9"
|
||||
},
|
||||
{
|
||||
"id": "6e6fd6d62ce54031a4e36f88c2d5b63d"
|
||||
},
|
||||
{
|
||||
"id": "8d0dd71f94cc4abba255f3cdd6c2d940"
|
||||
},
|
||||
{
|
||||
"id": "7c26258ff99c4adeb4b9b35001005453"
|
||||
},
|
||||
{
|
||||
"id": "d13c24c99c704e31bbb3aaa585884d56"
|
||||
},
|
||||
{
|
||||
"id": "dbfb81f100dc474f8267e6f9ab0ae51c"
|
||||
},
|
||||
{
|
||||
"id": "3b4a5a2f617b43c8aa37b0890f576749"
|
||||
},
|
||||
{
|
||||
"id": "c49a8bf2d0ad4afd9b0bb0d03c1b9d96"
|
||||
},
|
||||
{
|
||||
"id": "fbfb4851dd494e44b9a21475d7e6dbaa"
|
||||
},
|
||||
{
|
||||
"id": "e0ba941f8e2c4fa6a0dea2b562b0e4be"
|
||||
},
|
||||
{
|
||||
"id": "7c9ec2b1cd8d4a8eaa6e1154e6fb40c4"
|
||||
},
|
||||
{
|
||||
"id": "4ae645d2721c4751b1e71c83870bbe16"
|
||||
},
|
||||
{
|
||||
"id": "c7bd32aee95f485d85d0b6749d542c51"
|
||||
},
|
||||
{
|
||||
"id": "20e140b44e0f4a6484e9cf776b09679b"
|
||||
},
|
||||
{
|
||||
"id": "5deb6bfcdd8c486d9f5f630309968763"
|
||||
},
|
||||
{
|
||||
"id": "18d1fcc3ed394b23ada799aad3beca90"
|
||||
},
|
||||
{
|
||||
"id": "74a655bcaead4857b92a49ec5f93ee23"
|
||||
},
|
||||
{
|
||||
"id": "bfec094ae9da4aaebd91713b39d3008e"
|
||||
},
|
||||
{
|
||||
"id": "80ebe13e36eb413a9acaace84648b36d"
|
||||
},
|
||||
{
|
||||
"id": "982e414b371248788c72732975949957"
|
||||
},
|
||||
{
|
||||
"id": "963ad06f3e314d3b99e0b6b9eaeaf2d8"
|
||||
},
|
||||
{
|
||||
"id": "90b6478bd7ae4145976c474dc42da80f"
|
||||
},
|
||||
{
|
||||
"id": "ea475dcd08d54da6896f56e330fa2a1c"
|
||||
},
|
||||
{
|
||||
"id": "64899cac91e5423fa3617bf576e6df4d"
|
||||
},
|
||||
{
|
||||
"id": "6231d57d040143e3bc56cafaabaa0975"
|
||||
},
|
||||
{
|
||||
"id": "3fdb66ab2b0a49318b053edda6633237"
|
||||
},
|
||||
{
|
||||
"id": "7dff080db1b8438e9ed737df1d7c32e3"
|
||||
},
|
||||
{
|
||||
"id": "878108ec7e9742a090175bd33390e2c5"
|
||||
},
|
||||
{
|
||||
"id": "3b2acfd62bdf4361aa4e51895d12df8b"
|
||||
},
|
||||
{
|
||||
"id": "0335847f4a44451db5a1b775cc019765"
|
||||
},
|
||||
{
|
||||
"id": "b6bfa34339c649398ae3b4540ed95fcf"
|
||||
},
|
||||
{
|
||||
"id": "82fd2c61b9a64b5f8971b28a1cfd8c1b"
|
||||
},
|
||||
{
|
||||
"id": "a9385b35a3a5492283aa461b93fdb361"
|
||||
},
|
||||
{
|
||||
"id": "847f117d4638487d92539d52eb07a999"
|
||||
},
|
||||
{
|
||||
"id": "de103586389245e7858b26b9b9a1ba57"
|
||||
},
|
||||
{
|
||||
"id": "82d95f6b3df64bb48668f7e98ddf5b9c"
|
||||
},
|
||||
{
|
||||
"id": "b1f6907b8f0f41ec82dc2eae7f84666b"
|
||||
},
|
||||
{
|
||||
"id": "1f232b1de1f34793b6557c8688bf4d9c"
|
||||
},
|
||||
{
|
||||
"id": "c16f718bd6554e998a95fe1d07623a13"
|
||||
},
|
||||
{
|
||||
"id": "dc2add9004fd40e69fcbd2eea80c513c"
|
||||
},
|
||||
{
|
||||
"id": "464e204e258340b69eca2b90451f06b8"
|
||||
},
|
||||
{
|
||||
"id": "c12fd75ea3db450bbfcabeb4f62f7e8f"
|
||||
},
|
||||
{
|
||||
"id": "5dbcadd728fa4956a4f739c1168fbf18"
|
||||
},
|
||||
{
|
||||
"id": "1730220302104d54ba2bb781ca219e52"
|
||||
},
|
||||
{
|
||||
"id": "ffcf25fe8cce4198bd483f2faaa17889"
|
||||
},
|
||||
{
|
||||
"id": "98ed5d627d0746e0aecb42eae39c14dc"
|
||||
},
|
||||
{
|
||||
"id": "30718db44c7f44a5b4b0c68842b1053b"
|
||||
},
|
||||
{
|
||||
"id": "4e90a93f4548458bb5d35ac9c767233b"
|
||||
},
|
||||
{
|
||||
"id": "dabccc32402a43a7ab9c82408f3603a1"
|
||||
},
|
||||
{
|
||||
"id": "80a8a1ea275944228e71fd0dfc74fb74"
|
||||
},
|
||||
{
|
||||
"id": "687434d75186430896ea636fec156e6d"
|
||||
},
|
||||
{
|
||||
"id": "28e5233352ba44ea9f7a8d4cf7f6b86a"
|
||||
},
|
||||
{
|
||||
"id": "b11499ee2a1f4a2b8fa8c9f561136430"
|
||||
},
|
||||
{
|
||||
"id": "6efcba5974464dcbaf5d9c5e9322bd9c"
|
||||
},
|
||||
{
|
||||
"id": "a1d460152c3840d98ed3bc45bf80d5c1"
|
||||
},
|
||||
{
|
||||
"id": "0d482f0e297544dcb2ce1e87e4c3b4b4"
|
||||
},
|
||||
{
|
||||
"id": "1a5baeb7a51d453d892ba70a9de2146d"
|
||||
},
|
||||
{
|
||||
"id": "ab84169c66b2415180563269295f53be"
|
||||
},
|
||||
{
|
||||
"id": "08c0e9678f1548aebeada27ab7b954e1"
|
||||
},
|
||||
{
|
||||
"id": "efd2d367aeec4cc4ae72cd3eaa040a0f"
|
||||
},
|
||||
{
|
||||
"id": "ae54626131ed42eaa55755f10ee64a76"
|
||||
},
|
||||
{
|
||||
"id": "1787cbe3e160427ca4e61931daf66e51"
|
||||
},
|
||||
{
|
||||
"id": "24143f01e7dd455e86c34a4a6f4ed3b9"
|
||||
},
|
||||
{
|
||||
"id": "5e92b869dd0b4ecbaf8694041891eb75"
|
||||
},
|
||||
{
|
||||
"id": "8d130f744b214055ad0c4cdce459a4b9"
|
||||
},
|
||||
{
|
||||
"id": "87913f6be2004e938ba60cda550bc675"
|
||||
},
|
||||
{
|
||||
"id": "68c7f892cbd94fc59ebe4ef1e1ad0c89"
|
||||
},
|
||||
{
|
||||
"id": "f7bf36628c7540da907dda84b48aac82"
|
||||
},
|
||||
{
|
||||
"id": "357af030b55046fc856e03fdac36032a"
|
||||
},
|
||||
{
|
||||
"id": "3a90f09909344dfdbb1a73f80c983a87"
|
||||
},
|
||||
{
|
||||
"id": "44fd76ce5f864cc2aef99c75cfd4904e"
|
||||
},
|
||||
{
|
||||
"id": "95e54b962ce1420d82c716ff428d84fe"
|
||||
},
|
||||
{
|
||||
"id": "8e22ba3b6fc34886a86d60dd1a1db025"
|
||||
},
|
||||
{
|
||||
"id": "162917ea12b24d0e874540feecf9b445"
|
||||
},
|
||||
{
|
||||
"id": "c8e87f5da41c47c7a3116bd1ed0cc34d"
|
||||
},
|
||||
{
|
||||
"id": "a3458bd73564491585df515468475a66"
|
||||
},
|
||||
{
|
||||
"id": "e27a4c1fb25c48809feb3cc438b5eb3c"
|
||||
},
|
||||
{
|
||||
"id": "f36338c39e2846959491ea48cbce07f6"
|
||||
},
|
||||
{
|
||||
"id": "eae01f064523415c95e57ff1e8dadc43"
|
||||
},
|
||||
{
|
||||
"id": "8b845d50f998467a898367107377c64b"
|
||||
},
|
||||
{
|
||||
"id": "a50ae9e7cb6a4f1e8dad07a9bbace810"
|
||||
},
|
||||
{
|
||||
"id": "5c2009e4f1384b27b4b4fe9422f048ed"
|
||||
},
|
||||
{
|
||||
"id": "0baaa0fe4fde4bae99bc6d2be230d525"
|
||||
},
|
||||
{
|
||||
"id": "d2563a0ca702457497995a6959ff1c61"
|
||||
},
|
||||
{
|
||||
"id": "b07871d2991d47e899cfa68a91f16b52"
|
||||
},
|
||||
{
|
||||
"id": "b745738e5b3c482aa2ae94ad5688246b"
|
||||
},
|
||||
{
|
||||
"id": "d66f1f0158054f638ff1b8acf3e18a8a"
|
||||
},
|
||||
{
|
||||
"id": "ede0705d606d4435a300e06beab2322f"
|
||||
},
|
||||
{
|
||||
"id": "e9d415f928644b59b5ed19b7011f627d"
|
||||
},
|
||||
{
|
||||
"id": "1a78a8cf476c4a0dbf551558da1d879b"
|
||||
},
|
||||
{
|
||||
"id": "ea7c66d5dae241eda155763541749cd1"
|
||||
},
|
||||
{
|
||||
"id": "78ae271549144680a42139693d1117e9"
|
||||
},
|
||||
{
|
||||
"id": "7eb9a69a23b8469b9f20f1915abff3be"
|
||||
},
|
||||
{
|
||||
"id": "db73ca2786684b9e92deb1fcc796effd"
|
||||
},
|
||||
{
|
||||
"id": "a876fb1accf64f4eb1b20b4a0ee089db"
|
||||
},
|
||||
{
|
||||
"id": "71d6dabd553c471b98debe941341a4c0"
|
||||
},
|
||||
{
|
||||
"id": "1c6add1cbd464f34b6e90485780f1c3e"
|
||||
},
|
||||
{
|
||||
"id": "5e9b0b44825e44d89ba0501c3b819685"
|
||||
},
|
||||
{
|
||||
"id": "4cfc841cb81e4c90822cfb7c0ec9f86e"
|
||||
},
|
||||
{
|
||||
"id": "2933df89dd6146c791eac1a85f0d02c8"
|
||||
},
|
||||
{
|
||||
"id": "5af06dca120a46889964109e94fff0a0"
|
||||
},
|
||||
{
|
||||
"id": "77970683cb6444388e5cfdd743784db6"
|
||||
},
|
||||
{
|
||||
"id": "3e40d5d91615427d976a56932be3c955"
|
||||
},
|
||||
{
|
||||
"id": "70bc65e0c17041729a4eabe86ff36d53"
|
||||
},
|
||||
{
|
||||
"id": "b913e55670474b69a70736d314822155"
|
||||
},
|
||||
{
|
||||
"id": "ada9cbe2dd8e48deb55171645e47a24a"
|
||||
},
|
||||
{
|
||||
"id": "fba813b8837249cbb39a1996dc657ef0"
|
||||
},
|
||||
{
|
||||
"id": "f7ecc32a212e4c3b8471ce26c32f3135"
|
||||
},
|
||||
{
|
||||
"id": "015641bf588b43aebba1aee07a8afde0"
|
||||
},
|
||||
{
|
||||
"id": "bcecb67a14f34fc6896bc5d65c58405a"
|
||||
}
|
||||
]
|
||||
@@ -1,69 +0,0 @@
|
||||
# Contributing to SkillOpt
|
||||
|
||||
Thank you for your interest in contributing to SkillOpt! This guide covers how to get started.
|
||||
|
||||
## Development Setup
|
||||
|
||||
```bash
|
||||
git clone https://github.com/microsoft/SkillOpt.git
|
||||
cd SkillOpt
|
||||
pip install -e ".[dev]"
|
||||
```
|
||||
|
||||
## Ways to Contribute
|
||||
|
||||
### 🐛 Bug Reports
|
||||
|
||||
Open an issue with:
|
||||
- Steps to reproduce
|
||||
- Expected vs actual behavior
|
||||
- Config file used (sanitize API keys)
|
||||
- Python version and OS
|
||||
|
||||
### 🔧 New Benchmark
|
||||
|
||||
See [Add a New Benchmark](guide/new-benchmark.md) for the implementation guide.
|
||||
|
||||
**Checklist:**
|
||||
- [ ] Data loader in `skillopt/envs/<benchmark>/loader.py`
|
||||
- [ ] Environment adapter in `skillopt/envs/<benchmark>/env.py`
|
||||
- [ ] Config file in `configs/<benchmark>/default.yaml`
|
||||
- [ ] Registration in `skillopt/envs/__init__.py`
|
||||
- [ ] Documentation page in `docs/`
|
||||
|
||||
### 🤖 New Model Backend
|
||||
|
||||
See [Add a New Model Backend](guide/new-backend.md) for the implementation guide.
|
||||
|
||||
**Checklist:**
|
||||
- [ ] Backend in `skillopt/model/<backend>.py`
|
||||
- [ ] Registration in `skillopt/model/__init__.py`
|
||||
- [ ] API key entry in `.env.example`
|
||||
- [ ] Documentation update
|
||||
|
||||
### 📝 Documentation
|
||||
|
||||
Documentation is built with MkDocs Material:
|
||||
|
||||
```bash
|
||||
pip install -e ".[docs]"
|
||||
mkdocs serve # Preview at http://localhost:8000
|
||||
```
|
||||
|
||||
## Code Style
|
||||
|
||||
- Follow existing patterns in the codebase
|
||||
- Use type hints for function signatures
|
||||
- Keep docstrings concise
|
||||
|
||||
## Pull Request Process
|
||||
|
||||
1. Fork the repository
|
||||
2. Create a feature branch: `git checkout -b feature/my-benchmark`
|
||||
3. Make your changes
|
||||
4. Test with an existing benchmark config
|
||||
5. Submit a PR with a clear description
|
||||
|
||||
## License
|
||||
|
||||
By contributing, you agree that your contributions will be licensed under the MIT License.
|
||||
@@ -1,109 +0,0 @@
|
||||
# Configuration Guide
|
||||
|
||||
SkillOpt uses YAML configuration files with a hierarchical override system.
|
||||
|
||||
## Config Structure
|
||||
|
||||
```
|
||||
configs/
|
||||
├── _base_/
|
||||
│ └── default.yaml # Global defaults
|
||||
├── searchqa/
|
||||
│ └── default.yaml # SearchQA overrides
|
||||
├── docvqa/
|
||||
│ └── default.yaml # DocVQA overrides
|
||||
└── alfworld/
|
||||
└── default.yaml # ALFWorld overrides
|
||||
```
|
||||
|
||||
Benchmark configs inherit from `_base_/default.yaml` and override specific values.
|
||||
|
||||
## Key Parameters
|
||||
|
||||
### Model
|
||||
|
||||
```yaml
|
||||
model:
|
||||
backend: azure_openai # azure_openai | openai_chat | claude_code_exec | qwen
|
||||
optimizer: gpt-5.5 # Optimizer model (for reflection)
|
||||
target: gpt-5.5 # Target model (for rollout)
|
||||
```
|
||||
|
||||
### Training
|
||||
|
||||
```yaml
|
||||
train:
|
||||
num_epochs: 4 # Number of training epochs
|
||||
batch_size: 40 # Tasks per step (batch size)
|
||||
accumulation: 1 # Gradient accumulation
|
||||
seed: 42
|
||||
```
|
||||
|
||||
### Gradient (Reflection)
|
||||
|
||||
```yaml
|
||||
gradient:
|
||||
minibatch_size: 8 # Reflect minibatch size
|
||||
analyst_workers: 16 # Parallel reflection workers
|
||||
max_analyst_rounds: 3 # Max rounds of analyst reflection
|
||||
failure_only: false # Only reflect on failures
|
||||
```
|
||||
|
||||
### Optimizer
|
||||
|
||||
```yaml
|
||||
optimizer:
|
||||
learning_rate: 4 # Max edits per step (edit budget)
|
||||
min_learning_rate: 2 # Min edits for decay schedulers
|
||||
lr_scheduler: cosine # constant | linear | cosine | autonomous
|
||||
use_slow_update: true # Momentum-like blending at epoch boundary
|
||||
slow_update_samples: 20 # Samples for slow update evaluation
|
||||
use_meta_skill: true # Cross-epoch strategy memory
|
||||
```
|
||||
|
||||
### Evaluation
|
||||
|
||||
```yaml
|
||||
evaluation:
|
||||
use_gate: true # Validation gating (accept/reject updates)
|
||||
eval_test: true # Run test evaluation after training
|
||||
```
|
||||
|
||||
### Environment (Data)
|
||||
|
||||
```yaml
|
||||
env:
|
||||
name: searchqa # Benchmark name
|
||||
split_mode: ratio # ratio | split_dir
|
||||
split_ratio: "2:1:7" # train:val:test ratio
|
||||
data_path: "" # Path to dataset
|
||||
exec_timeout: 120 # Per-task timeout (seconds)
|
||||
```
|
||||
|
||||
## CLI Overrides
|
||||
|
||||
Override any config value from the command line:
|
||||
|
||||
```bash
|
||||
python scripts/train.py \
|
||||
--config configs/searchqa/default.yaml \
|
||||
optimizer.learning_rate=16 \
|
||||
optimizer.lr_scheduler=linear \
|
||||
gradient.analyst_workers=8
|
||||
```
|
||||
|
||||
## Environment Variables
|
||||
|
||||
Model credentials are loaded from environment variables:
|
||||
|
||||
| Variable | Backend | Description |
|
||||
|---|---|---|
|
||||
| `AZURE_OPENAI_ENDPOINT` | azure_openai | Azure resource endpoint |
|
||||
| `AZURE_OPENAI_API_KEY` | azure_openai | Azure API key |
|
||||
| `OPENAI_API_KEY` | openai | OpenAI API key |
|
||||
| `ANTHROPIC_API_KEY` | claude | Anthropic API key |
|
||||
| `QWEN_API_BASE` | qwen | Local Qwen vLLM endpoint |
|
||||
|
||||
## Full Reference
|
||||
|
||||
See [Configuration Reference](../reference/config.md) for the complete parameter list.
|
||||
@@ -1,51 +0,0 @@
|
||||
# Deep Learning ↔ SkillOpt Analogy
|
||||
|
||||
SkillOpt is designed around a core insight: **optimizing natural-language prompts follows the same structure as training neural networks**. This page maps every DL concept to its SkillOpt counterpart.
|
||||
|
||||
## Complete Mapping
|
||||
|
||||
| Deep Learning | SkillOpt | Description |
|
||||
|---|---|---|
|
||||
| **Model weights** | Skill document (Markdown) | The thing being optimized |
|
||||
| **Forward pass** | Rollout | Target executes tasks using current skill |
|
||||
| **Loss function** | Task evaluator | Scores task execution quality |
|
||||
| **Backpropagation** | Reflect | Optimizer analyzes failures → edit patches |
|
||||
| **Gradients** | Edit patches | Proposed changes to the skill |
|
||||
| **Gradient aggregation** | Patch aggregation | Merge similar edits |
|
||||
| **Gradient clipping** | Edit selection | Cap max edits per step |
|
||||
| **Learning rate** | `learning_rate` | Max number of edits applied per step |
|
||||
| **LR scheduler** | `lr_scheduler` | Decay schedule: cosine, linear, constant |
|
||||
| **SGD step** | Skill update | Apply selected patches to document |
|
||||
| **Validation set** | Selection split | Gate checks improvement before accepting |
|
||||
| **Early stopping** | Gate patience | Reject updates that don't improve |
|
||||
| **Training step** | Step | One rollout → reflect → update cycle |
|
||||
| **Epoch** | Epoch | Full pass with slow update + meta memory |
|
||||
| **Momentum** | Slow update | Longitudinal comparison at epoch boundary |
|
||||
| **Meta-learning** | Meta skill | Cross-epoch optimizer strategy memory |
|
||||
| **Batch size** | `batch_size` | Tasks sampled per rollout |
|
||||
| **Data parallelism** | `analyst_workers` | Parallel reflection workers |
|
||||
| **Training set** | Train split | Items used for rollout |
|
||||
| **Test set** | Test split | Held-out final evaluation |
|
||||
| **Warm-up** | (implicit) | High LR early steps explore broadly |
|
||||
| **Checkpointing** | Skill snapshots | Saved after each accepted step |
|
||||
| **Transfer learning** | Seed skill / cross-benchmark init | Start from pre-trained skill |
|
||||
|
||||
## Why This Analogy Matters
|
||||
|
||||
1. **Familiar mental model**: ML practitioners immediately understand how to tune SkillOpt
|
||||
2. **Principled hyperparameter search**: Grid search over `learning_rate` × `lr_scheduler` works just like in DL
|
||||
3. **Proven mechanisms**: Gating ≈ validation-based selection, patience ≈ early stopping, slow update ≈ momentum — all with strong theoretical motivation
|
||||
|
||||
## Hyperparameter Transfer Rules
|
||||
|
||||
From our experiments, these DL intuitions transfer well:
|
||||
|
||||
!!! success "What transfers"
|
||||
- **Cosine schedule > constant** — same as in DL, cosine annealing helps convergence
|
||||
- **Moderate LR (4-16) > very high/low** — too few edits = slow learning, too many = noisy
|
||||
- **Slow update helps** — longitudinal comparison prevents catastrophic forgetting across epochs
|
||||
- **Meta skill memory improves reflection** — optimizer benefits from cross-epoch strategy notes
|
||||
|
||||
!!! warning "What doesn't transfer"
|
||||
- **Batch size ≠ better** — larger rollout batches have diminishing returns due to API costs
|
||||
- **More epochs ≠ better** — skills converge faster than neural networks (2-4 epochs usually enough)
|
||||
@@ -1,110 +0,0 @@
|
||||
# Your First Experiment
|
||||
|
||||
This guide walks through running a complete SkillOpt training on SearchQA.
|
||||
|
||||
## 1. Choose a Benchmark
|
||||
|
||||
SkillOpt includes ready-to-use configs for several benchmarks:
|
||||
|
||||
| Benchmark | Difficulty | Typical Runtime |
|
||||
|---|---|---|
|
||||
| SearchQA | ⭐ Easy | ~30 min |
|
||||
| DocVQA | ⭐⭐ Medium | ~2 hours |
|
||||
| ALFWorld | ⭐⭐⭐ Hard | ~3 hours |
|
||||
|
||||
We'll use **SearchQA** as it's the fastest to complete.
|
||||
|
||||
## 2. Configure
|
||||
|
||||
Review the config file:
|
||||
|
||||
```bash
|
||||
cat configs/searchqa/default.yaml
|
||||
```
|
||||
|
||||
Key parameters (deep learning analogy in parentheses):
|
||||
|
||||
```yaml
|
||||
train:
|
||||
num_epochs: 4 # (epochs)
|
||||
batch_size: 40 # (batch size)
|
||||
|
||||
optimizer:
|
||||
learning_rate: 4 # (max edits per step)
|
||||
lr_scheduler: cosine # (learning rate schedule)
|
||||
use_slow_update: true # (momentum at epoch boundary)
|
||||
use_meta_skill: true # (cross-epoch optimizer memory)
|
||||
|
||||
gradient:
|
||||
analyst_workers: 16 # (parallel reflection workers)
|
||||
|
||||
evaluation:
|
||||
use_gate: true # (validation gating)
|
||||
```
|
||||
|
||||
## 3. Train
|
||||
|
||||
```bash
|
||||
python scripts/train.py --config configs/searchqa/default.yaml
|
||||
```
|
||||
|
||||
You'll see output like:
|
||||
|
||||
```
|
||||
[Step 1/8] Rollout: 20 items, 4 workers...
|
||||
[Step 1/8] Score: 0.65 → Reflect...
|
||||
[Step 1/8] 6 edit patches generated
|
||||
[Step 1/8] Selected 4 edits (lr=8, cosine → 7.7)
|
||||
[Step 1/8] Gate: val score 0.68 > 0.65 ✓ ACCEPT
|
||||
[Step 2/8] ...
|
||||
```
|
||||
|
||||
## 4. Monitor
|
||||
|
||||
Training outputs are saved to `outputs/<benchmark>/<run_id>/`:
|
||||
|
||||
```
|
||||
outputs/searchqa/2024-01-15_10-30-00/
|
||||
├── steps/
|
||||
│ ├── step_0001/
|
||||
│ │ ├── candidate_skill.md
|
||||
│ │ ├── step_record.json
|
||||
│ │ └── trajectory_digest.json
|
||||
│ └── step_0002/
|
||||
├── slow_update/
|
||||
│ └── epoch_02/
|
||||
├── meta_skill/
|
||||
│ └── epoch_02/
|
||||
├── skills/
|
||||
│ └── step_0001.md
|
||||
├── best_skill.md
|
||||
├── history.json
|
||||
└── config.yaml
|
||||
```
|
||||
|
||||
## 5. Evaluate
|
||||
|
||||
Evaluate the best skill on the test split:
|
||||
|
||||
```bash
|
||||
python scripts/eval_only.py \
|
||||
--config configs/searchqa/default.yaml \
|
||||
--skill outputs/searchqa/<run_id>/skills/best_skill.md
|
||||
```
|
||||
|
||||
## WebUI
|
||||
|
||||
Prefer a graphical interface? Launch the WebUI:
|
||||
|
||||
```bash
|
||||
pip install -e ".[webui]"
|
||||
python -m skillopt_webui.app
|
||||
```
|
||||
|
||||
Then open `http://localhost:7860` in your browser to configure parameters and launch training.
|
||||
|
||||
## Next Steps
|
||||
|
||||
- [Understand the training loop](training-loop.md)
|
||||
- [Configuration reference](../reference/config.md)
|
||||
- [Add a new benchmark](new-benchmark.md)
|
||||
@@ -1,89 +0,0 @@
|
||||
# Installation
|
||||
|
||||
## Requirements
|
||||
|
||||
- Python ≥ 3.10
|
||||
- At least one model API key (Azure OpenAI, OpenAI, Anthropic, or local Qwen)
|
||||
|
||||
## Quick Install
|
||||
|
||||
```bash
|
||||
git clone https://github.com/microsoft/SkillOpt.git
|
||||
cd SkillOpt
|
||||
pip install -e .
|
||||
```
|
||||
|
||||
## Optional Dependencies
|
||||
|
||||
Install extras for specific benchmarks or backends:
|
||||
|
||||
=== "ALFWorld"
|
||||
|
||||
```bash
|
||||
pip install -e ".[alfworld]"
|
||||
```
|
||||
|
||||
=== "Claude Backend"
|
||||
|
||||
```bash
|
||||
pip install -e ".[claude]"
|
||||
```
|
||||
|
||||
=== "Qwen (Local)"
|
||||
|
||||
```bash
|
||||
pip install -e ".[qwen]"
|
||||
```
|
||||
|
||||
=== "WebUI"
|
||||
|
||||
```bash
|
||||
pip install -e ".[webui]"
|
||||
```
|
||||
|
||||
=== "Development"
|
||||
|
||||
```bash
|
||||
pip install -e ".[dev]"
|
||||
```
|
||||
|
||||
=== "All"
|
||||
|
||||
```bash
|
||||
pip install -e ".[alfworld,claude,qwen,webui,dev]"
|
||||
```
|
||||
|
||||
## Environment Variables
|
||||
|
||||
Copy the example `.env` file and fill in your credentials:
|
||||
|
||||
```bash
|
||||
cp .env.example .env
|
||||
```
|
||||
|
||||
Edit `.env` with your API keys:
|
||||
|
||||
```ini
|
||||
# Azure OpenAI (default backend)
|
||||
AZURE_OPENAI_ENDPOINT=https://your-resource.openai.azure.com/
|
||||
AZURE_OPENAI_API_KEY=your-key
|
||||
|
||||
# Or use OpenAI directly
|
||||
OPENAI_API_KEY=sk-...
|
||||
|
||||
# Or Anthropic Claude
|
||||
ANTHROPIC_API_KEY=sk-ant-...
|
||||
```
|
||||
|
||||
!!! tip
|
||||
You only need credentials for the backend you plan to use. Azure OpenAI is the default.
|
||||
|
||||
## Verify Installation
|
||||
|
||||
```bash
|
||||
python -c "import skillopt; print('SkillOpt ready!')"
|
||||
```
|
||||
|
||||
## Next Steps
|
||||
|
||||
→ [Run your first experiment](first-experiment.md)
|
||||
@@ -1,143 +0,0 @@
|
||||
# Local Environment Smoke Tests
|
||||
|
||||
This guide describes a lightweight pattern for testing a custom SkillOpt environment before connecting it to expensive model calls or a full benchmark dataset.
|
||||
|
||||
The goal is to validate the training loop plumbing first:
|
||||
|
||||
- config loading
|
||||
- adapter construction
|
||||
- dataloader splits
|
||||
- rollout output shape
|
||||
- reflection patch shape
|
||||
- merge/rank/update control flow
|
||||
- artifact creation under `out_root`
|
||||
|
||||
Once those are stable, you can switch the same environment to real model calls and larger evaluation splits.
|
||||
|
||||
## 1. Add a tiny fixture split
|
||||
|
||||
Start with a handful of deterministic examples that cover the expected pass/fail cases for your environment. Keep them small enough that a single training step can run locally.
|
||||
|
||||
A minimal fixture item usually needs:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "example-1",
|
||||
"split": "train",
|
||||
"question": "...",
|
||||
"expected": "..."
|
||||
}
|
||||
```
|
||||
|
||||
Use the split names your adapter maps to SkillOpt phases:
|
||||
|
||||
- `train` for optimization rollouts
|
||||
- `val` or `valid_seen` for selection/gating
|
||||
- `test` or `valid_unseen` for final evaluation
|
||||
|
||||
## 2. Support an offline mock mode
|
||||
|
||||
Add a configuration flag such as `mock: true` to your adapter. In mock mode, `rollout()` should return deterministic responses without calling external model APIs.
|
||||
|
||||
This lets you verify the SkillOpt loop with a fast command such as:
|
||||
|
||||
```bash
|
||||
python scripts/train.py \
|
||||
--config configs/myenv/tiny_mock.yaml
|
||||
```
|
||||
|
||||
Mock mode should still write the same artifacts as a real run, for example:
|
||||
|
||||
- `responses.json`
|
||||
- `rollout_results.json`
|
||||
- `ranked_edits.json`
|
||||
- `candidate_skill.md`
|
||||
- `summary.json`
|
||||
|
||||
## 3. Keep the smoke config tiny
|
||||
|
||||
A CI-friendly smoke config should run a single small step:
|
||||
|
||||
```yaml
|
||||
train:
|
||||
num_epochs: 1
|
||||
train_size: 3
|
||||
batch_size: 3
|
||||
|
||||
gradient:
|
||||
minibatch_size: 1
|
||||
merge_batch_size: 2
|
||||
analyst_workers: 1
|
||||
max_analyst_rounds: 1
|
||||
|
||||
optimizer:
|
||||
learning_rate: 1
|
||||
min_learning_rate: 1
|
||||
lr_scheduler: constant
|
||||
skill_update_mode: patch
|
||||
use_slow_update: false
|
||||
|
||||
evaluation:
|
||||
use_gate: true
|
||||
sel_env_num: 2
|
||||
test_env_num: 2
|
||||
eval_test: false
|
||||
|
||||
env:
|
||||
name: myenv
|
||||
out_root: outputs/myenv_tiny_mock
|
||||
mock: true
|
||||
```
|
||||
|
||||
Prefer a mock config that runs without credentials. That makes it useful for contributors and CI.
|
||||
|
||||
## 4. Validate optimizer JSON before returning it
|
||||
|
||||
If your environment or extension asks an LLM to merge or rank skill edits, validate the returned JSON before passing it back into SkillOpt. This avoids silent fallbacks from empty, malformed, or out-of-range responses.
|
||||
|
||||
Useful checks for edit payloads:
|
||||
|
||||
- response is a JSON object
|
||||
- `edits` is a non-empty list
|
||||
- every edit is an object
|
||||
- every edit has an allowed operation
|
||||
- required fields such as `content` or `target` are present for that operation
|
||||
|
||||
Useful checks for ranking payloads:
|
||||
|
||||
- `selected_indices` exists
|
||||
- indices are integers
|
||||
- indices are unique
|
||||
- indices are within the candidate edit range
|
||||
- selected count does not exceed the edit budget
|
||||
|
||||
On failure, retry with a compact prompt that includes the schema error. If retries fail, raise an explicit error instead of silently accepting malformed output.
|
||||
|
||||
## 5. Run progressively stronger checks
|
||||
|
||||
A good development sequence is:
|
||||
|
||||
```bash
|
||||
python -m py_compile scripts/train.py skillopt/envs/myenv/adapter.py
|
||||
python scripts/train.py --config configs/myenv/tiny_mock.yaml
|
||||
python scripts/train.py --config configs/myenv/tiny.yaml
|
||||
```
|
||||
|
||||
For the real tiny run, verify that:
|
||||
|
||||
- the run completes
|
||||
- `summary.json` is written
|
||||
- `ranked_edits.json` contains the expected ranking metadata
|
||||
- any optimizer bridge log marks the response schema as valid
|
||||
- no generated files are written outside `out_root`
|
||||
|
||||
## 6. Keep custom environments isolated
|
||||
|
||||
When adding a custom environment to the registry, avoid side effects for existing benchmarks:
|
||||
|
||||
- lazy-import optional dependencies
|
||||
- install environment-specific hooks only when `cfg["env"]` matches your environment
|
||||
- keep mock behavior behind an explicit config flag
|
||||
- write generated artifacts only under `out_root`
|
||||
|
||||
This makes it easier to review and test a custom integration without affecting the built-in benchmarks.
|
||||
@@ -1,130 +0,0 @@
|
||||
# Add a New Model Backend
|
||||
|
||||
SkillOpt supports multiple LLM backends. This guide shows how to add your own.
|
||||
|
||||
## Backend Architecture
|
||||
|
||||
```
|
||||
skillopt/model/
|
||||
├── base.py # Abstract base class
|
||||
├── azure_openai.py # Azure OpenAI backend
|
||||
├── openai_model.py # Direct OpenAI backend
|
||||
├── claude.py # Anthropic Claude backend
|
||||
├── qwen.py # Local Qwen (vLLM) backend
|
||||
└── your_backend.py # Your new backend
|
||||
```
|
||||
|
||||
## Step 1: Create the Backend
|
||||
|
||||
Create `skillopt/model/your_backend.py`:
|
||||
|
||||
```python
|
||||
from skillopt.model.base import ModelBackend, ModelResponse
|
||||
|
||||
class YourBackend(ModelBackend):
|
||||
"""Your custom model backend."""
|
||||
|
||||
def __init__(self, cfg: dict):
|
||||
super().__init__(cfg)
|
||||
self.model_name = cfg.get('model_name', 'your-default-model')
|
||||
self.api_key = os.environ.get('YOUR_API_KEY', '')
|
||||
self.client = self._init_client()
|
||||
|
||||
def _init_client(self):
|
||||
"""Initialize API client."""
|
||||
# TODO: Set up your API client
|
||||
pass
|
||||
|
||||
async def generate(
|
||||
self,
|
||||
messages: list[dict],
|
||||
temperature: float = 0.7,
|
||||
max_tokens: int = 4096,
|
||||
**kwargs
|
||||
) -> ModelResponse:
|
||||
"""
|
||||
Generate a completion.
|
||||
|
||||
Args:
|
||||
messages: Chat messages [{"role": "...", "content": "..."}]
|
||||
temperature: Sampling temperature
|
||||
max_tokens: Maximum tokens in response
|
||||
|
||||
Returns:
|
||||
ModelResponse with content, usage, and metadata
|
||||
"""
|
||||
response = await self.client.chat(
|
||||
model=self.model_name,
|
||||
messages=messages,
|
||||
temperature=temperature,
|
||||
max_tokens=max_tokens,
|
||||
)
|
||||
|
||||
return ModelResponse(
|
||||
content=response.text,
|
||||
usage={
|
||||
'prompt_tokens': response.usage.input,
|
||||
'completion_tokens': response.usage.output,
|
||||
},
|
||||
model=self.model_name,
|
||||
)
|
||||
|
||||
async def generate_with_tools(
|
||||
self,
|
||||
messages: list[dict],
|
||||
tools: list[dict],
|
||||
**kwargs
|
||||
) -> ModelResponse:
|
||||
"""Generate with tool/function calling support."""
|
||||
# Optional: implement if your model supports tool use
|
||||
raise NotImplementedError("Tool use not supported")
|
||||
```
|
||||
|
||||
## Step 2: Register the Backend
|
||||
|
||||
Add to `skillopt/model/__init__.py`:
|
||||
|
||||
```python
|
||||
from .your_backend import YourBackend
|
||||
|
||||
BACKEND_REGISTRY = {
|
||||
# ... existing backends ...
|
||||
'your_backend': YourBackend,
|
||||
}
|
||||
```
|
||||
|
||||
## Step 3: Configure
|
||||
|
||||
Use your backend in any config:
|
||||
|
||||
```yaml
|
||||
model:
|
||||
backend: your_backend
|
||||
model_name: your-model-id
|
||||
temperature: 0.7
|
||||
max_tokens: 4096
|
||||
```
|
||||
|
||||
Set credentials via environment variable:
|
||||
|
||||
```bash
|
||||
export YOUR_API_KEY="your-key"
|
||||
```
|
||||
|
||||
## Required Interface
|
||||
|
||||
Your backend must implement these methods:
|
||||
|
||||
| Method | Required | Description |
|
||||
|---|---|---|
|
||||
| `generate()` | ✅ | Basic text generation |
|
||||
| `generate_with_tools()` | Optional | Tool/function calling |
|
||||
| `count_tokens()` | Optional | Token counting for context management |
|
||||
|
||||
## Tips
|
||||
|
||||
!!! tip
|
||||
- Test your backend with `python -c "from skillopt.model.your_backend import YourBackend"` first
|
||||
- Use `async` methods for all API calls — SkillOpt uses asyncio throughout
|
||||
- Implement retry logic with exponential backoff for production use
|
||||
- Add your API key to `.env.example` when submitting a PR
|
||||
@@ -1,181 +0,0 @@
|
||||
# Add a New Benchmark
|
||||
|
||||
Extend SkillOpt with your own benchmark in ~100 lines of code.
|
||||
|
||||
## Overview
|
||||
|
||||
To add a benchmark, you need:
|
||||
|
||||
1. **Data Loader** — Loads and splits your dataset
|
||||
2. **Environment Adapter** — Executes tasks and returns scores
|
||||
3. **Config** — YAML configuration file
|
||||
|
||||
## Step 1: Create the Benchmark Package
|
||||
|
||||
```bash
|
||||
mkdir -p skillopt/envs/my_benchmark
|
||||
touch skillopt/envs/my_benchmark/__init__.py
|
||||
```
|
||||
|
||||
## Step 2: Implement the Data Loader
|
||||
|
||||
Create `skillopt/envs/my_benchmark/loader.py`:
|
||||
|
||||
```python
|
||||
from skillopt.data.base import DataLoader, DataItem
|
||||
|
||||
class MyBenchmarkDataLoader(DataLoader):
|
||||
"""Load and split your benchmark data."""
|
||||
|
||||
def __init__(self, data_dir: str, **kwargs):
|
||||
super().__init__(**kwargs)
|
||||
self.data_dir = data_dir
|
||||
|
||||
def setup(self, cfg: dict):
|
||||
"""Initialize splits based on config."""
|
||||
self.split_mode = cfg.get('split_mode', 'ratio')
|
||||
# Load your data here
|
||||
self.items = self._load_items()
|
||||
self._create_splits(cfg)
|
||||
|
||||
def _load_items(self) -> list[DataItem]:
|
||||
"""Load raw data into DataItem objects."""
|
||||
items = []
|
||||
# TODO: Load your data
|
||||
for entry in your_data:
|
||||
items.append(DataItem(
|
||||
id=entry['id'],
|
||||
input=entry['question'],
|
||||
ground_truth=entry['answer'],
|
||||
metadata=entry.get('metadata', {})
|
||||
))
|
||||
return items
|
||||
|
||||
def get_split_items(self, split: str) -> list[DataItem]:
|
||||
"""Return items for a given split (train/valid/test)."""
|
||||
return self.splits[split]
|
||||
```
|
||||
|
||||
## Step 3: Implement the Environment Adapter
|
||||
|
||||
Create `skillopt/envs/my_benchmark/env.py`:
|
||||
|
||||
```python
|
||||
from skillopt.envs.base import EnvAdapter, TaskResult
|
||||
|
||||
class MyBenchmarkEnv(EnvAdapter):
|
||||
"""Execute tasks and evaluate results."""
|
||||
|
||||
def __init__(self, cfg: dict):
|
||||
super().__init__(cfg)
|
||||
|
||||
async def execute(self, item: DataItem, skill: str, model) -> TaskResult:
|
||||
"""
|
||||
Execute a single task.
|
||||
|
||||
Args:
|
||||
item: The data item to process
|
||||
skill: Current skill document content
|
||||
model: The target model instance
|
||||
|
||||
Returns:
|
||||
TaskResult with prediction, score, and trajectory
|
||||
"""
|
||||
# Build prompt with skill document
|
||||
prompt = self.build_prompt(item, skill)
|
||||
|
||||
# Get model response
|
||||
response = await model.generate(prompt)
|
||||
|
||||
# Extract prediction
|
||||
prediction = self.parse_response(response)
|
||||
|
||||
# Score against ground truth
|
||||
score = self.evaluate(prediction, item.ground_truth)
|
||||
|
||||
return TaskResult(
|
||||
item_id=item.id,
|
||||
prediction=prediction,
|
||||
score=score,
|
||||
trajectory=[
|
||||
{"role": "system", "content": skill},
|
||||
{"role": "user", "content": item.input},
|
||||
{"role": "assistant", "content": response}
|
||||
]
|
||||
)
|
||||
|
||||
def evaluate(self, prediction: str, ground_truth: str) -> float:
|
||||
"""
|
||||
Score a prediction against ground truth.
|
||||
|
||||
Returns:
|
||||
Float between 0.0 and 1.0
|
||||
"""
|
||||
# TODO: Implement your scoring logic
|
||||
# Examples: exact match, F1, ANLS, etc.
|
||||
return float(prediction.strip() == ground_truth.strip())
|
||||
|
||||
def build_prompt(self, item, skill: str) -> str:
|
||||
"""Combine skill document with task input."""
|
||||
return f"{skill}\n\n---\n\nQuestion: {item.input}"
|
||||
|
||||
def parse_response(self, response: str) -> str:
|
||||
"""Extract the answer from model response."""
|
||||
return response.strip()
|
||||
```
|
||||
|
||||
## Step 4: Register the Benchmark
|
||||
|
||||
Add to `skillopt/envs/__init__.py`:
|
||||
|
||||
```python
|
||||
from .my_benchmark.env import MyBenchmarkEnv
|
||||
from .my_benchmark.loader import MyBenchmarkDataLoader
|
||||
|
||||
BENCHMARK_REGISTRY = {
|
||||
# ... existing benchmarks ...
|
||||
'my_benchmark': {
|
||||
'env': MyBenchmarkEnv,
|
||||
'loader': MyBenchmarkDataLoader,
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
## Step 5: Create Config
|
||||
|
||||
Create `configs/my_benchmark/default.yaml`:
|
||||
|
||||
```yaml
|
||||
_base_: ['../_base_/default.yaml']
|
||||
|
||||
env:
|
||||
name: my_benchmark
|
||||
data_path: data/my_benchmark
|
||||
split_mode: ratio
|
||||
split_ratio: "2:1:7"
|
||||
|
||||
train:
|
||||
num_epochs: 4
|
||||
batch_size: 40
|
||||
|
||||
optimizer:
|
||||
learning_rate: 4
|
||||
lr_scheduler: cosine
|
||||
use_slow_update: true
|
||||
use_meta_skill: true
|
||||
|
||||
gradient:
|
||||
analyst_workers: 16
|
||||
```
|
||||
|
||||
## Step 6: Run
|
||||
|
||||
```bash
|
||||
python scripts/train.py --config configs/my_benchmark/default.yaml
|
||||
```
|
||||
|
||||
## Tips
|
||||
|
||||
!!! tip
|
||||
- Use a small `batch_size` (10-20) for initial testing
|
||||
- The `evaluate()` method is critical — a noisy metric will confuse the optimizer
|
||||
@@ -1,78 +0,0 @@
|
||||
# Skill Document
|
||||
|
||||
A **skill document** is a Markdown file that serves as the "prompt weights" of your agent. SkillOpt trains this document through iterative optimization.
|
||||
|
||||
## What is a Skill Document?
|
||||
|
||||
A skill document is a structured set of instructions that tells a language model **how** to approach a specific type of task. It's analogous to learned weights in a neural network — encoding task-specific knowledge in natural language rather than floating-point parameters.
|
||||
|
||||
## Structure
|
||||
|
||||
A typical skill document contains:
|
||||
|
||||
```markdown
|
||||
# Task Strategy
|
||||
|
||||
## General Approach
|
||||
- Break complex problems into sub-steps
|
||||
- Always verify intermediate results
|
||||
|
||||
## Common Patterns
|
||||
- When you see X, try approach Y
|
||||
- Avoid Z because it leads to errors
|
||||
|
||||
## Edge Cases
|
||||
- If the input contains A, handle it specially by...
|
||||
- Watch out for B — it requires C
|
||||
|
||||
## Output Format
|
||||
- Always include reasoning before the answer
|
||||
- Format numbers with proper units
|
||||
```
|
||||
|
||||
## How It Evolves
|
||||
|
||||
During training, the skill document is modified by **edit patches**:
|
||||
|
||||
1. **Additions**: New rules or strategies discovered from failed trajectories
|
||||
2. **Modifications**: Refining existing rules that are partially correct
|
||||
3. **Deletions**: Removing rules that consistently lead to errors
|
||||
|
||||
Each edit is validated through the **gate** mechanism before being permanently accepted.
|
||||
|
||||
## Initial Skill
|
||||
|
||||
You can start training with:
|
||||
|
||||
- **Empty skill**: The system learns everything from scratch
|
||||
- **Seed skill**: Provide initial instructions to bootstrap training
|
||||
- **Pre-trained skill**: Transfer a skill from a related benchmark
|
||||
|
||||
Configure the initial skill in your YAML:
|
||||
|
||||
```yaml
|
||||
train:
|
||||
init_skill: "path/to/initial_skill.md" # or omit for empty
|
||||
```
|
||||
|
||||
## Skill Quality Metrics
|
||||
|
||||
Track your skill's evolution through:
|
||||
|
||||
- **Validation score**: Primary metric on the selection split
|
||||
- **Test score**: Final metric on held-out test data
|
||||
- **Skill length**: Total tokens in the document
|
||||
- **Edit acceptance rate**: Fraction of proposed edits that pass gating
|
||||
|
||||
## Best Practices
|
||||
|
||||
!!! tip "Tips for better skills"
|
||||
1. **Start with a seed skill** (`env.skill_init`) if you have domain knowledge — it converges faster
|
||||
2. **Use cosine LR schedule** — aggressive early exploration + careful late refinement
|
||||
3. **Enable slow update** (`use_slow_update: true`) to prevent forgetting across epochs
|
||||
4. **Enable meta skill** (`use_meta_skill: true`) so the optimizer accumulates strategy memory
|
||||
|
||||
## Next Steps
|
||||
|
||||
- [Deep Learning Analogy](dl-analogy.md)
|
||||
- [Configuration Reference](../reference/config.md)
|
||||
@@ -1,92 +0,0 @@
|
||||
# The Training Loop
|
||||
|
||||
SkillOpt's core insight: **optimizing natural-language skill documents follows the same structure as training neural networks**.
|
||||
|
||||
## Overview
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────┐
|
||||
│ Training Loop │
|
||||
│ │
|
||||
│ for epoch in epochs: │
|
||||
│ for step in steps: │
|
||||
│ 1. Rollout — Target executes tasks │
|
||||
│ 2. Reflect — Optimizer analyzes trajectories │
|
||||
│ 3. Aggregate — Hierarchical merge of patches │
|
||||
│ 4. Select — Rank & clip edits (learning rate) │
|
||||
│ 5. Update — Apply patches to skill doc │
|
||||
│ 6. Gate — Validate & accept/reject │
|
||||
│ │
|
||||
│ Epoch Boundary: │
|
||||
│ • Slow Update (longitudinal comparison & guidance) │
|
||||
│ • Meta Skill (cross-epoch strategy memory) │
|
||||
└─────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Stage Details
|
||||
|
||||
### 1. Rollout (Forward Pass)
|
||||
|
||||
The **target** model executes tasks using the current skill document as its prompt. Each task produces a trajectory and a score.
|
||||
|
||||
```python
|
||||
# Analogy: forward pass through the network
|
||||
predictions = model(input, skill_document)
|
||||
scores = evaluate(predictions, ground_truth)
|
||||
```
|
||||
|
||||
### 2. Reflect (Backward Pass)
|
||||
|
||||
The **optimizer** model analyzes failed trajectories and produces **edit patches** — structured suggestions for improving the skill document.
|
||||
|
||||
Two modes:
|
||||
|
||||
- **Shallow**: Analyze each trajectory independently
|
||||
- **Deep**: Cross-reference multiple failures to find systemic issues
|
||||
|
||||
```python
|
||||
# Analogy: computing gradients
|
||||
gradients = loss.backward() # → edit patches
|
||||
```
|
||||
|
||||
### 3. Aggregate
|
||||
|
||||
Semantically similar edit patches are merged to avoid redundant edits.
|
||||
|
||||
### 4. Select (Gradient Clipping)
|
||||
|
||||
Edits are ranked by relevance score. The `learning_rate` parameter caps how many edits are applied per step — just like gradient clipping prevents overshooting.
|
||||
|
||||
```python
|
||||
# Analogy: gradient clipping + optimizer step size
|
||||
selected = top_k(edits, k=learning_rate)
|
||||
```
|
||||
|
||||
The `lr_scheduler` adjusts this over training:
|
||||
|
||||
- **cosine**: Start aggressive, taper smoothly
|
||||
- **linear**: Linear decay
|
||||
- **constant**: Fixed rate
|
||||
|
||||
### 5. Update (Parameter Update)
|
||||
|
||||
Selected edits are applied to the skill document, producing a new version.
|
||||
|
||||
### 6. Gate (Validation)
|
||||
|
||||
The updated skill is evaluated on a **selection split** (analogous to a validation set). The update is only accepted if performance improves.
|
||||
|
||||
## Epoch Boundary Mechanisms
|
||||
|
||||
### Slow Update
|
||||
|
||||
At the end of each epoch (starting from epoch 2), the system performs a **longitudinal comparison**: it rolls out both the previous epoch's skill and the current skill on the same samples, categorizes items as improved/regressed/persistent_fail/stable_success, then generates high-level **guidance** that is injected into the skill document. This prevents catastrophic forgetting of earlier improvements.
|
||||
|
||||
### Meta Skill
|
||||
|
||||
A **meta-skill memory** accumulates high-level strategy notes across the entire training run. At the end of each epoch, the optimizer reflects on what changed between epochs and produces a compact memory that is provided as additional context during future reflection steps.
|
||||
|
||||
## Next Steps
|
||||
|
||||
- [Understand Skill Documents](skill-document.md)
|
||||
- [DL ↔ SkillOpt analogy table](dl-analogy.md)
|
||||
@@ -1,911 +0,0 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>SkillOpt — Documentation & Reproduction Guide</title>
|
||||
<meta name="description" content="Complete documentation and reproduction guide for SkillOpt: installation, data preparation, training, configuration reference, framework internals, and API reference.">
|
||||
<link rel="icon" type="image/svg+xml" href="data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 23 23'%3E%3Crect width='10' height='10' fill='%23F25022'/%3E%3Crect x='13' width='10' height='10' fill='%237FBA00'/%3E%3Crect y='13' width='10' height='10' fill='%2300A4EF'/%3E%3Crect x='13' y='13' width='10' height='10' fill='%23FFB900'/%3E%3C/svg%3E">
|
||||
<style>
|
||||
:root {
|
||||
--bg: #ffffff;
|
||||
--bg-soft: #f7f8fb;
|
||||
--sidebar-bg: #fbfcfe;
|
||||
--ink: #1f2733;
|
||||
--muted: #5b6675;
|
||||
--quiet: #8a94a3;
|
||||
--line: #e6e9ef;
|
||||
--line-strong: #d3d9e3;
|
||||
--brand: #4f46e5;
|
||||
--brand-soft: #eef0fe;
|
||||
--accent: #0ea5e9;
|
||||
--green: #16a34a;
|
||||
--amber: #d97706;
|
||||
--red: #dc2626;
|
||||
--code-bg: #0f172a;
|
||||
--code-ink: #e2e8f0;
|
||||
--inline-code-bg: #eef1f6;
|
||||
--inline-code-ink: #b3146b;
|
||||
--sidebar-w: 300px;
|
||||
--toc-w: 220px;
|
||||
--mono: "SFMono-Regular", "JetBrains Mono", Consolas, "Liberation Mono", monospace;
|
||||
--sans: "Inter", -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif;
|
||||
}
|
||||
* { box-sizing: border-box; }
|
||||
html { scroll-behavior: smooth; }
|
||||
body {
|
||||
margin: 0;
|
||||
font-family: var(--sans);
|
||||
color: var(--ink);
|
||||
background: var(--bg);
|
||||
font-size: 15px;
|
||||
line-height: 1.65;
|
||||
-webkit-font-smoothing: antialiased;
|
||||
}
|
||||
|
||||
/* ── Top bar ─────────────────────────────────────────── */
|
||||
header.topbar {
|
||||
position: sticky; top: 0; z-index: 40;
|
||||
height: 56px;
|
||||
display: flex; align-items: center; gap: 14px;
|
||||
padding: 0 20px;
|
||||
background: rgba(255,255,255,0.92);
|
||||
backdrop-filter: blur(8px);
|
||||
border-bottom: 1px solid var(--line);
|
||||
}
|
||||
.topbar .logo { width: 22px; height: 22px; flex: none; }
|
||||
.topbar .brand { font-weight: 700; font-size: 16px; letter-spacing: -0.01em; }
|
||||
.topbar .brand span { color: var(--brand); }
|
||||
.topbar .tag { color: var(--quiet); font-size: 13px; border-left: 1px solid var(--line-strong); padding-left: 14px; }
|
||||
.topbar .spacer { flex: 1; }
|
||||
.topbar a.gh {
|
||||
display: inline-flex; align-items: center; gap: 6px;
|
||||
font-size: 13px; font-weight: 600; color: var(--muted);
|
||||
text-decoration: none; padding: 6px 12px; border: 1px solid var(--line-strong);
|
||||
border-radius: 8px;
|
||||
}
|
||||
.topbar a.gh:hover { color: var(--brand); border-color: var(--brand); }
|
||||
#menuBtn {
|
||||
display: none; background: none; border: 1px solid var(--line-strong);
|
||||
border-radius: 8px; width: 38px; height: 34px; cursor: pointer; font-size: 18px; color: var(--muted);
|
||||
}
|
||||
|
||||
/* ── Layout ──────────────────────────────────────────── */
|
||||
.layout { display: flex; align-items: flex-start; }
|
||||
|
||||
/* ── Sidebar (left nav) ──────────────────────────────── */
|
||||
nav.sidebar {
|
||||
position: sticky; top: 56px;
|
||||
width: var(--sidebar-w); flex: none;
|
||||
height: calc(100vh - 56px);
|
||||
overflow-y: auto;
|
||||
background: var(--sidebar-bg);
|
||||
border-right: 1px solid var(--line);
|
||||
padding: 22px 14px 60px 20px;
|
||||
}
|
||||
nav.sidebar .group { margin-bottom: 22px; }
|
||||
nav.sidebar .group > .glabel {
|
||||
display: flex; align-items: center; gap: 8px;
|
||||
font-size: 11.5px; font-weight: 700; text-transform: uppercase;
|
||||
letter-spacing: 0.07em; color: var(--quiet);
|
||||
margin: 0 0 8px 2px;
|
||||
}
|
||||
nav.sidebar .group > .glabel .num {
|
||||
display: inline-flex; align-items: center; justify-content: center;
|
||||
width: 18px; height: 18px; border-radius: 5px;
|
||||
background: var(--brand-soft); color: var(--brand);
|
||||
font-size: 11px; font-weight: 700;
|
||||
}
|
||||
nav.sidebar a {
|
||||
display: block; text-decoration: none;
|
||||
color: var(--muted); font-size: 13.5px;
|
||||
padding: 5px 10px; border-radius: 7px; margin: 1px 0;
|
||||
border-left: 2px solid transparent;
|
||||
}
|
||||
nav.sidebar a:hover { background: #eef1f6; color: var(--ink); }
|
||||
nav.sidebar a.active {
|
||||
color: var(--brand); background: var(--brand-soft);
|
||||
border-left-color: var(--brand); font-weight: 600;
|
||||
}
|
||||
|
||||
/* ── Content ─────────────────────────────────────────── */
|
||||
main.content {
|
||||
flex: 1; min-width: 0;
|
||||
padding: 38px 46px 120px;
|
||||
max-width: 900px;
|
||||
}
|
||||
main.content section { scroll-margin-top: 72px; }
|
||||
main h1 { font-size: 30px; line-height: 1.2; letter-spacing: -0.02em; margin: 0 0 8px; }
|
||||
main h2 {
|
||||
font-size: 23px; letter-spacing: -0.015em; margin: 52px 0 14px;
|
||||
padding-bottom: 10px; border-bottom: 1px solid var(--line);
|
||||
}
|
||||
main section:first-of-type h2 { margin-top: 8px; }
|
||||
main h3 { font-size: 17.5px; margin: 30px 0 10px; letter-spacing: -0.01em; }
|
||||
main h4 { font-size: 15px; margin: 22px 0 8px; color: var(--ink); }
|
||||
main p { margin: 12px 0; color: #2c3645; }
|
||||
main ul, main ol { margin: 12px 0; padding-left: 22px; }
|
||||
main li { margin: 5px 0; }
|
||||
main a { color: var(--brand); text-decoration: none; }
|
||||
main a:hover { text-decoration: underline; }
|
||||
.lead { font-size: 16.5px; color: var(--muted); margin: 6px 0 4px; }
|
||||
.eyebrow { color: var(--brand); font-weight: 700; font-size: 12.5px; letter-spacing: 0.08em; text-transform: uppercase; }
|
||||
|
||||
/* code */
|
||||
code {
|
||||
font-family: var(--mono); font-size: 0.86em;
|
||||
background: var(--inline-code-bg); color: var(--inline-code-ink);
|
||||
padding: 2px 6px; border-radius: 5px;
|
||||
}
|
||||
pre {
|
||||
background: var(--code-bg); color: var(--code-ink);
|
||||
border-radius: 12px; padding: 16px 18px; overflow-x: auto;
|
||||
font-family: var(--mono); font-size: 13px; line-height: 1.6;
|
||||
margin: 14px 0; border: 1px solid #1e293b;
|
||||
}
|
||||
pre code { background: none; color: inherit; padding: 0; font-size: inherit; }
|
||||
.tok-c { color: #7c8aa5; } /* comment */
|
||||
.tok-k { color: #c4b5fd; } /* keyword */
|
||||
.tok-s { color: #86efac; } /* string */
|
||||
.tok-f { color: #93c5fd; } /* flag/path */
|
||||
.tok-n { color: #fca5a5; } /* number/value */
|
||||
|
||||
/* tables */
|
||||
.table-wrap { overflow-x: auto; margin: 16px 0; border: 1px solid var(--line); border-radius: 12px; }
|
||||
table { border-collapse: collapse; width: 100%; font-size: 13.5px; }
|
||||
th, td { text-align: left; padding: 9px 13px; border-bottom: 1px solid var(--line); vertical-align: top; }
|
||||
thead th { background: var(--bg-soft); font-weight: 700; color: var(--ink); white-space: nowrap; }
|
||||
tbody tr:last-child td { border-bottom: none; }
|
||||
td code { white-space: nowrap; }
|
||||
td.def { color: var(--muted); font-family: var(--mono); font-size: 12px; }
|
||||
|
||||
/* callouts */
|
||||
.note { border-radius: 10px; padding: 12px 16px; margin: 16px 0; border: 1px solid; font-size: 14px; }
|
||||
.note p { margin: 4px 0; }
|
||||
.note .nh { font-weight: 700; display: block; margin-bottom: 2px; }
|
||||
.note.info { background: #eff6ff; border-color: #bfdbfe; }
|
||||
.note.info .nh { color: #1d4ed8; }
|
||||
.note.tip { background: #ecfdf5; border-color: #a7f3d0; }
|
||||
.note.tip .nh { color: #047857; }
|
||||
.note.warn { background: #fffbeb; border-color: #fde68a; }
|
||||
.note.warn .nh { color: #b45309; }
|
||||
|
||||
.pill { display:inline-block; font-size: 11px; font-weight:700; padding: 1px 8px; border-radius: 999px; vertical-align: middle; }
|
||||
.pill.def { background:#eef2ff; color:#4338ca; }
|
||||
.pill.opt { background:#f1f5f9; color:#475569; }
|
||||
|
||||
/* card grid */
|
||||
.cards { display: grid; grid-template-columns: repeat(auto-fit, minmax(230px,1fr)); gap: 14px; margin: 18px 0; }
|
||||
.card { border: 1px solid var(--line); border-radius: 12px; padding: 16px; background: var(--bg-soft); }
|
||||
.card h4 { margin: 0 0 6px; font-size: 14.5px; }
|
||||
.card p { margin: 0; font-size: 13px; color: var(--muted); }
|
||||
|
||||
/* anchor link on hover */
|
||||
.anchor { color: var(--quiet); text-decoration: none; font-weight: 400; opacity: 0; margin-left: 8px; font-size: 0.8em; }
|
||||
h2:hover .anchor, h3:hover .anchor { opacity: 1; }
|
||||
|
||||
/* ── Right TOC ───────────────────────────────────────── */
|
||||
aside.toc {
|
||||
position: sticky; top: 56px;
|
||||
width: var(--toc-w); flex: none;
|
||||
height: calc(100vh - 56px); overflow-y: auto;
|
||||
padding: 38px 18px; border-left: 1px solid var(--line);
|
||||
}
|
||||
aside.toc .tl { font-size: 11.5px; font-weight: 700; text-transform: uppercase; letter-spacing: 0.07em; color: var(--quiet); margin-bottom: 10px; }
|
||||
aside.toc a { display: block; color: var(--muted); text-decoration: none; font-size: 12.5px; padding: 4px 8px; border-left: 2px solid var(--line); line-height: 1.45; }
|
||||
aside.toc a:hover { color: var(--ink); }
|
||||
aside.toc a.active { color: var(--brand); border-left-color: var(--brand); font-weight: 600; }
|
||||
|
||||
.footer-note { margin-top: 60px; padding-top: 20px; border-top: 1px solid var(--line); color: var(--quiet); font-size: 13px; }
|
||||
|
||||
/* responsive */
|
||||
@media (max-width: 1180px) { aside.toc { display: none; } }
|
||||
@media (max-width: 860px) {
|
||||
#menuBtn { display: inline-block; }
|
||||
nav.sidebar {
|
||||
position: fixed; left: 0; top: 56px; z-index: 35;
|
||||
transform: translateX(-100%); transition: transform 0.22s ease;
|
||||
box-shadow: 0 16px 40px rgba(15,23,42,0.18);
|
||||
}
|
||||
nav.sidebar.open { transform: translateX(0); }
|
||||
main.content { padding: 28px 20px 100px; }
|
||||
.topbar .tag { display: none; }
|
||||
}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
|
||||
<header class="topbar">
|
||||
<button id="menuBtn" aria-label="Toggle navigation">☰</button>
|
||||
<svg class="logo" viewBox="0 0 23 23"><rect width="10" height="10" fill="#F25022"/><rect x="13" width="10" height="10" fill="#7FBA00"/><rect y="13" width="10" height="10" fill="#00A4EF"/><rect x="13" y="13" width="10" height="10" fill="#FFB900"/></svg>
|
||||
<span class="brand">Skill<span>Opt</span></span>
|
||||
<span class="tag">Documentation & Reproduction Guide</span>
|
||||
<span class="spacer"></span>
|
||||
<a class="gh" href="https://github.com/microsoft/SkillOpt" target="_blank" rel="noopener">GitHub ↗</a>
|
||||
<a class="gh" href="https://arxiv.org/abs/2605.23904" target="_blank" rel="noopener">Paper ↗</a>
|
||||
</header>
|
||||
|
||||
<div class="layout">
|
||||
|
||||
<!-- ───────────── LEFT NAV ───────────── -->
|
||||
<nav class="sidebar" id="sidebar">
|
||||
<div class="group">
|
||||
<div class="glabel"><span class="num">1</span> Overview</div>
|
||||
<a href="#what-is">What is SkillOpt</a>
|
||||
<a href="#analogy">DL ↔ SkillOpt analogy</a>
|
||||
<a href="#features">Key features</a>
|
||||
<a href="#layout">Repository layout</a>
|
||||
</div>
|
||||
<div class="group">
|
||||
<div class="glabel"><span class="num">2</span> Installation</div>
|
||||
<a href="#requirements">Requirements</a>
|
||||
<a href="#install">Install the package</a>
|
||||
<a href="#credentials">Configure credentials</a>
|
||||
<a href="#verify">Verify installation</a>
|
||||
</div>
|
||||
<div class="group">
|
||||
<div class="glabel"><span class="num">3</span> Data Preparation</div>
|
||||
<a href="#split-dir">Split directory format</a>
|
||||
<a href="#item-schema">Item JSON schema</a>
|
||||
<a href="#split-modes">Split modes</a>
|
||||
</div>
|
||||
<div class="group">
|
||||
<div class="glabel"><span class="num">4</span> Quick Start</div>
|
||||
<a href="#train">Train a skill</a>
|
||||
<a href="#eval">Evaluate a skill</a>
|
||||
<a href="#outputs">Output structure</a>
|
||||
<a href="#resume">Auto-resume</a>
|
||||
</div>
|
||||
<div class="group">
|
||||
<div class="glabel"><span class="num">5</span> How It Works</div>
|
||||
<a href="#loop">The training loop</a>
|
||||
<a href="#stages">The six per-step stages</a>
|
||||
<a href="#gate">Validation gate</a>
|
||||
<a href="#slow-update">Slow update (momentum)</a>
|
||||
<a href="#meta-skill">Meta skill (memory)</a>
|
||||
<a href="#skill-doc">Skill document anatomy</a>
|
||||
</div>
|
||||
<div class="group">
|
||||
<div class="glabel"><span class="num">6</span> Configuration</div>
|
||||
<a href="#config-system">Config system</a>
|
||||
<a href="#cfg-model">model.*</a>
|
||||
<a href="#cfg-train">train.*</a>
|
||||
<a href="#cfg-gradient">gradient.*</a>
|
||||
<a href="#cfg-optimizer">optimizer.*</a>
|
||||
<a href="#cfg-evaluation">evaluation.*</a>
|
||||
<a href="#cfg-env">env.*</a>
|
||||
</div>
|
||||
<div class="group">
|
||||
<div class="glabel"><span class="num">7</span> Benchmarks</div>
|
||||
<a href="#bench-list">Supported benchmarks</a>
|
||||
<a href="#bench-new">Add a new benchmark</a>
|
||||
</div>
|
||||
<div class="group">
|
||||
<div class="glabel"><span class="num">8</span> API Reference</div>
|
||||
<a href="#module-map">Module map</a>
|
||||
<a href="#functions">Core functions</a>
|
||||
<a href="#cli">CLI scripts</a>
|
||||
<a href="#webui">WebUI</a>
|
||||
</div>
|
||||
</nav>
|
||||
|
||||
<!-- ───────────── MAIN CONTENT ───────────── -->
|
||||
<main class="content">
|
||||
|
||||
<span class="eyebrow">Microsoft Research</span>
|
||||
<h1>SkillOpt Documentation & Reproduction Guide</h1>
|
||||
<p class="lead">Train agent skills like you train neural networks — with epochs, (mini-)batch size, learning rates, and validation gates — but without touching any model weights.</p>
|
||||
<p>This guide walks you from a clean checkout to a reproduced result and a full reference for every configuration knob and core function. It is generated from, and kept consistent with, the current state of the codebase.</p>
|
||||
|
||||
<!-- ===================== 1. OVERVIEW ===================== -->
|
||||
<section id="what-is">
|
||||
<h2>1.1 What is SkillOpt <a class="anchor" href="#what-is">#</a></h2>
|
||||
<p><strong>SkillOpt</strong> is a text-space optimizer that improves a <em>frozen</em> language agent by iteratively editing a natural-language <strong>skill document</strong> — never the model weights. The skill document is a Markdown file that conditions a target model as it executes tasks. SkillOpt treats this document as the "weights" and runs a training loop that mirrors deep-learning training: rollout (forward pass), reflect (backward pass / gradients), select & apply edits (optimizer step), and a validation gate (accept/reject).</p>
|
||||
<p>Two roles split every model call:</p>
|
||||
<ul>
|
||||
<li><strong>Target</strong> — executes tasks using the current skill document (the agent being improved).</li>
|
||||
<li><strong>Optimizer</strong> — analyzes the target's trajectories and proposes edits to the skill document.</li>
|
||||
</ul>
|
||||
<p>The same loop drives six benchmarks out of the box (QA, document QA, embodied agents, math, spreadsheet code generation, and tool-augmented QA).</p>
|
||||
</section>
|
||||
|
||||
<section id="analogy">
|
||||
<h2>1.2 Deep-Learning ↔ SkillOpt Analogy <a class="anchor" href="#analogy">#</a></h2>
|
||||
<p>Every concept below maps to a concrete code construct, so deep-learning intuitions transfer directly to hyperparameter tuning.</p>
|
||||
<div class="table-wrap">
|
||||
<table>
|
||||
<thead><tr><th>Deep learning</th><th>SkillOpt</th><th>Where it lives</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td>Model weights</td><td>Skill document (Markdown)</td><td><code>skillopt/optimizer/skill.py</code></td></tr>
|
||||
<tr><td>Forward pass</td><td>Rollout — target runs tasks</td><td><code>envs/<bench>/rollout.py</code></td></tr>
|
||||
<tr><td>Loss / score</td><td>Task evaluator</td><td><code>envs/<bench>/evaluator.py</code></td></tr>
|
||||
<tr><td>Backprop / gradients</td><td>Reflect → edit patches</td><td><code>gradient/reflect.py</code></td></tr>
|
||||
<tr><td>Gradient aggregation</td><td>Hierarchical patch merge</td><td><code>gradient/aggregate.py</code></td></tr>
|
||||
<tr><td>Gradient clipping</td><td>Rank & select top-k edits</td><td><code>optimizer/clip.py</code></td></tr>
|
||||
<tr><td>Learning rate</td><td><code>optimizer.learning_rate</code> (edits/step)</td><td><code>optimizer/scheduler.py</code></td></tr>
|
||||
<tr><td>LR scheduler</td><td><code>lr_scheduler</code> (cosine/linear/…)</td><td><code>optimizer/scheduler.py</code></td></tr>
|
||||
<tr><td>Optimizer step</td><td>Apply patches to the document</td><td><code>optimizer/skill.py</code></td></tr>
|
||||
<tr><td>Validation set</td><td>Selection split (<code>valid_seen</code>)</td><td><code>evaluation/gate.py</code></td></tr>
|
||||
<tr><td>Early stopping / accept</td><td>Validation gate</td><td><code>evaluation/gate.py</code></td></tr>
|
||||
<tr><td>Momentum</td><td>Slow update (epoch boundary)</td><td><code>optimizer/slow_update.py</code></td></tr>
|
||||
<tr><td>Meta-learning</td><td>Meta skill (cross-epoch memory)</td><td><code>optimizer/meta_skill.py</code></td></tr>
|
||||
<tr><td>Batch / minibatch</td><td><code>batch_size</code> / <code>minibatch_size</code></td><td><code>engine/trainer.py</code></td></tr>
|
||||
<tr><td>Epoch</td><td>Epoch (+ slow update & meta skill)</td><td><code>engine/trainer.py</code></td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
<div class="note tip"><span class="nh">What transfers from DL</span>
|
||||
<p>Cosine schedule tends to beat constant; moderate learning rates (≈4–16 edits/step) beat very high/low; slow update curbs cross-epoch forgetting; meta-skill memory improves reflection quality. Conversely, bigger rollout batches and many epochs show diminishing returns — skills converge in ~2–4 epochs.</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section id="features">
|
||||
<h2>1.3 Key Features <a class="anchor" href="#features">#</a></h2>
|
||||
<div class="cards">
|
||||
<div class="card"><h4>Validation gating</h4><p>Every candidate skill is scored on a held-out selection split and only accepted if it beats the current/best skill.</p></div>
|
||||
<div class="card"><h4>Slow update</h4><p>Epoch-boundary longitudinal comparison writes guidance into a protected region — momentum against forgetting. Force-injected or selection-gated.</p></div>
|
||||
<div class="card"><h4>Meta skill</h4><p>Optimizer-side memory that reflects on what worked across epochs and feeds back into reflection.</p></div>
|
||||
<div class="card"><h4>Pluggable backends</h4><p>OpenAI / Azure OpenAI, Anthropic Claude, local Qwen (vLLM), plus Codex/Claude-Code exec backends for the target.</p></div>
|
||||
<div class="card"><h4>Six benchmarks</h4><p>SearchQA, DocVQA, ALFWorld, LiveMathematicianBench, SpreadsheetBench, OfficeQA — each a self-contained env module.</p></div>
|
||||
<div class="card"><h4>Auto-resume</h4><p>Every run is checkpointed step-by-step; re-running the same command continues from the last completed step.</p></div>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section id="layout">
|
||||
<h2>1.4 Repository Layout <a class="anchor" href="#layout">#</a></h2>
|
||||
<pre><code><span class="tok-c"># top level</span>
|
||||
configs/ <span class="tok-c"># YAML configs (_base_ + per-benchmark)</span>
|
||||
scripts/ <span class="tok-c"># train.py, eval_only.py CLIs</span>
|
||||
ckpt/ <span class="tok-c"># packaged reference skills (e.g. gpt5.5_skill.md)</span>
|
||||
docs/ <span class="tok-c"># this guide + mkdocs sources</span>
|
||||
skillopt/ <span class="tok-c"># the package</span>
|
||||
├─ config.py <span class="tok-c"># YAML loading, _base_ inheritance, flatten</span>
|
||||
├─ engine/trainer.py<span class="tok-c"># the training loop (ReflACTTrainer)</span>
|
||||
├─ gradient/ <span class="tok-c"># reflect.py (analyst), aggregate.py (merge)</span>
|
||||
├─ optimizer/ <span class="tok-c"># skill edits, scheduler, clip, slow_update, meta_skill</span>
|
||||
├─ evaluation/ <span class="tok-c"># gate.py (accept/reject logic)</span>
|
||||
├─ model/ <span class="tok-c"># backend clients + routing</span>
|
||||
└─ envs/<benchmark>/ <span class="tok-c"># adapter, dataloader, rollout, evaluator, reflect</span></code></pre>
|
||||
</section>
|
||||
|
||||
<!-- ===================== 2. INSTALLATION ===================== -->
|
||||
<section id="requirements">
|
||||
<h2>2.1 Requirements <a class="anchor" href="#requirements">#</a></h2>
|
||||
<ul>
|
||||
<li>Python ≥ 3.10</li>
|
||||
<li>Credentials for at least one model backend (Azure OpenAI, OpenAI-compatible, Anthropic, or a local Qwen server)</li>
|
||||
<li>Benchmark datasets are <strong>not</strong> bundled — prepare your own splits (see §3)</li>
|
||||
</ul>
|
||||
</section>
|
||||
|
||||
<section id="install">
|
||||
<h2>2.2 Install the Package <a class="anchor" href="#install">#</a></h2>
|
||||
<pre><code><span class="tok-k">git</span> clone https://github.com/microsoft/SkillOpt.git
|
||||
<span class="tok-k">cd</span> SkillOpt
|
||||
<span class="tok-k">pip</span> install -e .
|
||||
|
||||
<span class="tok-c"># Optional extras (install only what you need):</span>
|
||||
<span class="tok-k">pip</span> install -e <span class="tok-s">".[alfworld]"</span> <span class="tok-c"># ALFWorld benchmark</span>
|
||||
<span class="tok-k">pip</span> install -e <span class="tok-s">".[claude]"</span> <span class="tok-c"># Anthropic Claude backend</span>
|
||||
<span class="tok-k">pip</span> install -e <span class="tok-s">".[qwen]"</span> <span class="tok-c"># local Qwen backend</span>
|
||||
<span class="tok-k">pip</span> install -e <span class="tok-s">".[webui]"</span> <span class="tok-c"># monitoring dashboard</span>
|
||||
|
||||
<span class="tok-c"># ALFWorld also needs its data assets:</span>
|
||||
<span class="tok-k">alfworld-download</span></code></pre>
|
||||
</section>
|
||||
|
||||
<section id="credentials">
|
||||
<h2>2.3 Configure Credentials <a class="anchor" href="#credentials">#</a></h2>
|
||||
<p>Copy the template and fill in whichever backend you will use:</p>
|
||||
<pre><code><span class="tok-k">cp</span> .env.example .env
|
||||
<span class="tok-c"># edit .env, then:</span>
|
||||
<span class="tok-k">set</span> -a; <span class="tok-k">source</span> .env; <span class="tok-k">set</span> +a</code></pre>
|
||||
<div class="note info"><span class="nh">One env-var family for all OpenAI modes</span>
|
||||
<p>SkillOpt reuses the <code>AZURE_OPENAI_*</code> variable names even for plain OpenAI — there is no separate <code>OPENAI_API_KEY</code> knob. <code>AZURE_OPENAI_ENDPOINT</code> is required for every OpenAI auth mode.</p>
|
||||
</div>
|
||||
<h4>Azure OpenAI (default)</h4>
|
||||
<pre><code><span class="tok-k">export</span> AZURE_OPENAI_ENDPOINT=<span class="tok-s">"https://your-resource.openai.azure.com/"</span>
|
||||
<span class="tok-k">export</span> AZURE_OPENAI_API_VERSION=<span class="tok-s">"2024-12-01-preview"</span>
|
||||
<span class="tok-c"># Auth option 1 — API key:</span>
|
||||
<span class="tok-k">export</span> AZURE_OPENAI_API_KEY=<span class="tok-s">"your-key"</span>
|
||||
<span class="tok-c"># Auth option 2 — Azure CLI (no key; recommended on Azure VMs):</span>
|
||||
<span class="tok-k">export</span> AZURE_OPENAI_AUTH_MODE=azure_cli
|
||||
<span class="tok-c"># Auth option 3 — Managed Identity:</span>
|
||||
<span class="tok-k">export</span> AZURE_OPENAI_AUTH_MODE=managed_identity
|
||||
<span class="tok-k">export</span> AZURE_OPENAI_MANAGED_IDENTITY_CLIENT_ID=<span class="tok-s">"your-client-id"</span></code></pre>
|
||||
<h4>OpenAI-compatible endpoint</h4>
|
||||
<pre><code><span class="tok-k">export</span> AZURE_OPENAI_ENDPOINT=<span class="tok-s">"https://api.openai.com/v1"</span>
|
||||
<span class="tok-k">export</span> AZURE_OPENAI_API_KEY=<span class="tok-s">"sk-..."</span>
|
||||
<span class="tok-k">export</span> AZURE_OPENAI_AUTH_MODE=openai_compatible</code></pre>
|
||||
<h4>Anthropic Claude / local Qwen</h4>
|
||||
<pre><code><span class="tok-k">export</span> ANTHROPIC_API_KEY=<span class="tok-s">"sk-ant-..."</span> <span class="tok-c"># claude_chat backend</span>
|
||||
|
||||
<span class="tok-k">export</span> QWEN_CHAT_BASE_URL=<span class="tok-s">"http://localhost:8000/v1"</span> <span class="tok-c"># local vLLM</span>
|
||||
<span class="tok-k">export</span> QWEN_CHAT_MODEL=<span class="tok-s">"Qwen/Qwen3.5-4B"</span></code></pre>
|
||||
</section>
|
||||
|
||||
<section id="verify">
|
||||
<h2>2.4 Verify Installation <a class="anchor" href="#verify">#</a></h2>
|
||||
<pre><code><span class="tok-k">python</span> -c <span class="tok-s">"import skillopt; print('SkillOpt ready!')"</span></code></pre>
|
||||
</section>
|
||||
|
||||
<!-- ===================== 3. DATA ===================== -->
|
||||
<section id="split-dir">
|
||||
<h2>3.1 Split Directory Format <a class="anchor" href="#split-dir">#</a></h2>
|
||||
<p>With <code>env.split_mode: split_dir</code> (the recommended, deterministic mode), SkillOpt reads a directory containing <code>train/</code>, <code>val/</code>, and <code>test/</code> subfolders, each holding a JSON array of task items:</p>
|
||||
<pre><code>data/my_split/
|
||||
├─ train/items.json <span class="tok-c"># used for rollout (the "train split")</span>
|
||||
├─ val/items.json <span class="tok-c"># selection split → validation gate (valid_seen)</span>
|
||||
└─ test/items.json <span class="tok-c"># held-out final eval (valid_unseen)</span></code></pre>
|
||||
<div class="note info"><span class="nh">Split naming</span>
|
||||
<p>Internally the splits are referred to as <code>train</code>, <code>valid_seen</code> (validation/selection), and <code>valid_unseen</code> (test). The <code>--split</code> flag of <code>eval_only.py</code> uses these names.</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section id="item-schema">
|
||||
<h2>3.2 Item JSON Schema <a class="anchor" href="#item-schema">#</a></h2>
|
||||
<p>Required fields depend on the benchmark; consult <code>skillopt/envs/<benchmark>/dataloader.py</code> for the exact contract. A SearchQA item, for example:</p>
|
||||
<pre><code>[
|
||||
{
|
||||
<span class="tok-f">"id"</span>: <span class="tok-s">"unique_item_id"</span>,
|
||||
<span class="tok-f">"question"</span>: <span class="tok-s">"Who wrote the novel ..."</span>,
|
||||
<span class="tok-f">"context"</span>: <span class="tok-s">"[DOC] relevant passage text ..."</span>,
|
||||
<span class="tok-f">"answers"</span>: [<span class="tok-s">"expected answer"</span>]
|
||||
}
|
||||
]</code></pre>
|
||||
<div class="note warn"><span class="nh">Datasets not included</span>
|
||||
<p>This repository ships no benchmark data. Prepare your own splits in the format above before training.</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section id="split-modes">
|
||||
<h2>3.3 Split Modes <a class="anchor" href="#split-modes">#</a></h2>
|
||||
<div class="table-wrap"><table>
|
||||
<thead><tr><th><code>env.split_mode</code></th><th>Behavior</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td><code>split_dir</code></td><td>Use a pre-built directory with explicit <code>train/val/test</code> folders (set <code>env.split_dir</code>). Deterministic and reproducible.</td></tr>
|
||||
<tr><td><code>ratio</code></td><td>Build a deterministic split on the fly from a single <code>env.data_path</code>, using <code>split_seed</code> (and a train:val:test ratio). Convenient for quick experiments.</td></tr>
|
||||
</tbody>
|
||||
</table></div>
|
||||
</section>
|
||||
|
||||
<!-- ===================== 4. QUICK START ===================== -->
|
||||
<section id="train">
|
||||
<h2>4.1 Train a Skill <a class="anchor" href="#train">#</a></h2>
|
||||
<pre><code><span class="tok-c"># Minimal SearchQA run</span>
|
||||
<span class="tok-k">python</span> scripts/train.py \
|
||||
<span class="tok-f">--config</span> configs/searchqa/default.yaml \
|
||||
<span class="tok-f">--split_dir</span> /path/to/your/searchqa_split \
|
||||
<span class="tok-f">--azure_openai_endpoint</span> https://your-resource.openai.azure.com/ \
|
||||
<span class="tok-f">--optimizer_model</span> gpt-5.5 \
|
||||
<span class="tok-f">--target_model</span> gpt-5.5</code></pre>
|
||||
<p>Swap the config for another benchmark (e.g. <code>configs/livemathematicianbench/default.yaml</code>, <code>configs/alfworld/default.yaml</code>). Common CLI arguments:</p>
|
||||
<div class="table-wrap"><table>
|
||||
<thead><tr><th>Argument</th><th>Description</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td><code>--config</code></td><td>Benchmark config YAML (required)</td></tr>
|
||||
<tr><td><code>--split_dir</code></td><td>Path to the data split directory</td></tr>
|
||||
<tr><td><code>--azure_openai_endpoint</code></td><td>Azure OpenAI endpoint URL</td></tr>
|
||||
<tr><td><code>--optimizer_model</code> / <code>--target_model</code></td><td>Deployment names for optimizer / target</td></tr>
|
||||
<tr><td><code>--num_epochs</code> / <code>--batch_size</code></td><td>Epochs and rollout batch size</td></tr>
|
||||
<tr><td><code>--out_root</code></td><td>Output directory</td></tr>
|
||||
<tr><td><code>--cfg-options k=v ...</code></td><td>Override any config key (see §6.1)</td></tr>
|
||||
</tbody>
|
||||
</table></div>
|
||||
</section>
|
||||
|
||||
<section id="eval">
|
||||
<h2>4.2 Evaluate a Skill <a class="anchor" href="#eval">#</a></h2>
|
||||
<p>Evaluate any skill document (a packaged reference skill, or a trained run's <code>best_skill.md</code>) without training:</p>
|
||||
<pre><code><span class="tok-c"># Evaluate the packaged GPT-5.5 SearchQA skill on the test split</span>
|
||||
<span class="tok-k">python</span> scripts/eval_only.py \
|
||||
<span class="tok-f">--config</span> configs/searchqa/default.yaml \
|
||||
<span class="tok-f">--skill</span> ckpt/searchqa/gpt5.5_skill.md \
|
||||
<span class="tok-f">--split</span> valid_unseen \
|
||||
<span class="tok-f">--split_dir</span> /path/to/searchqa_split \
|
||||
<span class="tok-f">--azure_openai_endpoint</span> https://your-resource.openai.azure.com/</code></pre>
|
||||
<div class="table-wrap"><table>
|
||||
<thead><tr><th><code>--split</code></th><th>Meaning</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td><code>valid_unseen</code></td><td>Test set (held-out)</td></tr>
|
||||
<tr><td><code>valid_seen</code></td><td>Validation / selection set</td></tr>
|
||||
<tr><td><code>train</code></td><td>Training set</td></tr>
|
||||
<tr><td><code>all</code></td><td>All splits combined (default)</td></tr>
|
||||
</tbody>
|
||||
</table></div>
|
||||
</section>
|
||||
|
||||
<section id="outputs">
|
||||
<h2>4.3 Output Structure <a class="anchor" href="#outputs">#</a></h2>
|
||||
<pre><code>outputs/<run_name>/
|
||||
├─ config.json <span class="tok-c"># flattened runtime config</span>
|
||||
├─ history.json <span class="tok-c"># per-step training history</span>
|
||||
├─ runtime_state.json <span class="tok-c"># resume checkpoint</span>
|
||||
├─ best_skill.md <span class="tok-c"># best validated skill document</span>
|
||||
├─ skills/skill_vXXXX.md<span class="tok-c"># skill snapshot per step</span>
|
||||
├─ steps/step_XXXX/ <span class="tok-c"># per-step artifacts (patches, evals)</span>
|
||||
├─ slow_update/epoch_XX/<span class="tok-c"># slow-update logs & rollouts</span>
|
||||
└─ meta_skill/epoch_XX/ <span class="tok-c"># meta-skill logs</span></code></pre>
|
||||
</section>
|
||||
|
||||
<section id="resume">
|
||||
<h2>4.4 Auto-Resume <a class="anchor" href="#resume">#</a></h2>
|
||||
<p>Each completed step persists its state to <code>runtime_state.json</code> and a <code>steps/step_XXXX/</code> directory. Re-running the <em>same command</em> against the same <code>out_root</code> detects finished work and continues from the last completed step — including epoch-boundary slow-update and meta-skill stages.</p>
|
||||
</section>
|
||||
|
||||
<!-- ===================== 5. HOW IT WORKS ===================== -->
|
||||
<section id="loop">
|
||||
<h2>5.1 The Training Loop <a class="anchor" href="#loop">#</a></h2>
|
||||
<p>The loop lives in <code>ReflACTTrainer</code> (<code>skillopt/engine/trainer.py</code>). Each epoch runs a series of optimization steps over rollout batches, then performs two epoch-boundary stages.</p>
|
||||
<pre><code><span class="tok-k">for</span> epoch <span class="tok-k">in</span> epochs:
|
||||
<span class="tok-k">for</span> step <span class="tok-k">in</span> steps:
|
||||
1. Rollout <span class="tok-c"># target executes a batch of tasks</span>
|
||||
2. Reflect <span class="tok-c"># optimizer analyzes trajectories → edit patches</span>
|
||||
3. Aggregate <span class="tok-c"># hierarchically merge similar patches</span>
|
||||
4. Select <span class="tok-c"># rank & clip edits to the learning rate</span>
|
||||
5. Update <span class="tok-c"># apply patches → candidate skill</span>
|
||||
6. Gate <span class="tok-c"># score on selection split → accept / reject</span>
|
||||
|
||||
<span class="tok-c"># epoch boundary (from epoch 2 onward)</span>
|
||||
Slow update <span class="tok-c"># longitudinal comparison → protected guidance</span>
|
||||
Meta skill <span class="tok-c"># cross-epoch optimizer memory</span></code></pre>
|
||||
</section>
|
||||
|
||||
<section id="stages">
|
||||
<h2>5.2 The Six Per-Step Stages <a class="anchor" href="#stages">#</a></h2>
|
||||
<div class="table-wrap"><table>
|
||||
<thead><tr><th>Stage</th><th>What happens</th><th>Source</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td><strong>1. Rollout</strong></td><td>The target model runs each task in the batch with the current skill as context, producing trajectories and scores.</td><td><code>envs/<b>/rollout.py</code></td></tr>
|
||||
<tr><td><strong>2. Reflect</strong></td><td>The optimizer runs an error analyst (and optional success analyst) over minibatches of trajectories, emitting structured edit patches. Runs in parallel across <code>analyst_workers</code>.</td><td><code>gradient/reflect.py</code></td></tr>
|
||||
<tr><td><strong>3. Aggregate</strong></td><td>Semantically similar patches are merged hierarchically to remove redundancy.</td><td><code>gradient/aggregate.py</code> → <code>merge_patches</code></td></tr>
|
||||
<tr><td><strong>4. Select</strong></td><td>Patches are ranked and clipped to the current learning rate (max edits this step), set by the scheduler.</td><td><code>optimizer/clip.py</code> → <code>rank_and_select</code></td></tr>
|
||||
<tr><td><strong>5. Update</strong></td><td>Selected edits are applied to the skill document, producing a candidate skill (patch / rewrite modes).</td><td><code>optimizer/skill.py</code>, <code>update_modes.py</code></td></tr>
|
||||
<tr><td><strong>6. Gate</strong></td><td>The candidate is scored on the selection split and accepted only if it improves (see §5.3).</td><td><code>evaluation/gate.py</code> → <code>evaluate_gate</code></td></tr>
|
||||
</tbody>
|
||||
</table></div>
|
||||
</section>
|
||||
|
||||
<section id="gate">
|
||||
<h2>5.3 Validation Gate <a class="anchor" href="#gate">#</a></h2>
|
||||
<p><code>evaluate_gate</code> is a pure decision function. It compares the candidate's selection-set score against the <em>current</em> and <em>best</em> skills:</p>
|
||||
<ul>
|
||||
<li><strong>accept_new_best</strong> — candidate > current <em>and</em> candidate > best → becomes both current and best.</li>
|
||||
<li><strong>accept</strong> — candidate > current but ≤ best → becomes current only.</li>
|
||||
<li><strong>reject</strong> — candidate ≤ current → discarded; current/best unchanged.</li>
|
||||
</ul>
|
||||
<p>The comparison metric is configurable via <code>evaluation.gate_metric</code>:</p>
|
||||
<div class="table-wrap"><table>
|
||||
<thead><tr><th>Metric</th><th>Score used</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td><code>hard</code> <span class="pill def">default</span></td><td>Exact-match / discrete score</td></tr>
|
||||
<tr><td><code>soft</code></td><td>Partial-credit / continuous score</td></tr>
|
||||
<tr><td><code>mixed</code></td><td>Weighted blend, controlled by <code>gate_mixed_weight</code></td></tr>
|
||||
</tbody>
|
||||
</table></div>
|
||||
<div class="note info"><span class="nh">When to use soft/mixed</span>
|
||||
<p>The <code>soft</code>/<code>mixed</code> metrics (contributed config <code>configs/examples/soft_gate.yaml</code>) help when the selection split is small and rewards are continuous, where a discrete hard gate may reject every candidate and stall training. Paper numbers use the default <code>hard</code> gate.</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section id="slow-update">
|
||||
<h2>5.4 Slow Update (Momentum) <a class="anchor" href="#slow-update">#</a></h2>
|
||||
<p>At each epoch boundary (from epoch 2), the slow update rolls out both the <em>previous</em> epoch's skill and the <em>current</em> skill on the same sampled tasks, categorizes items (improved / regressed / persistent-fail / stable-success), and asks the optimizer to write a free-form <strong>guidance</strong> block. This guidance lands in a <strong>protected region</strong> of the skill that step-level edits cannot touch — only the slow update overwrites it. It is SkillOpt's analogue of momentum, countering cross-epoch forgetting.</p>
|
||||
<p>Acceptance has two modes, selected by <code>optimizer.slow_update_gate_with_selection</code>:</p>
|
||||
<div class="table-wrap"><table>
|
||||
<thead><tr><th>Mode</th><th>Behavior</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td><code>false</code> <span class="pill def">default</span> — force-injected</td><td>Guidance is injected into both current and best skills unconditionally. The longitudinal guidance always persists; it is not gated by step-level selection scores.</td></tr>
|
||||
<tr><td><code>true</code> — gated</td><td>The slow-update candidate is scored on the selection split and accepted/rejected through the same validation gate as step-level updates.</td></tr>
|
||||
</tbody>
|
||||
</table></div>
|
||||
</section>
|
||||
|
||||
<section id="meta-skill">
|
||||
<h2>5.5 Meta Skill (Optimizer Memory) <a class="anchor" href="#meta-skill">#</a></h2>
|
||||
<p>The meta skill is <strong>optimizer-side memory</strong> — it never modifies the target skill document. At the end of each epoch (skipped for epoch 1), the optimizer compares the previous and current epoch's last-step skills on the same sampled tasks and writes a compact, evidence-based reflection on what kind of edits helped or hurt. That memory is then injected as extra context into the next epoch's reflect / merge / learning-rate / ranking stages, so the optimizer accumulates strategy across the run.</p>
|
||||
</section>
|
||||
|
||||
<section id="skill-doc">
|
||||
<h2>5.6 Skill Document Anatomy <a class="anchor" href="#skill-doc">#</a></h2>
|
||||
<p>A skill document is plain Markdown. Initial skills can be empty (learn from scratch) or seeded with domain knowledge via <code>env.skill_init</code>. During training the document accrues rules, patterns, and edge-case handling through accepted edit patches. A dedicated protected region holds the slow-update guidance, delimited by HTML-comment markers:</p>
|
||||
<pre><code><span class="tok-c"># Question Answering Skill</span>
|
||||
|
||||
<span class="tok-c">## Learned rules ...</span>
|
||||
- When the context contains multiple candidates, prefer ...
|
||||
|
||||
<span class="tok-c"><!-- SLOW_UPDATE_START --></span>
|
||||
<span class="tok-c"># (epoch-level longitudinal guidance — only the slow update writes here)</span>
|
||||
<span class="tok-c"><!-- SLOW_UPDATE_END --></span></code></pre>
|
||||
<p>Helpers in <code>optimizer/slow_update.py</code> manage this region: <code>inject_empty_slow_update_field</code> (placeholder at epoch 1), <code>extract_slow_update_field</code> (read), and <code>replace_slow_update_field</code> (overwrite). Step-level edits are blocked from modifying anything inside the markers.</p>
|
||||
</section>
|
||||
|
||||
<!-- ===================== 6. CONFIGURATION ===================== -->
|
||||
<section id="config-system">
|
||||
<h2>6.1 Configuration System <a class="anchor" href="#config-system">#</a></h2>
|
||||
<p>Configs are <strong>structured YAML</strong> with section blocks (<code>model</code>, <code>train</code>, <code>gradient</code>, <code>optimizer</code>, <code>evaluation</code>, <code>env</code>) and <code>_base_</code> inheritance. A benchmark config inherits the shared defaults and overrides only what differs:</p>
|
||||
<pre><code><span class="tok-c"># configs/searchqa/default.yaml</span>
|
||||
<span class="tok-f">_base_</span>: ../_base_/default.yaml
|
||||
<span class="tok-f">train</span>:
|
||||
<span class="tok-f">train_size</span>: <span class="tok-n">400</span>
|
||||
<span class="tok-f">batch_size</span>: <span class="tok-n">40</span>
|
||||
<span class="tok-f">optimizer</span>:
|
||||
<span class="tok-f">learning_rate</span>: <span class="tok-n">4</span>
|
||||
<span class="tok-f">env</span>:
|
||||
<span class="tok-f">name</span>: searchqa
|
||||
<span class="tok-f">split_dir</span>: data/searchqa_split</code></pre>
|
||||
<p>Override any key at the command line without editing files:</p>
|
||||
<pre><code><span class="tok-k">python</span> scripts/train.py --config configs/searchqa/default.yaml \
|
||||
<span class="tok-f">--cfg-options</span> optimizer.learning_rate=<span class="tok-n">16</span> optimizer.lr_scheduler=linear</code></pre>
|
||||
<div class="note info"><span class="nh">Reading the tables below</span>
|
||||
<p>Each section lists the key (relative to its YAML block), type, default (from <code>configs/_base_/default.yaml</code>), allowed values, and meaning. Defaults shown are the shipped base defaults.</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section id="cfg-model">
|
||||
<h2>6.2 <code>model.*</code> <a class="anchor" href="#cfg-model">#</a></h2>
|
||||
<div class="table-wrap"><table>
|
||||
<thead><tr><th>Key</th><th>Type</th><th>Default</th><th>Description / options</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td><code>backend</code></td><td>str</td><td class="def">azure_openai</td><td>High-level backend label for the run.</td></tr>
|
||||
<tr><td><code>optimizer</code></td><td>str</td><td class="def">gpt-5.5</td><td>Optimizer model deployment (writes skill edits).</td></tr>
|
||||
<tr><td><code>target</code></td><td>str</td><td class="def">gpt-5.5</td><td>Target model deployment (executes tasks).</td></tr>
|
||||
<tr><td><code>optimizer_backend</code></td><td>str</td><td class="def">openai_chat</td><td>Client path for the optimizer: <code>openai_chat</code> or <code>claude_chat</code>.</td></tr>
|
||||
<tr><td><code>target_backend</code></td><td>str</td><td class="def">openai_chat</td><td>Client path for the target: <code>openai_chat</code> / <code>claude_chat</code> / <code>qwen_chat</code> / <code>codex_exec</code> / <code>claude_code_exec</code>.</td></tr>
|
||||
<tr><td><code>reasoning_effort</code></td><td>str</td><td class="def">medium</td><td><code>low</code> / <code>medium</code> / <code>high</code> / <code>xhigh</code> / <code>max</code> (or empty).</td></tr>
|
||||
<tr><td><code>rewrite_reasoning_effort</code></td><td>str</td><td class="def">""</td><td>Override effort for full-rewrite calls (empty = inherit).</td></tr>
|
||||
<tr><td><code>rewrite_max_completion_tokens</code></td><td>int</td><td class="def">64000</td><td>Token cap for full-rewrite optimizer calls.</td></tr>
|
||||
<tr><td><code>azure_openai_endpoint</code></td><td>str</td><td class="def">""</td><td>Azure resource URL (or via <code>AZURE_OPENAI_ENDPOINT</code>).</td></tr>
|
||||
<tr><td><code>azure_openai_api_version</code></td><td>str</td><td class="def">2024-12-01-preview</td><td>Azure API version header.</td></tr>
|
||||
<tr><td><code>azure_openai_auth_mode</code></td><td>str</td><td class="def">""</td><td><code>api_key</code> / <code>azure_cli</code> / <code>managed_identity</code> / <code>openai_compatible</code> (empty → env default).</td></tr>
|
||||
</tbody>
|
||||
</table></div>
|
||||
<div class="note info"><span class="nh">Separate optimizer / target endpoints</span>
|
||||
<p>Every <code>azure_openai_*</code> key also has <code>optimizer_azure_openai_*</code> and <code>target_azure_openai_*</code> variants, letting you point the optimizer and target at different Azure resources. Exec backends (<code>codex_exec</code>, <code>claude_code_exec</code>) add their own <code>codex_exec_*</code> / <code>claude_code_exec_*</code> knobs (sandbox, reasoning effort, SDK mode, etc.).</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section id="cfg-train">
|
||||
<h2>6.3 <code>train.*</code> <a class="anchor" href="#cfg-train">#</a></h2>
|
||||
<div class="table-wrap"><table>
|
||||
<thead><tr><th>Key</th><th>Type</th><th>Default</th><th>DL analogy</th><th>Description</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td><code>num_epochs</code></td><td>int</td><td class="def">4</td><td>Epochs</td><td>Number of training epochs.</td></tr>
|
||||
<tr><td><code>train_size</code></td><td>int</td><td class="def">0</td><td>Train-set size</td><td>0 = derive from the dataset split. (Fixed by split size when using <code>split_dir</code>.)</td></tr>
|
||||
<tr><td><code>batch_size</code></td><td>int</td><td class="def">40</td><td>Batch size</td><td>Tasks rolled out per optimization step.</td></tr>
|
||||
<tr><td><code>accumulation</code></td><td>int</td><td class="def">1</td><td>Grad accumulation</td><td>Accumulation rounds per step.</td></tr>
|
||||
<tr><td><code>seed</code></td><td>int</td><td class="def">42</td><td>Random seed</td><td>Reproducibility seed.</td></tr>
|
||||
</tbody>
|
||||
</table></div>
|
||||
</section>
|
||||
|
||||
<section id="cfg-gradient">
|
||||
<h2>6.4 <code>gradient.*</code> <a class="anchor" href="#cfg-gradient">#</a></h2>
|
||||
<div class="table-wrap"><table>
|
||||
<thead><tr><th>Key</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td><code>minibatch_size</code></td><td>int</td><td class="def">8</td><td>Trajectories per reflect minibatch.</td></tr>
|
||||
<tr><td><code>merge_batch_size</code></td><td>int</td><td class="def">8</td><td>Patches per merge batch during aggregation.</td></tr>
|
||||
<tr><td><code>analyst_workers</code></td><td>int</td><td class="def">16</td><td>Parallel reflection workers (data parallelism).</td></tr>
|
||||
<tr><td><code>max_analyst_rounds</code></td><td>int</td><td class="def">3</td><td>Max rounds of analyst reflection per step.</td></tr>
|
||||
<tr><td><code>failure_only</code></td><td>bool</td><td class="def">false</td><td>Reflect only on failed trajectories when true.</td></tr>
|
||||
</tbody>
|
||||
</table></div>
|
||||
</section>
|
||||
|
||||
<section id="cfg-optimizer">
|
||||
<h2>6.5 <code>optimizer.*</code> <a class="anchor" href="#cfg-optimizer">#</a></h2>
|
||||
<div class="table-wrap"><table>
|
||||
<thead><tr><th>Key</th><th>Type</th><th>Default</th><th>DL analogy</th><th>Description / options</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td><code>learning_rate</code></td><td>int</td><td class="def">4</td><td>Learning rate</td><td>Max edit patches applied per step (the "edit budget").</td></tr>
|
||||
<tr><td><code>min_learning_rate</code></td><td>int</td><td class="def">2</td><td>Min LR</td><td>Floor edit budget for decaying schedulers.</td></tr>
|
||||
<tr><td><code>lr_scheduler</code></td><td>str</td><td class="def">cosine</td><td>LR schedule</td><td><code>constant</code> / <code>linear</code> / <code>cosine</code> / <code>autonomous</code>.</td></tr>
|
||||
<tr><td><code>lr_control_mode</code></td><td>str</td><td class="def">fixed</td><td>—</td><td><code>fixed</code> / <code>autonomous</code> / <code>none</code>.</td></tr>
|
||||
<tr><td><code>skill_update_mode</code></td><td>str</td><td class="def">patch</td><td>—</td><td><code>patch</code> / <code>rewrite_from_suggestions</code> / <code>full_rewrite_minibatch</code>.</td></tr>
|
||||
<tr><td><code>use_slow_update</code></td><td>bool</td><td class="def">true</td><td>Momentum</td><td>Enable epoch-boundary slow update.</td></tr>
|
||||
<tr><td><code>slow_update_samples</code></td><td>int</td><td class="def">20</td><td>—</td><td>Tasks sampled for the longitudinal comparison.</td></tr>
|
||||
<tr><td><code>slow_update_gate_with_selection</code></td><td>bool</td><td class="def">false</td><td>—</td><td><code>false</code> = force-inject guidance; <code>true</code> = gate it on the selection split (see §5.4).</td></tr>
|
||||
<tr><td><code>longitudinal_pair_policy</code></td><td>str</td><td class="def">mixed</td><td>—</td><td><code>mixed</code> / <code>changed</code> / <code>unchanged</code> — which comparison pairs to keep.</td></tr>
|
||||
<tr><td><code>use_meta_skill</code></td><td>bool</td><td class="def">true</td><td>Meta-learning</td><td>Enable cross-epoch optimizer memory.</td></tr>
|
||||
</tbody>
|
||||
</table></div>
|
||||
</section>
|
||||
|
||||
<section id="cfg-evaluation">
|
||||
<h2>6.6 <code>evaluation.*</code> <a class="anchor" href="#cfg-evaluation">#</a></h2>
|
||||
<div class="table-wrap"><table>
|
||||
<thead><tr><th>Key</th><th>Type</th><th>Default</th><th>Description / options</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td><code>use_gate</code></td><td>bool</td><td class="def">true</td><td>Validation gating is mandatory in this branch (must remain <code>true</code>).</td></tr>
|
||||
<tr><td><code>gate_metric</code></td><td>str</td><td class="def">hard</td><td><code>hard</code> / <code>soft</code> / <code>mixed</code> — score used by the gate (see §5.3).</td></tr>
|
||||
<tr><td><code>gate_mixed_weight</code></td><td>float</td><td class="def">0.5</td><td>Weight on the soft score when <code>gate_metric = mixed</code>.</td></tr>
|
||||
<tr><td><code>sel_env_num</code></td><td>int</td><td class="def">0</td><td>Selection-split eval size (0 = use full split).</td></tr>
|
||||
<tr><td><code>test_env_num</code></td><td>int</td><td class="def">0</td><td>Test-split eval size (0 = use full split).</td></tr>
|
||||
<tr><td><code>eval_test</code></td><td>bool</td><td class="def">true</td><td>Run a final test evaluation after training.</td></tr>
|
||||
</tbody>
|
||||
</table></div>
|
||||
<div class="note warn"><span class="nh">Gate is required</span>
|
||||
<p>Setting <code>evaluation.use_gate: false</code> raises an error — validation gating cannot be disabled in this branch.</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section id="cfg-env">
|
||||
<h2>6.7 <code>env.*</code> <a class="anchor" href="#cfg-env">#</a></h2>
|
||||
<div class="table-wrap"><table>
|
||||
<thead><tr><th>Key</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td><code>name</code></td><td>str</td><td class="def">""</td><td>Benchmark name (<code>searchqa</code>, <code>docvqa</code>, <code>alfworld</code>, …). Selects the env module.</td></tr>
|
||||
<tr><td><code>skill_init</code></td><td>str</td><td class="def">""</td><td>Path to a seed skill (empty = start from scratch).</td></tr>
|
||||
<tr><td><code>split_mode</code></td><td>str</td><td class="def">ratio</td><td><code>ratio</code> or <code>split_dir</code> (see §3.3).</td></tr>
|
||||
<tr><td><code>split_dir</code></td><td>str</td><td class="def">""</td><td>Pre-split directory (when <code>split_mode = split_dir</code>).</td></tr>
|
||||
<tr><td><code>data_path</code></td><td>str</td><td class="def">""</td><td>Single dataset path (when <code>split_mode = ratio</code>).</td></tr>
|
||||
<tr><td><code>split_seed</code></td><td>int</td><td class="def">42</td><td>Seed for deterministic ratio splitting.</td></tr>
|
||||
<tr><td><code>exec_timeout</code></td><td>int</td><td class="def">120</td><td>Per-task target/code-agent timeout (seconds).</td></tr>
|
||||
<tr><td><code>out_root</code></td><td>str</td><td class="def">""</td><td>Output directory for the run.</td></tr>
|
||||
</tbody>
|
||||
</table></div>
|
||||
<div class="note info"><span class="nh">Benchmark-specific env keys</span>
|
||||
<p>Env blocks may carry extra benchmark-specific keys (e.g. <code>max_turns</code>, <code>workers</code>, <code>max_completion_tokens</code>, <code>limit</code>). Unmapped env keys are passed straight through to the benchmark adapter — check the relevant <code>configs/<benchmark>/default.yaml</code>.</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<!-- ===================== 7. BENCHMARKS ===================== -->
|
||||
<section id="bench-list">
|
||||
<h2>7.1 Supported Benchmarks <a class="anchor" href="#bench-list">#</a></h2>
|
||||
<div class="table-wrap"><table>
|
||||
<thead><tr><th>Benchmark</th><th>Type</th><th>Config</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td>SearchQA</td><td>Question answering</td><td><code>configs/searchqa/default.yaml</code></td></tr>
|
||||
<tr><td>DocVQA</td><td>Document QA</td><td><code>configs/docvqa/default.yaml</code></td></tr>
|
||||
<tr><td>ALFWorld</td><td>Embodied agent</td><td><code>configs/alfworld/default.yaml</code></td></tr>
|
||||
<tr><td>LiveMathematicianBench</td><td>Math reasoning</td><td><code>configs/livemathematicianbench/default.yaml</code></td></tr>
|
||||
<tr><td>SpreadsheetBench</td><td>Spreadsheet code generation</td><td><code>configs/spreadsheetbench/default.yaml</code></td></tr>
|
||||
<tr><td>OfficeQA</td><td>Tool-augmented QA</td><td><code>configs/officeqa/default.yaml</code></td></tr>
|
||||
</tbody>
|
||||
</table></div>
|
||||
<p>Each benchmark is a self-contained module under <code>skillopt/envs/<benchmark>/</code> with an <code>adapter.py</code>, <code>dataloader.py</code>, <code>rollout.py</code>, and <code>evaluator.py</code> (some add a custom <code>reflect.py</code>). Packaged reference skills live in <code>ckpt/<benchmark>/</code>.</p>
|
||||
</section>
|
||||
|
||||
<section id="bench-new">
|
||||
<h2>7.2 Add a New Benchmark <a class="anchor" href="#bench-new">#</a></h2>
|
||||
<p>Use <code>skillopt/envs/_template/</code> as a starting point. At minimum, implement:</p>
|
||||
<ol>
|
||||
<li><strong>Dataloader</strong> — read your item JSON into the framework's item dicts (<code>dataloader.py</code>).</li>
|
||||
<li><strong>Rollout</strong> — run the target on one item with the current skill and return a trajectory + score (<code>rollout.py</code>).</li>
|
||||
<li><strong>Evaluator</strong> — score predictions against ground truth (<code>evaluator.py</code>).</li>
|
||||
<li><strong>Adapter</strong> — wire the above into the trainer's expected interface and register the env name (<code>adapter.py</code>).</li>
|
||||
</ol>
|
||||
<p>Then add a <code>configs/<name>/default.yaml</code> inheriting <code>_base_/default.yaml</code> and set <code>env.name</code> to your new benchmark.</p>
|
||||
</section>
|
||||
|
||||
<!-- ===================== 8. API REFERENCE ===================== -->
|
||||
<section id="module-map">
|
||||
<h2>8.1 Module Map <a class="anchor" href="#module-map">#</a></h2>
|
||||
<div class="table-wrap"><table>
|
||||
<thead><tr><th>Module</th><th>Responsibility</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td><code>skillopt/config.py</code></td><td>Load structured YAML, resolve <code>_base_</code> inheritance, flatten to the trainer's flat dict, apply CLI overrides.</td></tr>
|
||||
<tr><td><code>skillopt/engine/trainer.py</code></td><td><code>ReflACTTrainer</code> — orchestrates the whole loop, gating, slow update, meta skill, resume, and artifact writing.</td></tr>
|
||||
<tr><td><code>skillopt/gradient/</code></td><td>Reflection ("backward pass"): <code>reflect.py</code> analysts, <code>aggregate.py</code> patch merging.</td></tr>
|
||||
<tr><td><code>skillopt/optimizer/</code></td><td>The "optimizer": edit application, learning-rate scheduling, edit selection, slow update, meta skill, rewrite modes.</td></tr>
|
||||
<tr><td><code>skillopt/evaluation/gate.py</code></td><td>Pure accept/reject decision and metric selection.</td></tr>
|
||||
<tr><td><code>skillopt/model/</code></td><td>Backend clients (OpenAI/Azure, Claude, Qwen, Codex/Claude-Code exec) and routing.</td></tr>
|
||||
<tr><td><code>skillopt/envs/<b>/</code></td><td>Per-benchmark dataloader, rollout, evaluator, adapter.</td></tr>
|
||||
</tbody>
|
||||
</table></div>
|
||||
</section>
|
||||
|
||||
<section id="functions">
|
||||
<h2>8.2 Core Functions <a class="anchor" href="#functions">#</a></h2>
|
||||
<div class="table-wrap"><table>
|
||||
<thead><tr><th>Function</th><th>File</th><th>Purpose</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td><code>load_config</code> / <code>flatten_config</code> / <code>apply_overrides</code></td><td><code>config.py</code></td><td>Load YAML with inheritance; flatten sections; apply <code>key=value</code> overrides.</td></tr>
|
||||
<tr><td><code>run_minibatch_reflect</code></td><td><code>gradient/reflect.py</code></td><td>Run error/success analysts over trajectory minibatches → edit patches.</td></tr>
|
||||
<tr><td><code>merge_patches</code></td><td><code>gradient/aggregate.py</code></td><td>Hierarchically merge semantically similar patches.</td></tr>
|
||||
<tr><td><code>rank_and_select</code></td><td><code>optimizer/clip.py</code></td><td>Rank edits and clip to the learning-rate budget.</td></tr>
|
||||
<tr><td><code>build_scheduler</code></td><td><code>optimizer/scheduler.py</code></td><td>Construct the LR (edit-budget) scheduler: constant/linear/cosine/autonomous.</td></tr>
|
||||
<tr><td><code>decide_autonomous_learning_rate</code></td><td><code>optimizer/lr_autonomous.py</code></td><td>Let the optimizer pick the next learning rate (autonomous mode).</td></tr>
|
||||
<tr><td><code>apply_patch</code> / <code>apply_edit</code></td><td><code>optimizer/skill.py</code></td><td>Apply edits to the skill document (respecting the protected region).</td></tr>
|
||||
<tr><td><code>rewrite_skill_from_suggestions</code></td><td><code>optimizer/rewrite.py</code></td><td>Full-rewrite update mode from accumulated suggestions.</td></tr>
|
||||
<tr><td><code>evaluate_gate</code> / <code>select_gate_score</code></td><td><code>evaluation/gate.py</code></td><td>Accept/reject decision; compute hard/soft/mixed score.</td></tr>
|
||||
<tr><td><code>run_slow_update</code></td><td><code>optimizer/slow_update.py</code></td><td>Produce epoch-boundary longitudinal guidance.</td></tr>
|
||||
<tr><td><code>replace_slow_update_field</code> / <code>extract_slow_update_field</code></td><td><code>optimizer/slow_update.py</code></td><td>Read/overwrite the protected guidance region.</td></tr>
|
||||
<tr><td><code>run_meta_skill</code> / <code>format_meta_skill_context</code></td><td><code>optimizer/meta_skill.py</code></td><td>Generate cross-epoch optimizer memory and render it into reflection context.</td></tr>
|
||||
</tbody>
|
||||
</table></div>
|
||||
</section>
|
||||
|
||||
<section id="cli">
|
||||
<h2>8.3 CLI Scripts <a class="anchor" href="#cli">#</a></h2>
|
||||
<h4>scripts/train.py</h4>
|
||||
<p>Runs a full training loop. Required: <code>--config</code>. Override config via <code>--cfg-options section.key=value …</code> or legacy flat flags (<code>--num_epochs</code>, <code>--batch_size</code>, <code>--optimizer_model</code>, <code>--target_model</code>, <code>--lr_scheduler</code>, <code>--edit_budget</code>, <code>--split_dir</code>, …).</p>
|
||||
<h4>scripts/eval_only.py</h4>
|
||||
<p>Evaluates a skill document without training. Required: <code>--config</code> and <code>--skill</code>. Use <code>--split</code> to choose <code>train</code> / <code>valid_seen</code> / <code>valid_unseen</code> / <code>all</code>.</p>
|
||||
<pre><code><span class="tok-k">python</span> scripts/eval_only.py \
|
||||
--config configs/searchqa/default.yaml \
|
||||
--skill outputs/my_run/best_skill.md \
|
||||
--split valid_unseen</code></pre>
|
||||
</section>
|
||||
|
||||
<section id="webui">
|
||||
<h2>8.4 WebUI <a class="anchor" href="#webui">#</a></h2>
|
||||
<p>An optional Gradio dashboard to configure parameters and monitor runs:</p>
|
||||
<pre><code><span class="tok-k">pip</span> install -e <span class="tok-s">".[webui]"</span>
|
||||
<span class="tok-k">python</span> -m skillopt_webui.app <span class="tok-c"># http://localhost:7860</span>
|
||||
<span class="tok-k">python</span> -m skillopt_webui.app --share <span class="tok-c"># public share link</span></code></pre>
|
||||
<div class="table-wrap"><table>
|
||||
<thead><tr><th>Flag</th><th>Default</th><th>Description</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td><code>--port</code></td><td class="def">7860</td><td>Server port.</td></tr>
|
||||
<tr><td><code>--host</code></td><td class="def">0.0.0.0</td><td>Bind address.</td></tr>
|
||||
<tr><td><code>--share</code></td><td class="def">off</td><td>Create a public Gradio share link.</td></tr>
|
||||
</tbody>
|
||||
</table></div>
|
||||
|
||||
<div class="footer-note">
|
||||
SkillOpt — Executive Strategy for Self-Evolving Agent Skills ·
|
||||
<a href="https://github.com/microsoft/SkillOpt">github.com/microsoft/SkillOpt</a> ·
|
||||
<a href="https://arxiv.org/abs/2605.23904">arXiv:2605.23904</a><br>
|
||||
This guide reflects the current configuration defaults in <code>configs/_base_/default.yaml</code>. When in doubt, the code is the source of truth.
|
||||
</div>
|
||||
</section>
|
||||
|
||||
</main>
|
||||
|
||||
<!-- ───────────── RIGHT TOC ───────────── -->
|
||||
<aside class="toc" id="toc">
|
||||
<div class="tl">On this page</div>
|
||||
<div id="tocLinks"></div>
|
||||
</aside>
|
||||
|
||||
</div>
|
||||
|
||||
<script>
|
||||
(function () {
|
||||
// Build right-hand "On this page" from <h2> elements
|
||||
var sections = Array.prototype.slice.call(document.querySelectorAll('main.content section[id]'));
|
||||
var tocLinks = document.getElementById('tocLinks');
|
||||
var h2s = Array.prototype.slice.call(document.querySelectorAll('main.content h2'));
|
||||
h2s.forEach(function (h) {
|
||||
var sec = h.closest('section');
|
||||
if (!sec || !sec.id) return;
|
||||
var a = document.createElement('a');
|
||||
a.href = '#' + sec.id;
|
||||
a.textContent = h.textContent.replace(/#$/, '').trim();
|
||||
a.dataset.target = sec.id;
|
||||
tocLinks.appendChild(a);
|
||||
});
|
||||
|
||||
var sideLinks = Array.prototype.slice.call(document.querySelectorAll('nav.sidebar a'));
|
||||
var tocAnchors = Array.prototype.slice.call(tocLinks.querySelectorAll('a'));
|
||||
|
||||
function setActive(id) {
|
||||
sideLinks.forEach(function (a) {
|
||||
a.classList.toggle('active', a.getAttribute('href') === '#' + id);
|
||||
});
|
||||
tocAnchors.forEach(function (a) {
|
||||
a.classList.toggle('active', a.dataset.target === id);
|
||||
});
|
||||
}
|
||||
|
||||
// Scroll spy
|
||||
var observer = new IntersectionObserver(function (entries) {
|
||||
entries.forEach(function (e) {
|
||||
if (e.isIntersecting) setActive(e.target.id);
|
||||
});
|
||||
}, { rootMargin: '-64px 0px -75% 0px', threshold: 0 });
|
||||
sections.forEach(function (s) { observer.observe(s); });
|
||||
|
||||
// Mobile sidebar toggle
|
||||
var btn = document.getElementById('menuBtn');
|
||||
var sidebar = document.getElementById('sidebar');
|
||||
btn.addEventListener('click', function () { sidebar.classList.toggle('open'); });
|
||||
sideLinks.forEach(function (a) {
|
||||
a.addEventListener('click', function () { sidebar.classList.remove('open'); });
|
||||
});
|
||||
})();
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
-170
@@ -1,170 +0,0 @@
|
||||
---
|
||||
hide:
|
||||
- navigation
|
||||
---
|
||||
|
||||
<div class="hero" markdown>
|
||||
|
||||
# SkillOpt
|
||||
|
||||
### Train Agent Skills Like Neural Networks
|
||||
|
||||
*Optimize natural-language skill documents through iterative rollout, reflection, and gated validation — with epochs, learning rates, and validation gates — without touching model weights.*
|
||||
|
||||
[Get Started :material-rocket-launch:](guide/installation.md){ .md-button .md-button--primary }
|
||||
[View on GitHub :material-github:](https://github.com/microsoft/SkillOpt){ .md-button }
|
||||
|
||||
</div>
|
||||
|
||||
---
|
||||
|
||||
## How It Works
|
||||
|
||||
<div class="pipeline-container" markdown>
|
||||
<div class="pipeline-wrapper">
|
||||
|
||||
<div class="pipeline-stage" id="stage-rollout">
|
||||
<div class="stage-icon">🎯</div>
|
||||
<div class="stage-label">Rollout</div>
|
||||
<div class="stage-desc">Target executes tasks</div>
|
||||
</div>
|
||||
|
||||
<div class="pipeline-arrow"><div class="flow-line"></div></div>
|
||||
|
||||
<div class="pipeline-stage" id="stage-reflect">
|
||||
<div class="stage-icon">🔍</div>
|
||||
<div class="stage-label">Reflect</div>
|
||||
<div class="stage-desc">Optimizer analyzes trajectories</div>
|
||||
</div>
|
||||
|
||||
<div class="pipeline-arrow"><div class="flow-line"></div></div>
|
||||
|
||||
<div class="pipeline-stage" id="stage-aggregate">
|
||||
<div class="stage-icon">🔗</div>
|
||||
<div class="stage-label">Aggregate</div>
|
||||
<div class="stage-desc">Merge edit patches</div>
|
||||
</div>
|
||||
|
||||
<div class="pipeline-arrow"><div class="flow-line"></div></div>
|
||||
|
||||
<div class="pipeline-stage" id="stage-select">
|
||||
<div class="stage-icon">✂️</div>
|
||||
<div class="stage-label">Select</div>
|
||||
<div class="stage-desc">Rank & clip edits</div>
|
||||
</div>
|
||||
|
||||
<div class="pipeline-arrow"><div class="flow-line"></div></div>
|
||||
|
||||
<div class="pipeline-stage" id="stage-update">
|
||||
<div class="stage-icon">📝</div>
|
||||
<div class="stage-label">Update</div>
|
||||
<div class="stage-desc">Apply to skill doc</div>
|
||||
</div>
|
||||
|
||||
<div class="pipeline-arrow"><div class="flow-line"></div></div>
|
||||
|
||||
<div class="pipeline-stage" id="stage-gate">
|
||||
<div class="stage-icon">🚦</div>
|
||||
<div class="stage-label">Gate</div>
|
||||
<div class="stage-desc">Validate & accept</div>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
|
||||
<div class="pipeline-epoch-bar">
|
||||
<div class="epoch-mechanism">🔄 Slow Update</div>
|
||||
<div class="epoch-mechanism">🧠 Meta Skill</div>
|
||||
<div class="epoch-label">Epoch Boundary</div>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
|
||||
---
|
||||
|
||||
## Deep Learning Analogy
|
||||
|
||||
SkillOpt brings the familiar deep-learning training paradigm to agentic prompt optimization:
|
||||
|
||||
| Deep Learning | SkillOpt |
|
||||
|---|---|
|
||||
| Model weights | Skill document (Markdown) |
|
||||
| Forward pass | Rollout (target executes tasks) |
|
||||
| Loss / gradient | Reflect (optimizer produces edit patches) |
|
||||
| Gradient clipping | Edit selection (`learning_rate` = max edits) |
|
||||
| SGD step | Patch application to skill |
|
||||
| Validation set | Gated evaluation on selection split |
|
||||
| LR schedule | `lr_scheduler`: cosine, linear, constant |
|
||||
| Epochs | Multi-epoch with slow update & meta skill memory |
|
||||
|
||||
---
|
||||
|
||||
## Supported Benchmarks
|
||||
|
||||
| Benchmark | Type | Config |
|
||||
|---|---|---|
|
||||
| **DocVQA** | Document QA | `configs/docvqa/` |
|
||||
| **ALFWorld** | Embodied AI | `configs/alfworld/` |
|
||||
| **OfficeQA** | Enterprise QA | `configs/officeqa/` |
|
||||
| **SearchQA** | Open-domain QA | `configs/searchqa/` |
|
||||
| **LiveMathBench** | Math reasoning | `configs/livemathematicianbench/` |
|
||||
| **SWEBench** | Software Engineering | `configs/swebench/` |
|
||||
| + 5 more | Various | See [docs](guide/first-experiment.md) |
|
||||
|
||||
---
|
||||
|
||||
## Quick Example
|
||||
|
||||
```bash
|
||||
# Install
|
||||
pip install -e .
|
||||
|
||||
# Configure credentials
|
||||
export AZURE_OPENAI_ENDPOINT="https://your-resource.openai.azure.com/"
|
||||
export AZURE_OPENAI_API_KEY="your-key"
|
||||
|
||||
# Train on SearchQA
|
||||
python scripts/train.py --config configs/searchqa/default.yaml
|
||||
|
||||
# Evaluate best skill
|
||||
python scripts/eval_only.py \
|
||||
--config configs/searchqa/default.yaml \
|
||||
--skill outputs/best_skill.md
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
<div class="grid cards" markdown>
|
||||
|
||||
- :material-book-open-variant:{ .lg .middle } **Getting Started**
|
||||
|
||||
---
|
||||
|
||||
Install SkillOpt, configure your API keys, and run your first experiment in 5 minutes.
|
||||
|
||||
[:octicons-arrow-right-24: Installation](guide/installation.md)
|
||||
|
||||
- :material-puzzle:{ .lg .middle } **Add a Benchmark**
|
||||
|
||||
---
|
||||
|
||||
Extend SkillOpt with your own benchmark in ~100 lines of code.
|
||||
|
||||
[:octicons-arrow-right-24: Extension Guide](guide/new-benchmark.md)
|
||||
|
||||
- :material-cog:{ .lg .middle } **Configuration**
|
||||
|
||||
---
|
||||
|
||||
Full reference for all hyperparameters with deep learning analogies.
|
||||
|
||||
[:octicons-arrow-right-24: Config Reference](reference/config.md)
|
||||
|
||||
- :material-monitor-dashboard:{ .lg .middle } **WebUI**
|
||||
|
||||
---
|
||||
|
||||
Configure, launch, and monitor training from your browser.
|
||||
|
||||
[:octicons-arrow-right-24: WebUI Guide](guide/first-experiment.md#webui)
|
||||
|
||||
</div>
|
||||
@@ -1,81 +0,0 @@
|
||||
# API Reference
|
||||
|
||||
## Core Classes
|
||||
|
||||
### `EnvAdapter`
|
||||
|
||||
Abstract base class for benchmark environments.
|
||||
|
||||
```python
|
||||
class EnvAdapter(ABC):
|
||||
async def execute(self, item, skill, model) -> TaskResult
|
||||
def evaluate(self, prediction, ground_truth) -> float
|
||||
def build_prompt(self, item, skill) -> str
|
||||
```
|
||||
|
||||
### `DataLoader`
|
||||
|
||||
Abstract base class for data loading and splitting.
|
||||
|
||||
```python
|
||||
class DataLoader(ABC):
|
||||
def setup(self, cfg: dict) -> None
|
||||
def get_split_items(self, split: str) -> list[DataItem]
|
||||
```
|
||||
|
||||
### `ModelBackend`
|
||||
|
||||
Abstract base class for LLM backends.
|
||||
|
||||
```python
|
||||
class ModelBackend(ABC):
|
||||
async def generate(self, messages, **kwargs) -> ModelResponse
|
||||
async def generate_with_tools(self, messages, tools, **kwargs) -> ModelResponse
|
||||
```
|
||||
|
||||
### `Trainer`
|
||||
|
||||
Main training loop orchestrator.
|
||||
|
||||
```python
|
||||
class Trainer:
|
||||
def __init__(self, cfg: dict)
|
||||
async def train(self) -> TrainResult
|
||||
async def evaluate(self, skill: str, split: str) -> EvalResult
|
||||
```
|
||||
|
||||
## Data Classes
|
||||
|
||||
### `DataItem`
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class DataItem:
|
||||
id: str
|
||||
input: str
|
||||
ground_truth: str
|
||||
metadata: dict = field(default_factory=dict)
|
||||
```
|
||||
|
||||
### `TaskResult`
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class TaskResult:
|
||||
item_id: str
|
||||
prediction: str
|
||||
score: float
|
||||
trajectory: list[dict]
|
||||
```
|
||||
|
||||
### `ModelResponse`
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class ModelResponse:
|
||||
content: str
|
||||
usage: dict
|
||||
model: str
|
||||
```
|
||||
|
||||
For detailed source code, see the [`skillopt/`](https://github.com/microsoft/SkillOpt/tree/main/skillopt) directory.
|
||||
@@ -1,71 +0,0 @@
|
||||
# CLI Reference
|
||||
|
||||
## Training
|
||||
|
||||
```bash
|
||||
python scripts/train.py --config <config.yaml> [overrides...]
|
||||
```
|
||||
|
||||
### Arguments
|
||||
|
||||
| Argument | Description |
|
||||
|---|---|
|
||||
| `--config` | Path to YAML config file (required) |
|
||||
| `key=value` | Override any config parameter |
|
||||
|
||||
### Examples
|
||||
|
||||
```bash
|
||||
# Basic training
|
||||
python scripts/train.py --config configs/searchqa/default.yaml
|
||||
|
||||
# With overrides
|
||||
python scripts/train.py \
|
||||
--config configs/searchqa/default.yaml \
|
||||
--cfg-options optimizer.learning_rate=16 optimizer.lr_scheduler=linear
|
||||
|
||||
# With custom initial skill
|
||||
python scripts/train.py \
|
||||
--config configs/searchqa/default.yaml \
|
||||
--cfg-options env.skill_init=skills/my_seed.md
|
||||
```
|
||||
|
||||
## Evaluation
|
||||
|
||||
```bash
|
||||
python scripts/eval_only.py --config <config.yaml> --skill <skill.md>
|
||||
```
|
||||
|
||||
### Arguments
|
||||
|
||||
| Argument | Description |
|
||||
|---|---|
|
||||
| `--config` | Path to YAML config file (required) |
|
||||
| `--skill` | Path to skill document to evaluate (required) |
|
||||
| `--split` | Evaluation split: `test` (default), `valid`, `train` |
|
||||
|
||||
### Examples
|
||||
|
||||
```bash
|
||||
# Evaluate best skill on test set
|
||||
python scripts/eval_only.py \
|
||||
--config configs/searchqa/default.yaml \
|
||||
--skill outputs/searchqa/run_001/skills/best_skill.md
|
||||
|
||||
# Evaluate on validation set
|
||||
python scripts/eval_only.py \
|
||||
--config configs/searchqa/default.yaml \
|
||||
--skill outputs/searchqa/run_001/skills/best_skill.md \
|
||||
--split valid
|
||||
```
|
||||
|
||||
## WebUI
|
||||
|
||||
```bash
|
||||
python -m skillopt_webui.app [--port PORT] [--share]
|
||||
```
|
||||
|
||||
| Argument | Default | Description |
|
||||
|---|---|---|
|
||||
| `--port` | 7860 | Port number |
|
||||
| `--share` | false | Create public Gradio link |
|
||||
@@ -1,72 +0,0 @@
|
||||
# Configuration Reference
|
||||
|
||||
Complete reference for all SkillOpt configuration parameters.
|
||||
|
||||
## Model
|
||||
|
||||
| Parameter | Type | Default | Description |
|
||||
|---|---|---|---|
|
||||
| `model.backend` | str | `azure_openai` | Backend: `azure_openai` / `openai_chat` / `claude_code_exec` / `qwen` |
|
||||
| `model.optimizer` | str | `gpt-5.5` | Optimizer model (for reflection & slow update) |
|
||||
| `model.target` | str | `gpt-5.5` | Target model (for rollout execution) |
|
||||
| `model.reasoning_effort` | str | `medium` | Reasoning effort level |
|
||||
|
||||
## Training (`train`)
|
||||
|
||||
| Parameter | Type | Default | DL Analogy | Description |
|
||||
|---|---|---|---|---|
|
||||
| `train.num_epochs` | int | 4 | Epochs | Number of training epochs |
|
||||
| `train.batch_size` | int | 40 | Batch size | Tasks sampled per step |
|
||||
| `train.accumulation` | int | 1 | Gradient accumulation | Accumulation rounds per step |
|
||||
| `train.seed` | int | 42 | Random seed | Reproducibility seed |
|
||||
|
||||
## Gradient / Reflection (`gradient`)
|
||||
|
||||
| Parameter | Type | Default | Description |
|
||||
|---|---|---|---|
|
||||
| `gradient.minibatch_size` | int | 8 | Reflect minibatch size |
|
||||
| `gradient.merge_batch_size` | int | 8 | Patch merge batch size |
|
||||
| `gradient.analyst_workers` | int | 16 | Parallel reflection workers |
|
||||
| `gradient.max_analyst_rounds` | int | 3 | Max rounds of analyst reflection |
|
||||
| `gradient.failure_only` | bool | `false` | Only reflect on failures |
|
||||
|
||||
## Optimizer (`optimizer`)
|
||||
|
||||
| Parameter | Type | Default | DL Analogy | Description |
|
||||
|---|---|---|---|---|
|
||||
| `optimizer.learning_rate` | int | 4 | Learning rate | Max edit patches per step (edit budget) |
|
||||
| `optimizer.min_learning_rate` | int | 2 | Min LR | Min edits for decay schedulers |
|
||||
| `optimizer.lr_scheduler` | str | `cosine` | LR schedule | `constant` / `linear` / `cosine` / `autonomous` |
|
||||
| `optimizer.skill_update_mode` | str | `patch` | — | `patch` / `rewrite_from_suggestions` / `full_rewrite_minibatch` |
|
||||
| `optimizer.use_slow_update` | bool | `true` | Momentum | Epoch-boundary longitudinal comparison & guidance |
|
||||
| `optimizer.slow_update_samples` | int | 20 | — | Samples for slow update evaluation |
|
||||
| `optimizer.use_meta_skill` | bool | `true` | Meta-learning | Cross-epoch optimizer-side strategy memory |
|
||||
| `optimizer.longitudinal_pair_policy` | str | `mixed` | — | `mixed` / `changed` / `unchanged` |
|
||||
|
||||
## Evaluation (`evaluation`)
|
||||
|
||||
| Parameter | Type | Default | Description |
|
||||
|---|---|---|---|
|
||||
| `evaluation.use_gate` | bool | `true` | Enable validation gating (accept/reject updates) |
|
||||
| `evaluation.eval_test` | bool | `true` | Run test evaluation after training |
|
||||
|
||||
## Environment (`env`)
|
||||
|
||||
| Parameter | Type | Default | Description |
|
||||
|---|---|---|---|
|
||||
| `env.name` | str | — | Benchmark name (e.g., `searchqa`, `docvqa`) |
|
||||
| `env.data_path` | str | — | Path to dataset |
|
||||
| `env.skill_init` | str | — | Path to initial seed skill (optional) |
|
||||
| `env.split_mode` | str | `ratio` | `ratio` or `split_dir` |
|
||||
| `env.split_ratio` | str | `2:1:7` | Train:val:test ratio |
|
||||
| `env.exec_timeout` | int | 120 | Per-task timeout in seconds |
|
||||
| `env.out_root` | str | — | Output directory |
|
||||
|
||||
## Azure OpenAI Credentials
|
||||
|
||||
| Variable | Description |
|
||||
|---|---|
|
||||
| `AZURE_OPENAI_ENDPOINT` / `model.azure_openai_endpoint` | Azure resource endpoint |
|
||||
| `AZURE_OPENAI_API_KEY` / `model.azure_openai_api_key` | Azure API key |
|
||||
| `OPENAI_API_KEY` | OpenAI API key (for `openai_chat` backend) |
|
||||
| `ANTHROPIC_API_KEY` | Anthropic API key (for `claude_code_exec` backend) |
|
||||
-2739
File diff suppressed because it is too large
Load Diff
-78
@@ -1,78 +0,0 @@
|
||||
site_name: SkillOpt Documentation
|
||||
site_url: https://microsoft.github.io/SkillOpt
|
||||
site_description: "SkillOpt: Agentic Skill Optimization via Reflective Training Loops"
|
||||
repo_url: https://github.com/microsoft/SkillOpt
|
||||
repo_name: microsoft/SkillOpt
|
||||
|
||||
theme:
|
||||
name: material
|
||||
palette:
|
||||
- scheme: default
|
||||
primary: indigo
|
||||
accent: deep purple
|
||||
toggle:
|
||||
icon: material/brightness-7
|
||||
name: Switch to dark mode
|
||||
- scheme: slate
|
||||
primary: indigo
|
||||
accent: deep purple
|
||||
toggle:
|
||||
icon: material/brightness-4
|
||||
name: Switch to light mode
|
||||
features:
|
||||
- navigation.instant
|
||||
- navigation.tracking
|
||||
- navigation.sections
|
||||
- navigation.expand
|
||||
- navigation.top
|
||||
- content.code.copy
|
||||
- content.tabs.link
|
||||
- search.suggest
|
||||
- search.highlight
|
||||
icon:
|
||||
repo: fontawesome/brands/github
|
||||
font:
|
||||
text: Inter
|
||||
code: JetBrains Mono
|
||||
|
||||
|
||||
|
||||
nav:
|
||||
- Home: index.md
|
||||
- Getting Started:
|
||||
- Installation: guide/installation.md
|
||||
- First Experiment: guide/first-experiment.md
|
||||
- Configuration: guide/configuration.md
|
||||
- Core Concepts:
|
||||
- Training Loop: guide/training-loop.md
|
||||
- Skill Document: guide/skill-document.md
|
||||
- Deep Learning Analogy: guide/dl-analogy.md
|
||||
- Extension Guides:
|
||||
- Add a New Benchmark: guide/new-benchmark.md
|
||||
- Local Environment Smoke Tests: guide/local-env-smoke.md
|
||||
- Add a New Model Backend: guide/new-backend.md
|
||||
- Reference:
|
||||
- Configuration Reference: reference/config.md
|
||||
- CLI Reference: reference/cli.md
|
||||
- API Reference: reference/api.md
|
||||
- Contributing: contributing.md
|
||||
|
||||
markdown_extensions:
|
||||
- admonition
|
||||
- pymdownx.details
|
||||
- pymdownx.superfences
|
||||
- pymdownx.tabbed:
|
||||
alternate_style: true
|
||||
- pymdownx.highlight:
|
||||
anchor_linenums: true
|
||||
- pymdownx.inlinehilite
|
||||
- pymdownx.emoji:
|
||||
emoji_index: !!python/name:material.extensions.emoji.twemoji
|
||||
emoji_generator: !!python/name:material.extensions.emoji.to_svg
|
||||
- attr_list
|
||||
- md_in_html
|
||||
- toc:
|
||||
permalink: true
|
||||
|
||||
plugins:
|
||||
- search
|
||||
@@ -1,75 +0,0 @@
|
||||
[build-system]
|
||||
requires = ["setuptools>=68.0", "wheel"]
|
||||
build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "skillopt"
|
||||
version = "0.1.0"
|
||||
description = "SkillOpt: Agentic Skill Optimization via Reflective Training Loops"
|
||||
readme = "README.md"
|
||||
license = {text = "MIT"}
|
||||
requires-python = ">=3.10"
|
||||
authors = [
|
||||
{name = "SkillOpt Team"},
|
||||
]
|
||||
keywords = ["agent", "prompt-optimization", "skill-learning", "LLM", "agentic"]
|
||||
classifiers = [
|
||||
"Development Status :: 3 - Alpha",
|
||||
"Intended Audience :: Science/Research",
|
||||
"License :: OSI Approved :: MIT License",
|
||||
"Programming Language :: Python :: 3",
|
||||
"Programming Language :: Python :: 3.10",
|
||||
"Programming Language :: Python :: 3.11",
|
||||
"Programming Language :: Python :: 3.12",
|
||||
"Topic :: Scientific/Engineering :: Artificial Intelligence",
|
||||
]
|
||||
dependencies = [
|
||||
"openai>=1.30.0",
|
||||
"pyyaml>=6.0",
|
||||
"numpy>=1.24.0",
|
||||
"openpyxl>=3.1.0",
|
||||
"azure-identity>=1.15.0",
|
||||
"azure-core>=1.30.0",
|
||||
"httpx>=0.27.0",
|
||||
]
|
||||
|
||||
[project.optional-dependencies]
|
||||
# Benchmark-specific dependencies
|
||||
alfworld = ["alfworld>=0.4.0", "gymnasium>=0.29.0"]
|
||||
# Claude model backend
|
||||
claude = ["claude-agent-sdk>=0.1.0"]
|
||||
# Qwen local model backend (via vLLM)
|
||||
qwen = ["vllm>=0.4.0"]
|
||||
# Documentation site
|
||||
docs = ["mkdocs-material>=9.5.0", "mkdocstrings[python]>=0.24.0"]
|
||||
# WebUI dashboard
|
||||
webui = ["gradio>=4.0.0"]
|
||||
# Development tools
|
||||
dev = ["ruff>=0.4.0", "pytest>=8.0.0"]
|
||||
# All optional dependencies (except docs/dev/webui)
|
||||
all = [
|
||||
"alfworld>=0.4.0",
|
||||
"gymnasium>=0.29.0",
|
||||
"claude-agent-sdk>=0.1.0",
|
||||
]
|
||||
|
||||
[project.scripts]
|
||||
skillopt-train = "scripts.train:main"
|
||||
skillopt-eval = "scripts.eval_only:main"
|
||||
|
||||
[project.urls]
|
||||
Homepage = "https://github.com/microsoft/SkillOpt"
|
||||
Documentation = "https://microsoft.github.io/SkillOpt"
|
||||
Repository = "https://github.com/microsoft/SkillOpt"
|
||||
Issues = "https://github.com/microsoft/SkillOpt/issues"
|
||||
|
||||
[tool.setuptools.packages.find]
|
||||
include = ["skillopt*", "scripts*"]
|
||||
|
||||
[tool.ruff]
|
||||
line-length = 120
|
||||
target-version = "py310"
|
||||
|
||||
[tool.ruff.lint]
|
||||
select = ["E", "F", "I", "W"]
|
||||
ignore = ["E501"]
|
||||
@@ -1,25 +0,0 @@
|
||||
# ── Core ──────────────────────────────────────────
|
||||
openai>=1.30.0
|
||||
pyyaml>=6.0
|
||||
numpy>=1.24.0
|
||||
openpyxl>=3.1.0
|
||||
azure-identity>=1.15.0
|
||||
azure-core>=1.30.0
|
||||
httpx>=0.27.0
|
||||
|
||||
# ── Optional: ALFWorld benchmark ──────────────────
|
||||
# alfworld>=0.4.0
|
||||
# gymnasium>=0.29.0
|
||||
|
||||
# ── Optional: Claude model backend ────────────────
|
||||
# claude-agent-sdk>=0.1.0
|
||||
|
||||
# ── Optional: Qwen local model (via vLLM) ────────
|
||||
# vllm>=0.4.0
|
||||
|
||||
# ── Optional: WebUI dashboard ────────────────────
|
||||
# gradio>=4.0.0
|
||||
|
||||
# ── Optional: Documentation site ─────────────────
|
||||
# mkdocs-material>=9.5.0
|
||||
# mkdocstrings[python]>=0.24.0
|
||||
@@ -1,451 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""SkillOpt eval-only: run a single skill on a dataset without training.
|
||||
|
||||
Usage
|
||||
-----
|
||||
python scripts/eval_only.py \
|
||||
--config configs/spreadsheetbench/default.yaml \
|
||||
--skill skillopt/envs/spreadsheetbench/skills/initial.md \
|
||||
--split_dir /path/to/split \
|
||||
--out_root outputs/eval_skill0
|
||||
|
||||
All YAML keys can be overridden from the CLI, same as train.py.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import datetime
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
|
||||
_SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__))
|
||||
_PROJECT_ROOT = os.path.dirname(_SCRIPT_DIR)
|
||||
if _PROJECT_ROOT not in sys.path:
|
||||
sys.path.insert(0, _PROJECT_ROOT)
|
||||
|
||||
from skillopt.model import (
|
||||
configure_azure_openai,
|
||||
configure_claude_code_exec,
|
||||
configure_codex_exec,
|
||||
set_reasoning_effort,
|
||||
set_target_backend,
|
||||
set_target_deployment,
|
||||
set_optimizer_backend,
|
||||
set_optimizer_deployment,
|
||||
)
|
||||
from skillopt.model.common import default_model_for_backend, normalize_backend_name
|
||||
|
||||
_OPENAI_DEFAULT_MODEL_SENTINELS = {"gpt-5.4", "gpt-5.5"}
|
||||
from skillopt.utils import compute_score
|
||||
|
||||
|
||||
# ── Reuse registry from train.py ───────────────────────────────────────────
|
||||
|
||||
_ENV_REGISTRY: dict[str, type] = {}
|
||||
|
||||
|
||||
def _register_builtins() -> None:
|
||||
try:
|
||||
from skillopt.envs.alfworld.adapter import ALFWorldAdapter
|
||||
_ENV_REGISTRY["alfworld"] = ALFWorldAdapter
|
||||
except ImportError:
|
||||
pass
|
||||
try:
|
||||
from skillopt.envs.searchqa.adapter import SearchQAAdapter
|
||||
_ENV_REGISTRY["searchqa"] = SearchQAAdapter
|
||||
except ImportError:
|
||||
pass
|
||||
try:
|
||||
from skillopt.envs.livemathematicianbench.adapter import LiveMathematicianBenchAdapter
|
||||
_ENV_REGISTRY["livemathematicianbench"] = LiveMathematicianBenchAdapter
|
||||
except ImportError:
|
||||
pass
|
||||
try:
|
||||
from skillopt.envs.babyvision.adapter import BabyVisionAdapter
|
||||
_ENV_REGISTRY["babyvision"] = BabyVisionAdapter
|
||||
except ImportError:
|
||||
pass
|
||||
try:
|
||||
from skillopt.envs.spreadsheetbench.adapter import SpreadsheetBenchAdapter
|
||||
_ENV_REGISTRY["spreadsheetbench"] = SpreadsheetBenchAdapter
|
||||
except ImportError:
|
||||
pass
|
||||
try:
|
||||
from skillopt.envs.mmrb.adapter import MMRBAdapter
|
||||
_ENV_REGISTRY["mmrb"] = MMRBAdapter
|
||||
except ImportError:
|
||||
pass
|
||||
try:
|
||||
from skillopt.envs.docvqa.adapter import DocVQAAdapter
|
||||
_ENV_REGISTRY["docvqa"] = DocVQAAdapter
|
||||
except ImportError:
|
||||
pass
|
||||
try:
|
||||
from skillopt.envs.mathverse.adapter import MathVerseAdapter
|
||||
_ENV_REGISTRY["mathverse"] = MathVerseAdapter
|
||||
except ImportError:
|
||||
pass
|
||||
try:
|
||||
from skillopt.envs.officeqa.adapter import OfficeQAAdapter
|
||||
_ENV_REGISTRY["officeqa"] = OfficeQAAdapter
|
||||
except ImportError:
|
||||
pass
|
||||
try:
|
||||
from skillopt.envs.sealqa.adapter import SealQAAdapter
|
||||
_ENV_REGISTRY["sealqa"] = SealQAAdapter
|
||||
except ImportError:
|
||||
pass
|
||||
try:
|
||||
from skillopt.envs.swebench.adapter import SWEBenchAdapter
|
||||
_ENV_REGISTRY["swebench"] = SWEBenchAdapter
|
||||
except ImportError:
|
||||
pass
|
||||
|
||||
|
||||
def get_adapter(cfg: dict):
|
||||
_register_builtins()
|
||||
env_name = cfg.get("env", "alfworld")
|
||||
if env_name not in _ENV_REGISTRY:
|
||||
raise ValueError(
|
||||
f"Unknown environment '{env_name}'. "
|
||||
f"Available: {list(_ENV_REGISTRY.keys())}"
|
||||
)
|
||||
adapter_cls = _ENV_REGISTRY[env_name]
|
||||
|
||||
import inspect
|
||||
sig = inspect.signature(adapter_cls.__init__)
|
||||
accepted = set(sig.parameters.keys()) - {"self"}
|
||||
adapter_kwargs = {k: cfg[k] for k in accepted if k in cfg}
|
||||
return adapter_cls(**adapter_kwargs)
|
||||
|
||||
|
||||
# ── CLI ────────────────────────────────────────────────────────────────────
|
||||
|
||||
_BOOL = lambda x: str(x).lower() in ("true", "1", "yes") # noqa: E731
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
p = argparse.ArgumentParser(description="SkillOpt eval-only")
|
||||
p.add_argument("--config", type=str, required=True)
|
||||
p.add_argument("--skill", type=str, required=True,
|
||||
help="Path to skill .md file to evaluate")
|
||||
p.add_argument("--split", type=str, default="all",
|
||||
help="Which split to eval: train/valid_seen/valid_unseen/all (default: all)")
|
||||
p.add_argument("--cfg-options", nargs="+", default=[],
|
||||
help="Override config: section.key=value")
|
||||
# Legacy flat overrides
|
||||
p.add_argument("--env", type=str)
|
||||
p.add_argument("--backend", type=str,
|
||||
choices=["azure_openai", "codex", "codex_exec", "claude", "claude_chat", "claude_code_exec"])
|
||||
p.add_argument("--optimizer_model", type=str)
|
||||
p.add_argument("--target_model", type=str)
|
||||
p.add_argument("--optimizer_backend", type=str)
|
||||
p.add_argument("--target_backend", type=str)
|
||||
p.add_argument("--reasoning_effort", type=str,
|
||||
choices=["", "low", "medium", "high", "xhigh", "max"])
|
||||
p.add_argument("--azure_endpoint", type=str)
|
||||
p.add_argument("--azure_api_version", type=str)
|
||||
p.add_argument("--azure_api_key", type=str)
|
||||
p.add_argument("--azure_openai_endpoint", type=str)
|
||||
p.add_argument("--azure_openai_api_version", type=str)
|
||||
p.add_argument("--azure_openai_api_key", type=str)
|
||||
p.add_argument("--azure_openai_auth_mode", type=str)
|
||||
p.add_argument("--azure_openai_ad_scope", type=str)
|
||||
p.add_argument("--azure_openai_managed_identity_client_id", type=str)
|
||||
p.add_argument("--optimizer_azure_openai_endpoint", type=str)
|
||||
p.add_argument("--optimizer_azure_openai_api_version", type=str)
|
||||
p.add_argument("--optimizer_azure_openai_api_key", type=str)
|
||||
p.add_argument("--optimizer_azure_openai_auth_mode", type=str)
|
||||
p.add_argument("--optimizer_azure_openai_ad_scope", type=str)
|
||||
p.add_argument("--optimizer_azure_openai_managed_identity_client_id", type=str)
|
||||
p.add_argument("--target_azure_openai_endpoint", type=str)
|
||||
p.add_argument("--target_azure_openai_api_version", type=str)
|
||||
p.add_argument("--target_azure_openai_api_key", type=str)
|
||||
p.add_argument("--target_azure_openai_auth_mode", type=str)
|
||||
p.add_argument("--target_azure_openai_ad_scope", type=str)
|
||||
p.add_argument("--target_azure_openai_managed_identity_client_id", type=str)
|
||||
p.add_argument("--codex_exec_path", type=str)
|
||||
p.add_argument("--codex_exec_sandbox", type=str)
|
||||
p.add_argument("--codex_exec_profile", type=str)
|
||||
p.add_argument("--codex_exec_full_auto", type=_BOOL)
|
||||
p.add_argument("--codex_exec_reasoning_effort", type=str)
|
||||
p.add_argument("--codex_exec_use_sdk", type=str)
|
||||
p.add_argument("--codex_exec_network_access", type=_BOOL)
|
||||
p.add_argument("--codex_exec_web_search", type=_BOOL)
|
||||
p.add_argument("--codex_exec_approval_policy", type=str)
|
||||
p.add_argument("--claude_code_exec_path", type=str)
|
||||
p.add_argument("--claude_code_exec_profile", type=str)
|
||||
p.add_argument("--claude_code_exec_use_sdk", type=str)
|
||||
p.add_argument("--claude_code_exec_effort", type=str)
|
||||
p.add_argument("--claude_code_exec_max_thinking_tokens", type=int)
|
||||
p.add_argument("--out_root", type=str)
|
||||
p.add_argument("--data_path", type=str)
|
||||
p.add_argument("--split_mode", type=str,
|
||||
choices=["ratio", "split_dir"])
|
||||
p.add_argument("--split_ratio", type=str)
|
||||
p.add_argument("--split_seed", type=int)
|
||||
p.add_argument("--split_dir", type=str)
|
||||
p.add_argument("--split_output_dir", type=str)
|
||||
p.add_argument("--data_root", type=str)
|
||||
p.add_argument("--max_turns", type=int)
|
||||
p.add_argument("--workers", type=int)
|
||||
p.add_argument("--max_api_workers", type=int)
|
||||
p.add_argument("--seed", type=int)
|
||||
p.add_argument("--test_env_num", type=int)
|
||||
p.add_argument("--mode", type=str,
|
||||
help="SpreadsheetBench: single/multi/react (default comes from config)")
|
||||
return p.parse_args()
|
||||
|
||||
|
||||
def main() -> None:
|
||||
args = parse_args()
|
||||
|
||||
from skillopt.config import load_config as _load, flatten_config, is_structured
|
||||
|
||||
cfg = _load(args.config, overrides=args.cfg_options)
|
||||
structured = is_structured(cfg)
|
||||
|
||||
# Apply legacy --key value overrides
|
||||
cli = {k: v for k, v in vars(args).items()
|
||||
if v is not None and k not in ("config", "skill", "split", "cfg_options")}
|
||||
if cli:
|
||||
if structured:
|
||||
from skillopt.config import apply_overrides
|
||||
_MAP = {
|
||||
"backend": "model.backend",
|
||||
"optimizer_model": "model.optimizer",
|
||||
"target_model": "model.target",
|
||||
"optimizer_backend": "model.optimizer_backend",
|
||||
"target_backend": "model.target_backend",
|
||||
"reasoning_effort": "model.reasoning_effort",
|
||||
"azure_endpoint": "model.azure_endpoint",
|
||||
"azure_api_version": "model.azure_api_version",
|
||||
"azure_api_key": "model.azure_api_key",
|
||||
"azure_openai_endpoint": "model.azure_openai_endpoint",
|
||||
"azure_openai_api_version": "model.azure_openai_api_version",
|
||||
"azure_openai_api_key": "model.azure_openai_api_key",
|
||||
"azure_openai_auth_mode": "model.azure_openai_auth_mode",
|
||||
"azure_openai_ad_scope": "model.azure_openai_ad_scope",
|
||||
"azure_openai_managed_identity_client_id": "model.azure_openai_managed_identity_client_id",
|
||||
"optimizer_azure_openai_endpoint": "model.optimizer_azure_openai_endpoint",
|
||||
"optimizer_azure_openai_api_version": "model.optimizer_azure_openai_api_version",
|
||||
"optimizer_azure_openai_api_key": "model.optimizer_azure_openai_api_key",
|
||||
"optimizer_azure_openai_auth_mode": "model.optimizer_azure_openai_auth_mode",
|
||||
"optimizer_azure_openai_ad_scope": "model.optimizer_azure_openai_ad_scope",
|
||||
"optimizer_azure_openai_managed_identity_client_id": "model.optimizer_azure_openai_managed_identity_client_id",
|
||||
"target_azure_openai_endpoint": "model.target_azure_openai_endpoint",
|
||||
"target_azure_openai_api_version": "model.target_azure_openai_api_version",
|
||||
"target_azure_openai_api_key": "model.target_azure_openai_api_key",
|
||||
"target_azure_openai_auth_mode": "model.target_azure_openai_auth_mode",
|
||||
"target_azure_openai_ad_scope": "model.target_azure_openai_ad_scope",
|
||||
"target_azure_openai_managed_identity_client_id": "model.target_azure_openai_managed_identity_client_id",
|
||||
"codex_exec_path": "model.codex_exec_path",
|
||||
"codex_exec_sandbox": "model.codex_exec_sandbox",
|
||||
"codex_exec_profile": "model.codex_exec_profile",
|
||||
"codex_exec_full_auto": "model.codex_exec_full_auto",
|
||||
"codex_exec_reasoning_effort": "model.codex_exec_reasoning_effort",
|
||||
"codex_exec_use_sdk": "model.codex_exec_use_sdk",
|
||||
"codex_exec_network_access": "model.codex_exec_network_access",
|
||||
"codex_exec_web_search": "model.codex_exec_web_search",
|
||||
"codex_exec_approval_policy": "model.codex_exec_approval_policy",
|
||||
"claude_code_exec_path": "model.claude_code_exec_path",
|
||||
"claude_code_exec_profile": "model.claude_code_exec_profile",
|
||||
"claude_code_exec_use_sdk": "model.claude_code_exec_use_sdk",
|
||||
"claude_code_exec_effort": "model.claude_code_exec_effort",
|
||||
"claude_code_exec_max_thinking_tokens": "model.claude_code_exec_max_thinking_tokens",
|
||||
"seed": "train.seed",
|
||||
"test_env_num": "evaluation.test_env_num",
|
||||
"env": "env.name",
|
||||
"out_root": "env.out_root",
|
||||
}
|
||||
mapped = []
|
||||
for k, v in cli.items():
|
||||
dotted = _MAP.get(k)
|
||||
if dotted:
|
||||
mapped.append(f"{dotted}={v}")
|
||||
else:
|
||||
mapped.append(f"env.{k}={v}")
|
||||
apply_overrides(cfg, mapped)
|
||||
else:
|
||||
cfg.update(cli)
|
||||
|
||||
cfg = flatten_config(cfg) if structured else cfg
|
||||
|
||||
for new_key, old_key in (
|
||||
("azure_openai_endpoint", "azure_endpoint"),
|
||||
("azure_openai_api_version", "azure_api_version"),
|
||||
("azure_openai_api_key", "azure_api_key"),
|
||||
):
|
||||
if cfg.get(new_key) in (None, "") and cfg.get(old_key) not in (None, ""):
|
||||
cfg[new_key] = cfg[old_key]
|
||||
|
||||
explicit_backend = getattr(args, "backend", None)
|
||||
if explicit_backend is None:
|
||||
for option in args.cfg_options or []:
|
||||
key = str(option).split("=", 1)[0].strip()
|
||||
if key == "model.backend":
|
||||
explicit_backend = str(option).split("=", 1)[1].strip()
|
||||
break
|
||||
|
||||
backend = normalize_backend_name(cfg.get("model_backend") or cfg.get("target_backend") or "azure_openai")
|
||||
|
||||
def _has_model_override(dotted_key: str, legacy_key: str) -> bool:
|
||||
if getattr(args, legacy_key, None) is not None:
|
||||
return True
|
||||
for option in args.cfg_options or []:
|
||||
key = str(option).split("=", 1)[0].strip()
|
||||
if key == dotted_key:
|
||||
return True
|
||||
return False
|
||||
|
||||
if explicit_backend is not None:
|
||||
backend = normalize_backend_name(explicit_backend)
|
||||
cfg["model_backend"] = backend
|
||||
if backend in {"claude", "claude_chat"}:
|
||||
cfg.setdefault("optimizer_backend", "claude_chat")
|
||||
cfg.setdefault("target_backend", "claude_chat")
|
||||
elif backend in {"codex", "codex_exec"}:
|
||||
cfg.setdefault("optimizer_backend", "openai_chat")
|
||||
cfg.setdefault("target_backend", "codex_exec")
|
||||
elif backend == "claude_code_exec":
|
||||
cfg.setdefault("optimizer_backend", "openai_chat")
|
||||
cfg.setdefault("target_backend", "claude_code_exec")
|
||||
else:
|
||||
cfg.setdefault("optimizer_backend", "openai_chat")
|
||||
cfg.setdefault("target_backend", "openai_chat")
|
||||
else:
|
||||
cfg.setdefault("optimizer_backend", "openai_chat")
|
||||
cfg.setdefault("target_backend", "openai_chat")
|
||||
|
||||
if cfg.get("optimizer_backend") == "claude_chat":
|
||||
if (
|
||||
str(cfg.get("optimizer_model", "") or "").strip() in _OPENAI_DEFAULT_MODEL_SENTINELS
|
||||
and not _has_model_override("model.optimizer", "optimizer_model")
|
||||
):
|
||||
cfg["optimizer_model"] = default_model_for_backend("claude_chat")
|
||||
if cfg.get("target_backend") == "claude_chat":
|
||||
if (
|
||||
str(cfg.get("target_model", "") or "").strip() in _OPENAI_DEFAULT_MODEL_SENTINELS
|
||||
and not _has_model_override("model.target", "target_model")
|
||||
):
|
||||
cfg["target_model"] = default_model_for_backend("claude_chat")
|
||||
if cfg.get("target_backend") == "claude_code_exec":
|
||||
if (
|
||||
str(cfg.get("target_model", "") or "").strip() in _OPENAI_DEFAULT_MODEL_SENTINELS
|
||||
and not _has_model_override("model.target", "target_model")
|
||||
):
|
||||
cfg["target_model"] = default_model_for_backend("claude_chat")
|
||||
|
||||
if not cfg.get("out_root"):
|
||||
env = cfg.get("env", "unknown")
|
||||
model = cfg.get("target_model", "unknown").replace("/", "-")
|
||||
ts = datetime.datetime.now().strftime("%Y%m%d_%H%M%S")
|
||||
cfg["out_root"] = os.path.join("outputs", f"eval_{env}_{model}_{ts}")
|
||||
|
||||
cfg["out_root"] = os.path.abspath(cfg["out_root"])
|
||||
|
||||
out_root = cfg["out_root"]
|
||||
os.makedirs(out_root, exist_ok=True)
|
||||
|
||||
# Load skill
|
||||
skill_path = os.path.abspath(args.skill)
|
||||
with open(skill_path) as f:
|
||||
skill_content = f.read()
|
||||
print(f" [skill] {skill_path} ({len(skill_content)} chars)")
|
||||
|
||||
# Configure models
|
||||
configure_azure_openai(
|
||||
endpoint=(cfg.get("azure_openai_endpoint") or cfg.get("azure_endpoint") or None),
|
||||
api_version=(cfg.get("azure_openai_api_version") or cfg.get("azure_api_version") or None),
|
||||
api_key=(cfg.get("azure_openai_api_key") or cfg.get("azure_api_key") or None),
|
||||
auth_mode=cfg.get("azure_openai_auth_mode") or None,
|
||||
ad_scope=cfg.get("azure_openai_ad_scope") or None,
|
||||
managed_identity_client_id=cfg.get("azure_openai_managed_identity_client_id") or None,
|
||||
optimizer_endpoint=cfg.get("optimizer_azure_openai_endpoint") or None,
|
||||
optimizer_api_version=cfg.get("optimizer_azure_openai_api_version") or None,
|
||||
optimizer_api_key=cfg.get("optimizer_azure_openai_api_key") or None,
|
||||
optimizer_auth_mode=cfg.get("optimizer_azure_openai_auth_mode") or None,
|
||||
optimizer_ad_scope=cfg.get("optimizer_azure_openai_ad_scope") or None,
|
||||
optimizer_managed_identity_client_id=(
|
||||
cfg.get("optimizer_azure_openai_managed_identity_client_id") or None
|
||||
),
|
||||
target_endpoint=cfg.get("target_azure_openai_endpoint") or None,
|
||||
target_api_version=cfg.get("target_azure_openai_api_version") or None,
|
||||
target_api_key=cfg.get("target_azure_openai_api_key") or None,
|
||||
target_auth_mode=cfg.get("target_azure_openai_auth_mode") or None,
|
||||
target_ad_scope=cfg.get("target_azure_openai_ad_scope") or None,
|
||||
target_managed_identity_client_id=(
|
||||
cfg.get("target_azure_openai_managed_identity_client_id") or None
|
||||
),
|
||||
)
|
||||
set_optimizer_backend(cfg.get("optimizer_backend", "openai_chat"))
|
||||
set_target_backend(cfg.get("target_backend", "openai_chat"))
|
||||
set_optimizer_deployment(cfg.get("optimizer_model", default_model_for_backend(backend)))
|
||||
set_target_deployment(cfg.get("target_model", default_model_for_backend(backend)))
|
||||
configure_codex_exec(
|
||||
path=cfg.get("codex_exec_path", "codex"),
|
||||
sandbox=cfg.get("codex_exec_sandbox", "workspace-write"),
|
||||
profile=cfg.get("codex_exec_profile", ""),
|
||||
full_auto=cfg.get("codex_exec_full_auto", False),
|
||||
reasoning_effort=cfg.get("codex_exec_reasoning_effort", "none"),
|
||||
use_sdk=cfg.get("codex_exec_use_sdk", None),
|
||||
network_access=cfg.get("codex_exec_network_access", False),
|
||||
web_search=cfg.get("codex_exec_web_search", False),
|
||||
approval_policy=cfg.get("codex_exec_approval_policy", "never"),
|
||||
)
|
||||
configure_claude_code_exec(
|
||||
path=cfg.get("claude_code_exec_path", "claude"),
|
||||
profile=cfg.get("claude_code_exec_profile", ""),
|
||||
use_sdk=cfg.get("claude_code_exec_use_sdk", None),
|
||||
effort=cfg.get("claude_code_exec_effort", cfg.get("reasoning_effort", "medium")),
|
||||
max_thinking_tokens=cfg.get("claude_code_exec_max_thinking_tokens", 16384),
|
||||
)
|
||||
set_reasoning_effort(cfg.get("reasoning_effort", "") or None)
|
||||
|
||||
# Build adapter
|
||||
adapter = get_adapter(cfg)
|
||||
adapter.setup(cfg)
|
||||
|
||||
seed = cfg.get("seed", 42)
|
||||
split = args.split or "all"
|
||||
|
||||
if split == "all":
|
||||
items = (
|
||||
adapter.build_eval_env(0, "train", seed)
|
||||
+ adapter.build_eval_env(0, "valid_seen", seed)
|
||||
+ adapter.build_eval_env(0, "valid_unseen", seed)
|
||||
)
|
||||
else:
|
||||
env_num = cfg.get("test_env_num", 0)
|
||||
items = adapter.build_eval_env(env_num, split, seed)
|
||||
|
||||
print(f"\n [eval] split={split} items={len(items)}")
|
||||
print(f" [eval] out_root={out_root}")
|
||||
print(f"{'='*60}")
|
||||
|
||||
# Run rollout
|
||||
results = adapter.rollout(items, skill_content, out_root)
|
||||
|
||||
# Score
|
||||
hard, soft = compute_score(results)
|
||||
print(f"\n{'='*60}")
|
||||
print(f" Results: hard={hard:.4f} soft={soft:.4f} (n={len(results)})")
|
||||
print(f"{'='*60}")
|
||||
|
||||
# Save summary
|
||||
summary = {
|
||||
"skill": skill_path,
|
||||
"split": split,
|
||||
"n_items": len(results),
|
||||
"hard": hard,
|
||||
"soft": soft,
|
||||
}
|
||||
with open(os.path.join(out_root, "eval_summary.json"), "w") as f:
|
||||
json.dump(summary, f, indent=2, ensure_ascii=False)
|
||||
|
||||
print(f" Saved to: {out_root}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,60 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# ──────────────────────────────────────────────────────────────────────────────
|
||||
# SkillOpt — ALFWorld training launch script
|
||||
#
|
||||
# Prerequisites:
|
||||
# pip install -e ".[alfworld]"
|
||||
# pip install alfworld[full] && alfworld-download
|
||||
#
|
||||
# Usage:
|
||||
# bash scripts/run_alfworld.sh
|
||||
# bash scripts/run_alfworld.sh --num_epochs 2 --edit_budget 6
|
||||
# bash scripts/run_alfworld.sh --split_dir /path/to/alfworld_split
|
||||
# ──────────────────────────────────────────────────────────────────────────────
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
PROJECT_ROOT="$(dirname "${SCRIPT_DIR}")"
|
||||
|
||||
export PYTHONPATH="${PROJECT_ROOT}:${PYTHONPATH:-}"
|
||||
|
||||
# ALFWorld data — uses ~/.cache/alfworld by default
|
||||
export ALFWORLD_DATA="${ALFWORLD_DATA:-${HOME}/.cache/alfworld}"
|
||||
|
||||
if [ ! -d "${ALFWORLD_DATA}/json_2.1.1" ]; then
|
||||
echo "ERROR: ALFWorld data not found at ${ALFWORLD_DATA}/json_2.1.1"
|
||||
echo ""
|
||||
echo "To download ALFWorld data, run:"
|
||||
echo " pip install alfworld[full]"
|
||||
echo " alfworld-download"
|
||||
echo ""
|
||||
echo "Or set ALFWORLD_DATA to the directory containing json_2.1.1/"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
OPTIMIZER_MODEL="${OPTIMIZER_MODEL:-gpt-5.5}"
|
||||
TARGET_MODEL="${TARGET_MODEL:-gpt-5.5}"
|
||||
|
||||
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
|
||||
DEFAULT_OUT_ROOT="${PROJECT_ROOT}/outputs/skillopt_alfworld_${TARGET_MODEL}_${TIMESTAMP}"
|
||||
|
||||
echo "============================================================"
|
||||
echo " SkillOpt — ALFWorld Training"
|
||||
echo "============================================================"
|
||||
echo " Optimizer: ${OPTIMIZER_MODEL}"
|
||||
echo " Target: ${TARGET_MODEL}"
|
||||
echo " ALFWORLD_DATA: ${ALFWORLD_DATA}"
|
||||
echo " Output: ${DEFAULT_OUT_ROOT}"
|
||||
echo "============================================================"
|
||||
|
||||
cd "${PROJECT_ROOT}"
|
||||
|
||||
python scripts/train.py \
|
||||
--config configs/alfworld/default.yaml \
|
||||
--optimizer_model "${OPTIMIZER_MODEL}" \
|
||||
--target_model "${TARGET_MODEL}" \
|
||||
--out_root "${DEFAULT_OUT_ROOT}" \
|
||||
"$@"
|
||||
|
||||
echo ""
|
||||
echo "Done! Results saved to: ${DEFAULT_OUT_ROOT}"
|
||||
@@ -1,40 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# ──────────────────────────────────────────────────────────────────────────────
|
||||
# SkillOpt — SearchQA training launch script
|
||||
#
|
||||
# Usage:
|
||||
# bash scripts/run_searchqa.sh
|
||||
# bash scripts/run_searchqa.sh --num_epochs 2 --edit_budget 6
|
||||
# bash scripts/run_searchqa.sh --split_dir /path/to/searchqa_split
|
||||
# ──────────────────────────────────────────────────────────────────────────────
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
PROJECT_ROOT="$(dirname "${SCRIPT_DIR}")"
|
||||
|
||||
export PYTHONPATH="${PROJECT_ROOT}:${PYTHONPATH:-}"
|
||||
|
||||
OPTIMIZER_MODEL="${OPTIMIZER_MODEL:-gpt-5.5}"
|
||||
TARGET_MODEL="${TARGET_MODEL:-gpt-5.5}"
|
||||
|
||||
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
|
||||
DEFAULT_OUT_ROOT="${PROJECT_ROOT}/outputs/skillopt_searchqa_${TARGET_MODEL}_${TIMESTAMP}"
|
||||
|
||||
echo "============================================================"
|
||||
echo " SkillOpt — SearchQA Training"
|
||||
echo "============================================================"
|
||||
echo " Optimizer: ${OPTIMIZER_MODEL}"
|
||||
echo " Target: ${TARGET_MODEL}"
|
||||
echo "============================================================"
|
||||
|
||||
cd "${PROJECT_ROOT}"
|
||||
|
||||
python scripts/train.py \
|
||||
--config configs/searchqa/default.yaml \
|
||||
--optimizer_model "${OPTIMIZER_MODEL}" \
|
||||
--target_model "${TARGET_MODEL}" \
|
||||
--out_root "${DEFAULT_OUT_ROOT}" \
|
||||
"$@"
|
||||
|
||||
echo ""
|
||||
echo "Done! Results saved to: ${DEFAULT_OUT_ROOT}"
|
||||
@@ -1,39 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# ──────────────────────────────────────────────────────────────────────────────
|
||||
# SkillOpt — SpreadsheetBench training launch script
|
||||
#
|
||||
# Usage:
|
||||
# bash scripts/run_spreadsheetbench.sh --split_dir /path/to/split --data_root /path/to/data
|
||||
# bash scripts/run_spreadsheetbench.sh --num_epochs 2 --edit_budget 6
|
||||
# ──────────────────────────────────────────────────────────────────────────────
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
PROJECT_ROOT="$(dirname "${SCRIPT_DIR}")"
|
||||
|
||||
export PYTHONPATH="${PROJECT_ROOT}:${PYTHONPATH:-}"
|
||||
|
||||
OPTIMIZER_MODEL="${OPTIMIZER_MODEL:-gpt-5.5}"
|
||||
TARGET_MODEL="${TARGET_MODEL:-gpt-5.5}"
|
||||
|
||||
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
|
||||
DEFAULT_OUT_ROOT="${PROJECT_ROOT}/outputs/skillopt_spreadsheetbench_${TARGET_MODEL}_${TIMESTAMP}"
|
||||
|
||||
echo "============================================================"
|
||||
echo " SkillOpt — SpreadsheetBench Training"
|
||||
echo "============================================================"
|
||||
echo " Optimizer: ${OPTIMIZER_MODEL}"
|
||||
echo " Target: ${TARGET_MODEL}"
|
||||
echo "============================================================"
|
||||
|
||||
cd "${PROJECT_ROOT}"
|
||||
|
||||
python scripts/train.py \
|
||||
--config configs/spreadsheetbench/default.yaml \
|
||||
--optimizer_model "${OPTIMIZER_MODEL}" \
|
||||
--target_model "${TARGET_MODEL}" \
|
||||
--out_root "${DEFAULT_OUT_ROOT}" \
|
||||
"$@"
|
||||
|
||||
echo ""
|
||||
echo "Done! Results saved to: ${DEFAULT_OUT_ROOT}"
|
||||
@@ -1,494 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""SkillOpt unified training entry point.
|
||||
|
||||
Usage
|
||||
-----
|
||||
python scripts/train.py --config configs/alfworld/default.yaml
|
||||
|
||||
Any YAML key can be overridden from the command line::
|
||||
|
||||
python scripts/train.py --config configs/alfworld/default.yaml \\
|
||||
--batch_size 40 --num_epochs 2 --seed 123
|
||||
|
||||
Run ``python scripts/train.py --help`` for a full list of options.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import datetime
|
||||
import os
|
||||
import sys
|
||||
|
||||
# Ensure the project root is on sys.path so ``import skillopt`` works
|
||||
# regardless of where the script is invoked from.
|
||||
_SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__))
|
||||
_PROJECT_ROOT = os.path.dirname(_SCRIPT_DIR)
|
||||
if _PROJECT_ROOT not in sys.path:
|
||||
sys.path.insert(0, _PROJECT_ROOT)
|
||||
|
||||
from skillopt.model.common import default_model_for_backend, normalize_backend_name
|
||||
|
||||
_OPENAI_DEFAULT_MODEL_SENTINELS = {"gpt-5.4", "gpt-5.5"}
|
||||
|
||||
|
||||
# ── Environment registry ────────────────────────────────────────────────────
|
||||
|
||||
_ENV_REGISTRY: dict[str, type] = {}
|
||||
|
||||
|
||||
def _register_builtins() -> None:
|
||||
"""Lazy-import built-in adapters so we don't pull heavy deps at CLI parse time."""
|
||||
try:
|
||||
from skillopt.envs.alfworld.adapter import ALFWorldAdapter
|
||||
_ENV_REGISTRY["alfworld"] = ALFWorldAdapter
|
||||
except ImportError:
|
||||
pass # ALFWorld deps not installed — skip
|
||||
try:
|
||||
from skillopt.envs.searchqa.adapter import SearchQAAdapter
|
||||
_ENV_REGISTRY["searchqa"] = SearchQAAdapter
|
||||
except ImportError:
|
||||
pass
|
||||
try:
|
||||
from skillopt.envs.livemathematicianbench.adapter import LiveMathematicianBenchAdapter
|
||||
_ENV_REGISTRY["livemathematicianbench"] = LiveMathematicianBenchAdapter
|
||||
except ImportError:
|
||||
pass
|
||||
try:
|
||||
from skillopt.envs.babyvision.adapter import BabyVisionAdapter
|
||||
_ENV_REGISTRY["babyvision"] = BabyVisionAdapter
|
||||
except ImportError:
|
||||
pass
|
||||
try:
|
||||
from skillopt.envs.spreadsheetbench.adapter import SpreadsheetBenchAdapter
|
||||
_ENV_REGISTRY["spreadsheetbench"] = SpreadsheetBenchAdapter
|
||||
except ImportError:
|
||||
pass
|
||||
try:
|
||||
from skillopt.envs.mmrb.adapter import MMRBAdapter
|
||||
_ENV_REGISTRY["mmrb"] = MMRBAdapter
|
||||
except ImportError:
|
||||
pass
|
||||
try:
|
||||
from skillopt.envs.docvqa.adapter import DocVQAAdapter
|
||||
_ENV_REGISTRY["docvqa"] = DocVQAAdapter
|
||||
except ImportError:
|
||||
pass
|
||||
try:
|
||||
from skillopt.envs.mathverse.adapter import MathVerseAdapter
|
||||
_ENV_REGISTRY["mathverse"] = MathVerseAdapter
|
||||
except ImportError:
|
||||
pass
|
||||
try:
|
||||
from skillopt.envs.officeqa.adapter import OfficeQAAdapter
|
||||
_ENV_REGISTRY["officeqa"] = OfficeQAAdapter
|
||||
except ImportError:
|
||||
pass
|
||||
try:
|
||||
from skillopt.envs.sealqa.adapter import SealQAAdapter
|
||||
_ENV_REGISTRY["sealqa"] = SealQAAdapter
|
||||
except ImportError:
|
||||
pass
|
||||
try:
|
||||
from skillopt.envs.swebench.adapter import SWEBenchAdapter
|
||||
_ENV_REGISTRY["swebench"] = SWEBenchAdapter
|
||||
except ImportError:
|
||||
pass
|
||||
|
||||
|
||||
def get_adapter(cfg: dict):
|
||||
"""Instantiate the environment adapter specified in ``cfg["env"]``."""
|
||||
_register_builtins()
|
||||
env_name = cfg.get("env", "alfworld")
|
||||
if env_name not in _ENV_REGISTRY:
|
||||
raise ValueError(
|
||||
f"Unknown environment '{env_name}'. "
|
||||
f"Available: {list(_ENV_REGISTRY.keys())}"
|
||||
)
|
||||
adapter_cls = _ENV_REGISTRY[env_name]
|
||||
|
||||
# Inspect adapter __init__ signature and only pass accepted kwargs
|
||||
import inspect
|
||||
sig = inspect.signature(adapter_cls.__init__)
|
||||
accepted = set(sig.parameters.keys()) - {"self"}
|
||||
adapter_kwargs: dict = {}
|
||||
for key in accepted:
|
||||
if key in cfg:
|
||||
adapter_kwargs[key] = cfg[key]
|
||||
|
||||
return adapter_cls(**adapter_kwargs)
|
||||
|
||||
|
||||
# ── CLI ──────────────────────────────────────────────────────────────────────
|
||||
|
||||
_BOOL = lambda x: x.lower() in ("true", "1", "yes") # noqa: E731
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
p = argparse.ArgumentParser(
|
||||
description="SkillOpt: Executive Strategy for Self-Evolving Agent Skills",
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter,
|
||||
epilog=__doc__,
|
||||
)
|
||||
p.add_argument("--config", type=str, required=True,
|
||||
help="Path to YAML config file")
|
||||
p.add_argument("--cfg-options", nargs="+", default=[],
|
||||
help="Override config: section.key=value (e.g. train.batch_size=40)")
|
||||
|
||||
# Legacy flat CLI overrides (still work, prefer --cfg-options for new usage)
|
||||
p.add_argument("--env", type=str)
|
||||
p.add_argument("--backend", type=str,
|
||||
choices=["azure_openai", "codex", "codex_exec", "claude", "claude_chat", "claude_code_exec", "qwen", "qwen_chat"])
|
||||
p.add_argument("--optimizer_model", type=str)
|
||||
p.add_argument("--target_model", type=str)
|
||||
p.add_argument("--optimizer_backend", type=str)
|
||||
p.add_argument("--target_backend", type=str)
|
||||
p.add_argument("--reasoning_effort", type=str,
|
||||
choices=["", "low", "medium", "high", "xhigh", "max"])
|
||||
p.add_argument("--rewrite_reasoning_effort", type=str)
|
||||
p.add_argument("--rewrite_max_completion_tokens", type=int)
|
||||
p.add_argument("--azure_endpoint", type=str)
|
||||
p.add_argument("--azure_api_version", type=str)
|
||||
p.add_argument("--azure_api_key", type=str)
|
||||
p.add_argument("--azure_openai_endpoint", type=str)
|
||||
p.add_argument("--azure_openai_api_version", type=str)
|
||||
p.add_argument("--azure_openai_api_key", type=str)
|
||||
p.add_argument("--azure_openai_auth_mode", type=str)
|
||||
p.add_argument("--azure_openai_ad_scope", type=str)
|
||||
p.add_argument("--azure_openai_managed_identity_client_id", type=str)
|
||||
p.add_argument("--optimizer_azure_openai_endpoint", type=str)
|
||||
p.add_argument("--optimizer_azure_openai_api_version", type=str)
|
||||
p.add_argument("--optimizer_azure_openai_api_key", type=str)
|
||||
p.add_argument("--optimizer_azure_openai_auth_mode", type=str)
|
||||
p.add_argument("--optimizer_azure_openai_ad_scope", type=str)
|
||||
p.add_argument("--optimizer_azure_openai_managed_identity_client_id", type=str)
|
||||
p.add_argument("--target_azure_openai_endpoint", type=str)
|
||||
p.add_argument("--target_azure_openai_api_version", type=str)
|
||||
p.add_argument("--target_azure_openai_api_key", type=str)
|
||||
p.add_argument("--target_azure_openai_auth_mode", type=str)
|
||||
p.add_argument("--target_azure_openai_ad_scope", type=str)
|
||||
p.add_argument("--target_azure_openai_managed_identity_client_id", type=str)
|
||||
p.add_argument("--qwen_chat_base_url", type=str)
|
||||
p.add_argument("--qwen_chat_api_key", type=str)
|
||||
p.add_argument("--qwen_chat_temperature", type=float)
|
||||
p.add_argument("--qwen_chat_timeout_seconds", type=float)
|
||||
p.add_argument("--qwen_chat_max_tokens", type=int)
|
||||
p.add_argument("--qwen_chat_enable_thinking", type=_BOOL)
|
||||
p.add_argument("--codex_exec_path", type=str)
|
||||
p.add_argument("--codex_exec_sandbox", type=str)
|
||||
p.add_argument("--codex_exec_profile", type=str)
|
||||
p.add_argument("--codex_exec_full_auto", type=_BOOL)
|
||||
p.add_argument("--codex_exec_reasoning_effort", type=str)
|
||||
p.add_argument("--codex_exec_use_sdk", type=str)
|
||||
p.add_argument("--codex_exec_network_access", type=_BOOL)
|
||||
p.add_argument("--codex_exec_web_search", type=_BOOL)
|
||||
p.add_argument("--codex_exec_approval_policy", type=str)
|
||||
p.add_argument("--claude_code_exec_path", type=str)
|
||||
p.add_argument("--claude_code_exec_profile", type=str)
|
||||
p.add_argument("--claude_code_exec_use_sdk", type=str)
|
||||
p.add_argument("--claude_code_exec_effort", type=str)
|
||||
p.add_argument("--claude_code_exec_max_thinking_tokens", type=int)
|
||||
p.add_argument("--codex_trace_to_optimizer", type=_BOOL)
|
||||
p.add_argument("--skill_init", type=str)
|
||||
p.add_argument("--num_epochs", type=int)
|
||||
p.add_argument("--train_size", type=int)
|
||||
p.add_argument("--steps_per_epoch", type=int)
|
||||
p.add_argument("--batch_size", type=int)
|
||||
p.add_argument("--accumulation", type=int)
|
||||
p.add_argument("--seed", type=int)
|
||||
p.add_argument("--edit_budget", type=int)
|
||||
p.add_argument("--min_edit_budget", type=int)
|
||||
p.add_argument("--lr_scheduler", type=str,
|
||||
choices=["constant", "linear", "cosine", "autonomous"])
|
||||
p.add_argument("--lr_control_mode", type=str,
|
||||
choices=["fixed", "autonomous", "none"])
|
||||
p.add_argument("--merge_batch_size", type=int)
|
||||
p.add_argument("--max_analyst_rounds", type=int)
|
||||
p.add_argument("--sel_env_num", type=int)
|
||||
p.add_argument("--test_env_num", type=int)
|
||||
p.add_argument("--eval_test", type=_BOOL)
|
||||
p.add_argument("--use_gate", type=_BOOL)
|
||||
p.add_argument("--max_steps", type=int)
|
||||
p.add_argument("--max_api_workers", type=int)
|
||||
p.add_argument("--analyst_workers", type=int)
|
||||
p.add_argument("--failure_only", type=_BOOL)
|
||||
p.add_argument("--minibatch_size", type=int)
|
||||
p.add_argument("--skill_update_mode", type=str,
|
||||
choices=[
|
||||
"patch",
|
||||
"rewrite_from_suggestions",
|
||||
"rewrite",
|
||||
"suggestions",
|
||||
"full_rewrite",
|
||||
"full_rewrite_minibatch",
|
||||
"minibatch_full_rewrite",
|
||||
])
|
||||
p.add_argument("--use_slow_update", type=_BOOL)
|
||||
p.add_argument("--slow_update_samples", type=int)
|
||||
p.add_argument("--longitudinal_pair_policy", type=str,
|
||||
choices=["mixed", "changed", "unchanged"])
|
||||
p.add_argument("--use_meta_skill", type=_BOOL)
|
||||
p.add_argument("--data_path", type=str)
|
||||
p.add_argument("--split_mode", type=str,
|
||||
choices=["ratio", "split_dir"])
|
||||
p.add_argument("--split_ratio", type=str)
|
||||
p.add_argument("--split_seed", type=int)
|
||||
p.add_argument("--split_dir", type=str)
|
||||
p.add_argument("--split_output_dir", type=str)
|
||||
p.add_argument("--data_root", type=str)
|
||||
p.add_argument("--max_turns", type=int)
|
||||
p.add_argument("--workers", type=int)
|
||||
p.add_argument("--limit", type=int)
|
||||
p.add_argument("--shuffle_choices", type=_BOOL)
|
||||
p.add_argument("--use_theorem", type=_BOOL)
|
||||
p.add_argument("--use_sketch", type=_BOOL)
|
||||
p.add_argument("--image_detail", type=str)
|
||||
p.add_argument("--judge_model", type=str)
|
||||
p.add_argument("--judge_max_completion_tokens", type=int)
|
||||
p.add_argument("--judge_retries", type=int)
|
||||
p.add_argument("--out_root", type=str)
|
||||
p.add_argument("--mode", type=str)
|
||||
|
||||
return p.parse_args()
|
||||
|
||||
|
||||
# ── Flat key → structured path mapping (for legacy CLI → structured config) ──
|
||||
|
||||
_LEGACY_TO_STRUCTURED: dict[str, str] = {
|
||||
"backend": "model.backend",
|
||||
"optimizer_model": "model.optimizer",
|
||||
"target_model": "model.target",
|
||||
"optimizer_backend": "model.optimizer_backend",
|
||||
"target_backend": "model.target_backend",
|
||||
"reasoning_effort": "model.reasoning_effort",
|
||||
"rewrite_reasoning_effort": "model.rewrite_reasoning_effort",
|
||||
"rewrite_max_completion_tokens": "model.rewrite_max_completion_tokens",
|
||||
"azure_endpoint": "model.azure_endpoint",
|
||||
"azure_api_version": "model.azure_api_version",
|
||||
"azure_api_key": "model.azure_api_key",
|
||||
"azure_openai_endpoint": "model.azure_openai_endpoint",
|
||||
"azure_openai_api_version": "model.azure_openai_api_version",
|
||||
"azure_openai_api_key": "model.azure_openai_api_key",
|
||||
"azure_openai_auth_mode": "model.azure_openai_auth_mode",
|
||||
"azure_openai_ad_scope": "model.azure_openai_ad_scope",
|
||||
"azure_openai_managed_identity_client_id": "model.azure_openai_managed_identity_client_id",
|
||||
"optimizer_azure_openai_endpoint": "model.optimizer_azure_openai_endpoint",
|
||||
"optimizer_azure_openai_api_version": "model.optimizer_azure_openai_api_version",
|
||||
"optimizer_azure_openai_api_key": "model.optimizer_azure_openai_api_key",
|
||||
"optimizer_azure_openai_auth_mode": "model.optimizer_azure_openai_auth_mode",
|
||||
"optimizer_azure_openai_ad_scope": "model.optimizer_azure_openai_ad_scope",
|
||||
"optimizer_azure_openai_managed_identity_client_id": "model.optimizer_azure_openai_managed_identity_client_id",
|
||||
"target_azure_openai_endpoint": "model.target_azure_openai_endpoint",
|
||||
"target_azure_openai_api_version": "model.target_azure_openai_api_version",
|
||||
"target_azure_openai_api_key": "model.target_azure_openai_api_key",
|
||||
"target_azure_openai_auth_mode": "model.target_azure_openai_auth_mode",
|
||||
"target_azure_openai_ad_scope": "model.target_azure_openai_ad_scope",
|
||||
"target_azure_openai_managed_identity_client_id": "model.target_azure_openai_managed_identity_client_id",
|
||||
"qwen_chat_base_url": "model.qwen_chat_base_url",
|
||||
"qwen_chat_api_key": "model.qwen_chat_api_key",
|
||||
"qwen_chat_temperature": "model.qwen_chat_temperature",
|
||||
"qwen_chat_timeout_seconds": "model.qwen_chat_timeout_seconds",
|
||||
"qwen_chat_max_tokens": "model.qwen_chat_max_tokens",
|
||||
"qwen_chat_enable_thinking": "model.qwen_chat_enable_thinking",
|
||||
"codex_exec_path": "model.codex_exec_path",
|
||||
"codex_exec_sandbox": "model.codex_exec_sandbox",
|
||||
"codex_exec_profile": "model.codex_exec_profile",
|
||||
"codex_exec_full_auto": "model.codex_exec_full_auto",
|
||||
"codex_exec_reasoning_effort": "model.codex_exec_reasoning_effort",
|
||||
"codex_exec_use_sdk": "model.codex_exec_use_sdk",
|
||||
"codex_exec_network_access": "model.codex_exec_network_access",
|
||||
"codex_exec_web_search": "model.codex_exec_web_search",
|
||||
"codex_exec_approval_policy": "model.codex_exec_approval_policy",
|
||||
"claude_code_exec_path": "model.claude_code_exec_path",
|
||||
"claude_code_exec_profile": "model.claude_code_exec_profile",
|
||||
"claude_code_exec_use_sdk": "model.claude_code_exec_use_sdk",
|
||||
"claude_code_exec_effort": "model.claude_code_exec_effort",
|
||||
"claude_code_exec_max_thinking_tokens": "model.claude_code_exec_max_thinking_tokens",
|
||||
"codex_trace_to_optimizer": "model.codex_trace_to_optimizer",
|
||||
"num_epochs": "train.num_epochs",
|
||||
"train_size": "train.train_size",
|
||||
"steps_per_epoch": "train.steps_per_epoch",
|
||||
"batch_size": "train.batch_size",
|
||||
"accumulation": "train.accumulation",
|
||||
"seed": "train.seed",
|
||||
"minibatch_size": "gradient.minibatch_size",
|
||||
"merge_batch_size": "gradient.merge_batch_size",
|
||||
"analyst_workers": "gradient.analyst_workers",
|
||||
"max_analyst_rounds": "gradient.max_analyst_rounds",
|
||||
"failure_only": "gradient.failure_only",
|
||||
"edit_budget": "optimizer.learning_rate",
|
||||
"min_edit_budget": "optimizer.min_learning_rate",
|
||||
"lr_scheduler": "optimizer.lr_scheduler",
|
||||
"lr_control_mode": "optimizer.lr_control_mode",
|
||||
"skill_update_mode": "optimizer.skill_update_mode",
|
||||
"use_slow_update": "optimizer.use_slow_update",
|
||||
"slow_update_samples": "optimizer.slow_update_samples",
|
||||
"longitudinal_pair_policy": "optimizer.longitudinal_pair_policy",
|
||||
"use_meta_skill": "optimizer.use_meta_skill",
|
||||
"use_gate": "evaluation.use_gate",
|
||||
"sel_env_num": "evaluation.sel_env_num",
|
||||
"test_env_num": "evaluation.test_env_num",
|
||||
"eval_test": "evaluation.eval_test",
|
||||
"env": "env.name",
|
||||
"skill_init": "env.skill_init",
|
||||
"out_root": "env.out_root",
|
||||
}
|
||||
|
||||
|
||||
def load_config(args: argparse.Namespace) -> dict:
|
||||
"""Load config with _base_ inheritance, then apply CLI overrides."""
|
||||
from skillopt.config import load_config as _load, flatten_config, is_structured
|
||||
|
||||
cfg = _load(args.config, overrides=args.cfg_options)
|
||||
structured = is_structured(cfg)
|
||||
|
||||
# Apply legacy --key value overrides
|
||||
cli = {k: v for k, v in vars(args).items()
|
||||
if v is not None and k not in ("config", "cfg_options")}
|
||||
if cli:
|
||||
if structured:
|
||||
from skillopt.config import apply_overrides
|
||||
mapped = []
|
||||
for k, v in cli.items():
|
||||
dotted = _LEGACY_TO_STRUCTURED.get(k)
|
||||
if dotted:
|
||||
mapped.append(f"{dotted}={v}")
|
||||
else:
|
||||
mapped.append(f"env.{k}={v}")
|
||||
apply_overrides(cfg, mapped)
|
||||
else:
|
||||
cfg.update(cli)
|
||||
|
||||
# Flatten structured config → flat dict for trainer/adapter
|
||||
flat = flatten_config(cfg) if structured else cfg
|
||||
|
||||
for new_key, old_key in (
|
||||
("azure_openai_endpoint", "azure_endpoint"),
|
||||
("azure_openai_api_version", "azure_api_version"),
|
||||
("azure_openai_api_key", "azure_api_key"),
|
||||
):
|
||||
if flat.get(new_key) in (None, "") and flat.get(old_key) not in (None, ""):
|
||||
flat[new_key] = flat[old_key]
|
||||
|
||||
explicit_backend = getattr(args, "backend", None)
|
||||
if explicit_backend is None:
|
||||
for option in args.cfg_options or []:
|
||||
key = str(option).split("=", 1)[0].strip()
|
||||
if key == "model.backend":
|
||||
explicit_backend = str(option).split("=", 1)[1].strip()
|
||||
break
|
||||
|
||||
backend = normalize_backend_name(flat.get("model_backend") or flat.get("target_backend") or "azure_openai")
|
||||
|
||||
def _has_model_override(dotted_key: str, legacy_key: str) -> bool:
|
||||
if getattr(args, legacy_key, None) is not None:
|
||||
return True
|
||||
for option in args.cfg_options or []:
|
||||
key = str(option).split("=", 1)[0].strip()
|
||||
if key == dotted_key:
|
||||
return True
|
||||
return False
|
||||
|
||||
if explicit_backend is not None:
|
||||
backend = normalize_backend_name(explicit_backend)
|
||||
flat["model_backend"] = backend
|
||||
if backend in {"claude", "claude_chat"}:
|
||||
flat.setdefault("optimizer_backend", "claude_chat")
|
||||
flat.setdefault("target_backend", "claude_chat")
|
||||
elif backend in {"codex", "codex_exec"}:
|
||||
flat.setdefault("optimizer_backend", "openai_chat")
|
||||
flat.setdefault("target_backend", "codex_exec")
|
||||
elif backend == "claude_code_exec":
|
||||
flat.setdefault("optimizer_backend", "openai_chat")
|
||||
flat.setdefault("target_backend", "claude_code_exec")
|
||||
elif backend in {"qwen", "qwen_chat"}:
|
||||
flat.setdefault("optimizer_backend", "openai_chat")
|
||||
flat.setdefault("target_backend", "qwen_chat")
|
||||
else:
|
||||
flat.setdefault("optimizer_backend", "openai_chat")
|
||||
flat.setdefault("target_backend", "openai_chat")
|
||||
else:
|
||||
flat.setdefault("optimizer_backend", "openai_chat")
|
||||
flat.setdefault("target_backend", "openai_chat")
|
||||
|
||||
if flat.get("optimizer_backend") == "claude_chat":
|
||||
if (
|
||||
str(flat.get("optimizer_model", "") or "").strip() in _OPENAI_DEFAULT_MODEL_SENTINELS
|
||||
and not _has_model_override("model.optimizer", "optimizer_model")
|
||||
):
|
||||
flat["optimizer_model"] = default_model_for_backend("claude_chat")
|
||||
if flat.get("target_backend") == "claude_chat":
|
||||
if (
|
||||
str(flat.get("target_model", "") or "").strip() in _OPENAI_DEFAULT_MODEL_SENTINELS
|
||||
and not _has_model_override("model.target", "target_model")
|
||||
):
|
||||
flat["target_model"] = default_model_for_backend("claude_chat")
|
||||
if flat.get("target_backend") == "claude_code_exec":
|
||||
if (
|
||||
str(flat.get("target_model", "") or "").strip() in _OPENAI_DEFAULT_MODEL_SENTINELS
|
||||
and not _has_model_override("model.target", "target_model")
|
||||
):
|
||||
flat["target_model"] = default_model_for_backend("claude_chat")
|
||||
if flat.get("target_backend") == "qwen_chat":
|
||||
if (
|
||||
str(flat.get("target_model", "") or "").strip() in _OPENAI_DEFAULT_MODEL_SENTINELS
|
||||
and not _has_model_override("model.target", "target_model")
|
||||
):
|
||||
flat["target_model"] = default_model_for_backend("qwen_chat")
|
||||
|
||||
# Auto-generate output root
|
||||
if not flat.get("out_root"):
|
||||
env = flat.get("env", "unknown")
|
||||
model = flat.get("optimizer_model", "unknown").replace("/", "-")
|
||||
ts = datetime.datetime.now().strftime("%Y%m%d_%H%M%S")
|
||||
flat["out_root"] = os.path.join("outputs", f"skillopt_{env}_{model}_{ts}")
|
||||
|
||||
flat["out_root"] = os.path.abspath(flat["out_root"])
|
||||
return flat
|
||||
|
||||
|
||||
# ── Main ─────────────────────────────────────────────────────────────────────
|
||||
|
||||
def main() -> None:
|
||||
args = parse_args()
|
||||
cfg = load_config(args)
|
||||
|
||||
print(f"\n{'='*60}")
|
||||
print(f" SkillOpt — Executive Strategy for Self-Evolving Agent Skills")
|
||||
print(f"{'='*60}")
|
||||
print(f" env: {cfg.get('env')}")
|
||||
print(f" optimizer_model: {cfg.get('optimizer_model')}")
|
||||
print(f" target_model: {cfg.get('target_model')}")
|
||||
print(f" optimizer_backend:{cfg.get('optimizer_backend', 'openai_chat')}")
|
||||
print(f" target_backend:{cfg.get('target_backend', 'openai_chat')}")
|
||||
print(f" reasoning: {cfg.get('reasoning_effort') or 'off'}")
|
||||
print(f" rewrite_effort: {cfg.get('rewrite_reasoning_effort') or 'off'}")
|
||||
print(f" epochs: {cfg.get('num_epochs')}")
|
||||
print(f" train_size: {cfg.get('train_size') or 'from dataset'}")
|
||||
print(f" steps/epoch: auto")
|
||||
print(f" batch_size: {cfg.get('batch_size')}")
|
||||
print(f" edit_budget: {cfg.get('edit_budget')}")
|
||||
print(f" lr_scheduler: {cfg.get('lr_scheduler', 'constant')}")
|
||||
print(f" update_mode: {cfg.get('skill_update_mode', 'patch')}")
|
||||
print(f" min_edit_budget:{cfg.get('min_edit_budget', 2)}")
|
||||
print(f" minibatch_size: {cfg.get('minibatch_size')}")
|
||||
print(f" seed: {cfg.get('seed')}")
|
||||
print(f" meta_skill: {cfg.get('use_meta_skill', False)}")
|
||||
print(f" slow_update: {cfg.get('use_slow_update', False)}")
|
||||
print(f" out_root: {cfg.get('out_root')}")
|
||||
print(f"{'='*60}\n")
|
||||
|
||||
# Build adapter
|
||||
adapter = get_adapter(cfg)
|
||||
|
||||
# Build trainer and run
|
||||
from skillopt.engine.trainer import ReflACTTrainer
|
||||
trainer = ReflACTTrainer(cfg, adapter)
|
||||
summary = trainer.train()
|
||||
|
||||
print(f"\n Output saved to: {cfg['out_root']}")
|
||||
if summary.get("test_hard") is not None:
|
||||
print(f" Final test: {summary['test_hard']:.4f}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1 +0,0 @@
|
||||
<svg id="logomark" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 74.492 100.25"><g id="tiny_-_black" data-name="tiny - black"><path d="M586.72,255.616a3.377,3.377,0,0,1,.448.031,5.917,5.917,0,0,1,3.581,2.79c.454,1.116.314,2.023-1.315,4.141L563.168,293.6l-8.558-10.047,29.348-26.616a4.406,4.406,0,0,1,2.762-1.321m0-1.5a5.766,5.766,0,0,0-3.69,1.643l-.041.032-.038.035L553.6,282.442l-1.077.977.943,1.107,8.558,10.047,1.145,1.344,1.141-1.348,26.267-31.022.022-.027.022-.028c1.574-2.046,2.327-3.622,1.516-5.619a7.309,7.309,0,0,0-4.779-3.714,5.083,5.083,0,0,0-.64-.043Z" transform="translate(-526.086 -245.559)"/><path d="M553.423,284.593l8.977,10.558L597.911,337.9c.873,1.093,1.419,2.186,1.047,3.418a4.092,4.092,0,0,1-2.721,2.837,3.557,3.557,0,0,1-1.045.159,4,4,0,0,1-2.687-1.124L548.01,300.808c-3.5-3.5-2.971-8.151.436-11.558l4.977-4.657m.124-2.17L552.4,283.5l-4.976,4.656c-4.192,4.191-4.372,9.816-.473,13.714l44.521,42.4a5.485,5.485,0,0,0,3.722,1.538,5.1,5.1,0,0,0,1.483-.224,5.59,5.59,0,0,0,3.719-3.838,5.176,5.176,0,0,0-1.31-4.788l-35.53-42.767-8.988-10.571-1.019-1.2Z" transform="translate(-526.086 -245.559)"/><path d="M562.4,295.151l9.556,11.5,5.761-5.356a7.926,7.926,0,0,0,.041-11.743l-43.7-41.923s-1.671-2.029-3.437-2.071a4.49,4.49,0,0,0-4.23,2.718c-.688,1.651-.194,2.809,1.315,4.97l29.306,35.565Z" transform="translate(-526.086 -245.559)"/><path d="M553.7,306.223l-17.116,21.024c-1.255,1.337-2.032,3.683-1.331,5.367a4.587,4.587,0,0,0,4.287,2.841,4.087,4.087,0,0,0,3.082-1.523l20.328-18.9Z" transform="translate(-526.086 -245.559)"/><path d="M592.074,250.547" transform="translate(-526.086 -245.559)" fill="#fff" stroke="#000" stroke-miterlimit="10" stroke-width="0.25"/></g></svg>
|
||||
|
Before Width: | Height: | Size: 1.6 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 78 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 11 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 390 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 10 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 407 KiB |
-2739
File diff suppressed because it is too large
Load Diff
@@ -1,28 +0,0 @@
|
||||
"""ReflACT: Reflective Agent Tuning.
|
||||
|
||||
A general-purpose framework for iteratively optimizing LLM agent skills
|
||||
through structured reflection and self-improvement.
|
||||
|
||||
Pipeline stages:
|
||||
1. Rollout — execute episodes with current skill
|
||||
2. Reflect — analyze trajectories, generate patches
|
||||
3. Aggregate — hierarchical merge of patches
|
||||
4. Select — rank and select top edits
|
||||
5. Update — apply edits to skill document
|
||||
6. Evaluate — validate candidate skill, accept/reject
|
||||
"""
|
||||
|
||||
__version__ = "0.1.0"
|
||||
|
||||
from skillopt.types import ( # noqa: F401
|
||||
BatchSpec,
|
||||
Edit,
|
||||
EditOp,
|
||||
FailureSummaryEntry,
|
||||
GateAction,
|
||||
GateResult,
|
||||
Patch,
|
||||
RawPatch,
|
||||
RolloutResult,
|
||||
SlowUpdateResult,
|
||||
)
|
||||
@@ -1,268 +0,0 @@
|
||||
"""ReflACT config loading engine — structured YAML with inheritance.
|
||||
|
||||
Supports two config formats:
|
||||
1. **Structured** (new): sections like ``model``, ``train``, ``gradient``,
|
||||
``optimizer``, ``evaluation``, ``env`` — with ``_base_`` inheritance.
|
||||
2. **Flat** (legacy): all keys at top level — fully backward compatible.
|
||||
|
||||
Usage::
|
||||
|
||||
from skillopt.config import load_config, flatten_config
|
||||
|
||||
cfg = load_config("configs/searchqa_default.yaml")
|
||||
flat = flatten_config(cfg) # always returns flat dict for trainer
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import copy
|
||||
import os
|
||||
from typing import Any
|
||||
|
||||
import yaml
|
||||
|
||||
# ── Section names that indicate a structured config ──────────────────────
|
||||
|
||||
_STRUCTURED_SECTIONS = frozenset({
|
||||
"model", "train", "gradient", "optimizer", "evaluation", "env",
|
||||
})
|
||||
|
||||
# ── Structured → flat key mapping ────────────────────────────────────────
|
||||
|
||||
_FLATTEN_MAP: dict[str, str] = {
|
||||
"model.backend": "model_backend",
|
||||
"model.optimizer": "optimizer_model",
|
||||
"model.target": "target_model",
|
||||
"model.optimizer_backend": "optimizer_backend",
|
||||
"model.target_backend": "target_backend",
|
||||
"model.reasoning_effort": "reasoning_effort",
|
||||
"model.rewrite_reasoning_effort": "rewrite_reasoning_effort",
|
||||
"model.rewrite_max_completion_tokens": "rewrite_max_completion_tokens",
|
||||
"model.codex_exec_path": "codex_exec_path",
|
||||
"model.codex_exec_sandbox": "codex_exec_sandbox",
|
||||
"model.codex_exec_profile": "codex_exec_profile",
|
||||
"model.codex_exec_full_auto": "codex_exec_full_auto",
|
||||
"model.codex_exec_reasoning_effort": "codex_exec_reasoning_effort",
|
||||
"model.codex_exec_use_sdk": "codex_exec_use_sdk",
|
||||
"model.codex_exec_network_access": "codex_exec_network_access",
|
||||
"model.codex_exec_web_search": "codex_exec_web_search",
|
||||
"model.codex_exec_approval_policy": "codex_exec_approval_policy",
|
||||
"model.claude_code_exec_path": "claude_code_exec_path",
|
||||
"model.claude_code_exec_profile": "claude_code_exec_profile",
|
||||
"model.claude_code_exec_use_sdk": "claude_code_exec_use_sdk",
|
||||
"model.claude_code_exec_effort": "claude_code_exec_effort",
|
||||
"model.claude_code_exec_max_thinking_tokens": "claude_code_exec_max_thinking_tokens",
|
||||
"model.codex_trace_to_optimizer": "codex_trace_to_optimizer",
|
||||
"model.azure_endpoint": "azure_endpoint",
|
||||
"model.azure_api_version": "azure_api_version",
|
||||
"model.azure_api_key": "azure_api_key",
|
||||
"model.azure_openai_endpoint": "azure_openai_endpoint",
|
||||
"model.azure_openai_api_version": "azure_openai_api_version",
|
||||
"model.azure_openai_api_key": "azure_openai_api_key",
|
||||
"model.azure_openai_auth_mode": "azure_openai_auth_mode",
|
||||
"model.azure_openai_ad_scope": "azure_openai_ad_scope",
|
||||
"model.azure_openai_managed_identity_client_id": "azure_openai_managed_identity_client_id",
|
||||
"model.optimizer_azure_openai_endpoint": "optimizer_azure_openai_endpoint",
|
||||
"model.optimizer_azure_openai_api_version": "optimizer_azure_openai_api_version",
|
||||
"model.optimizer_azure_openai_api_key": "optimizer_azure_openai_api_key",
|
||||
"model.optimizer_azure_openai_auth_mode": "optimizer_azure_openai_auth_mode",
|
||||
"model.optimizer_azure_openai_ad_scope": "optimizer_azure_openai_ad_scope",
|
||||
"model.optimizer_azure_openai_managed_identity_client_id": "optimizer_azure_openai_managed_identity_client_id",
|
||||
"model.target_azure_openai_endpoint": "target_azure_openai_endpoint",
|
||||
"model.target_azure_openai_api_version": "target_azure_openai_api_version",
|
||||
"model.target_azure_openai_api_key": "target_azure_openai_api_key",
|
||||
"model.target_azure_openai_auth_mode": "target_azure_openai_auth_mode",
|
||||
"model.target_azure_openai_ad_scope": "target_azure_openai_ad_scope",
|
||||
"model.target_azure_openai_managed_identity_client_id": "target_azure_openai_managed_identity_client_id",
|
||||
"model.qwen_chat_base_url": "qwen_chat_base_url",
|
||||
"model.qwen_chat_api_key": "qwen_chat_api_key",
|
||||
"model.qwen_chat_temperature": "qwen_chat_temperature",
|
||||
"model.qwen_chat_timeout_seconds": "qwen_chat_timeout_seconds",
|
||||
"model.qwen_chat_max_tokens": "qwen_chat_max_tokens",
|
||||
"model.qwen_chat_enable_thinking": "qwen_chat_enable_thinking",
|
||||
"train.num_epochs": "num_epochs",
|
||||
"train.train_size": "train_size",
|
||||
"train.steps_per_epoch": "steps_per_epoch",
|
||||
"train.batch_size": "batch_size",
|
||||
"train.accumulation": "accumulation",
|
||||
"train.seed": "seed",
|
||||
"gradient.minibatch_size": "minibatch_size",
|
||||
"gradient.merge_batch_size": "merge_batch_size",
|
||||
"gradient.analyst_workers": "analyst_workers",
|
||||
"gradient.failure_only": "failure_only",
|
||||
"gradient.max_analyst_rounds": "max_analyst_rounds",
|
||||
"optimizer.learning_rate": "edit_budget",
|
||||
"optimizer.min_learning_rate": "min_edit_budget",
|
||||
"optimizer.lr_scheduler": "lr_scheduler",
|
||||
"optimizer.lr_control_mode": "lr_control_mode",
|
||||
"optimizer.skill_update_mode": "skill_update_mode",
|
||||
"optimizer.meta_learning_rate": "meta_edit_budget",
|
||||
"optimizer.use_slow_update": "use_slow_update",
|
||||
"optimizer.slow_update_samples": "slow_update_samples",
|
||||
"optimizer.slow_update_gate_with_selection": "slow_update_gate_with_selection",
|
||||
"optimizer.longitudinal_pair_policy": "longitudinal_pair_policy",
|
||||
"optimizer.use_meta_skill": "use_meta_skill",
|
||||
"evaluation.use_gate": "use_gate",
|
||||
"evaluation.gate_metric": "gate_metric",
|
||||
"evaluation.gate_mixed_weight": "gate_mixed_weight",
|
||||
"evaluation.sel_env_num": "sel_env_num",
|
||||
"evaluation.test_env_num": "test_env_num",
|
||||
"evaluation.eval_test": "eval_test",
|
||||
"env.name": "env",
|
||||
"env.skill_init": "skill_init",
|
||||
"env.out_root": "out_root",
|
||||
}
|
||||
|
||||
|
||||
# ── Deep merge ───────────────────────────────────────────────────────────
|
||||
|
||||
def _deep_merge(base: dict, override: dict) -> dict:
|
||||
"""Recursively merge *override* into *base* (returns new dict)."""
|
||||
result = copy.deepcopy(base)
|
||||
for key, val in override.items():
|
||||
if key in result and isinstance(result[key], dict) and isinstance(val, dict):
|
||||
result[key] = _deep_merge(result[key], val)
|
||||
else:
|
||||
result[key] = copy.deepcopy(val)
|
||||
return result
|
||||
|
||||
|
||||
# ── YAML loading with _base_ inheritance ─────────────────────────────────
|
||||
|
||||
def _load_yaml(path: str, _visited: set[str] | None = None) -> dict:
|
||||
"""Load a YAML file, resolving ``_base_`` inheritance recursively."""
|
||||
abs_path = os.path.abspath(path)
|
||||
if _visited is None:
|
||||
_visited = set()
|
||||
if abs_path in _visited:
|
||||
raise ValueError(f"Circular _base_ inheritance: {abs_path}")
|
||||
_visited.add(abs_path)
|
||||
|
||||
with open(abs_path) as f:
|
||||
cfg = yaml.safe_load(f) or {}
|
||||
|
||||
base_ref = cfg.pop("_base_", None)
|
||||
if base_ref:
|
||||
base_path = os.path.join(os.path.dirname(abs_path), base_ref)
|
||||
base_cfg = _load_yaml(base_path, _visited)
|
||||
cfg = _deep_merge(base_cfg, cfg)
|
||||
|
||||
return cfg
|
||||
|
||||
|
||||
# ── Format detection ─────────────────────────────────────────────────────
|
||||
|
||||
def is_structured(cfg: dict) -> bool:
|
||||
"""Return True if *cfg* uses the new structured section format."""
|
||||
return any(
|
||||
key in _STRUCTURED_SECTIONS and isinstance(cfg.get(key), dict)
|
||||
for key in cfg
|
||||
)
|
||||
|
||||
|
||||
# ── Flatten ──────────────────────────────────────────────────────────────
|
||||
|
||||
def flatten_config(cfg: dict) -> dict:
|
||||
"""Convert a structured config to the flat dict expected by the trainer.
|
||||
|
||||
If *cfg* is already flat, returns a shallow copy unchanged.
|
||||
"""
|
||||
if not is_structured(cfg):
|
||||
return dict(cfg)
|
||||
|
||||
flat: dict[str, Any] = {}
|
||||
|
||||
evaluation_section = cfg.get("evaluation", {})
|
||||
if isinstance(evaluation_section, dict) and evaluation_section.get("use_gate") is False:
|
||||
raise ValueError(
|
||||
"Gate validation is mandatory in this branch. Remove "
|
||||
"`evaluation.use_gate: false` from the config."
|
||||
)
|
||||
|
||||
# Apply the explicit mapping
|
||||
for dotted, flat_key in _FLATTEN_MAP.items():
|
||||
section, key = dotted.split(".", 1)
|
||||
section_dict = cfg.get(section, {})
|
||||
if isinstance(section_dict, dict) and key in section_dict:
|
||||
flat[flat_key] = section_dict[key]
|
||||
|
||||
# Pass through env-specific keys not in the explicit mapping
|
||||
env_section = cfg.get("env", {})
|
||||
if isinstance(env_section, dict):
|
||||
mapped_env_keys = {
|
||||
k.split(".", 1)[1]
|
||||
for k in _FLATTEN_MAP
|
||||
if k.startswith("env.")
|
||||
}
|
||||
for key, val in env_section.items():
|
||||
if key not in mapped_env_keys:
|
||||
flat[key] = val
|
||||
|
||||
return flat
|
||||
|
||||
|
||||
# ── Override application ─────────────────────────────────────────────────
|
||||
|
||||
def _cast_value(val_str: str) -> Any:
|
||||
"""Auto-cast a CLI string value to int / float / bool / str."""
|
||||
if val_str.lower() in ("true", "yes"):
|
||||
return True
|
||||
if val_str.lower() in ("false", "no"):
|
||||
return False
|
||||
try:
|
||||
return int(val_str)
|
||||
except ValueError:
|
||||
pass
|
||||
try:
|
||||
return float(val_str)
|
||||
except ValueError:
|
||||
pass
|
||||
return val_str
|
||||
|
||||
|
||||
def apply_overrides(cfg: dict, overrides: list[str]) -> None:
|
||||
"""Apply ``key=value`` overrides to a structured config (in place).
|
||||
|
||||
Supports both ``section.key=value`` (for structured configs) and
|
||||
``key=value`` (for flat configs or flat keys in env section).
|
||||
"""
|
||||
for item in overrides:
|
||||
if "=" not in item:
|
||||
raise ValueError(f"Invalid override (expected key=value): {item!r}")
|
||||
key, val_str = item.split("=", 1)
|
||||
val = _cast_value(val_str)
|
||||
|
||||
if "." in key:
|
||||
section, subkey = key.split(".", 1)
|
||||
if section in cfg and isinstance(cfg[section], dict):
|
||||
cfg[section][subkey] = val
|
||||
else:
|
||||
cfg.setdefault(section, {})[subkey] = val
|
||||
else:
|
||||
# Flat key — apply to top level (for legacy compat)
|
||||
cfg[key] = val
|
||||
|
||||
|
||||
# ── Public API ───────────────────────────────────────────────────────────
|
||||
|
||||
def load_config(
|
||||
path: str,
|
||||
overrides: list[str] | None = None,
|
||||
) -> dict:
|
||||
"""Load a config file with ``_base_`` inheritance and optional overrides.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
path : str
|
||||
Path to the YAML config file.
|
||||
overrides : list[str] | None
|
||||
``key=value`` strings from ``--cfg-options``.
|
||||
|
||||
Returns
|
||||
-------
|
||||
dict
|
||||
The merged config (structured or flat depending on the YAML).
|
||||
"""
|
||||
cfg = _load_yaml(path)
|
||||
if overrides:
|
||||
apply_overrides(cfg, overrides)
|
||||
return cfg
|
||||
@@ -1,7 +0,0 @@
|
||||
"""ReflACT Datasets -- task batch planning and data loading.
|
||||
|
||||
Analogous to the datasets and dataloaders in neural network training:
|
||||
provides batch sampling, epoch planning, and data management for the
|
||||
ReflACT training pipeline.
|
||||
"""
|
||||
from skillopt.datasets.base import BaseDataLoader, BatchSpec, SplitDataLoader # noqa: F401
|
||||
@@ -1,512 +0,0 @@
|
||||
"""Generic task dataloader abstractions for ReflACT.
|
||||
|
||||
ReflACT does not train model parameters directly. Instead, it iterates over
|
||||
task batches, rolls out the current skill, reflects on failures/successes,
|
||||
and updates the skill document. Because of that, the "dataloader" abstraction
|
||||
here is closer to a batch sampler / episode planner than a tensor loader.
|
||||
|
||||
Class hierarchy::
|
||||
|
||||
BaseDataLoader # abstract — simulator-backed envs (e.g. ALFWorld)
|
||||
└── SplitDataLoader # abstract — dataset-backed envs with split_dir
|
||||
|
||||
SplitDataLoader supports two dataset entry modes:
|
||||
|
||||
1. ``split_mode="split_dir"``: consume an existing split directory.
|
||||
2. ``split_mode="ratio"``: build a deterministic split directory from a raw
|
||||
dataset path using an explicit train:val:test ratio.
|
||||
|
||||
In either case, the standardised split layout is:
|
||||
|
||||
split_dir/
|
||||
├── train/ # training items
|
||||
├── val/ # validation / selection items (gate)
|
||||
└── test/ # held-out test items
|
||||
|
||||
Each subdirectory's contents are benchmark-specific. Subclasses only need
|
||||
to implement ``load_split_items(split_path)`` to teach the loader how to
|
||||
read items from one of those directories.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import glob
|
||||
import json
|
||||
import os
|
||||
import random
|
||||
from abc import ABC, abstractmethod
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class BatchSpec:
|
||||
"""A concrete batch request consumed by the training loop.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
phase : str
|
||||
``"train"`` or ``"eval"``.
|
||||
split : str
|
||||
Dataset split name, typically ``"train"`` or an eval split.
|
||||
seed : int
|
||||
Random seed used to construct the batch deterministically.
|
||||
batch_size : int
|
||||
Requested number of items / episodes in this batch.
|
||||
payload : object | None
|
||||
Environment-specific batch payload. For dataset-backed environments
|
||||
this is often a list of sampled items; for simulator-backed
|
||||
environments this may be ``None`` and the seed alone can define the
|
||||
batch.
|
||||
metadata : dict[str, Any]
|
||||
Optional structured metadata for logging, resume, or curriculum logic.
|
||||
"""
|
||||
|
||||
phase: str
|
||||
split: str
|
||||
seed: int
|
||||
batch_size: int
|
||||
payload: object | None = None
|
||||
metadata: dict[str, Any] = field(default_factory=dict)
|
||||
|
||||
|
||||
class BaseDataLoader(ABC):
|
||||
"""Abstract base class for task batch planning in ReflACT.
|
||||
|
||||
Subclasses are responsible for defining how a train or eval batch is
|
||||
sampled. The default implementation here provides deterministic epoch seed
|
||||
planning so all loaders share the same reproducibility behavior.
|
||||
"""
|
||||
|
||||
def setup(self, cfg: dict) -> None:
|
||||
"""Optional one-time initialization with the full trainer config."""
|
||||
|
||||
def set_out_root(self, out_root: str) -> None:
|
||||
"""Optional hook for loaders that persist split files or state."""
|
||||
|
||||
def state_dict(self) -> dict[str, Any]:
|
||||
"""Return serializable loader state for resume support."""
|
||||
return {}
|
||||
|
||||
def load_state_dict(self, state: dict[str, Any]) -> None:
|
||||
"""Restore loader state from :meth:`state_dict` output."""
|
||||
|
||||
def get_train_size(self) -> int | None:
|
||||
"""Return the size of the training pool when known."""
|
||||
return None
|
||||
|
||||
@staticmethod
|
||||
def make_base_seeds(steps_per_epoch: int, accumulation: int, seed: int) -> list[int]:
|
||||
"""Return the deterministic seed pool used to define train batches."""
|
||||
batches_per_epoch = steps_per_epoch * accumulation
|
||||
return [seed + i + 1 for i in range(batches_per_epoch)]
|
||||
|
||||
@staticmethod
|
||||
def shuffle_epoch_seeds(base_seeds: list[int], epoch: int, seed: int) -> list[int]:
|
||||
"""Return the per-epoch deterministic shuffle of *base_seeds*."""
|
||||
epoch_rng = random.Random(seed + epoch * 1000)
|
||||
shuffled = list(base_seeds)
|
||||
epoch_rng.shuffle(shuffled)
|
||||
return shuffled
|
||||
|
||||
def plan_train_epoch(
|
||||
self,
|
||||
*,
|
||||
epoch: int,
|
||||
steps_per_epoch: int,
|
||||
accumulation: int,
|
||||
batch_size: int,
|
||||
seed: int,
|
||||
**kwargs,
|
||||
) -> list[BatchSpec]:
|
||||
"""Build the full list of training batches for one epoch."""
|
||||
base_seeds = self.make_base_seeds(
|
||||
steps_per_epoch=steps_per_epoch,
|
||||
accumulation=accumulation,
|
||||
seed=seed,
|
||||
)
|
||||
shuffled_seeds = self.shuffle_epoch_seeds(base_seeds, epoch=epoch, seed=seed)
|
||||
return [
|
||||
self.build_train_batch(batch_size=batch_size, seed=batch_seed, **kwargs)
|
||||
for batch_seed in shuffled_seeds
|
||||
]
|
||||
|
||||
@abstractmethod
|
||||
def build_train_batch(self, batch_size: int, seed: int, **kwargs) -> BatchSpec:
|
||||
"""Construct one training batch specification."""
|
||||
|
||||
@abstractmethod
|
||||
def build_eval_batch(
|
||||
self,
|
||||
env_num: int,
|
||||
split: str,
|
||||
seed: int,
|
||||
**kwargs,
|
||||
) -> BatchSpec:
|
||||
"""Construct one evaluation batch specification."""
|
||||
|
||||
|
||||
# ── Split-based dataloader for dataset-backed environments ──────────────
|
||||
|
||||
# Canonical split names expected under split_dir/
|
||||
SPLIT_NAMES = ("train", "val", "test")
|
||||
|
||||
# Maps legacy / trainer split names → canonical directory names
|
||||
_SPLIT_ALIAS: dict[str, str] = {
|
||||
"train": "train",
|
||||
"valid_seen": "val",
|
||||
"selection": "val",
|
||||
"val": "val",
|
||||
"valid_unseen": "test",
|
||||
"test": "test",
|
||||
}
|
||||
|
||||
|
||||
def _load_json_or_jsonl(path: str) -> list[dict]:
|
||||
"""Load a list of items from a JSON or JSONL file."""
|
||||
with open(path, encoding="utf-8") as f:
|
||||
content = f.read().strip()
|
||||
if not content:
|
||||
return []
|
||||
|
||||
try:
|
||||
data = json.loads(content)
|
||||
except json.JSONDecodeError:
|
||||
data = None
|
||||
|
||||
if isinstance(data, list):
|
||||
return data
|
||||
if isinstance(data, dict):
|
||||
nested = data.get("data")
|
||||
if isinstance(nested, list):
|
||||
return nested
|
||||
return list(data.values())
|
||||
|
||||
items: list[dict] = []
|
||||
for line in content.splitlines():
|
||||
line = line.strip()
|
||||
if line:
|
||||
items.append(json.loads(line))
|
||||
return items
|
||||
|
||||
|
||||
def _parse_split_ratio(text: str) -> tuple[int, int, int]:
|
||||
parts = [part.strip() for part in str(text or "").split(":") if part.strip()]
|
||||
if len(parts) != 3:
|
||||
raise ValueError(
|
||||
f"split_ratio must be in train:val:test form, got {text!r}"
|
||||
)
|
||||
try:
|
||||
train, val, test = (int(part) for part in parts)
|
||||
except ValueError as exc:
|
||||
raise ValueError(
|
||||
f"split_ratio must contain integers, got {text!r}"
|
||||
) from exc
|
||||
if min(train, val, test) <= 0:
|
||||
raise ValueError(f"split_ratio parts must be positive, got {text!r}")
|
||||
return train, val, test
|
||||
|
||||
|
||||
def _compute_split_counts(total: int, ratio: tuple[int, int, int]) -> tuple[int, int, int]:
|
||||
weights = list(ratio)
|
||||
denom = sum(weights)
|
||||
raw = [total * weight / denom for weight in weights]
|
||||
counts = [int(value) for value in raw]
|
||||
remaining = total - sum(counts)
|
||||
order = sorted(
|
||||
range(len(raw)),
|
||||
key=lambda idx: (raw[idx] - counts[idx], weights[idx]),
|
||||
reverse=True,
|
||||
)
|
||||
for idx in order[:remaining]:
|
||||
counts[idx] += 1
|
||||
return counts[0], counts[1], counts[2]
|
||||
|
||||
|
||||
class SplitDataLoader(BaseDataLoader):
|
||||
"""Base class for dataset-backed environments.
|
||||
|
||||
Supported modes:
|
||||
|
||||
- ``split_mode="split_dir"``: load an existing ``train/``, ``val/``,
|
||||
``test/`` directory tree.
|
||||
- ``split_mode="ratio"``: load raw items from ``data_path`` and materialize
|
||||
a deterministic split directory with the requested ratio.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
split_dir: str = "",
|
||||
data_path: str = "",
|
||||
split_mode: str = "ratio",
|
||||
split_ratio: str = "2:1:7",
|
||||
split_seed: int = 42,
|
||||
split_output_dir: str = "",
|
||||
seed: int = 42,
|
||||
limit: int = 0,
|
||||
**kwargs,
|
||||
) -> None:
|
||||
self.split_dir = split_dir
|
||||
self.data_path = data_path
|
||||
self.split_mode = split_mode
|
||||
self.split_ratio = split_ratio
|
||||
self.split_seed = int(split_seed)
|
||||
self.split_output_dir = split_output_dir
|
||||
self.seed = seed
|
||||
self.limit = limit
|
||||
self._splits: dict[str, list[dict]] = {}
|
||||
|
||||
# ── Setup ────────────────────────────────────────────────────────────
|
||||
|
||||
def setup(self, cfg: dict) -> None:
|
||||
if not self.split_mode:
|
||||
self.split_mode = str(cfg.get("split_mode", "ratio") or "ratio")
|
||||
if not self.split_dir:
|
||||
self.split_dir = cfg.get("split_dir", "")
|
||||
if not self.data_path:
|
||||
self.data_path = cfg.get("data_path", "")
|
||||
if not self.split_output_dir:
|
||||
self.split_output_dir = cfg.get("split_output_dir", "")
|
||||
if "split_seed" in cfg and not self.split_seed:
|
||||
self.split_seed = int(cfg.get("split_seed", 0) or 0)
|
||||
if not self.split_seed:
|
||||
self.split_seed = self.seed
|
||||
if not self.split_ratio:
|
||||
self.split_ratio = str(cfg.get("split_ratio", "2:1:7") or "2:1:7")
|
||||
|
||||
mode = str(self.split_mode or "ratio").strip().lower()
|
||||
if mode not in {"ratio", "split_dir"}:
|
||||
raise ValueError(
|
||||
f"{type(self).__name__} split_mode must be 'ratio' or 'split_dir', "
|
||||
f"got {self.split_mode!r}"
|
||||
)
|
||||
self.split_mode = mode
|
||||
|
||||
if self.split_mode == "ratio":
|
||||
self.split_dir = self._materialize_ratio_split(cfg)
|
||||
if not self.split_dir:
|
||||
raise ValueError(
|
||||
f"{type(self).__name__} requires either "
|
||||
"`split_mode=ratio` with `data_path`, or `split_mode=split_dir` "
|
||||
f"with `split_dir` pointing to {'/'.join(SPLIT_NAMES)}/."
|
||||
)
|
||||
self._load_all_splits()
|
||||
|
||||
def _resolve_split_output_dir(self, cfg: dict) -> str:
|
||||
if self.split_output_dir:
|
||||
return os.path.abspath(self.split_output_dir)
|
||||
out_root = os.path.abspath(str(cfg.get("out_root") or os.getcwd()))
|
||||
env_name = str(cfg.get("env") or type(self).__name__.replace("DataLoader", "").lower())
|
||||
ratio_tag = str(self.split_ratio or "2:1:7").replace(":", "-")
|
||||
return os.path.join(out_root, "_generated_splits", f"{env_name}_{ratio_tag}_seed{self.split_seed}")
|
||||
|
||||
def load_raw_items(self, data_path: str) -> list[dict]:
|
||||
"""Load raw items from a dataset path before ratio splitting.
|
||||
|
||||
Subclasses can override when the raw dataset is not a single JSON/JSONL
|
||||
file or when directory layouts require custom normalization.
|
||||
"""
|
||||
if os.path.isdir(data_path):
|
||||
if any(os.path.isdir(os.path.join(data_path, name)) for name in SPLIT_NAMES):
|
||||
raise ValueError(
|
||||
f"{type(self).__name__} got a split directory as data_path. "
|
||||
"Use split_mode=split_dir and pass it as split_dir instead."
|
||||
)
|
||||
candidates = sorted(glob.glob(os.path.join(data_path, "*.json")))
|
||||
candidates += sorted(glob.glob(os.path.join(data_path, "*.jsonl")))
|
||||
if len(candidates) != 1:
|
||||
raise ValueError(
|
||||
f"{type(self).__name__} expected data_path to be one JSON/JSONL file "
|
||||
f"or a directory containing exactly one such file, got: {data_path}"
|
||||
)
|
||||
return _load_json_or_jsonl(candidates[0])
|
||||
return _load_json_or_jsonl(data_path)
|
||||
|
||||
def write_split_items(self, split_path: str, items: list[dict]) -> None:
|
||||
os.makedirs(split_path, exist_ok=True)
|
||||
out_path = os.path.join(split_path, "items.json")
|
||||
with open(out_path, "w", encoding="utf-8") as f:
|
||||
json.dump(items, f, ensure_ascii=False, indent=2)
|
||||
|
||||
def _materialize_ratio_split(self, cfg: dict) -> str:
|
||||
data_path = os.path.abspath(str(self.data_path or "").strip())
|
||||
if not data_path:
|
||||
raise ValueError(
|
||||
f"{type(self).__name__} requires data_path when split_mode=ratio."
|
||||
)
|
||||
|
||||
ratio = _parse_split_ratio(self.split_ratio)
|
||||
items = self.load_raw_items(data_path)
|
||||
if not isinstance(items, list) or not items:
|
||||
raise ValueError(f"No raw items available for ratio split from {data_path}")
|
||||
|
||||
shuffled = list(items)
|
||||
rng = random.Random(self.split_seed)
|
||||
rng.shuffle(shuffled)
|
||||
|
||||
train_n, val_n, test_n = _compute_split_counts(len(shuffled), ratio)
|
||||
train_items = shuffled[:train_n]
|
||||
val_items = shuffled[train_n: train_n + val_n]
|
||||
test_items = shuffled[train_n + val_n: train_n + val_n + test_n]
|
||||
|
||||
split_dir = self._resolve_split_output_dir(cfg)
|
||||
manifest = {
|
||||
"source_data_path": data_path,
|
||||
"split_mode": "ratio",
|
||||
"split_ratio": self.split_ratio,
|
||||
"split_seed": self.split_seed,
|
||||
"counts": {
|
||||
"train": len(train_items),
|
||||
"val": len(val_items),
|
||||
"test": len(test_items),
|
||||
},
|
||||
}
|
||||
os.makedirs(split_dir, exist_ok=True)
|
||||
self.write_split_items(os.path.join(split_dir, "train"), train_items)
|
||||
self.write_split_items(os.path.join(split_dir, "val"), val_items)
|
||||
self.write_split_items(os.path.join(split_dir, "test"), test_items)
|
||||
with open(os.path.join(split_dir, "split_manifest.json"), "w", encoding="utf-8") as f:
|
||||
json.dump(manifest, f, ensure_ascii=False, indent=2)
|
||||
print(
|
||||
f" [{type(self).__name__}] generated ratio split {self.split_ratio} "
|
||||
f"at {split_dir} from {data_path}"
|
||||
)
|
||||
return split_dir
|
||||
|
||||
def _load_all_splits(self) -> None:
|
||||
for name in SPLIT_NAMES:
|
||||
split_path = os.path.join(self.split_dir, name)
|
||||
if not os.path.isdir(split_path):
|
||||
raise ValueError(
|
||||
f"Missing '{name}/' subdirectory in split_dir: {self.split_dir}"
|
||||
)
|
||||
items = self.load_split_items(split_path)
|
||||
if self.limit:
|
||||
items = items[: self.limit]
|
||||
self._splits[name] = items
|
||||
|
||||
counts = " ".join(f"{k}={len(v)}" for k, v in self._splits.items())
|
||||
print(f" [{type(self).__name__}] {counts} (from {self.split_dir})")
|
||||
|
||||
def load_split_items(self, split_path: str) -> list[dict]:
|
||||
"""Load items from one split directory (e.g. ``split_dir/train/``).
|
||||
|
||||
Default: finds the first ``.json`` file in the directory and loads it
|
||||
as a JSON array. Subclasses can override for custom formats.
|
||||
"""
|
||||
json_files = sorted(glob.glob(os.path.join(split_path, "*.json")))
|
||||
if not json_files:
|
||||
raise FileNotFoundError(
|
||||
f"No .json file found in {split_path}"
|
||||
)
|
||||
with open(json_files[0], encoding="utf-8") as f:
|
||||
items = json.load(f)
|
||||
if not isinstance(items, list):
|
||||
raise ValueError(
|
||||
f"Expected JSON array in {json_files[0]}, got {type(items).__name__}"
|
||||
)
|
||||
return items
|
||||
|
||||
# ── Accessors ────────────────────────────────────────────────────────
|
||||
|
||||
@property
|
||||
def train_items(self) -> list[dict]:
|
||||
return self._splits.get("train", [])
|
||||
|
||||
@property
|
||||
def val_items(self) -> list[dict]:
|
||||
return self._splits.get("val", [])
|
||||
|
||||
@property
|
||||
def test_items(self) -> list[dict]:
|
||||
return self._splits.get("test", [])
|
||||
|
||||
def get_split_items(self, split: str) -> list[dict]:
|
||||
"""Resolve a split name (including legacy aliases) to its item list."""
|
||||
canonical = _SPLIT_ALIAS.get(split, split)
|
||||
return list(self._splits.get(canonical, self.val_items))
|
||||
|
||||
def get_train_size(self) -> int:
|
||||
return len(self.train_items)
|
||||
|
||||
def plan_train_epoch(
|
||||
self,
|
||||
*,
|
||||
epoch: int,
|
||||
steps_per_epoch: int,
|
||||
accumulation: int,
|
||||
batch_size: int,
|
||||
seed: int,
|
||||
**kwargs,
|
||||
) -> list[BatchSpec]:
|
||||
"""Build one full epoch that covers the train split in shuffled order.
|
||||
|
||||
For split-backed datasets, an epoch should correspond to one pass over
|
||||
the available training items rather than repeated independent sampling.
|
||||
"""
|
||||
epoch_rng = random.Random(seed + epoch * 1000)
|
||||
items = list(self.train_items)
|
||||
epoch_rng.shuffle(items)
|
||||
|
||||
total_batches = steps_per_epoch * accumulation
|
||||
if total_batches <= 0:
|
||||
return []
|
||||
|
||||
batches: list[BatchSpec] = []
|
||||
cursor = 0
|
||||
for batch_idx in range(total_batches):
|
||||
batch_items = items[cursor: cursor + batch_size]
|
||||
cursor += len(batch_items)
|
||||
|
||||
# Extremely small datasets can leave trailing empty microbatches
|
||||
# when accumulation > 1. Reuse the shuffled prefix in that case so
|
||||
# the trainer still receives the expected batch count.
|
||||
if not batch_items and items:
|
||||
refill_rng = random.Random(seed + epoch * 1000 + batch_idx + 1)
|
||||
batch_items = list(items)
|
||||
refill_rng.shuffle(batch_items)
|
||||
batch_items = batch_items[:batch_size]
|
||||
|
||||
batches.append(
|
||||
BatchSpec(
|
||||
phase="train",
|
||||
split="train",
|
||||
seed=seed + epoch * 1000 + batch_idx + 1,
|
||||
batch_size=len(batch_items),
|
||||
payload=batch_items,
|
||||
)
|
||||
)
|
||||
|
||||
return batches
|
||||
|
||||
# ── Batch construction ───────────────────────────────────────────────
|
||||
|
||||
def build_train_batch(self, batch_size: int, seed: int, **kwargs) -> BatchSpec:
|
||||
rng = random.Random(seed)
|
||||
items = list(self.train_items)
|
||||
rng.shuffle(items)
|
||||
items = items[:batch_size]
|
||||
return BatchSpec(
|
||||
phase="train",
|
||||
split="train",
|
||||
seed=seed,
|
||||
batch_size=len(items),
|
||||
payload=items,
|
||||
)
|
||||
|
||||
def build_eval_batch(
|
||||
self,
|
||||
env_num: int,
|
||||
split: str,
|
||||
seed: int,
|
||||
**kwargs,
|
||||
) -> BatchSpec:
|
||||
items = self.get_split_items(split)
|
||||
if env_num and env_num < len(items):
|
||||
items = items[:env_num]
|
||||
return BatchSpec(
|
||||
phase="eval",
|
||||
split=split,
|
||||
seed=seed,
|
||||
batch_size=len(items),
|
||||
payload=items,
|
||||
)
|
||||
@@ -1,9 +0,0 @@
|
||||
"""ReflACT Engine -- the training runner.
|
||||
|
||||
Analogous to the Runner in mmengine: orchestrates the full training pipeline
|
||||
including rollout, gradient computation, aggregation, optimization, and
|
||||
evaluation.
|
||||
"""
|
||||
from skillopt.engine.trainer import ReflACTTrainer # noqa: F401
|
||||
|
||||
__all__ = ["ReflACTTrainer"]
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1 +0,0 @@
|
||||
"""ReflACT environment adapters."""
|
||||
@@ -1,19 +0,0 @@
|
||||
# Benchmark Template
|
||||
|
||||
This directory provides scaffold files for adding a new benchmark to SkillOpt.
|
||||
|
||||
## Files
|
||||
|
||||
- `env_template.py` — Environment adapter template
|
||||
- `loader_template.py` — Data loader template
|
||||
- `config_template.yaml` — Config file template
|
||||
|
||||
## Usage
|
||||
|
||||
1. Copy this directory: `cp -r skillopt/envs/_template skillopt/envs/your_benchmark`
|
||||
2. Rename files: remove `_template` suffix
|
||||
3. Implement the `TODO` sections
|
||||
4. Register in `skillopt/envs/__init__.py`
|
||||
5. Create config at `configs/your_benchmark/default.yaml`
|
||||
|
||||
See the [documentation](../../docs/guide/new-benchmark.md) for the full guide.
|
||||
@@ -1,45 +0,0 @@
|
||||
# ──────────────────────────────────────────────────
|
||||
# SkillOpt Config Template — <Your Benchmark Name>
|
||||
# ──────────────────────────────────────────────────
|
||||
# Copy this file to configs/<your_benchmark>/default.yaml
|
||||
# and customize the values below.
|
||||
|
||||
# Inherit global defaults
|
||||
_base_: ['../_base_/default.yaml']
|
||||
|
||||
# ── Environment ──────────────────────────────────
|
||||
env:
|
||||
name: your_benchmark # Must match registry key
|
||||
data_path: data/your_benchmark # Path to your data
|
||||
split_mode: ratio # "ratio" or "split_dir"
|
||||
split_ratio: "2:1:7" # train:val:test
|
||||
exec_timeout: 120 # Per-task timeout (seconds)
|
||||
|
||||
# ── Training ─────────────────────────────────────
|
||||
train:
|
||||
num_epochs: 4 # Number of epochs
|
||||
batch_size: 40 # Tasks per step (batch size)
|
||||
seed: 42
|
||||
|
||||
# ── Gradient (Reflection) ───────────────────────
|
||||
gradient:
|
||||
analyst_workers: 16 # Parallel reflection workers
|
||||
minibatch_size: 8
|
||||
|
||||
# ── Optimizer ────────────────────────────────────
|
||||
optimizer:
|
||||
learning_rate: 4 # Max edits per step (edit budget)
|
||||
lr_scheduler: cosine # cosine | linear | constant | autonomous
|
||||
use_slow_update: true # Epoch-boundary momentum
|
||||
use_meta_skill: true # Cross-epoch optimizer memory
|
||||
|
||||
# ── Evaluation ───────────────────────────────────
|
||||
evaluation:
|
||||
use_gate: true # Validation gating
|
||||
eval_test: true # Run test eval after training
|
||||
|
||||
# ── Model ────────────────────────────────────────
|
||||
model:
|
||||
backend: azure_openai # azure_openai | openai_chat | claude_code_exec | qwen
|
||||
optimizer: gpt-4o
|
||||
target: gpt-4o
|
||||
@@ -1,92 +0,0 @@
|
||||
"""
|
||||
Benchmark Environment Template
|
||||
===============================
|
||||
Copy this file and implement the TODO sections to add a new benchmark.
|
||||
|
||||
The EnvAdapter is responsible for:
|
||||
1. Executing tasks using the target model + current skill document
|
||||
2. Evaluating predictions against ground truth
|
||||
3. Returning structured results for the training loop
|
||||
"""
|
||||
from skillopt.envs.base import EnvAdapter
|
||||
|
||||
|
||||
class TemplateBenchmarkEnv(EnvAdapter):
|
||||
"""
|
||||
Environment adapter for <Your Benchmark Name>.
|
||||
|
||||
Rename this class and implement the abstract methods below.
|
||||
"""
|
||||
|
||||
def __init__(self, cfg: dict):
|
||||
super().__init__(cfg)
|
||||
# TODO: Initialize benchmark-specific state
|
||||
# Example: self.tools = load_tools(cfg)
|
||||
|
||||
async def execute(self, item, skill: str, model):
|
||||
"""
|
||||
Execute a single task with the target model.
|
||||
|
||||
Args:
|
||||
item: DataItem with .id, .input, .ground_truth, .metadata
|
||||
skill: Current skill document content (Markdown string)
|
||||
model: Target model backend instance
|
||||
|
||||
Returns:
|
||||
TaskResult with prediction, score, and trajectory
|
||||
"""
|
||||
# Step 1: Build the prompt combining skill + task input
|
||||
prompt = self.build_prompt(item, skill)
|
||||
|
||||
# Step 2: Call the target model
|
||||
# TODO: Customize the message format for your benchmark
|
||||
messages = [
|
||||
{"role": "system", "content": skill},
|
||||
{"role": "user", "content": item.input},
|
||||
]
|
||||
response = await model.generate(messages)
|
||||
|
||||
# Step 3: Parse the model response into a prediction
|
||||
prediction = self.parse_response(response.content)
|
||||
|
||||
# Step 4: Score the prediction
|
||||
score = self.evaluate(prediction, item.ground_truth)
|
||||
|
||||
# Step 5: Return structured result
|
||||
return {
|
||||
"item_id": item.id,
|
||||
"prediction": prediction,
|
||||
"score": score,
|
||||
"trajectory": messages + [{"role": "assistant", "content": response.content}],
|
||||
}
|
||||
|
||||
def evaluate(self, prediction: str, ground_truth: str) -> float:
|
||||
"""
|
||||
Score a prediction against the ground truth.
|
||||
|
||||
Returns:
|
||||
Float between 0.0 (wrong) and 1.0 (correct)
|
||||
|
||||
TODO: Implement your scoring metric. Common options:
|
||||
- Exact match: float(pred.strip().lower() == gt.strip().lower())
|
||||
- F1 score: compute token overlap
|
||||
- ANLS: for document QA tasks
|
||||
- Custom: any float in [0, 1]
|
||||
"""
|
||||
# Placeholder — exact match
|
||||
return float(prediction.strip().lower() == ground_truth.strip().lower())
|
||||
|
||||
def build_prompt(self, item, skill: str) -> str:
|
||||
"""Combine skill document with task input."""
|
||||
return f"{skill}\n\n---\n\nQuestion: {item.input}"
|
||||
|
||||
def parse_response(self, response: str) -> str:
|
||||
"""
|
||||
Extract the answer from the model's raw response.
|
||||
|
||||
TODO: Implement extraction logic. For example:
|
||||
- Extract text after "Answer:"
|
||||
- Parse JSON output
|
||||
- Extract from code blocks
|
||||
"""
|
||||
return response.strip()
|
||||
@@ -1,103 +0,0 @@
|
||||
"""
|
||||
Benchmark Data Loader Template
|
||||
================================
|
||||
Copy this file and implement the TODO sections to load your benchmark data.
|
||||
|
||||
The DataLoader is responsible for:
|
||||
1. Loading raw data from disk
|
||||
2. Splitting into train / validation / test sets
|
||||
3. Providing DataItem objects to the training loop
|
||||
"""
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
class TemplateBenchmarkLoader:
|
||||
"""
|
||||
Data loader for <Your Benchmark Name>.
|
||||
|
||||
Rename this class and implement the methods below.
|
||||
"""
|
||||
|
||||
def __init__(self, data_dir: str = "data/your_benchmark", **kwargs):
|
||||
self.data_dir = Path(data_dir)
|
||||
self.items = []
|
||||
self.splits = {}
|
||||
|
||||
def setup(self, cfg: dict):
|
||||
"""
|
||||
Initialize the loader with config.
|
||||
|
||||
Called once before training starts.
|
||||
|
||||
Args:
|
||||
cfg: Dict with keys like 'split_mode', 'train_ratio', 'val_ratio', etc.
|
||||
"""
|
||||
# Step 1: Load raw data
|
||||
self.items = self._load_items()
|
||||
|
||||
# Step 2: Create splits
|
||||
split_mode = cfg.get("split_mode", "ratio")
|
||||
if split_mode == "ratio":
|
||||
self._split_by_ratio(
|
||||
train_ratio=cfg.get("train_ratio", 0.7),
|
||||
val_ratio=cfg.get("val_ratio", 0.15),
|
||||
)
|
||||
elif split_mode == "split_dir":
|
||||
self._load_predefined_splits(cfg.get("split_dir", self.data_dir))
|
||||
|
||||
def _load_items(self) -> list:
|
||||
"""
|
||||
Load raw data into structured items.
|
||||
|
||||
TODO: Implement data loading. Each item should have at minimum:
|
||||
- id: unique identifier
|
||||
- input: the task input (question, instruction, etc.)
|
||||
- ground_truth: the expected answer
|
||||
- metadata: optional dict with extra info
|
||||
|
||||
Example:
|
||||
items = []
|
||||
for path in self.data_dir.glob("*.json"):
|
||||
data = json.loads(path.read_text())
|
||||
for entry in data:
|
||||
items.append({
|
||||
"id": entry["id"],
|
||||
"input": entry["question"],
|
||||
"ground_truth": entry["answer"],
|
||||
"metadata": {"source": path.name},
|
||||
})
|
||||
return items
|
||||
"""
|
||||
raise NotImplementedError("Implement _load_items() for your benchmark")
|
||||
|
||||
def _split_by_ratio(self, train_ratio: float, val_ratio: float):
|
||||
"""Split items by ratio."""
|
||||
import random
|
||||
random.shuffle(self.items)
|
||||
n = len(self.items)
|
||||
n_train = int(n * train_ratio)
|
||||
n_val = int(n * val_ratio)
|
||||
self.splits = {
|
||||
"train": self.items[:n_train],
|
||||
"valid": self.items[n_train:n_train + n_val],
|
||||
"test": self.items[n_train + n_val:],
|
||||
}
|
||||
|
||||
def _load_predefined_splits(self, split_dir):
|
||||
"""Load from pre-split directories."""
|
||||
# TODO: Implement if your benchmark has pre-defined splits
|
||||
raise NotImplementedError
|
||||
|
||||
def get_split_items(self, split: str) -> list:
|
||||
"""
|
||||
Return items for a given split.
|
||||
|
||||
Args:
|
||||
split: One of "train", "valid", "test"
|
||||
|
||||
Returns:
|
||||
List of data items for the requested split
|
||||
"""
|
||||
if split not in self.splits:
|
||||
raise ValueError(f"Unknown split '{split}'. Available: {list(self.splits.keys())}")
|
||||
return self.splits[split]
|
||||
@@ -1,5 +0,0 @@
|
||||
"""ALFWorld environment adapter for ReflACT."""
|
||||
|
||||
from skillopt.envs.alfworld.adapter import ALFWorldAdapter
|
||||
|
||||
__all__ = ["ALFWorldAdapter"]
|
||||
@@ -1,459 +0,0 @@
|
||||
"""ALFWorld environment adapter for ReflACT.
|
||||
|
||||
Connects the ReflACT training loop to ALFWorld by implementing
|
||||
:class:`~skillopt.envs.base.EnvAdapter`.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
import json
|
||||
import os
|
||||
|
||||
from skillopt.datasets.base import BatchSpec
|
||||
from skillopt.envs.base import EnvAdapter
|
||||
from skillopt.envs.alfworld.dataloader import ALFWorldDataLoader
|
||||
from skillopt.envs.alfworld.rollout import (
|
||||
build_alfworld_env,
|
||||
run_alfworld_batch,
|
||||
TASKS,
|
||||
)
|
||||
from skillopt.gradient.reflect import run_minibatch_reflect
|
||||
from skillopt.utils import compute_score
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ALFWorldBatchRun:
|
||||
"""Lazy ALFWorld batch description.
|
||||
|
||||
The adapter materializes this in rollout chunks so a large evaluation set
|
||||
does not keep every ALFWorld simulator open at once.
|
||||
"""
|
||||
|
||||
env_num: int
|
||||
eval_dataset: str
|
||||
seed: int
|
||||
is_train: bool
|
||||
workers: int
|
||||
specific_gamefiles: list[str] | None = None
|
||||
result_ids: list[str] | None = None
|
||||
items: list[dict] | None = None
|
||||
|
||||
def __iter__(self):
|
||||
return iter(self.items or [])
|
||||
|
||||
def __len__(self) -> int:
|
||||
return int(self.env_num or 0)
|
||||
|
||||
|
||||
class ALFWorldAdapter(EnvAdapter):
|
||||
"""ALFWorld environment adapter.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
max_steps : int
|
||||
Maximum steps per ALFWorld episode (default 50).
|
||||
max_api_workers : int
|
||||
Maximum concurrent API calls during rollout (default 8).
|
||||
analyst_workers : int
|
||||
Parallel workers for analyst stage (default 16).
|
||||
failure_only : bool
|
||||
If True, only run error analyst (skip success analyst).
|
||||
minibatch_size : int
|
||||
Trajectories per analyst group, M (default 8).
|
||||
edit_budget : int
|
||||
Maximum edits per minibatch, L (default 4).
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
split_dir: str = "",
|
||||
data_path: str = "",
|
||||
split_mode: str = "split_dir",
|
||||
split_ratio: str = "2:1:7",
|
||||
split_seed: int = 42,
|
||||
split_output_dir: str = "",
|
||||
seed: int = 42,
|
||||
limit: int = 0,
|
||||
train_size: int = 0,
|
||||
max_steps: int = 50,
|
||||
workers: int = 8,
|
||||
max_api_workers: int = 8,
|
||||
analyst_workers: int = 16,
|
||||
failure_only: bool = False,
|
||||
minibatch_size: int = 8,
|
||||
edit_budget: int = 4,
|
||||
max_completion_tokens: int = 16384,
|
||||
) -> None:
|
||||
self.max_steps = max_steps
|
||||
self.workers = max(int(workers or 1), 1)
|
||||
self.max_api_workers = max_api_workers
|
||||
self.max_completion_tokens = int(max_completion_tokens)
|
||||
self.analyst_workers = analyst_workers
|
||||
self.failure_only = failure_only
|
||||
self.minibatch_size = minibatch_size
|
||||
self.edit_budget = edit_budget
|
||||
self.dataloader = ALFWorldDataLoader(
|
||||
split_dir=split_dir,
|
||||
data_path=data_path,
|
||||
split_mode=split_mode,
|
||||
split_ratio=split_ratio,
|
||||
split_seed=split_seed,
|
||||
split_output_dir=split_output_dir,
|
||||
seed=seed,
|
||||
limit=limit,
|
||||
train_size=train_size,
|
||||
)
|
||||
self._traj_cache: dict[str, dict | None] = {}
|
||||
|
||||
def setup(self, cfg: dict) -> None:
|
||||
super().setup(cfg)
|
||||
self.dataloader.setup(cfg)
|
||||
|
||||
def _load_traj_data(self, item: dict) -> dict | None:
|
||||
gamefile = str(item.get("gamefile") or "").strip()
|
||||
if not gamefile:
|
||||
return None
|
||||
if gamefile in self._traj_cache:
|
||||
return self._traj_cache[gamefile]
|
||||
|
||||
traj_path = os.path.join(os.path.dirname(gamefile), "traj_data.json")
|
||||
try:
|
||||
with open(traj_path, encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
except Exception:
|
||||
data = None
|
||||
self._traj_cache[gamefile] = data
|
||||
return data
|
||||
|
||||
@staticmethod
|
||||
def _unique_lines(values: list[str], *, limit: int = 0) -> list[str]:
|
||||
lines: list[str] = []
|
||||
seen: set[str] = set()
|
||||
for raw in values:
|
||||
line = str(raw or "").strip()
|
||||
if not line or line in seen:
|
||||
continue
|
||||
seen.add(line)
|
||||
lines.append(line)
|
||||
if limit > 0 and len(lines) >= limit:
|
||||
break
|
||||
return lines
|
||||
|
||||
@staticmethod
|
||||
def _format_high_pddl(high_pddl: list[dict]) -> list[str]:
|
||||
steps: list[str] = []
|
||||
for idx, step in enumerate(high_pddl or [], start=1):
|
||||
discrete = step.get("discrete_action") or {}
|
||||
action = str(discrete.get("action") or "").strip()
|
||||
args = [str(arg).strip() for arg in (discrete.get("args") or []) if str(arg).strip()]
|
||||
if action and args:
|
||||
text = f"{action}({', '.join(args)})"
|
||||
elif action:
|
||||
text = action
|
||||
else:
|
||||
planner_action = step.get("planner_action") or {}
|
||||
text = str(planner_action.get("action") or "").strip()
|
||||
if text:
|
||||
steps.append(f"{idx}. {text}")
|
||||
return steps
|
||||
|
||||
def _build_reference_bundle(self, item: dict) -> dict:
|
||||
data = self._load_traj_data(item)
|
||||
if not data:
|
||||
return {}
|
||||
|
||||
anns = ((data.get("turk_annotations") or {}).get("anns") or [])
|
||||
task_descs = self._unique_lines(
|
||||
[ann.get("task_desc", "") for ann in anns],
|
||||
limit=3,
|
||||
)
|
||||
high_descs = self._unique_lines(
|
||||
[step for ann in anns for step in (ann.get("high_descs") or [])],
|
||||
limit=12,
|
||||
)
|
||||
pddl_params = {
|
||||
key: value
|
||||
for key, value in (data.get("pddl_params") or {}).items()
|
||||
if value not in ("", None, [], {})
|
||||
}
|
||||
scene = data.get("scene") or {}
|
||||
scene_summary = {
|
||||
key: scene.get(key)
|
||||
for key in ("floor_plan", "scene_num", "dirty_and_empty")
|
||||
if scene.get(key) not in ("", None, [], {})
|
||||
}
|
||||
high_pddl = self._format_high_pddl((data.get("plan") or {}).get("high_pddl") or [])
|
||||
task_type = str(data.get("task_type") or item.get("task_type") or "").strip()
|
||||
return {
|
||||
"task_type": task_type,
|
||||
"task_descs": task_descs,
|
||||
"high_descs": high_descs,
|
||||
"pddl_params": pddl_params,
|
||||
"high_pddl": high_pddl,
|
||||
"scene_summary": scene_summary,
|
||||
}
|
||||
|
||||
def build_reference_text(self, item: dict) -> str:
|
||||
bundle = self._build_reference_bundle(item)
|
||||
if not bundle:
|
||||
return ""
|
||||
|
||||
parts: list[str] = []
|
||||
if bundle["task_type"]:
|
||||
parts.append(f"## Reference Task Type\n{bundle['task_type']}")
|
||||
if bundle["task_descs"]:
|
||||
parts.append(
|
||||
"## Reference Human Task Descriptions\n"
|
||||
+ "\n".join(f"- {line}" for line in bundle["task_descs"])
|
||||
)
|
||||
if bundle["high_descs"]:
|
||||
parts.append(
|
||||
"## Reference Human High-Level Steps\n"
|
||||
+ "\n".join(f"{idx}. {line}" for idx, line in enumerate(bundle["high_descs"], start=1))
|
||||
)
|
||||
if bundle["pddl_params"]:
|
||||
parts.append(
|
||||
"## Reference PDDL Params\n"
|
||||
+ "\n".join(f"- {key}: {value}" for key, value in bundle["pddl_params"].items())
|
||||
)
|
||||
if bundle["high_pddl"]:
|
||||
parts.append(
|
||||
"## Reference Planner High-Level Plan\n" + "\n".join(bundle["high_pddl"])
|
||||
)
|
||||
if bundle["scene_summary"]:
|
||||
parts.append(
|
||||
"## Reference Scene Summary\n"
|
||||
+ "\n".join(f"- {key}: {value}" for key, value in bundle["scene_summary"].items())
|
||||
)
|
||||
return "\n\n".join(parts)
|
||||
|
||||
def get_reference_metadata(self, item: dict) -> dict:
|
||||
bundle = self._build_reference_bundle(item)
|
||||
if not bundle:
|
||||
return {"fields": [], "preview": ""}
|
||||
|
||||
fields: list[str] = []
|
||||
previews: list[str] = []
|
||||
if bundle["task_type"]:
|
||||
fields.append("task_type")
|
||||
previews.append(f"[task_type] {bundle['task_type']}")
|
||||
if bundle["task_descs"]:
|
||||
fields.append("task_desc")
|
||||
previews.append("[task_desc]\n" + "\n".join(bundle["task_descs"][:2]))
|
||||
if bundle["high_descs"]:
|
||||
fields.append("high_descs")
|
||||
previews.append("[high_descs]\n" + "\n".join(bundle["high_descs"][:3]))
|
||||
if bundle["pddl_params"]:
|
||||
fields.append("pddl_params")
|
||||
previews.append(
|
||||
"[pddl_params]\n"
|
||||
+ "\n".join(
|
||||
f"{key}: {value}" for key, value in list(bundle["pddl_params"].items())[:4]
|
||||
)
|
||||
)
|
||||
if bundle["high_pddl"]:
|
||||
fields.append("plan.high_pddl")
|
||||
previews.append("[plan.high_pddl]\n" + "\n".join(bundle["high_pddl"][:3]))
|
||||
if bundle["scene_summary"]:
|
||||
fields.append("scene")
|
||||
previews.append(
|
||||
"[scene]\n"
|
||||
+ "\n".join(
|
||||
f"{key}: {value}" for key, value in bundle["scene_summary"].items()
|
||||
)
|
||||
)
|
||||
return {
|
||||
"fields": fields,
|
||||
"preview": "\n\n".join(previews)[:600],
|
||||
}
|
||||
|
||||
@staticmethod
|
||||
def _infer_dataset_from_gamefile(gamefile: str) -> tuple[str, bool]:
|
||||
path = str(gamefile or "")
|
||||
if "/valid_seen/" in path:
|
||||
return "eval_in_distribution", False
|
||||
if "/valid_unseen/" in path:
|
||||
return "eval_out_of_distribution", False
|
||||
return "train", True
|
||||
|
||||
def get_dataloader(self):
|
||||
return self.dataloader
|
||||
|
||||
def _comparison_items(self, items: list[dict]) -> list[dict]:
|
||||
enriched: list[dict] = []
|
||||
for item in items:
|
||||
row = dict(item)
|
||||
bundle = self._build_reference_bundle(row)
|
||||
if bundle.get("task_descs"):
|
||||
row["task_description"] = bundle["task_descs"][0]
|
||||
elif bundle.get("task_type"):
|
||||
row["task_description"] = bundle["task_type"]
|
||||
enriched.append(row)
|
||||
return enriched
|
||||
|
||||
def requires_ray(self) -> bool:
|
||||
return False
|
||||
|
||||
def build_env_from_batch(self, batch: BatchSpec, **kwargs):
|
||||
gamefiles = list(batch.metadata.get("gamefiles") or [])
|
||||
result_ids = list(batch.metadata.get("result_ids") or [])
|
||||
items = self._comparison_items(list(batch.payload or []))
|
||||
return ALFWorldBatchRun(
|
||||
env_num=batch.batch_size,
|
||||
eval_dataset=batch.metadata.get("eval_dataset", batch.split),
|
||||
seed=batch.seed,
|
||||
is_train=batch.metadata.get("is_train", batch.phase == "train"),
|
||||
specific_gamefiles=gamefiles or None,
|
||||
result_ids=result_ids or None,
|
||||
items=items,
|
||||
workers=self.workers,
|
||||
)
|
||||
|
||||
def build_train_env(self, batch_size: int, seed: int, **kwargs):
|
||||
batch = self.dataloader.build_train_batch(batch_size=batch_size, seed=seed, **kwargs)
|
||||
return self.build_env_from_batch(batch, **kwargs)
|
||||
|
||||
def build_eval_env(self, env_num: int, split: str, seed: int, **kwargs):
|
||||
batch = self.dataloader.build_eval_batch(env_num=env_num, split=split, seed=seed, **kwargs)
|
||||
return self.build_env_from_batch(batch, **kwargs)
|
||||
|
||||
def rollout(
|
||||
self,
|
||||
env_manager,
|
||||
skill_content: str,
|
||||
out_dir: str,
|
||||
**kwargs,
|
||||
) -> list[dict]:
|
||||
results_path = os.path.join(out_dir, "results.jsonl")
|
||||
os.makedirs(out_dir, exist_ok=True)
|
||||
|
||||
# Resume support
|
||||
if os.path.exists(results_path):
|
||||
existing: list[dict] = []
|
||||
with open(results_path) as f:
|
||||
for line in f:
|
||||
try:
|
||||
existing.append(json.loads(line))
|
||||
except Exception:
|
||||
pass
|
||||
if existing:
|
||||
return existing
|
||||
|
||||
if isinstance(env_manager, ALFWorldBatchRun):
|
||||
results = self._run_batch(
|
||||
env_manager,
|
||||
skill_content=skill_content,
|
||||
out_dir=out_dir,
|
||||
)
|
||||
else:
|
||||
results = run_alfworld_batch(
|
||||
env_manager=env_manager,
|
||||
skill_content=skill_content,
|
||||
max_steps=self.max_steps,
|
||||
out_root=out_dir,
|
||||
max_api_workers=self.max_api_workers,
|
||||
max_completion_tokens=self.max_completion_tokens,
|
||||
result_ids=getattr(env_manager, "_skillopt_result_ids", None),
|
||||
)
|
||||
|
||||
with open(results_path, "w") as f:
|
||||
for r in results:
|
||||
f.write(json.dumps(r, ensure_ascii=False) + "\n")
|
||||
|
||||
return results
|
||||
|
||||
@staticmethod
|
||||
def _close_env(env_manager) -> None:
|
||||
close = getattr(env_manager, "close", None)
|
||||
if callable(close):
|
||||
close()
|
||||
|
||||
def _run_batch(
|
||||
self,
|
||||
batch: ALFWorldBatchRun,
|
||||
skill_content: str,
|
||||
out_dir: str,
|
||||
*,
|
||||
diagnostic_mode: bool = False,
|
||||
diagnostic_instruction: str = "",
|
||||
) -> list[dict]:
|
||||
total = int(batch.env_num or 0)
|
||||
if total <= 0:
|
||||
return []
|
||||
|
||||
workers = max(1, min(int(batch.workers or self.workers), total))
|
||||
if total > workers:
|
||||
print(
|
||||
f" [alfworld rollout] episodes={total} "
|
||||
f"env_workers={workers} chunks={(total + workers - 1) // workers}"
|
||||
)
|
||||
|
||||
all_results: list[dict] = []
|
||||
for start in range(0, total, workers):
|
||||
chunk_size = min(workers, total - start)
|
||||
chunk_gamefiles = (
|
||||
batch.specific_gamefiles[start:start + chunk_size]
|
||||
if batch.specific_gamefiles
|
||||
else None
|
||||
)
|
||||
chunk_ids = (
|
||||
batch.result_ids[start:start + chunk_size]
|
||||
if batch.result_ids
|
||||
else [f"env_{idx:03d}" for idx in range(start, start + chunk_size)]
|
||||
)
|
||||
chunk_env = build_alfworld_env(
|
||||
env_num=chunk_size,
|
||||
eval_dataset=batch.eval_dataset,
|
||||
seed=batch.seed + start,
|
||||
is_train=batch.is_train,
|
||||
specific_gamefiles=chunk_gamefiles,
|
||||
)
|
||||
try:
|
||||
chunk_results = run_alfworld_batch(
|
||||
env_manager=chunk_env,
|
||||
skill_content=skill_content,
|
||||
max_steps=self.max_steps,
|
||||
out_root=out_dir,
|
||||
max_api_workers=min(self.max_api_workers, chunk_size),
|
||||
max_completion_tokens=self.max_completion_tokens,
|
||||
diagnostic_mode=diagnostic_mode,
|
||||
diagnostic_instruction=diagnostic_instruction,
|
||||
result_ids=chunk_ids,
|
||||
)
|
||||
finally:
|
||||
self._close_env(chunk_env)
|
||||
all_results.extend(chunk_results)
|
||||
return all_results
|
||||
|
||||
def reflect(
|
||||
self,
|
||||
results: list[dict],
|
||||
skill_content: str,
|
||||
out_dir: str,
|
||||
**kwargs,
|
||||
) -> list[dict | None]:
|
||||
prediction_dir = kwargs.get("prediction_dir", os.path.join(out_dir, "predictions"))
|
||||
patches_dir = kwargs.get("patches_dir", os.path.join(out_dir, "patches"))
|
||||
random_seed = kwargs.get("random_seed")
|
||||
step_buffer_context = kwargs.get("step_buffer_context", "")
|
||||
meta_skill_context = kwargs.get("meta_skill_context", "")
|
||||
|
||||
return run_minibatch_reflect(
|
||||
results=results,
|
||||
skill_content=skill_content,
|
||||
prediction_dir=prediction_dir,
|
||||
patches_dir=patches_dir,
|
||||
workers=self.analyst_workers,
|
||||
failure_only=self.failure_only,
|
||||
minibatch_size=self.minibatch_size,
|
||||
edit_budget=self.edit_budget,
|
||||
random_seed=random_seed,
|
||||
error_system=self.get_error_minibatch_prompt(),
|
||||
success_system=self.get_success_minibatch_prompt(),
|
||||
step_buffer_context=step_buffer_context,
|
||||
meta_skill_context=meta_skill_context,
|
||||
)
|
||||
|
||||
|
||||
def get_task_types(self) -> list[str]:
|
||||
return list(TASKS)
|
||||
@@ -1,123 +0,0 @@
|
||||
"""ALFWorld task dataloader."""
|
||||
from __future__ import annotations
|
||||
|
||||
from skillopt.datasets.base import BatchSpec, SplitDataLoader
|
||||
|
||||
|
||||
class ALFWorldDataLoader(SplitDataLoader):
|
||||
"""ALFWorld batch planner.
|
||||
|
||||
In split_dir mode, batches are fixed gamefile items so ablations differ
|
||||
only in how the same training set is batched.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
split_dir: str = "",
|
||||
data_path: str = "",
|
||||
split_mode: str = "split_dir",
|
||||
split_ratio: str = "2:1:7",
|
||||
split_seed: int = 42,
|
||||
split_output_dir: str = "",
|
||||
seed: int = 42,
|
||||
limit: int = 0,
|
||||
train_size: int = 0,
|
||||
**kwargs,
|
||||
) -> None:
|
||||
super().__init__(
|
||||
split_dir=split_dir,
|
||||
data_path=data_path,
|
||||
split_mode=split_mode,
|
||||
split_ratio=split_ratio,
|
||||
split_seed=split_seed,
|
||||
split_output_dir=split_output_dir,
|
||||
seed=seed,
|
||||
limit=limit,
|
||||
)
|
||||
self.train_size_override = int(train_size or 0)
|
||||
|
||||
@staticmethod
|
||||
def _metadata_for_items(items: list[dict], split: str, phase: str) -> dict:
|
||||
gamefiles = [str(item.get("gamefile") or "") for item in items]
|
||||
if any(not gamefile for gamefile in gamefiles):
|
||||
raise ValueError("ALFWorld split items must contain non-empty gamefile paths.")
|
||||
eval_dataset = "train"
|
||||
is_train = phase == "train"
|
||||
first = gamefiles[0] if gamefiles else ""
|
||||
if "/valid_seen/" in first:
|
||||
eval_dataset = "eval_in_distribution"
|
||||
is_train = False
|
||||
elif "/valid_unseen/" in first:
|
||||
eval_dataset = "eval_out_of_distribution"
|
||||
is_train = False
|
||||
return {
|
||||
"eval_dataset": eval_dataset,
|
||||
"is_train": is_train,
|
||||
"gamefiles": gamefiles,
|
||||
"result_ids": [str(item.get("id") or idx) for idx, item in enumerate(items)],
|
||||
}
|
||||
|
||||
def get_train_size(self) -> int:
|
||||
if self.train_size_override > 0:
|
||||
return self.train_size_override
|
||||
return super().get_train_size()
|
||||
|
||||
def build_train_batch(self, batch_size: int, seed: int, **kwargs) -> BatchSpec:
|
||||
batch = super().build_train_batch(batch_size=batch_size, seed=seed, **kwargs)
|
||||
items = list(batch.payload or [])
|
||||
batch.metadata.update(self._metadata_for_items(items, "train", "train"))
|
||||
return BatchSpec(
|
||||
phase="train",
|
||||
split="train",
|
||||
seed=seed,
|
||||
batch_size=len(items),
|
||||
payload=items,
|
||||
metadata=batch.metadata,
|
||||
)
|
||||
|
||||
def plan_train_epoch(
|
||||
self,
|
||||
*,
|
||||
epoch: int,
|
||||
steps_per_epoch: int,
|
||||
accumulation: int,
|
||||
batch_size: int,
|
||||
seed: int,
|
||||
**kwargs,
|
||||
) -> list[BatchSpec]:
|
||||
batches = super().plan_train_epoch(
|
||||
epoch=epoch,
|
||||
steps_per_epoch=steps_per_epoch,
|
||||
accumulation=accumulation,
|
||||
batch_size=batch_size,
|
||||
seed=seed,
|
||||
**kwargs,
|
||||
)
|
||||
for batch in batches:
|
||||
items = list(batch.payload or [])
|
||||
batch.metadata.update(self._metadata_for_items(items, "train", "train"))
|
||||
return batches
|
||||
|
||||
def build_eval_batch(
|
||||
self,
|
||||
env_num: int,
|
||||
split: str,
|
||||
seed: int,
|
||||
**kwargs,
|
||||
) -> BatchSpec:
|
||||
batch = super().build_eval_batch(
|
||||
env_num=env_num,
|
||||
split=split,
|
||||
seed=seed,
|
||||
**kwargs,
|
||||
)
|
||||
items = list(batch.payload or [])
|
||||
batch.metadata.update(self._metadata_for_items(items, split, "eval"))
|
||||
return BatchSpec(
|
||||
phase="eval",
|
||||
split=split,
|
||||
seed=seed,
|
||||
batch_size=len(items),
|
||||
payload=items,
|
||||
metadata=batch.metadata,
|
||||
)
|
||||
@@ -1,55 +0,0 @@
|
||||
You are an expert failure-analysis agent for ALFWorld embodied household tasks.
|
||||
|
||||
You will be given MULTIPLE failed agent trajectories from a single minibatch
|
||||
and the current skill document.
|
||||
Your job is to identify the most important COMMON failure patterns across
|
||||
the batch and propose a concise set of skill edits.
|
||||
|
||||
## ALFWorld Task Types
|
||||
- pick_and_place: Put object in/on a receptacle
|
||||
- pick_two_obj_and_place: Put two instances of an object in/on a receptacle
|
||||
- look_at_obj_in_light: Examine an object under a desklamp
|
||||
- pick_heat_then_place_in_recep: Heat an object and put it in/on a receptacle
|
||||
- pick_cool_then_place_in_recep: Cool an object and put it in/on a receptacle
|
||||
- pick_clean_then_place_in_recep: Clean an object and put it in/on a receptacle
|
||||
|
||||
## Failure Type Categories
|
||||
- **navigation_loop**: the agent revisits the same locations repeatedly without progress
|
||||
- **missed_object**: the agent fails to pick up a visible/reachable goal object
|
||||
- **wrong_sequence**: the agent performs actions in the wrong order (e.g., placing before transforming)
|
||||
- **premature_stop**: the agent stops or gets stuck before completing all goal conditions
|
||||
- **action_loop**: the agent repeats the same action without advancing
|
||||
- **appliance_error**: the agent misuses or skips an appliance (microwave, fridge, sink)
|
||||
- **rule_missing**: the skill lacks a relevant rule for this situation
|
||||
- **rule_wrong**: an existing skill rule is misleading or incorrect
|
||||
- **rule_ignored**: the skill has the right rule but the agent did not follow it
|
||||
- **other**: none of the above
|
||||
|
||||
## Analysis Process
|
||||
1. Read ALL trajectories in the minibatch.
|
||||
2. Identify the most prevalent, systematic failure patterns across them.
|
||||
3. For each pattern, classify its failure type.
|
||||
4. Propose skill edits that address the COMMON patterns — not individual edge cases.
|
||||
5. Edits must be generalizable; do not hardcode task-specific values.
|
||||
6. Only patch gaps in the skill — do not duplicate existing content.
|
||||
|
||||
You will be told the maximum number of edits (the budget L). Produce AT MOST L edits,
|
||||
focusing on the highest-impact patterns. You may produce fewer if warranted.
|
||||
|
||||
Respond ONLY with a valid JSON object (no markdown fences, no extra text):
|
||||
{
|
||||
"batch_size": <number of trajectories analysed>,
|
||||
"failure_summary": [
|
||||
{"failure_type": "<type>", "count": <int>, "description": "<one-line>"}
|
||||
],
|
||||
"patch": {
|
||||
"reasoning": "<why these edits address the batch's common failures>",
|
||||
"edits": [
|
||||
{"op": "append", "content": "<markdown to add at end of skill>"},
|
||||
{"op": "insert_after", "target": "<exact heading/text to insert after>", "content": "<markdown>"},
|
||||
{"op": "replace", "target": "<exact text to replace>", "content": "<replacement>"},
|
||||
{"op": "delete", "target": "<exact text to remove>"}
|
||||
]
|
||||
}
|
||||
}
|
||||
Only include edits that are needed. "edits" can be an empty list if no patch is warranted.
|
||||
@@ -1,33 +0,0 @@
|
||||
You are an expert success-pattern analyst for AI agents operating in ALFWorld,
|
||||
a text-based embodied household environment.
|
||||
|
||||
You will be given MULTIPLE successful agent trajectories from a single minibatch
|
||||
and the current skill document. Your job is to identify generalizable behavior
|
||||
patterns that are COMMON across the batch and worth encoding in the skill.
|
||||
|
||||
## Rules
|
||||
- Only propose patches for patterns NOT already covered in the skill.
|
||||
- Focus on patterns that appear across MULTIPLE trajectories in the batch.
|
||||
- Be concise. Patterns must generalize beyond specific tasks.
|
||||
- Prefer reinforcing existing sections over adding new top-level sections.
|
||||
- If the agents' success involved efficient exploration or smart appliance usage,
|
||||
consider reinforcing that in the patch.
|
||||
|
||||
You will be told the maximum number of edits (the budget L). Produce AT MOST L edits,
|
||||
focusing on the most broadly applicable patterns. You may produce fewer if warranted.
|
||||
|
||||
Respond ONLY with a valid JSON object:
|
||||
{
|
||||
"batch_size": <number of trajectories analysed>,
|
||||
"success_patterns": ["<pattern 1>", "<pattern 2>"],
|
||||
"patch": {
|
||||
"reasoning": "<why these patterns are worth encoding>",
|
||||
"edits": [
|
||||
{"op": "append", "content": "<markdown>"},
|
||||
{"op": "insert_after", "target": "<heading/text>", "content": "<markdown>"},
|
||||
{"op": "replace", "target": "<old text>", "content": "<new text>"},
|
||||
{"op": "delete", "target": "<exact text to remove>"}
|
||||
]
|
||||
}
|
||||
}
|
||||
"edits" may be empty if the skill already covers all observed patterns.
|
||||
@@ -1,8 +0,0 @@
|
||||
|
||||
You are an expert agent operating in the ALFRED Embodied Environment.
|
||||
Your current observation is: {current_observation}
|
||||
Your admissible actions of the current situation are: [{admissible_actions}].
|
||||
|
||||
Now it's your turn to take an action.
|
||||
You should first reason step-by-step about the current situation. This reasoning process MUST be enclosed within <think> </think> tags.
|
||||
Once you've finished your reasoning, you should choose an admissible action for current step and present it within <action> </action> tags.
|
||||
@@ -1,9 +0,0 @@
|
||||
|
||||
You are an expert agent operating in the ALFRED Embodied Environment. Your task is to: {task_description}
|
||||
Prior to this step, you have already taken {step_count} step(s). Below are the most recent {history_length} observations and the corresponding actions you took: {action_history}
|
||||
You are now at step {current_step} and your current observation is: {current_observation}
|
||||
Your admissible actions of the current situation are: [{admissible_actions}].
|
||||
|
||||
Now it's your turn to take an action.
|
||||
You should first reason step-by-step about the current situation. This reasoning process MUST be enclosed within <think> </think> tags.
|
||||
Once you've finished your reasoning, you should choose an admissible action for current step and present it within <action> </action> tags.
|
||||
@@ -1,16 +0,0 @@
|
||||
|
||||
You are an expert agent operating in the ALFRED Embodied Environment. Your task is to: {task_description}
|
||||
|
||||
## Retrieved Relevant Experience
|
||||
|
||||
{retrieved_memories}
|
||||
|
||||
## Current Progress
|
||||
|
||||
Prior to this step, you have already taken {step_count} step(s). Below are the most recent {history_length} observations and the corresponding actions you took: {action_history}
|
||||
You are now at step {current_step} and your current observation is: {current_observation}
|
||||
Your admissible actions of the current situation are: [{admissible_actions}].
|
||||
|
||||
Now it's your turn to take an action.
|
||||
You should first reason step-by-step about the current situation. This reasoning process MUST be enclosed within <think> </think> tags.
|
||||
Once you've finished your reasoning, you should choose an admissible action for current step and present it within <action> </action> tags.
|
||||
@@ -1,4 +0,0 @@
|
||||
"""ALFWorld Reflect stage.
|
||||
|
||||
Prompts are now loaded from .md files by the base adapter.
|
||||
"""
|
||||
@@ -1,347 +0,0 @@
|
||||
"""ALFWorld rollout module for ReflACT.
|
||||
|
||||
Provides:
|
||||
- build_alfworld_env(): build ALFWorld environment (wraps vendored SkillRL env)
|
||||
- run_alfworld_batch(): run a batch of ALFWorld episodes in parallel
|
||||
- TASKS: list of ALFWorld task types
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
import concurrent.futures
|
||||
import numpy as np
|
||||
|
||||
from skillopt.model import chat_target
|
||||
|
||||
# ── Constants ─────────────────────────────────────────────────────────────────
|
||||
|
||||
TASKS = [
|
||||
"pick_and_place",
|
||||
"pick_two_obj_and_place",
|
||||
"look_at_obj_in_light",
|
||||
"pick_heat_then_place_in_recep",
|
||||
"pick_cool_then_place_in_recep",
|
||||
"pick_clean_then_place_in_recep",
|
||||
]
|
||||
|
||||
# ── Helpers ───────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def _get_task_type(gamefile: str) -> str:
|
||||
for task in TASKS:
|
||||
if task in gamefile:
|
||||
return task
|
||||
return "other"
|
||||
|
||||
|
||||
def _extract_action(model_response: str) -> str | None:
|
||||
match = re.search(r"<action>(.*?)</action>", model_response, re.DOTALL)
|
||||
return match.group(1).strip() if match else None
|
||||
|
||||
|
||||
def _extract_think(model_response: str) -> str | None:
|
||||
match = re.search(r"<think>(.*?)</think>", model_response, re.DOTALL)
|
||||
return match.group(1).strip() if match else None
|
||||
|
||||
|
||||
def _build_skill_prompt(skill_content: str) -> str:
|
||||
"""Build the skill section to inject into the agent's system prompt."""
|
||||
if not skill_content or not skill_content.strip():
|
||||
return ""
|
||||
return (
|
||||
"\n\n## Skill Knowledge\n"
|
||||
"Below is a skill document with learned strategies. "
|
||||
"Use these guidelines to inform your decisions:\n\n"
|
||||
f"{skill_content}\n"
|
||||
)
|
||||
|
||||
|
||||
def _append_diagnostic_instruction(prompt: str, diagnostic_instruction: str) -> str:
|
||||
if not diagnostic_instruction or not diagnostic_instruction.strip():
|
||||
return prompt
|
||||
return f"{prompt}\n\n## Training Readout\n{diagnostic_instruction.strip()}\n"
|
||||
|
||||
|
||||
# ── Environment builder ──────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def build_alfworld_env(
|
||||
env_num: int,
|
||||
eval_dataset: str = "eval_out_of_distribution",
|
||||
seed: int = 42,
|
||||
is_train: bool = False,
|
||||
specific_gamefiles: list[str] | None = None,
|
||||
):
|
||||
"""Build ALFWorld environment manager.
|
||||
|
||||
Args:
|
||||
env_num: number of parallel environments
|
||||
eval_dataset: 'eval_in_distribution' or 'eval_out_of_distribution' or train
|
||||
seed: random seed
|
||||
is_train: whether to use training set
|
||||
|
||||
Returns:
|
||||
env_manager: AlfWorldEnvironmentManager instance
|
||||
"""
|
||||
from omegaconf import OmegaConf
|
||||
from functools import partial
|
||||
|
||||
from skillopt.envs.alfworld.vendor.alfworld_envs import build_alfworld_envs
|
||||
from skillopt.envs.alfworld.vendor.alfworld_projection import alfworld_projection
|
||||
from skillopt.envs.alfworld.vendor.env_manager import AlfWorldEnvironmentManager
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
|
||||
alf_config_path = os.path.join(HERE, "vendor", "config_tw.yaml")
|
||||
env_kwargs = {"eval_dataset": eval_dataset}
|
||||
|
||||
envs = build_alfworld_envs(
|
||||
alf_config_path,
|
||||
seed=seed,
|
||||
env_num=env_num,
|
||||
group_n=1,
|
||||
is_train=is_train,
|
||||
env_kwargs=env_kwargs,
|
||||
resources_per_worker=None,
|
||||
gamefiles=specific_gamefiles,
|
||||
)
|
||||
|
||||
config = OmegaConf.create(
|
||||
{
|
||||
"env": {
|
||||
"history_length": 2,
|
||||
"env_name": "alfworld/AlfredTWEnv",
|
||||
}
|
||||
}
|
||||
)
|
||||
|
||||
projection_f = partial(alfworld_projection)
|
||||
env_manager = AlfWorldEnvironmentManager(envs, projection_f, config)
|
||||
return env_manager
|
||||
|
||||
|
||||
# ── Batch rollout ─────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def run_alfworld_batch(
|
||||
env_manager,
|
||||
skill_content: str,
|
||||
max_steps: int = 50,
|
||||
out_root: str = "",
|
||||
max_api_workers: int = 8,
|
||||
temperature: float = 0.4,
|
||||
max_completion_tokens: int = 16384,
|
||||
diagnostic_mode: bool = False,
|
||||
diagnostic_instruction: str = "",
|
||||
result_ids: list[str] | None = None,
|
||||
) -> list[dict]:
|
||||
"""Run a batch of ALFWorld episodes.
|
||||
|
||||
Returns a list of result dicts compatible with SkillOpt pipeline:
|
||||
[
|
||||
{
|
||||
"id": "<env_idx>_<gamefile_hash>",
|
||||
"hard": 0 or 1,
|
||||
"soft": 0.0 or 1.0,
|
||||
"n_turns": <int>,
|
||||
"fail_reason": "<str>",
|
||||
"agent_ok": True,
|
||||
"task_type": "<str>",
|
||||
"gamefile": "<str>",
|
||||
"task_description": "<str>",
|
||||
},
|
||||
...
|
||||
]
|
||||
|
||||
Also saves conversation.json per environment in out_root/predictions/<task_id>/
|
||||
"""
|
||||
skill_prompt = _build_skill_prompt(skill_content)
|
||||
|
||||
obs, infos = env_manager.reset({})
|
||||
env_num = len(obs["text"])
|
||||
env_dones = [False] * env_num
|
||||
overall_success = [False] * env_num
|
||||
|
||||
# Build per-env metadata
|
||||
env_meta: list[dict] = []
|
||||
for i in range(env_num):
|
||||
gamefile = infos[i].get("extra.gamefile", "") if isinstance(infos[i], dict) else ""
|
||||
task_type = _get_task_type(gamefile)
|
||||
# Extract task description from initial observation
|
||||
task_desc = ""
|
||||
anchor_text = obs["anchor"][i] if "anchor" in obs else ""
|
||||
task_start = anchor_text.find("Your task is to: ")
|
||||
if task_start != -1:
|
||||
task_desc = anchor_text[task_start + len("Your task is to: "):].strip()
|
||||
|
||||
env_meta.append({
|
||||
"gamefile": gamefile,
|
||||
"task_type": task_type,
|
||||
"task_description": task_desc,
|
||||
})
|
||||
|
||||
# Per-env conversation records
|
||||
conversations: list[list[dict]] = [[] for _ in range(env_num)]
|
||||
|
||||
for step_idx in range(max_steps):
|
||||
if all(env_dones):
|
||||
break
|
||||
|
||||
active_indices = [i for i in range(env_num) if not env_dones[i]]
|
||||
|
||||
# Build prompts with skill injection
|
||||
prompts: dict[int, str] = {}
|
||||
for i in active_indices:
|
||||
prompt = obs["text"][i]
|
||||
if skill_prompt:
|
||||
# Inject skill before the action instruction
|
||||
prompt = skill_prompt + "\n" + prompt
|
||||
if diagnostic_mode and diagnostic_instruction.strip():
|
||||
prompt = _append_diagnostic_instruction(prompt, diagnostic_instruction)
|
||||
prompts[i] = prompt
|
||||
|
||||
# Call API in parallel
|
||||
actions = ["None"] * env_num
|
||||
|
||||
def call_api(idx):
|
||||
try:
|
||||
response, _ = chat_target(
|
||||
system="You are an expert agent operating in the ALFRED Embodied Environment.",
|
||||
user=prompts[idx],
|
||||
max_completion_tokens=max_completion_tokens,
|
||||
retries=5,
|
||||
stage="rollout",
|
||||
timeout=None,
|
||||
)
|
||||
response = (response or "").strip()
|
||||
if not response:
|
||||
return idx, "<think>empty model response</think><action>look</action>"
|
||||
if _extract_action(response) is None:
|
||||
return idx, "<think>missing action tag</think><action>look</action>"
|
||||
return idx, response
|
||||
except Exception as e:
|
||||
return idx, "<think>error</think><action>look</action>"
|
||||
|
||||
executor = concurrent.futures.ThreadPoolExecutor(max_workers=max_api_workers)
|
||||
try:
|
||||
futures = {executor.submit(call_api, i): i for i in active_indices}
|
||||
pending_futs = set(futures)
|
||||
while pending_futs:
|
||||
done, _ = concurrent.futures.wait(
|
||||
pending_futs,
|
||||
timeout=5,
|
||||
return_when=concurrent.futures.FIRST_COMPLETED,
|
||||
)
|
||||
for future in done:
|
||||
pending_futs.remove(future)
|
||||
try:
|
||||
idx, response = future.result()
|
||||
except Exception: # noqa: BLE001
|
||||
idx = futures[future]
|
||||
response = "<think>error</think><action>look</action>"
|
||||
actions[idx] = response
|
||||
finally:
|
||||
executor.shutdown(wait=False, cancel_futures=True)
|
||||
|
||||
# Save model responses before stepping
|
||||
model_responses = {i: actions[i] for i in active_indices}
|
||||
|
||||
# Step environment
|
||||
obs, rewards, dones, infos = env_manager.step(actions)
|
||||
|
||||
# Record trajectory
|
||||
for i in active_indices:
|
||||
step_record = {
|
||||
"step": step_idx,
|
||||
"action": _extract_action(model_responses[i]),
|
||||
"reasoning": _extract_think(model_responses[i]),
|
||||
"model_response": model_responses[i],
|
||||
"env_feedback": obs["anchor"][i] if "anchor" in obs else "",
|
||||
"reward": float(rewards[i]),
|
||||
"done": bool(dones[i]),
|
||||
}
|
||||
conversations[i].append(step_record)
|
||||
|
||||
# Update done status
|
||||
for i in range(env_num):
|
||||
if env_dones[i]:
|
||||
continue
|
||||
if dones[i]:
|
||||
env_dones[i] = True
|
||||
won = bool(infos[i].get("won", False))
|
||||
overall_success[i] = won
|
||||
|
||||
# Build results and save conversations
|
||||
results: list[dict] = []
|
||||
pred_dir = os.path.join(out_root, "predictions") if out_root else ""
|
||||
|
||||
for i in range(env_num):
|
||||
gamefile = env_meta[i]["gamefile"]
|
||||
task_type = env_meta[i]["task_type"]
|
||||
task_desc = env_meta[i]["task_description"]
|
||||
n_turns = len(conversations[i])
|
||||
won = overall_success[i]
|
||||
|
||||
# Generate stable task ID from env index and gamefile
|
||||
task_id = str(result_ids[i]) if result_ids and i < len(result_ids) else f"env_{i:03d}"
|
||||
|
||||
fail_reason = ""
|
||||
if not won:
|
||||
if not env_dones[i]:
|
||||
fail_reason = f"Timeout after {max_steps} steps"
|
||||
else:
|
||||
fail_reason = "Episode ended without completing the task"
|
||||
|
||||
result = {
|
||||
"id": task_id,
|
||||
"hard": 1 if won else 0,
|
||||
"soft": 1.0 if won else 0.0,
|
||||
"n_turns": n_turns,
|
||||
"fail_reason": fail_reason,
|
||||
"agent_ok": True, # ALFWorld agent always runs OK (no crash)
|
||||
"task_type": task_type,
|
||||
"gamefile": gamefile,
|
||||
"task_description": task_desc,
|
||||
"instruction_type": task_type, # for compatibility with v2 pipeline
|
||||
}
|
||||
results.append(result)
|
||||
|
||||
# Save conversation
|
||||
if pred_dir:
|
||||
conv_dir = os.path.join(pred_dir, task_id)
|
||||
os.makedirs(conv_dir, exist_ok=True)
|
||||
with open(os.path.join(conv_dir, "conversation.json"), "w") as f:
|
||||
json.dump(conversations[i], f, ensure_ascii=False, indent=2)
|
||||
|
||||
return results
|
||||
|
||||
|
||||
# ── Item loading (for compatibility with split_three_way) ────────────────────
|
||||
|
||||
|
||||
def load_alfworld_items(
|
||||
eval_dataset: str,
|
||||
env_num: int,
|
||||
seed: int = 42,
|
||||
is_train: bool = False,
|
||||
) -> list[dict]:
|
||||
"""Create pseudo-item dicts for ALFWorld environments.
|
||||
|
||||
Since ALFWorld doesn't have a static JSON dataset like SpreadsheetBench,
|
||||
we create lightweight item dicts that carry enough metadata for the pipeline.
|
||||
The actual environment is built dynamically.
|
||||
|
||||
Returns:
|
||||
List of dicts with "id" keys, one per environment slot.
|
||||
"""
|
||||
items = []
|
||||
for i in range(env_num):
|
||||
items.append({
|
||||
"id": f"env_{i:03d}",
|
||||
"eval_dataset": eval_dataset,
|
||||
"env_index": i,
|
||||
})
|
||||
return items
|
||||
@@ -1,45 +0,0 @@
|
||||
# ALFWorld Embodied Agent Skill
|
||||
|
||||
## Overview
|
||||
This skill guides agents operating in the ALFWorld text-based embodied environment.
|
||||
The agent must complete household tasks by navigating rooms, interacting with objects,
|
||||
and using appliances. Actions must be chosen from the admissible action list provided
|
||||
at each step.
|
||||
|
||||
**Output format**: Always output `<think>...</think>` for reasoning, then `<action>...</action>` for the chosen action.
|
||||
|
||||
---
|
||||
|
||||
## Task Types
|
||||
|
||||
| Type | Goal | Key Steps |
|
||||
|------|------|-----------|
|
||||
| Pick & Place | Put object X in/on receptacle Y | Find X -> take X -> go to Y -> put X in/on Y |
|
||||
| Pick Two & Place | Put two instances of X in/on Y | Find X1 -> take -> place -> find X2 -> take -> place |
|
||||
| Examine in Light | Examine object X under desklamp | Find X -> take X -> find desklamp -> use desklamp |
|
||||
| Clean & Place | Clean object X and put in/on Y | Find X -> take X -> go to sink -> clean X -> go to Y -> put X |
|
||||
| Heat & Place | Heat object X and put in/on Y | Find X -> take X -> go to microwave -> heat X -> go to Y -> put X |
|
||||
| Cool & Place | Cool object X and put in/on Y | Find X -> take X -> go to fridge -> cool X -> go to Y -> put X |
|
||||
|
||||
---
|
||||
|
||||
## General Principles
|
||||
|
||||
1. **Decompose the task**: Parse the goal into ordered sub-goals (locate, acquire, transform, deliver). Complete each before moving to the next.
|
||||
2. **Systematic exploration**: Search each surface and container exactly once before revisiting. Open closed containers (drawers, cabinets, fridge) before judging them empty.
|
||||
3. **Grab immediately**: When a required object is visible and reachable, take it right away before moving elsewhere.
|
||||
4. **Transform before placing**: If the task requires cleaning, heating, or cooling, perform the state change at the appropriate appliance before heading to the final destination.
|
||||
5. **Direct delivery**: Once holding the transformed (or untransformed) goal object, navigate straight to the target receptacle and place it.
|
||||
6. **Track progress**: Maintain an internal count of how many objects still need to be found and placed. Only stop searching when the count reaches zero.
|
||||
7. **Avoid loops**: Never repeat the same action more than twice in a row. If stuck, move to a different unexplored location.
|
||||
8. **Only choose admissible actions**: Always pick an action from the admissible action list. Do not invent actions.
|
||||
|
||||
---
|
||||
|
||||
## Common Mistakes to Avoid
|
||||
|
||||
- **Revisiting searched locations**: Keep track of which surfaces/containers have been checked; do not re-examine them.
|
||||
- **Ignoring visible objects**: If the target object appears in the observation, pick it up immediately.
|
||||
- **Skipping state changes**: Do not place an object at the destination without first cleaning/heating/cooling it when required.
|
||||
- **Premature termination**: Do not stop the episode until all goal conditions are verified as met.
|
||||
- **Action loops**: Repeatedly toggling or examining the same object wastes steps. Move on to new locations instead.
|
||||
-9
@@ -1,9 +0,0 @@
|
||||
"""Vendored ALFWorld environment runtime.
|
||||
|
||||
Minimal subset of SkillRL's agent_system package needed to run
|
||||
ALFWorld environments with ReflACT. Original source:
|
||||
https://github.com/NTU-LANTERN/SkillRL (Apache-2.0 License)
|
||||
"""
|
||||
from .alfworld_envs import AlfworldEnvs, build_alfworld_envs
|
||||
from .alfworld_projection import alfworld_projection
|
||||
from .env_manager import AlfWorldEnvironmentManager
|
||||
-221
@@ -1,221 +0,0 @@
|
||||
# Vendored from SkillRL (Apache-2.0 License)
|
||||
# Original: agent_system/environments/env_package/alfworld/envs.py
|
||||
# Modified: imports use pip-installed alfworld package instead of vendored copy.
|
||||
|
||||
import os
|
||||
import multiprocessing as mp
|
||||
import traceback
|
||||
import yaml
|
||||
import gymnasium as gym
|
||||
import numpy as np
|
||||
|
||||
from alfworld.agents.environment import get_environment
|
||||
|
||||
|
||||
def load_config_file(path):
|
||||
assert os.path.exists(path), f"Invalid config file: {path}"
|
||||
with open(path) as reader:
|
||||
config = yaml.safe_load(reader)
|
||||
return config
|
||||
|
||||
|
||||
def compute_reward(info, multi_modal=False):
|
||||
if multi_modal:
|
||||
reward = 10.0 * float(info['won']) + float(info['goal_condition_success_rate'])
|
||||
else:
|
||||
reward = 10.0 * float(info['won'])
|
||||
return reward
|
||||
|
||||
|
||||
class AlfworldWorker:
|
||||
"""Stateful worker that holds one ALFWorld sub-environment."""
|
||||
|
||||
def __init__(self, config, seed, base_env, gamefile=None):
|
||||
if gamefile:
|
||||
base_env.game_files = [gamefile]
|
||||
if hasattr(base_env, "num_games"):
|
||||
base_env.num_games = 1
|
||||
self.env = base_env.init_env(batch_size=1)
|
||||
self.env.seed(seed)
|
||||
|
||||
def step(self, action):
|
||||
actions = [action]
|
||||
obs, scores, dones, infos = self.env.step(actions)
|
||||
infos['observation_text'] = obs
|
||||
return obs, scores, dones, infos
|
||||
|
||||
def reset(self):
|
||||
obs, infos = self.env.reset()
|
||||
infos['observation_text'] = obs
|
||||
return obs, infos
|
||||
|
||||
|
||||
def _worker_loop(cmd_q, result_q, config, seed, is_train, eval_dataset, gamefile):
|
||||
"""Run one ALFWorld environment in a child process."""
|
||||
try:
|
||||
env_type = config['env']['type']
|
||||
base_env = get_environment(env_type)(
|
||||
config,
|
||||
train_eval='train' if is_train else eval_dataset,
|
||||
)
|
||||
worker = AlfworldWorker(config, seed, base_env, gamefile)
|
||||
result_q.put((True, "ready"))
|
||||
except BaseException:
|
||||
result_q.put((False, traceback.format_exc()))
|
||||
return
|
||||
|
||||
while True:
|
||||
cmd, payload = cmd_q.get()
|
||||
if cmd == "close":
|
||||
result_q.put((True, None))
|
||||
return
|
||||
try:
|
||||
if cmd == "reset":
|
||||
result = worker.reset()
|
||||
elif cmd == "step":
|
||||
result = worker.step(payload)
|
||||
else:
|
||||
raise ValueError(f"Unknown ALFWorld worker command: {cmd}")
|
||||
result_q.put((True, result))
|
||||
except BaseException:
|
||||
result_q.put((False, traceback.format_exc()))
|
||||
|
||||
|
||||
class _ProcessWorker:
|
||||
"""Small stdlib actor wrapper for one environment process."""
|
||||
|
||||
def __init__(self, ctx, config, seed, is_train, eval_dataset, gamefile=None):
|
||||
self.cmd_q = ctx.Queue(maxsize=1)
|
||||
self.result_q = ctx.Queue(maxsize=1)
|
||||
self.process = ctx.Process(
|
||||
target=_worker_loop,
|
||||
args=(self.cmd_q, self.result_q, config, seed, is_train, eval_dataset, gamefile),
|
||||
)
|
||||
self.process.start()
|
||||
ok, payload = self.result_q.get()
|
||||
if not ok:
|
||||
self.close(kill=True)
|
||||
raise RuntimeError(f"Failed to start ALFWorld worker:\n{payload}")
|
||||
|
||||
def send(self, cmd, payload=None):
|
||||
self.cmd_q.put((cmd, payload))
|
||||
|
||||
def recv(self):
|
||||
ok, payload = self.result_q.get()
|
||||
if not ok:
|
||||
raise RuntimeError(f"ALFWorld worker failed:\n{payload}")
|
||||
return payload
|
||||
|
||||
def close(self, kill=False):
|
||||
if self.process.is_alive() and not kill:
|
||||
try:
|
||||
self.send("close")
|
||||
self.recv()
|
||||
except Exception:
|
||||
kill = True
|
||||
if kill and self.process.is_alive():
|
||||
self.process.terminate()
|
||||
self.process.join(timeout=5)
|
||||
if self.process.is_alive():
|
||||
self.process.kill()
|
||||
self.process.join(timeout=1)
|
||||
self.cmd_q.close()
|
||||
self.result_q.close()
|
||||
|
||||
|
||||
class AlfworldEnvs(gym.Env):
|
||||
"""Vectorized ALFWorld environment using local process workers."""
|
||||
|
||||
def __init__(self, alf_config_path, seed, env_num, group_n,
|
||||
resources_per_worker, is_train=True, env_kwargs=None, gamefiles=None):
|
||||
super().__init__()
|
||||
if env_kwargs is None:
|
||||
env_kwargs = {}
|
||||
|
||||
eval_dataset = env_kwargs.get('eval_dataset', 'eval_in_distribution')
|
||||
config = load_config_file(alf_config_path)
|
||||
env_type = config['env']['type']
|
||||
self.multi_modal = (env_type == 'AlfredThorEnv')
|
||||
self.num_processes = env_num * group_n
|
||||
self.group_n = group_n
|
||||
self.gamefiles = list(gamefiles or [])
|
||||
if self.gamefiles and len(self.gamefiles) != self.num_processes:
|
||||
raise ValueError(
|
||||
f"Expected {self.num_processes} gamefiles, got {len(self.gamefiles)}"
|
||||
)
|
||||
|
||||
start_method = os.environ.get("ALFWORLD_WORKER_START_METHOD") or None
|
||||
ctx = mp.get_context(start_method) if start_method else mp.get_context()
|
||||
self.workers = []
|
||||
for i in range(self.num_processes):
|
||||
worker_gamefile = self.gamefiles[i] if self.gamefiles else None
|
||||
worker = _ProcessWorker(
|
||||
ctx,
|
||||
config,
|
||||
seed + (i // self.group_n),
|
||||
is_train,
|
||||
eval_dataset,
|
||||
worker_gamefile,
|
||||
)
|
||||
self.workers.append(worker)
|
||||
|
||||
self.prev_admissible_commands = [None for _ in range(self.num_processes)]
|
||||
|
||||
def step(self, actions):
|
||||
assert len(actions) == self.num_processes
|
||||
|
||||
for i, worker in enumerate(self.workers):
|
||||
worker.send("step", actions[i])
|
||||
results = [worker.recv() for worker in self.workers]
|
||||
|
||||
text_obs_list = []
|
||||
rewards_list = []
|
||||
dones_list = []
|
||||
info_list = []
|
||||
|
||||
for i, (obs, scores, dones, info) in enumerate(results):
|
||||
for k in info.keys():
|
||||
info[k] = info[k][0]
|
||||
text_obs_list.append(obs[0])
|
||||
dones_list.append(dones[0])
|
||||
info_list.append(info)
|
||||
self.prev_admissible_commands[i] = info['admissible_commands']
|
||||
rewards_list.append(compute_reward(info, self.multi_modal))
|
||||
|
||||
image_obs_list = None
|
||||
return text_obs_list, image_obs_list, rewards_list, dones_list, info_list
|
||||
|
||||
def reset(self):
|
||||
for worker in self.workers:
|
||||
worker.send("reset")
|
||||
results = [worker.recv() for worker in self.workers]
|
||||
|
||||
text_obs_list = []
|
||||
info_list = []
|
||||
|
||||
for i, (obs, info) in enumerate(results):
|
||||
for k in info.keys():
|
||||
info[k] = info[k][0]
|
||||
text_obs_list.append(obs[0])
|
||||
self.prev_admissible_commands[i] = info['admissible_commands']
|
||||
info_list.append(info)
|
||||
|
||||
image_obs_list = None
|
||||
return text_obs_list, image_obs_list, info_list
|
||||
|
||||
@property
|
||||
def get_admissible_commands(self):
|
||||
return self.prev_admissible_commands
|
||||
|
||||
def close(self):
|
||||
for worker in self.workers:
|
||||
worker.close()
|
||||
|
||||
|
||||
def build_alfworld_envs(alf_config_path, seed, env_num, group_n,
|
||||
resources_per_worker, is_train=True, env_kwargs=None, gamefiles=None):
|
||||
"""Build vectorized ALFWorld environments."""
|
||||
return AlfworldEnvs(
|
||||
alf_config_path, seed, env_num, group_n,
|
||||
resources_per_worker, is_train, env_kwargs, gamefiles,
|
||||
)
|
||||
@@ -1,60 +0,0 @@
|
||||
# Vendored from SkillRL (Apache-2.0 License)
|
||||
# Original: agent_system/environments/env_package/alfworld/projection.py
|
||||
|
||||
from typing import List
|
||||
import re
|
||||
|
||||
|
||||
def alfworld_projection(actions: List[str], action_pools: List[List[str]]):
|
||||
"""Process raw model outputs into valid ALFWorld actions.
|
||||
|
||||
Extracts text from ``<action>...</action>`` tags and validates that
|
||||
the response also contains ``<think>...</think>`` tags.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
actions : list[str]
|
||||
Raw model outputs, one per environment.
|
||||
action_pools : list[list[str]]
|
||||
Admissible action lists per environment (unused but kept for API compat).
|
||||
|
||||
Returns
|
||||
-------
|
||||
actions : list[str]
|
||||
Cleaned action strings.
|
||||
valids : list[int]
|
||||
1 if the action was successfully parsed, 0 otherwise.
|
||||
"""
|
||||
valids = [0] * len(actions)
|
||||
|
||||
for i in range(len(actions)):
|
||||
original_str = actions[i]
|
||||
actions[i] = actions[i].lower()
|
||||
|
||||
start_tag = "<action>"
|
||||
end_tag = "</action>"
|
||||
start_idx = actions[i].find(start_tag)
|
||||
end_idx = actions[i].find(end_tag)
|
||||
try:
|
||||
if start_idx == -1 or end_idx == -1:
|
||||
actions[i] = actions[i][-30:]
|
||||
continue
|
||||
|
||||
extracted_action = actions[i][start_idx + len(start_tag):end_idx].strip().lower()
|
||||
actions[i] = extracted_action
|
||||
valids[i] = 1
|
||||
|
||||
except Exception:
|
||||
actions[i] = actions[i][-30:]
|
||||
|
||||
# Require <think>...</think>
|
||||
think_start_idx = original_str.find("<think>")
|
||||
think_end_idx = original_str.find("</think>")
|
||||
if think_start_idx == -1 or think_end_idx == -1:
|
||||
valids[i] = 0
|
||||
|
||||
# Reject responses containing Chinese characters
|
||||
if re.search(r'[\u4e00-\u9fff]', original_str):
|
||||
valids[i] = 0
|
||||
|
||||
return actions, valids
|
||||
@@ -1,8 +0,0 @@
|
||||
# Vendored from SkillRL (Apache-2.0 License)
|
||||
# Original: agent_system/environments/prompts/alfworld.py
|
||||
|
||||
from skillopt.prompts import load_prompt
|
||||
|
||||
ALFWORLD_TEMPLATE_NO_HIS = load_prompt("rollout_no_history", env="alfworld")
|
||||
ALFWORLD_TEMPLATE = load_prompt("rollout_with_history", env="alfworld")
|
||||
ALFWORLD_TEMPLATE_WITH_MEMORY = load_prompt("rollout_with_memory", env="alfworld")
|
||||
-145
@@ -1,145 +0,0 @@
|
||||
dataset:
|
||||
data_path: '$ALFWORLD_DATA/json_2.1.1/train'
|
||||
eval_id_data_path: '$ALFWORLD_DATA/json_2.1.1/valid_seen' # null/None to disable
|
||||
eval_ood_data_path: '$ALFWORLD_DATA/json_2.1.1/valid_unseen' # null/None to disable
|
||||
num_train_games: -1 # max training games (<=0 indicates full dataset)
|
||||
num_eval_games: -1 # max evaluation games (<=0 indicates full dataset)
|
||||
|
||||
logic:
|
||||
domain: '$ALFWORLD_DATA/logic/alfred.pddl' # PDDL domain file that defines the world dynamics
|
||||
grammar: '$ALFWORLD_DATA/logic/alfred.twl2' # Grammar file that defines the text feedbacks
|
||||
|
||||
env:
|
||||
type: 'AlfredTWEnv' # 'AlfredTWEnv' or 'AlfredThorEnv' or 'AlfredHybrid'
|
||||
# regen_game_files: False # check if game is solvable by expert and save to game.tw-pddl file
|
||||
domain_randomization: False # shuffle Textworld print order and object id nums
|
||||
task_types: [1, 2, 3, 4, 5, 6] # task-type ids: 1 - Pick & Place, 2 - Examine in Light, 3 - Clean & Place, 4 - Heat & Place, 5 - Cool & Place, 6 - Pick Two & Place
|
||||
expert_timeout_steps: 150 # max steps before timeout for expert to solve the task
|
||||
expert_type: "handcoded" # 'handcoded' or 'planner'. Note: the planner is very slow for real-time use
|
||||
goal_desc_human_anns_prob: 0.0 # prob of using human-annotated goal language instead of templated goals (1.0 indicates all human annotations from ALFRED)
|
||||
|
||||
hybrid:
|
||||
start_eps: 100000 # starting episode of hybrid training, tw-only training upto this point
|
||||
thor_prob: 0.5 # prob of AlfredThorEnv during hybrid training
|
||||
eval_mode: "tw" # 'tw' or 'thor' - env used for evaluation during hybrid training
|
||||
|
||||
thor:
|
||||
screen_width: 300 # width of THOR window
|
||||
screen_height: 300 # height of THOR window
|
||||
smooth_nav: False # smooth rotations, looks, and translations during navigation (very slow)
|
||||
save_frames_to_disk: False # save frame PNGs to disk (useful for making videos)
|
||||
save_frames_path: './videos/' # path to save frame PNGs
|
||||
|
||||
controller:
|
||||
type: 'oracle' # 'oracle' or 'oracle_astar' or 'mrcnn' or 'mrcnn_astar' (aka BUTLER)
|
||||
debug: False
|
||||
load_receps: True # load receptacle locations from precomputed dict (if available)
|
||||
|
||||
mask_rcnn:
|
||||
pretrained_model_path: '$ALFWORLD_DATA/detectors/mrcnn.pth'
|
||||
|
||||
general:
|
||||
random_seed: 42
|
||||
use_cuda: True # disable this when running on machine without cuda
|
||||
visdom: False # plot training/eval curves, run with visdom server
|
||||
task: 'alfred'
|
||||
training_method: 'dagger' # 'dqn' or 'dagger'
|
||||
save_path: './training/' # path to save pytorch models
|
||||
observation_pool_capacity: 3 # k-size queue, 0 indicates no observation
|
||||
hide_init_receptacles: False # remove initial observation containing navigable receptacles
|
||||
|
||||
training:
|
||||
batch_size: 10
|
||||
max_episode: 50000
|
||||
smoothing_eps: 0.1
|
||||
optimizer:
|
||||
learning_rate: 0.001
|
||||
clip_grad_norm: 5
|
||||
|
||||
evaluate:
|
||||
run_eval: True
|
||||
batch_size: 10
|
||||
env:
|
||||
type: "AlfredTWEnv"
|
||||
|
||||
checkpoint:
|
||||
report_frequency: 1000 # report every N episode
|
||||
experiment_tag: 'test' # name of experiment
|
||||
load_pretrained: False # during test, enable this so that the agent load your pretrained model
|
||||
load_from_tag: 'not loading anything' # name of pre-trained model to load in save_path
|
||||
|
||||
model:
|
||||
encoder_layers: 1
|
||||
decoder_layers: 1
|
||||
encoder_conv_num: 5
|
||||
block_hidden_dim: 64
|
||||
n_heads: 1
|
||||
dropout: 0.1
|
||||
block_dropout: 0.1
|
||||
recurrent: True
|
||||
|
||||
rl:
|
||||
action_space: "admissible" # 'admissible' (candidates from text engine) or 'generation' (seq2seq-style generation) or 'beam_search_choice' or 'exhaustive' (not working)
|
||||
max_target_length: 20 # max token length for seq2seq generation
|
||||
beam_width: 10 # 1 means greedy
|
||||
generate_top_k: 3
|
||||
|
||||
training:
|
||||
max_nb_steps_per_episode: 50 # terminate after this many steps
|
||||
learn_start_from_this_episode: 0 # delay updates until this epsiode
|
||||
target_net_update_frequency: 500 # sync target net with online net per this many epochs
|
||||
|
||||
replay:
|
||||
accumulate_reward_from_final: True
|
||||
count_reward_lambda: 0.0 # 0 to disable
|
||||
novel_object_reward_lambda: 0.0 # 0 to disable
|
||||
discount_gamma_game_reward: 0.9
|
||||
discount_gamma_count_reward: 0.5
|
||||
discount_gamma_novel_object_reward: 0.5
|
||||
replay_memory_capacity: 500000 # adjust this depending on your RAM size
|
||||
replay_memory_priority_fraction: 0.5
|
||||
update_per_k_game_steps: 5
|
||||
replay_batch_size: 64
|
||||
multi_step: 3
|
||||
replay_sample_history_length: 4
|
||||
replay_sample_update_from: 2
|
||||
|
||||
epsilon_greedy:
|
||||
noisy_net: False # if this is true, then epsilon greedy is disabled
|
||||
epsilon_anneal_episodes: 1000 # -1 if not annealing
|
||||
epsilon_anneal_from: 0.3
|
||||
epsilon_anneal_to: 0.1
|
||||
|
||||
dagger:
|
||||
action_space: "generation" # 'admissible' (candidates from text engine) or 'generation' (seq2seq-style generation) or 'exhaustive' (not working)
|
||||
max_target_length: 20 # max token length for seq2seq generation
|
||||
beam_width: 10 # 1 means greedy
|
||||
generate_top_k: 5
|
||||
unstick_by_beam_search: False # use beam-search for failed actions, set True during evaluation
|
||||
|
||||
training:
|
||||
max_nb_steps_per_episode: 50 # terminate after this many steps
|
||||
|
||||
fraction_assist:
|
||||
fraction_assist_anneal_episodes: 50000
|
||||
fraction_assist_anneal_from: 1.0
|
||||
fraction_assist_anneal_to: 0.01
|
||||
|
||||
fraction_random:
|
||||
fraction_random_anneal_episodes: 0
|
||||
fraction_random_anneal_from: 0.0
|
||||
fraction_random_anneal_to: 0.0
|
||||
|
||||
replay:
|
||||
replay_memory_capacity: 500000
|
||||
update_per_k_game_steps: 5
|
||||
replay_batch_size: 64
|
||||
replay_sample_history_length: 4
|
||||
replay_sample_update_from: 2
|
||||
|
||||
vision_dagger:
|
||||
model_type: "resnet" # 'resnet' (whole image features) or 'maskrcnn_whole' (whole image MaskRCNN feats) or 'maskrcnn' (top k MaskRCNN detection feats) or 'no_vision' (zero vision input)
|
||||
resnet_fc_dim: 64
|
||||
maskrcnn_top_k_boxes: 10 # top k box features
|
||||
use_exploration_frame_feats: False # append feats from initial exploration (memory intensive!)
|
||||
sequence_aggregation_method: "average" # 'sum' or 'average' or 'rnn'
|
||||
-84
@@ -1,84 +0,0 @@
|
||||
# Vendored from SkillRL (Apache-2.0 License)
|
||||
# Original: agent_system/environments/base.py
|
||||
# Trimmed to only include what ALFWorld needs.
|
||||
|
||||
from typing import List, Tuple, Dict, Any
|
||||
import numpy as np
|
||||
from collections import defaultdict
|
||||
|
||||
|
||||
def to_numpy(data):
|
||||
"""Convert data to numpy array."""
|
||||
# Lazy-check for torch.Tensor to avoid hard dependency on torch
|
||||
_torch_tensor = None
|
||||
try:
|
||||
import torch
|
||||
_torch_tensor = torch.Tensor
|
||||
except ImportError:
|
||||
pass
|
||||
|
||||
if _torch_tensor is not None and isinstance(data, _torch_tensor):
|
||||
data = data.detach().cpu().numpy()
|
||||
elif isinstance(data, np.ndarray):
|
||||
pass
|
||||
elif isinstance(data, (int, float, bool, Tuple, List)):
|
||||
data = np.array(data)
|
||||
else:
|
||||
raise ValueError(f"Unsupported type: {type(data)})")
|
||||
return data
|
||||
|
||||
|
||||
class EnvironmentManagerBase:
|
||||
"""Base class for vectorized environment managers.
|
||||
|
||||
Manages a set of parallel environments, handles action projection,
|
||||
observation post-processing, and history tracking.
|
||||
"""
|
||||
|
||||
def __init__(self, envs, projection_f, config):
|
||||
self.envs = envs
|
||||
self.projection_f = projection_f
|
||||
self.config = config
|
||||
|
||||
def reset(self, kwargs) -> Dict[str, Any]:
|
||||
obs, infos = self.envs.reset()
|
||||
return {'text': None, 'image': obs, 'anchor': None}, infos
|
||||
|
||||
def step(self, text_actions: List[str]):
|
||||
actions, valids = self.projection_f(text_actions)
|
||||
next_obs, rewards, dones, infos = self.envs.step(actions)
|
||||
|
||||
next_observations = {
|
||||
'text': None,
|
||||
'image': next_obs,
|
||||
'anchor': None,
|
||||
}
|
||||
for i, info in enumerate(infos):
|
||||
info['is_action_valid'] = to_numpy(valids[i])
|
||||
|
||||
rewards = to_numpy(rewards)
|
||||
dones = to_numpy(dones)
|
||||
return next_observations, rewards, dones, infos
|
||||
|
||||
def close(self) -> None:
|
||||
self.envs.close()
|
||||
|
||||
def success_evaluator(self, *args, **kwargs) -> Dict[str, np.ndarray]:
|
||||
total_infos = kwargs['total_infos']
|
||||
total_batch_list = kwargs['total_batch_list']
|
||||
batch_size = len(total_batch_list)
|
||||
|
||||
success = defaultdict(list)
|
||||
for bs in range(batch_size):
|
||||
self._process_batch(bs, total_batch_list, total_infos, success)
|
||||
assert len(success['success_rate']) == batch_size
|
||||
return {key: np.array(value) for key, value in success.items()}
|
||||
|
||||
def _process_batch(self, batch_idx, total_batch_list, total_infos, success):
|
||||
for i in reversed(range(len(total_batch_list[batch_idx]))):
|
||||
batch_item = total_batch_list[batch_idx][i]
|
||||
if batch_item['active_masks']:
|
||||
info = total_infos[batch_idx][i]
|
||||
won_value = float(info['won'])
|
||||
success['success_rate'].append(won_value)
|
||||
return
|
||||
-139
@@ -1,139 +0,0 @@
|
||||
# Vendored from SkillRL (Apache-2.0 License)
|
||||
# Original: agent_system/environments/env_manager.py
|
||||
# Trimmed to only include AlfWorldEnvironmentManager and its helpers.
|
||||
|
||||
from typing import List, Dict, Any
|
||||
from collections import defaultdict
|
||||
import numpy as np
|
||||
|
||||
from skillopt.envs.alfworld.vendor.env_base import EnvironmentManagerBase, to_numpy
|
||||
from skillopt.envs.alfworld.vendor.alfworld_prompts import (
|
||||
ALFWORLD_TEMPLATE,
|
||||
ALFWORLD_TEMPLATE_NO_HIS,
|
||||
ALFWORLD_TEMPLATE_WITH_MEMORY,
|
||||
)
|
||||
from skillopt.envs.alfworld.vendor.memory import SimpleMemory
|
||||
|
||||
|
||||
def parse_gamefile(infos):
|
||||
gamefile = []
|
||||
for info in infos:
|
||||
if 'extra.gamefile' in info:
|
||||
gamefile.append(info['extra.gamefile'])
|
||||
else:
|
||||
gamefile.append(None)
|
||||
return gamefile
|
||||
|
||||
|
||||
def set_gamefile(infos, gamefile):
|
||||
for i in range(len(infos)):
|
||||
if 'extra.gamefile' in infos[i]:
|
||||
infos[i]['extra.gamefile'] = gamefile[i]
|
||||
else:
|
||||
infos[i]['extra.gamefile'] = None
|
||||
return infos
|
||||
|
||||
|
||||
class AlfWorldEnvironmentManager(EnvironmentManagerBase):
|
||||
"""Manages parallel ALFWorld environments with observation templating."""
|
||||
|
||||
def __init__(self, envs, projection_f, config):
|
||||
self.memory = SimpleMemory()
|
||||
self.retrieval_memory = None
|
||||
super().__init__(envs, projection_f, config)
|
||||
|
||||
def reset(self, kwargs):
|
||||
text_obs, image_obs, infos = self.envs.reset()
|
||||
self.gamefile = parse_gamefile(infos)
|
||||
self.memory.reset(batch_size=len(text_obs))
|
||||
self.tasks = []
|
||||
self.pre_text_obs = text_obs
|
||||
self.extract_task(text_obs)
|
||||
|
||||
full_text_obs = self.build_text_obs(text_obs, self.envs.get_admissible_commands, init=True)
|
||||
return {'text': full_text_obs, 'image': image_obs, 'anchor': text_obs}, infos
|
||||
|
||||
def step(self, text_actions: List[str]):
|
||||
actions, valids = self.projection_f(text_actions, self.envs.get_admissible_commands)
|
||||
text_obs, image_obs, rewards, dones, infos = self.envs.step(actions)
|
||||
self.memory.store({'text_obs': self.pre_text_obs, 'action': actions})
|
||||
self.pre_text_obs = text_obs
|
||||
|
||||
full_text_obs = self.build_text_obs(text_obs, self.envs.get_admissible_commands)
|
||||
if infos[0].get("extra.gamefile") is None:
|
||||
infos = set_gamefile(infos, self.gamefile)
|
||||
|
||||
for i, info in enumerate(infos):
|
||||
info['is_action_valid'] = to_numpy(valids[i])
|
||||
|
||||
next_observations = {'text': full_text_obs, 'image': image_obs, 'anchor': text_obs}
|
||||
rewards = to_numpy(rewards)
|
||||
dones = to_numpy(dones)
|
||||
return next_observations, rewards, dones, infos
|
||||
|
||||
def extract_task(self, text_obs: List[str]):
|
||||
for obs in text_obs:
|
||||
task_start = obs.find('Your task is to: ')
|
||||
if task_start != -1:
|
||||
self.tasks.append(obs[task_start + len('Your task is to: '):].strip())
|
||||
else:
|
||||
raise ValueError("Task description not found in text observation.")
|
||||
|
||||
def build_text_obs(self, text_obs: List[str], admissible_actions: List[List[str]], init: bool = False) -> List[str]:
|
||||
postprocess_text_obs = []
|
||||
if not init and self.config.env.history_length > 0:
|
||||
memory_contexts, valid_lens = self.memory.fetch(
|
||||
self.config.env.history_length,
|
||||
obs_key="text_obs",
|
||||
action_key="action",
|
||||
)
|
||||
|
||||
for i in range(len(text_obs)):
|
||||
reformatted_admissible_actions = "\n ".join(
|
||||
f"'{s}'" for s in admissible_actions[i] if s != 'help'
|
||||
)
|
||||
|
||||
if init or self.config.env.history_length <= 0:
|
||||
obs = ALFWORLD_TEMPLATE_NO_HIS.format(
|
||||
current_observation=text_obs[i],
|
||||
admissible_actions=reformatted_admissible_actions,
|
||||
)
|
||||
else:
|
||||
obs = ALFWORLD_TEMPLATE.format(
|
||||
task_description=self.tasks[i],
|
||||
step_count=len(self.memory[i]),
|
||||
history_length=valid_lens[i],
|
||||
action_history=memory_contexts[i],
|
||||
current_step=len(self.memory[i]) + 1,
|
||||
current_observation=text_obs[i],
|
||||
admissible_actions=reformatted_admissible_actions,
|
||||
)
|
||||
postprocess_text_obs.append(obs)
|
||||
return postprocess_text_obs
|
||||
|
||||
def _process_batch(self, batch_idx, total_batch_list, total_infos, success):
|
||||
for i in reversed(range(len(total_batch_list[batch_idx]))):
|
||||
batch_item = total_batch_list[batch_idx][i]
|
||||
if batch_item['active_masks']:
|
||||
info = total_infos[batch_idx][i]
|
||||
won_value = float(info['won'])
|
||||
success['success_rate'].append(won_value)
|
||||
|
||||
gamefile = info.get("extra.gamefile")
|
||||
if gamefile:
|
||||
self._process_gamefile(gamefile, won_value, success)
|
||||
return
|
||||
|
||||
def _process_gamefile(self, gamefile, won_value, success):
|
||||
tasks = [
|
||||
"pick_and_place",
|
||||
"pick_two_obj_and_place",
|
||||
"look_at_obj_in_light",
|
||||
"pick_heat_then_place_in_recep",
|
||||
"pick_cool_then_place_in_recep",
|
||||
"pick_clean_then_place_in_recep",
|
||||
]
|
||||
for task in tasks:
|
||||
if task in gamefile:
|
||||
success[f"{task}_success_rate"].append(won_value)
|
||||
break
|
||||
-87
@@ -1,87 +0,0 @@
|
||||
# Vendored from SkillRL (Apache-2.0 License)
|
||||
# Original: agent_system/memory/base.py + agent_system/memory/memory.py
|
||||
# Merged into a single file for simplicity.
|
||||
|
||||
from abc import ABC, abstractmethod
|
||||
from typing import List, Dict, Any, Tuple
|
||||
|
||||
|
||||
class BaseMemory(ABC):
|
||||
"""Base class for memory management."""
|
||||
|
||||
@abstractmethod
|
||||
def __len__(self):
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def __getitem__(self, idx: int):
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def reset(self, batch_size: int):
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def store(self, record: Dict[str, List[Any]]):
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def fetch(self, step: int):
|
||||
pass
|
||||
|
||||
|
||||
class SimpleMemory(BaseMemory):
|
||||
"""Per-environment history buffer for storing observations and actions."""
|
||||
|
||||
def __init__(self):
|
||||
self._data = None
|
||||
self.keys = None
|
||||
self.batch_size = 0
|
||||
|
||||
def __len__(self):
|
||||
return len(self._data)
|
||||
|
||||
def __getitem__(self, idx):
|
||||
return self._data[idx]
|
||||
|
||||
def reset(self, batch_size: int):
|
||||
if self._data is not None:
|
||||
self._data.clear()
|
||||
self._data = [[] for _ in range(batch_size)]
|
||||
self.batch_size = batch_size
|
||||
self.keys = None
|
||||
|
||||
def store(self, record: Dict[str, List[Any]]):
|
||||
if self.keys is None:
|
||||
self.keys = list(record.keys())
|
||||
assert self.keys == list(record.keys())
|
||||
|
||||
for env_idx in range(self.batch_size):
|
||||
self._data[env_idx].append({k: record[k][env_idx] for k in self.keys})
|
||||
|
||||
def fetch(
|
||||
self,
|
||||
history_length: int,
|
||||
obs_key: str = "text_obs",
|
||||
action_key: str = "action",
|
||||
) -> Tuple[List[str], List[int]]:
|
||||
memory_contexts, valid_lengths = [], []
|
||||
|
||||
for env_idx in range(self.batch_size):
|
||||
recent = self._data[env_idx][-history_length:]
|
||||
valid_len = len(recent)
|
||||
start_idx = len(self._data[env_idx]) - valid_len
|
||||
|
||||
lines = []
|
||||
for j, rec in enumerate(recent):
|
||||
step_num = start_idx + j + 1
|
||||
act = rec[action_key]
|
||||
obs = rec[obs_key]
|
||||
lines.append(
|
||||
f"[Observation {step_num}: '{obs}', Action {step_num}: '{act}']"
|
||||
)
|
||||
|
||||
memory_contexts.append("\n".join(lines))
|
||||
valid_lengths.append(valid_len)
|
||||
|
||||
return memory_contexts, valid_lengths
|
||||
@@ -1,309 +0,0 @@
|
||||
"""ReflACT environment adapter — abstract interface.
|
||||
|
||||
To connect ReflACT to a new environment (benchmark, simulator, etc.),
|
||||
implement a subclass of :class:`EnvAdapter` with environment-specific
|
||||
rollout and reflection logic.
|
||||
|
||||
Example::
|
||||
|
||||
class MyBenchAdapter(EnvAdapter):
|
||||
def build_train_env(self, batch_size, seed, **kw):
|
||||
return MyEnvManager(split="train", n=batch_size, seed=seed)
|
||||
|
||||
def build_eval_env(self, env_num, split, seed, **kw):
|
||||
return MyEnvManager(split=split, n=env_num, seed=seed)
|
||||
|
||||
def rollout(self, env_manager, skill_content, out_dir, **kw):
|
||||
# Run episodes, return [{"id": ..., "hard": 0/1, "soft": 0.0-1.0, ...}]
|
||||
...
|
||||
|
||||
def reflect(self, results, skill_content, out_dir, **kw):
|
||||
# Analyze trajectories, return list of patch dicts
|
||||
...
|
||||
|
||||
def get_task_types(self):
|
||||
return ["task_a", "task_b"]
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from abc import ABC, abstractmethod
|
||||
import os
|
||||
import random
|
||||
|
||||
from skillopt.datasets.base import BaseDataLoader, BatchSpec
|
||||
from skillopt.prompts import load_prompt
|
||||
|
||||
|
||||
class EnvAdapter(ABC):
|
||||
"""Abstract adapter for connecting ReflACT to any environment.
|
||||
|
||||
Subclasses must implement all abstract methods. The ReflACT trainer
|
||||
calls these methods at the appropriate pipeline stages.
|
||||
"""
|
||||
|
||||
# ── Lifecycle hooks ────────────────────────────────────────────────────
|
||||
|
||||
def setup(self, cfg: dict) -> None:
|
||||
"""Called once by the trainer before the training loop begins.
|
||||
|
||||
Override to perform one-time initialization that requires the full
|
||||
config (e.g., data loading, split creation). Default is a no-op.
|
||||
"""
|
||||
self._cfg = dict(cfg)
|
||||
|
||||
def get_dataloader(self) -> BaseDataLoader | None:
|
||||
"""Return the task dataloader used by this adapter, if any."""
|
||||
return None
|
||||
|
||||
def requires_ray(self) -> bool:
|
||||
"""Return whether this adapter requires Ray runtime initialization."""
|
||||
return False
|
||||
|
||||
def build_reference_text(self, item: dict) -> str:
|
||||
"""Return hidden reference material for reflection, if any."""
|
||||
return str(item.get("reference_text") or "").strip()
|
||||
|
||||
def get_reference_metadata(self, item: dict) -> dict:
|
||||
"""Return structured metadata about hidden reference material."""
|
||||
reference_text = self.build_reference_text(item)
|
||||
if not reference_text:
|
||||
return {"fields": [], "preview": ""}
|
||||
return {
|
||||
"fields": ["reference_text"],
|
||||
"preview": reference_text[:400],
|
||||
}
|
||||
|
||||
def attach_reference_context(
|
||||
self,
|
||||
results: list[dict],
|
||||
items: list[dict] | None,
|
||||
) -> list[dict]:
|
||||
"""Attach environment-specific hidden reference text to result dicts."""
|
||||
if not results or not items:
|
||||
return list(results)
|
||||
|
||||
item_by_id = {
|
||||
str(item.get("id")): item
|
||||
for item in items
|
||||
if isinstance(item, dict) and item.get("id") is not None
|
||||
}
|
||||
enriched: list[dict] = []
|
||||
for row in results:
|
||||
merged = dict(row)
|
||||
item = item_by_id.get(str(row.get("id")))
|
||||
if item:
|
||||
reference_text = self.build_reference_text(item)
|
||||
if reference_text:
|
||||
merged["reference_text"] = reference_text
|
||||
enriched.append(merged)
|
||||
return enriched
|
||||
|
||||
def select_representative_items(
|
||||
self,
|
||||
results: list[dict],
|
||||
items: list[dict] | None,
|
||||
*,
|
||||
n_failures: int,
|
||||
n_successes: int,
|
||||
seed: int | None = None,
|
||||
) -> list[dict]:
|
||||
"""Select a small diverse subset of current-batch items by outcome."""
|
||||
if not items:
|
||||
return []
|
||||
|
||||
item_by_id = {
|
||||
str(item.get("id")): item
|
||||
for item in items
|
||||
if isinstance(item, dict) and item.get("id") is not None
|
||||
}
|
||||
failures = [
|
||||
(result, item_by_id[str(result.get("id"))])
|
||||
for result in results
|
||||
if not result.get("hard") and str(result.get("id")) in item_by_id
|
||||
]
|
||||
successes = [
|
||||
(result, item_by_id[str(result.get("id"))])
|
||||
for result in results
|
||||
if result.get("hard") and str(result.get("id")) in item_by_id
|
||||
]
|
||||
|
||||
rng = random.Random(seed)
|
||||
|
||||
def _pick(pool: list[tuple[dict, dict]], quota: int) -> list[dict]:
|
||||
if quota <= 0 or not pool:
|
||||
return []
|
||||
shuffled = list(pool)
|
||||
rng.shuffle(shuffled)
|
||||
|
||||
picked_ids: set[str] = set()
|
||||
picked: list[dict] = []
|
||||
seen_types: set[str] = set()
|
||||
|
||||
for result, item in shuffled:
|
||||
task_type = str(result.get("task_type") or item.get("task_type") or item.get("subtype") or "unknown")
|
||||
item_id = str(item["id"])
|
||||
if task_type in seen_types or item_id in picked_ids:
|
||||
continue
|
||||
picked.append(item)
|
||||
picked_ids.add(item_id)
|
||||
seen_types.add(task_type)
|
||||
if len(picked) >= quota:
|
||||
return picked
|
||||
|
||||
for _, item in shuffled:
|
||||
item_id = str(item["id"])
|
||||
if item_id in picked_ids:
|
||||
continue
|
||||
picked.append(item)
|
||||
picked_ids.add(item_id)
|
||||
if len(picked) >= quota:
|
||||
break
|
||||
return picked
|
||||
|
||||
selected = _pick(failures, n_failures)
|
||||
selected_ids = {str(item["id"]) for item in selected}
|
||||
selected.extend(
|
||||
item for item in _pick(successes, n_successes)
|
||||
if str(item["id"]) not in selected_ids
|
||||
)
|
||||
return selected
|
||||
|
||||
def build_env_from_batch(self, batch: BatchSpec, **kwargs):
|
||||
"""Build an environment manager or item list from a :class:`BatchSpec`.
|
||||
|
||||
Default behavior preserves the legacy adapter API by routing training
|
||||
batches through :meth:`build_train_env` and evaluation batches through
|
||||
:meth:`build_eval_env`.
|
||||
"""
|
||||
if batch.phase == "train":
|
||||
return self.build_train_env(batch_size=batch.batch_size, seed=batch.seed, **kwargs)
|
||||
return self.build_eval_env(
|
||||
env_num=batch.batch_size,
|
||||
split=batch.split,
|
||||
seed=batch.seed,
|
||||
**kwargs,
|
||||
)
|
||||
|
||||
@abstractmethod
|
||||
def build_train_env(self, batch_size: int, seed: int, **kwargs):
|
||||
"""Build a training environment manager.
|
||||
|
||||
Returns
|
||||
-------
|
||||
object
|
||||
An environment manager that can be passed to :meth:`rollout`.
|
||||
"""
|
||||
|
||||
@abstractmethod
|
||||
def build_eval_env(self, env_num: int, split: str, seed: int, **kwargs):
|
||||
"""Build an evaluation environment manager.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
env_num : int
|
||||
Number of evaluation environments.
|
||||
split : str
|
||||
Dataset split (e.g. ``"valid_seen"``, ``"valid_unseen"``).
|
||||
seed : int
|
||||
Random seed for reproducibility.
|
||||
|
||||
Returns
|
||||
-------
|
||||
object
|
||||
An environment manager that can be passed to :meth:`rollout`.
|
||||
"""
|
||||
|
||||
@abstractmethod
|
||||
def rollout(
|
||||
self,
|
||||
env_manager,
|
||||
skill_content: str,
|
||||
out_dir: str,
|
||||
**kwargs,
|
||||
) -> list[dict]:
|
||||
"""Run a batch of episodes using the current skill.
|
||||
|
||||
Returns
|
||||
-------
|
||||
list[dict]
|
||||
Each dict conforms to :class:`~skillopt.types.RolloutResult`:
|
||||
must have ``"id"`` (str), ``"hard"`` (0/1), ``"soft"``
|
||||
(float 0-1). May include env-specific fields.
|
||||
"""
|
||||
|
||||
@abstractmethod
|
||||
def reflect(
|
||||
self,
|
||||
results: list[dict],
|
||||
skill_content: str,
|
||||
out_dir: str,
|
||||
**kwargs,
|
||||
) -> list[dict | None]:
|
||||
"""Analyze rollout results and produce patches.
|
||||
|
||||
Each returned dict conforms to :class:`~skillopt.types.RawPatch`:
|
||||
``"patch"`` (with ``"edits"`` list) + ``"source_type"``
|
||||
(``"failure"`` or ``"success"``).
|
||||
|
||||
Returns
|
||||
-------
|
||||
list[dict | None]
|
||||
Raw analyst outputs; ``None`` entries are filtered out.
|
||||
"""
|
||||
|
||||
@abstractmethod
|
||||
def get_task_types(self) -> list[str]:
|
||||
"""Return the list of task type names for this environment."""
|
||||
|
||||
# ── Prompt configuration (two-level priority) ────────────────────────
|
||||
#
|
||||
# Priority: env-specific prompt file > generic default prompt file.
|
||||
#
|
||||
# Prompts are loaded from ``.md`` files via ``load_prompt(name, env)``:
|
||||
# 1. ``skillopt/envs/<env>/prompts/<name>.md`` (env-specific)
|
||||
# 2. ``skillopt/prompts/<name>.md`` (generic fallback)
|
||||
#
|
||||
# Subclasses can still override ``get_*_prompt()`` for full control.
|
||||
|
||||
@property
|
||||
def _env_name(self) -> str:
|
||||
"""Derive the env directory name from this adapter's module path."""
|
||||
# e.g. "skillopt.envs.searchqa.adapter" → "searchqa"
|
||||
module = type(self).__module__
|
||||
parts = module.split(".")
|
||||
if len(parts) >= 3 and parts[-3] == "envs":
|
||||
return parts[-2]
|
||||
return ""
|
||||
|
||||
def _load_env_prompt(self, name: str) -> str | None:
|
||||
"""Load a prompt with env-specific override. Returns None if not found."""
|
||||
try:
|
||||
return load_prompt(name, env=self._env_name)
|
||||
except FileNotFoundError:
|
||||
return None
|
||||
|
||||
def get_error_minibatch_prompt(self) -> str | None:
|
||||
update_mode = getattr(self, "_cfg", {}).get("skill_update_mode", "patch")
|
||||
raw_mode = str(update_mode).strip().lower()
|
||||
if raw_mode in {"full_rewrite", "full_rewrite_minibatch", "minibatch_full_rewrite", "skill_rewrite_minibatch"}:
|
||||
prompt = self._load_env_prompt("analyst_error_full_rewrite")
|
||||
if prompt is not None:
|
||||
return prompt
|
||||
if raw_mode in {"rewrite", "rewrite_from_suggestions", "suggestions", "rewrite_suggestions"}:
|
||||
prompt = self._load_env_prompt("analyst_error_rewrite")
|
||||
if prompt is not None:
|
||||
return prompt
|
||||
return self._load_env_prompt("analyst_error")
|
||||
|
||||
def get_success_minibatch_prompt(self) -> str | None:
|
||||
update_mode = getattr(self, "_cfg", {}).get("skill_update_mode", "patch")
|
||||
raw_mode = str(update_mode).strip().lower()
|
||||
if raw_mode in {"full_rewrite", "full_rewrite_minibatch", "minibatch_full_rewrite", "skill_rewrite_minibatch"}:
|
||||
prompt = self._load_env_prompt("analyst_success_full_rewrite")
|
||||
if prompt is not None:
|
||||
return prompt
|
||||
if raw_mode in {"rewrite", "rewrite_from_suggestions", "suggestions", "rewrite_suggestions"}:
|
||||
prompt = self._load_env_prompt("analyst_success_rewrite")
|
||||
if prompt is not None:
|
||||
return prompt
|
||||
return self._load_env_prompt("analyst_success")
|
||||
@@ -1 +0,0 @@
|
||||
"""DocVQA environment package for ReflACT."""
|
||||
@@ -1,115 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
|
||||
from skillopt.datasets.base import BatchSpec
|
||||
from skillopt.envs.base import EnvAdapter
|
||||
from skillopt.envs.docvqa.dataloader import DocVQADataLoader
|
||||
from skillopt.envs.docvqa.rollout import run_batch
|
||||
from skillopt.gradient.reflect import run_minibatch_reflect
|
||||
|
||||
|
||||
class DocVQAAdapter(EnvAdapter):
|
||||
def __init__(
|
||||
self,
|
||||
split_dir: str = "",
|
||||
data_path: str = "",
|
||||
split_mode: str = "split_dir",
|
||||
split_ratio: str = "2:1:7",
|
||||
split_seed: int = 42,
|
||||
split_output_dir: str = "",
|
||||
max_turns: int = 1,
|
||||
exec_timeout: int = 120,
|
||||
workers: int = 16,
|
||||
analyst_workers: int = 16,
|
||||
failure_only: bool = False,
|
||||
minibatch_size: int = 8,
|
||||
edit_budget: int = 4,
|
||||
seed: int = 42,
|
||||
limit: int = 0,
|
||||
image_detail: str = "auto",
|
||||
max_completion_tokens: int = 16384,
|
||||
) -> None:
|
||||
self.max_turns = max_turns
|
||||
self.exec_timeout = exec_timeout
|
||||
self.workers = workers
|
||||
self.max_completion_tokens = int(max_completion_tokens)
|
||||
self.analyst_workers = analyst_workers
|
||||
self.failure_only = failure_only
|
||||
self.minibatch_size = minibatch_size
|
||||
self.edit_budget = edit_budget
|
||||
self.image_detail = image_detail
|
||||
self.dataloader = DocVQADataLoader(
|
||||
split_dir=split_dir,
|
||||
data_path=data_path,
|
||||
split_mode=split_mode,
|
||||
split_ratio=split_ratio,
|
||||
split_seed=split_seed,
|
||||
split_output_dir=split_output_dir,
|
||||
seed=seed,
|
||||
limit=limit,
|
||||
)
|
||||
|
||||
def setup(self, cfg: dict) -> None:
|
||||
super().setup(cfg)
|
||||
self.dataloader.setup(cfg)
|
||||
|
||||
def get_dataloader(self):
|
||||
return self.dataloader
|
||||
|
||||
def build_env_from_batch(self, batch: BatchSpec, **kwargs):
|
||||
return list(batch.payload or [])
|
||||
|
||||
def build_train_env(self, batch_size: int, seed: int, **kwargs):
|
||||
batch = self.dataloader.build_train_batch(batch_size=batch_size, seed=seed, **kwargs)
|
||||
return self.build_env_from_batch(batch, **kwargs)
|
||||
|
||||
def build_eval_env(self, env_num: int, split: str, seed: int, **kwargs):
|
||||
batch = self.dataloader.build_eval_batch(env_num=env_num, split=split, seed=seed, **kwargs)
|
||||
return self.build_env_from_batch(batch, **kwargs)
|
||||
|
||||
def rollout(self, env_manager, skill_content: str, out_dir: str, **kwargs) -> list[dict]:
|
||||
items: list[dict] = env_manager
|
||||
return run_batch(
|
||||
items=items,
|
||||
out_root=out_dir,
|
||||
skill_content=skill_content,
|
||||
max_turns=self.max_turns,
|
||||
exec_timeout=self.exec_timeout,
|
||||
workers=self.workers,
|
||||
image_detail=self.image_detail,
|
||||
max_completion_tokens=self.max_completion_tokens,
|
||||
diagnostic_mode=kwargs.get("diagnostic_mode", False),
|
||||
diagnostic_instruction=kwargs.get("diagnostic_instruction", ""),
|
||||
task_timeout=self.exec_timeout,
|
||||
)
|
||||
|
||||
def reflect(self, results: list[dict], skill_content: str, out_dir: str, **kwargs) -> list[dict | None]:
|
||||
prediction_dir = kwargs.get("prediction_dir", os.path.join(out_dir, "predictions"))
|
||||
patches_dir = kwargs.get("patches_dir", os.path.join(out_dir, "patches"))
|
||||
random_seed = kwargs.get("random_seed")
|
||||
step_buffer_context = kwargs.get("step_buffer_context", "")
|
||||
return run_minibatch_reflect(
|
||||
results=results,
|
||||
skill_content=skill_content,
|
||||
prediction_dir=prediction_dir,
|
||||
patches_dir=patches_dir,
|
||||
workers=self.analyst_workers,
|
||||
failure_only=self.failure_only,
|
||||
minibatch_size=self.minibatch_size,
|
||||
edit_budget=self.edit_budget,
|
||||
random_seed=random_seed,
|
||||
error_system=self.get_error_minibatch_prompt(),
|
||||
success_system=self.get_success_minibatch_prompt(),
|
||||
step_buffer_context=step_buffer_context,
|
||||
update_mode=getattr(self, "_cfg", {}).get("skill_update_mode", "patch"),
|
||||
)
|
||||
|
||||
|
||||
def get_task_types(self) -> list[str]:
|
||||
seen: list[str] = []
|
||||
for item in self.dataloader.train_items + self.dataloader.val_items + self.dataloader.test_items:
|
||||
task_type = str(item.get("task_type") or "docvqa")
|
||||
if task_type not in seen:
|
||||
seen.append(task_type)
|
||||
return seen or ["docvqa"]
|
||||
@@ -1,61 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import ast
|
||||
import csv
|
||||
from pathlib import Path
|
||||
|
||||
from skillopt.datasets.base import SplitDataLoader
|
||||
|
||||
|
||||
def _parse_answers(raw: str) -> list[str]:
|
||||
text = str(raw or "").strip()
|
||||
if not text:
|
||||
return []
|
||||
try:
|
||||
parsed = ast.literal_eval(text)
|
||||
except Exception:
|
||||
return [text]
|
||||
if isinstance(parsed, list):
|
||||
return [str(item).strip() for item in parsed if str(item).strip()]
|
||||
return [str(parsed).strip()]
|
||||
|
||||
|
||||
def _extract_document_path(question: str) -> tuple[str, str]:
|
||||
marker = "document_path:"
|
||||
if marker not in question:
|
||||
return question.strip(), ""
|
||||
main, tail = question.split(marker, 1)
|
||||
return main.strip(), tail.strip()
|
||||
|
||||
|
||||
def _normalize_row(row: dict[str, str]) -> dict:
|
||||
question_text, document_path = _extract_document_path(str(row.get("question") or ""))
|
||||
answers = _parse_answers(row.get("answer") or row.get("ground_truth") or "")
|
||||
image_path = str(row.get("image_path") or document_path or "").strip()
|
||||
task_type = str(row.get("topic") or row.get("category") or "docvqa").strip() or "docvqa"
|
||||
return {
|
||||
"id": str(row.get("questionId") or row.get("id") or "").strip(),
|
||||
"question": question_text,
|
||||
"answer": answers[0] if answers else "",
|
||||
"answers": answers,
|
||||
"task_type": task_type,
|
||||
"subtask": task_type,
|
||||
"image_paths": [image_path] if image_path else [],
|
||||
"image_path": image_path,
|
||||
"questionId": str(row.get("questionId") or "").strip(),
|
||||
"docId": str(row.get("docId") or "").strip(),
|
||||
"ucsf_document_id": str(row.get("ucsf_document_id") or "").strip(),
|
||||
"ucsf_document_page_no": str(row.get("ucsf_document_page_no") or "").strip(),
|
||||
"source_split": str(row.get("source_split") or "").strip(),
|
||||
}
|
||||
|
||||
|
||||
class DocVQADataLoader(SplitDataLoader):
|
||||
def load_split_items(self, split_path: str) -> list[dict]:
|
||||
path = Path(split_path)
|
||||
csv_files = sorted(path.glob("*.csv"))
|
||||
if not csv_files:
|
||||
raise FileNotFoundError(f"No .csv file found in {split_path}")
|
||||
with csv_files[0].open(encoding="utf-8", newline="") as f:
|
||||
reader = csv.DictReader(f)
|
||||
return [_normalize_row(row) for row in reader]
|
||||
@@ -1,113 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import ast
|
||||
import json
|
||||
from collections.abc import Iterable
|
||||
from typing import Any
|
||||
|
||||
DEFAULT_ANLS_THRESHOLD = 0.5
|
||||
|
||||
|
||||
def _normalize_text(value: Any) -> str:
|
||||
if value is None:
|
||||
return ""
|
||||
text = str(value).strip().lower()
|
||||
return " ".join(text.split())
|
||||
|
||||
|
||||
def _levenshtein_distance(a: str, b: str) -> int:
|
||||
if a == b:
|
||||
return 0
|
||||
if not a:
|
||||
return len(b)
|
||||
if not b:
|
||||
return len(a)
|
||||
if len(a) > len(b):
|
||||
a, b = b, a
|
||||
previous = list(range(len(b) + 1))
|
||||
for i, char_a in enumerate(a, start=1):
|
||||
current = [i]
|
||||
for j, char_b in enumerate(b, start=1):
|
||||
insert_cost = current[j - 1] + 1
|
||||
delete_cost = previous[j] + 1
|
||||
replace_cost = previous[j - 1] + (char_a != char_b)
|
||||
current.append(min(insert_cost, delete_cost, replace_cost))
|
||||
previous = current
|
||||
return previous[-1]
|
||||
|
||||
|
||||
def _score_single_answer(predicted: Any, target: Any, threshold: float) -> float:
|
||||
predicted_norm = _normalize_text(predicted)
|
||||
target_norm = _normalize_text(target)
|
||||
if not predicted_norm and not target_norm:
|
||||
return 1.0
|
||||
if not predicted_norm or not target_norm:
|
||||
return 0.0
|
||||
distance = _levenshtein_distance(predicted_norm, target_norm)
|
||||
normalized_distance = distance / max(len(predicted_norm), len(target_norm))
|
||||
if normalized_distance >= threshold:
|
||||
return 0.0
|
||||
return 1.0 - normalized_distance
|
||||
|
||||
|
||||
def _extract_answer_strings(raw: Any) -> list[str]:
|
||||
if raw is None:
|
||||
return [""]
|
||||
if isinstance(raw, str):
|
||||
text = raw.strip()
|
||||
if not text:
|
||||
return [""]
|
||||
parsed = None
|
||||
if text[0] in "[{":
|
||||
try:
|
||||
parsed = json.loads(text)
|
||||
except json.JSONDecodeError:
|
||||
try:
|
||||
parsed = ast.literal_eval(text)
|
||||
except (ValueError, SyntaxError):
|
||||
parsed = None
|
||||
if parsed is None:
|
||||
return [text]
|
||||
return _extract_answer_strings(parsed)
|
||||
if isinstance(raw, dict):
|
||||
for key in ("answers", "ground_truth", "answer"):
|
||||
if key in raw:
|
||||
return _extract_answer_strings(raw[key])
|
||||
return [str(raw)]
|
||||
if isinstance(raw, Iterable) and not isinstance(raw, (bytes, bytearray)):
|
||||
answers: list[str] = []
|
||||
for item in raw:
|
||||
if isinstance(item, dict):
|
||||
for key in ("text", "answer", "value"):
|
||||
if key in item:
|
||||
answers.extend(_extract_answer_strings(item[key]))
|
||||
break
|
||||
else:
|
||||
answers.append(str(item))
|
||||
continue
|
||||
answers.append(str(item))
|
||||
return answers or [""]
|
||||
return [str(raw)]
|
||||
|
||||
|
||||
def extract_answer(text: str) -> str:
|
||||
lower = text.lower()
|
||||
start = lower.rfind("<answer>")
|
||||
end = lower.rfind("</answer>")
|
||||
if start != -1 and end != -1 and end > start:
|
||||
return text[start + len("<answer>"):end].strip()
|
||||
lines = [line.strip() for line in text.splitlines() if line.strip()]
|
||||
return lines[-1] if lines else text.strip()
|
||||
|
||||
|
||||
def evaluate(prediction_text: str, gold_answers: Any) -> dict:
|
||||
answer = extract_answer(prediction_text)
|
||||
answers = _extract_answer_strings(gold_answers)
|
||||
score = 0.0
|
||||
for target in answers:
|
||||
score = max(score, _score_single_answer(answer, target, DEFAULT_ANLS_THRESHOLD))
|
||||
return {
|
||||
"anls": score,
|
||||
"predicted_answer": answer,
|
||||
"gold_answers": answers,
|
||||
}
|
||||
@@ -1,35 +0,0 @@
|
||||
You are an expert failure-analysis agent for visual document question answering tasks.
|
||||
|
||||
You will be given MULTIPLE failed DocVQA trajectories from a single minibatch and the current skill document. Each trajectory includes the model response and an evaluation result scored with ANLS against one or more acceptable answers.
|
||||
|
||||
Your job is to identify the most important COMMON failure patterns across the batch and propose concise skill edits.
|
||||
|
||||
## Failure Type Categories
|
||||
- evidence_miss: the model overlooked the relevant visible region or line
|
||||
- near_match_confusion: the model selected a nearby but incorrect text span
|
||||
- normalization_error: the answer differed mainly in formatting, spacing, punctuation, or minor text normalization
|
||||
- reading_error: the model misread the document content
|
||||
- other: none of the above
|
||||
|
||||
## Rules
|
||||
- Focus on common, reusable reading and extraction behaviors.
|
||||
- Do not hardcode image-specific answers.
|
||||
- Prefer concise edits that improve evidence selection and exact span extraction.
|
||||
|
||||
Respond ONLY with a valid JSON object (no markdown fences, no extra text):
|
||||
{
|
||||
"batch_size": <number of trajectories analysed>,
|
||||
"failure_summary": [
|
||||
{"failure_type": "<type>", "count": <int>, "description": "<one-line>"}
|
||||
],
|
||||
"patch": {
|
||||
"reasoning": "<why these edits address the batch's common failures>",
|
||||
"edits": [
|
||||
{"op": "append", "content": "<markdown to add at end of skill>"},
|
||||
{"op": "insert_after", "target": "<exact heading/text to insert after>", "content": "<markdown>"},
|
||||
{"op": "replace", "target": "<exact text to replace>", "content": "<replacement>"},
|
||||
{"op": "delete", "target": "<exact text to remove>"}
|
||||
]
|
||||
}
|
||||
}
|
||||
Only include edits that are needed. "edits" can be an empty list if no patch is warranted.
|
||||
@@ -1,24 +0,0 @@
|
||||
You are an expert success-pattern analyst for visual document question answering tasks.
|
||||
|
||||
You will be given MULTIPLE successful DocVQA trajectories from a single minibatch and the current skill document. Your job is to identify common visual reading and exact-answer extraction behaviors worth encoding in the skill.
|
||||
|
||||
## Rules
|
||||
- Focus on patterns shared across multiple successful trajectories.
|
||||
- Reinforce reusable behaviors like locating the right region, copying exact spans, and preferring the shortest exact answer over paraphrase.
|
||||
- Only propose patches for patterns not already captured by the current skill.
|
||||
|
||||
Respond ONLY with a valid JSON object:
|
||||
{
|
||||
"batch_size": <number of trajectories analysed>,
|
||||
"success_patterns": ["<pattern 1>", "<pattern 2>"],
|
||||
"patch": {
|
||||
"reasoning": "<why these patterns are worth encoding>",
|
||||
"edits": [
|
||||
{"op": "append", "content": "<markdown>"},
|
||||
{"op": "insert_after", "target": "<heading/text>", "content": "<markdown>"},
|
||||
{"op": "replace", "target": "<old text>", "content": "<new text>"},
|
||||
{"op": "delete", "target": "<exact text to remove>"}
|
||||
]
|
||||
}
|
||||
}
|
||||
"edits" may be empty if the skill already covers all observed patterns.
|
||||
@@ -1,12 +0,0 @@
|
||||
You are an expert visual document question answering agent.
|
||||
|
||||
{skill_section}You will receive a document image and a question about the document.
|
||||
Read the visual evidence carefully and answer concisely.
|
||||
|
||||
Rules:
|
||||
- Ground the answer in the visible document content.
|
||||
- Prefer exact spans, numbers, dates, and names from the document.
|
||||
- Do not invent content that is not visible.
|
||||
- If multiple near-matches exist, choose the one best supported by the document.
|
||||
|
||||
Return the final answer inside <answer>...</answer>.
|
||||
@@ -1,391 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import time
|
||||
from concurrent.futures import FIRST_COMPLETED, ThreadPoolExecutor, wait
|
||||
|
||||
from skillopt.envs.docvqa.evaluator import evaluate
|
||||
from skillopt.model import chat_target_messages, get_target_backend, is_target_exec_backend
|
||||
from skillopt.model.codex_harness import prepare_workspace, render_skill_md, run_target_exec
|
||||
from skillopt.prompts import load_prompt
|
||||
|
||||
|
||||
def _build_system(skill_content: str) -> str:
|
||||
if skill_content.strip():
|
||||
skill_section = f"## Skill\n{skill_content.strip()}\n\n"
|
||||
else:
|
||||
skill_section = ""
|
||||
return load_prompt("rollout_system", env="docvqa").format(skill_section=skill_section)
|
||||
|
||||
|
||||
def _image_to_data_uri(path: str) -> str:
|
||||
import base64
|
||||
import mimetypes
|
||||
|
||||
mime = mimetypes.guess_type(path)[0] or "image/png"
|
||||
with open(path, "rb") as f:
|
||||
encoded = base64.b64encode(f.read()).decode("ascii")
|
||||
return f"data:{mime};base64,{encoded}"
|
||||
|
||||
|
||||
def _build_messages(
|
||||
item: dict,
|
||||
skill_content: str,
|
||||
image_detail: str,
|
||||
*,
|
||||
diagnostic_mode: bool = False,
|
||||
diagnostic_instruction: str = "",
|
||||
) -> tuple[list[dict], str, str]:
|
||||
system = _build_system(skill_content)
|
||||
user_text = item["question"] + "\n\nReturn the final answer inside <answer>...</answer>."
|
||||
if diagnostic_mode and diagnostic_instruction.strip():
|
||||
user_text += f"\n\n## Training Readout\n{diagnostic_instruction.strip()}"
|
||||
image_url = {"url": _image_to_data_uri(item["image_path"])}
|
||||
if image_detail and image_detail != "auto":
|
||||
image_url["detail"] = image_detail
|
||||
messages = [
|
||||
{"role": "system", "content": system},
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "text", "text": user_text},
|
||||
{"type": "image_url", "image_url": image_url},
|
||||
],
|
||||
},
|
||||
]
|
||||
return messages, system, user_text
|
||||
|
||||
|
||||
def _build_codex_skill(skill_content: str) -> str:
|
||||
return render_skill_md(
|
||||
skill_content,
|
||||
description="Dynamic ReflACT skill for solving the current DocVQA document-image question.",
|
||||
preamble=(
|
||||
"Use this skill when answering the current DocVQA question.\n"
|
||||
"Inspect the attached document image carefully and return the final answer inside <answer>...</answer>."
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def _run_codex_once(
|
||||
*,
|
||||
pred_dir: str,
|
||||
item: dict,
|
||||
skill_content: str,
|
||||
model: str,
|
||||
timeout: int,
|
||||
image_detail: str,
|
||||
diagnostic_mode: bool = False,
|
||||
diagnostic_instruction: str = "",
|
||||
previous_response: str = "",
|
||||
) -> tuple[str, str, str, str]:
|
||||
_ = image_detail
|
||||
_messages, _system, user_text = _build_messages(
|
||||
item,
|
||||
skill_content,
|
||||
image_detail,
|
||||
diagnostic_mode=diagnostic_mode,
|
||||
diagnostic_instruction=diagnostic_instruction,
|
||||
)
|
||||
task_parts = [user_text]
|
||||
image_abs = os.path.abspath(item["image_path"])
|
||||
task_parts.append(
|
||||
"## Document Image\n"
|
||||
"The document image is available in this workspace via `ATTACHMENTS.md`.\n"
|
||||
f"Original image path: `{image_abs}`\n"
|
||||
"Open or inspect that image before answering; do not answer from memory."
|
||||
)
|
||||
if previous_response:
|
||||
task_parts.append(
|
||||
"## Previous Attempt\n"
|
||||
f"{previous_response}\n\n"
|
||||
"Review the same document image carefully and correct the answer if needed."
|
||||
)
|
||||
task_text = "\n\n".join(task_parts)
|
||||
skill_md = _build_codex_skill(skill_content)
|
||||
work_dir = os.path.join(pred_dir, "codex_exec")
|
||||
prepare_workspace(
|
||||
work_dir=work_dir,
|
||||
skill_md=skill_md,
|
||||
task_text=task_text,
|
||||
images=[item["image_path"]],
|
||||
)
|
||||
prompt = (
|
||||
"Use the `skillopt-target` skill available in this workspace.\n"
|
||||
"Read `task.md`, inspect the attached document image, and answer the DocVQA question.\n"
|
||||
"Return the final answer inside <answer>...</answer>."
|
||||
)
|
||||
final_message, raw = run_target_exec(
|
||||
work_dir=work_dir,
|
||||
prompt=prompt,
|
||||
model=model,
|
||||
timeout=timeout,
|
||||
images=[item["image_path"]],
|
||||
)
|
||||
return final_message or raw, raw, skill_md, task_text
|
||||
|
||||
|
||||
def process_one(
|
||||
item: dict,
|
||||
out_root: str,
|
||||
skill_content: str,
|
||||
*,
|
||||
max_turns: int = 1,
|
||||
exec_timeout: int = 120,
|
||||
image_detail: str = "auto",
|
||||
max_completion_tokens: int = 16384,
|
||||
diagnostic_mode: bool = False,
|
||||
diagnostic_instruction: str = "",
|
||||
) -> dict:
|
||||
item_id = str(item["id"])
|
||||
result = {
|
||||
"id": item_id,
|
||||
"question": item["question"],
|
||||
"task_type": item.get("subtask") or item.get("task_type") or "docvqa",
|
||||
"task_description": item["question"],
|
||||
"hard": 0,
|
||||
"soft": 0.0,
|
||||
"predicted_answer": "",
|
||||
"response": "",
|
||||
"fail_reason": "",
|
||||
"agent_ok": False,
|
||||
"n_turns": 0,
|
||||
"image_paths": item.get("image_paths", []),
|
||||
"gold_answer": item.get("answers", []),
|
||||
}
|
||||
try:
|
||||
response = ""
|
||||
system_prompt = ""
|
||||
user_text = ""
|
||||
conversation: list[dict] = []
|
||||
if is_target_exec_backend():
|
||||
from skillopt.model import azure_openai as _llm
|
||||
|
||||
conversation = [
|
||||
{
|
||||
"role": "user",
|
||||
"content": item["question"] + "\n\n" + f"[image] {os.path.basename(item['image_path'])}",
|
||||
}
|
||||
]
|
||||
for turn in range(max_turns):
|
||||
response, _raw, system_prompt, user_text = _run_codex_once(
|
||||
pred_dir=os.path.join(out_root, "predictions", item_id),
|
||||
item=item,
|
||||
skill_content=skill_content,
|
||||
model=_llm.TARGET_DEPLOYMENT,
|
||||
timeout=exec_timeout,
|
||||
image_detail=image_detail,
|
||||
diagnostic_mode=diagnostic_mode if turn == 0 else False,
|
||||
diagnostic_instruction=diagnostic_instruction if turn == 0 else "",
|
||||
previous_response=response if turn > 0 else "",
|
||||
)
|
||||
conversation.append({"type": "message", "turn": turn + 1, "content": response})
|
||||
if "<answer>" in response.lower():
|
||||
break
|
||||
else:
|
||||
messages, system_prompt, user_text = _build_messages(
|
||||
item,
|
||||
skill_content,
|
||||
image_detail,
|
||||
diagnostic_mode=diagnostic_mode,
|
||||
diagnostic_instruction=diagnostic_instruction,
|
||||
)
|
||||
conversation = [
|
||||
{
|
||||
"role": "user",
|
||||
"content": user_text + "\n\n" + f"[image] {os.path.basename(item['image_path'])}",
|
||||
}
|
||||
]
|
||||
for turn in range(max_turns):
|
||||
if turn == 0:
|
||||
resp_text, _ = chat_target_messages(
|
||||
messages=messages,
|
||||
max_completion_tokens=max_completion_tokens,
|
||||
retries=5,
|
||||
stage="rollout",
|
||||
timeout=exec_timeout,
|
||||
)
|
||||
else:
|
||||
refinement_messages = [
|
||||
messages[0],
|
||||
messages[1],
|
||||
{"role": "assistant", "content": response},
|
||||
{"role": "user", "content": "Review the same image carefully and answer again. Keep the final answer inside <answer>...</answer>."},
|
||||
]
|
||||
resp_text, _ = chat_target_messages(
|
||||
messages=refinement_messages,
|
||||
max_completion_tokens=max_completion_tokens,
|
||||
retries=5,
|
||||
stage="rollout",
|
||||
timeout=exec_timeout,
|
||||
)
|
||||
response = resp_text
|
||||
conversation.append({"type": "message", "turn": turn + 1, "content": resp_text})
|
||||
if "<answer>" in resp_text.lower():
|
||||
break
|
||||
|
||||
result["response"] = response
|
||||
result["agent_ok"] = True
|
||||
result["n_turns"] = len(conversation) - 1
|
||||
|
||||
pred_dir = os.path.join(out_root, "predictions", item_id)
|
||||
os.makedirs(pred_dir, exist_ok=True)
|
||||
with open(os.path.join(pred_dir, "target_system_prompt.txt"), "w", encoding="utf-8") as f:
|
||||
f.write(system_prompt)
|
||||
with open(os.path.join(pred_dir, "target_user_prompt.txt"), "w", encoding="utf-8") as f:
|
||||
f.write(user_text)
|
||||
|
||||
eval_result = evaluate(response, item.get("answers", []))
|
||||
result["predicted_answer"] = eval_result["predicted_answer"]
|
||||
result["hard"] = int(eval_result["anls"] >= 0.999)
|
||||
result["soft"] = eval_result["anls"]
|
||||
if result["soft"] <= 0.0:
|
||||
result["fail_reason"] = f"predicted '{eval_result['predicted_answer']}' but expected one of {item.get('answers', [])}"
|
||||
|
||||
eval_detail = (
|
||||
"[EVALUATION RESULT]\n"
|
||||
f"Question: {item['question']}\n"
|
||||
f"Predicted answer: {eval_result['predicted_answer']!r}\n"
|
||||
f"Gold answers: {item.get('answers', [])!r}\n"
|
||||
f"ANLS: {eval_result['anls']:.4f}"
|
||||
)
|
||||
conversation.append({"role": "system", "content": eval_detail})
|
||||
with open(os.path.join(pred_dir, "conversation.json"), "w", encoding="utf-8") as f:
|
||||
json.dump(conversation, f, ensure_ascii=False, indent=2)
|
||||
except Exception as e: # noqa: BLE001
|
||||
result["fail_reason"] = f"error: {e}"
|
||||
return result
|
||||
|
||||
|
||||
def run_batch(
|
||||
items: list[dict],
|
||||
out_root: str,
|
||||
skill_content: str,
|
||||
*,
|
||||
max_turns: int = 1,
|
||||
exec_timeout: int = 120,
|
||||
workers: int = 16,
|
||||
image_detail: str = "auto",
|
||||
max_completion_tokens: int = 16384,
|
||||
diagnostic_mode: bool = False,
|
||||
diagnostic_instruction: str = "",
|
||||
task_timeout: int = 600,
|
||||
) -> list[dict]:
|
||||
task_timeout = max(int(task_timeout), int(exec_timeout) + 60)
|
||||
results_path = os.path.join(out_root, "results.jsonl")
|
||||
os.makedirs(out_root, exist_ok=True)
|
||||
|
||||
done_ids: set[str] = set()
|
||||
existing: list[dict] = []
|
||||
if os.path.exists(results_path):
|
||||
with open(results_path, encoding="utf-8") as f:
|
||||
for line in f:
|
||||
try:
|
||||
row = json.loads(line)
|
||||
except Exception:
|
||||
continue
|
||||
done_ids.add(str(row["id"]))
|
||||
existing.append(row)
|
||||
|
||||
pending = [item for item in items if str(item["id"]) not in done_ids]
|
||||
if not pending:
|
||||
return existing
|
||||
|
||||
def _timeout_result(item: dict) -> dict:
|
||||
return {
|
||||
"id": str(item["id"]),
|
||||
"question": item.get("question", ""),
|
||||
"task_type": item.get("subtask") or item.get("task_type") or "docvqa",
|
||||
"task_description": item.get("question", ""),
|
||||
"hard": 0,
|
||||
"soft": 0.0,
|
||||
"predicted_answer": "",
|
||||
"response": "",
|
||||
"fail_reason": f"task-timeout-{task_timeout}s",
|
||||
"agent_ok": False,
|
||||
"n_turns": 0,
|
||||
"image_paths": item.get("image_paths", []),
|
||||
"gold_answer": item.get("answers", []),
|
||||
"phase": "timeout",
|
||||
}
|
||||
|
||||
def _error_result(item: dict, exc: Exception) -> dict:
|
||||
row = _timeout_result(item)
|
||||
row["phase"] = "error"
|
||||
row["fail_reason"] = f"unexpected: {type(exc).__name__}: {exc}"
|
||||
return row
|
||||
|
||||
started_at: dict[str, float] = {}
|
||||
|
||||
def _run_one(item: dict) -> dict:
|
||||
started_at[str(item["id"])] = time.time()
|
||||
return process_one(
|
||||
item,
|
||||
out_root,
|
||||
skill_content,
|
||||
max_turns=max_turns,
|
||||
exec_timeout=exec_timeout,
|
||||
image_detail=image_detail,
|
||||
max_completion_tokens=max_completion_tokens,
|
||||
diagnostic_mode=diagnostic_mode,
|
||||
diagnostic_instruction=diagnostic_instruction,
|
||||
)
|
||||
|
||||
total = len(existing) + len(pending)
|
||||
completed = len(existing)
|
||||
correct = sum(1 for r in existing if r.get("hard", 0))
|
||||
if existing:
|
||||
print(f" [rollout] resuming: {completed}/{total} already done", flush=True)
|
||||
|
||||
results = list(existing)
|
||||
with open(results_path, "a", encoding="utf-8") as outf:
|
||||
ex = ThreadPoolExecutor(max_workers=workers)
|
||||
try:
|
||||
futs = {ex.submit(_run_one, item): item for item in pending}
|
||||
pending_futs = set(futs)
|
||||
while pending_futs:
|
||||
done, _ = wait(pending_futs, timeout=5, return_when=FIRST_COMPLETED)
|
||||
now = time.time()
|
||||
timed_out = [
|
||||
fut for fut in pending_futs - done
|
||||
if str(futs[fut]["id"]) in started_at
|
||||
and now - started_at[str(futs[fut]["id"])] >= task_timeout
|
||||
]
|
||||
for fut in done:
|
||||
pending_futs.remove(fut)
|
||||
item = futs[fut]
|
||||
try:
|
||||
res = fut.result()
|
||||
except Exception as exc: # noqa: BLE001
|
||||
res = _error_result(item, exc)
|
||||
results.append(res)
|
||||
completed += 1
|
||||
if res.get("hard", 0):
|
||||
correct += 1
|
||||
acc = correct / completed if completed else 0
|
||||
print(
|
||||
f" [rollout] {completed}/{total} "
|
||||
f"(acc={acc:.3f}) id={res['id']} "
|
||||
f"hard={res.get('hard', '?')}",
|
||||
flush=True,
|
||||
)
|
||||
outf.write(json.dumps(res, ensure_ascii=False) + "\n")
|
||||
outf.flush()
|
||||
for fut in timed_out:
|
||||
pending_futs.remove(fut)
|
||||
fut.cancel()
|
||||
res = _timeout_result(futs[fut])
|
||||
results.append(res)
|
||||
completed += 1
|
||||
acc = correct / completed if completed else 0
|
||||
print(
|
||||
f" [rollout] {completed}/{total} "
|
||||
f"(acc={acc:.3f}) id={res['id']} TIMEOUT",
|
||||
flush=True,
|
||||
)
|
||||
outf.write(json.dumps(res, ensure_ascii=False) + "\n")
|
||||
outf.flush()
|
||||
finally:
|
||||
ex.shutdown(wait=False, cancel_futures=True)
|
||||
return results
|
||||
@@ -1,11 +0,0 @@
|
||||
# DocVQA Skill
|
||||
|
||||
## Visual Evidence Discipline
|
||||
- Read the document carefully before answering.
|
||||
- Prefer the smallest exact text span that answers the question.
|
||||
- When several nearby strings look similar, choose the one whose surrounding labels or layout best match the question.
|
||||
|
||||
## Exact Answer Discipline
|
||||
- Copy names, numbers, and dates exactly from the document whenever possible.
|
||||
- Prefer direct extraction over paraphrase.
|
||||
- Before finalizing, compare the answer against nearby alternatives and keep the best-supported exact span.
|
||||
@@ -1 +0,0 @@
|
||||
"""LiveMathematicianBench environment package for ReflACT."""
|
||||
@@ -1,162 +0,0 @@
|
||||
"""LiveMathematicianBench environment adapter for ReflACT."""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
|
||||
from skillopt.datasets.base import BatchSpec
|
||||
from skillopt.gradient.reflect import run_minibatch_reflect
|
||||
from skillopt.envs.base import EnvAdapter
|
||||
from skillopt.envs.livemathematicianbench.dataloader import LiveMathematicianBenchDataLoader
|
||||
from skillopt.envs.livemathematicianbench.rollout import run_batch
|
||||
from skillopt.model import get_target_backend
|
||||
|
||||
|
||||
class LiveMathematicianBenchAdapter(EnvAdapter):
|
||||
"""LiveMathematicianBench adapter."""
|
||||
|
||||
def build_reference_text(self, item: dict) -> str:
|
||||
parts: list[str] = []
|
||||
theorem = str(item.get("theorem") or "").strip()
|
||||
sketch = str(item.get("sketch") or "").strip()
|
||||
if theorem:
|
||||
parts.append(f"## Reference Theorem\n{theorem}")
|
||||
if sketch:
|
||||
parts.append(f"## Reference Sketch\n{sketch}")
|
||||
return "\n\n".join(parts)
|
||||
|
||||
def get_reference_metadata(self, item: dict) -> dict:
|
||||
fields: list[str] = []
|
||||
previews: list[str] = []
|
||||
theorem = str(item.get("theorem") or "").strip()
|
||||
sketch = str(item.get("sketch") or "").strip()
|
||||
if theorem:
|
||||
fields.append("theorem")
|
||||
previews.append(f"[theorem]\n{theorem[:220]}")
|
||||
if sketch:
|
||||
fields.append("sketch")
|
||||
previews.append(f"[sketch]\n{sketch[:220]}")
|
||||
return {
|
||||
"fields": fields,
|
||||
"preview": "\n\n".join(previews)[:500],
|
||||
}
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
split_dir: str = "",
|
||||
data_path: str = "",
|
||||
split_mode: str = "ratio",
|
||||
split_ratio: str = "2:1:7",
|
||||
split_seed: int = 42,
|
||||
split_output_dir: str = "",
|
||||
max_turns: int = 1,
|
||||
exec_timeout: int = 600,
|
||||
workers: int = 64,
|
||||
analyst_workers: int = 16,
|
||||
failure_only: bool = False,
|
||||
minibatch_size: int = 8,
|
||||
edit_budget: int = 4,
|
||||
seed: int = 42,
|
||||
limit: int = 0,
|
||||
shuffle_choices: bool = True,
|
||||
use_theorem: bool = False,
|
||||
use_sketch: bool = False,
|
||||
max_completion_tokens: int = 16384,
|
||||
) -> None:
|
||||
self.max_turns = max_turns
|
||||
self.exec_timeout = exec_timeout
|
||||
self.workers = workers
|
||||
self.max_completion_tokens = int(max_completion_tokens)
|
||||
self.analyst_workers = analyst_workers
|
||||
self.failure_only = failure_only
|
||||
self.minibatch_size = minibatch_size
|
||||
self.edit_budget = edit_budget
|
||||
self.use_theorem = use_theorem
|
||||
self.use_sketch = use_sketch
|
||||
self.dataloader = LiveMathematicianBenchDataLoader(
|
||||
split_dir=split_dir,
|
||||
data_path=data_path,
|
||||
split_mode=split_mode,
|
||||
split_ratio=split_ratio,
|
||||
split_seed=split_seed,
|
||||
split_output_dir=split_output_dir,
|
||||
seed=seed,
|
||||
limit=limit,
|
||||
shuffle_choices=shuffle_choices,
|
||||
)
|
||||
|
||||
def setup(self, cfg: dict) -> None:
|
||||
super().setup(cfg)
|
||||
self.dataloader.setup(cfg)
|
||||
|
||||
def get_dataloader(self):
|
||||
return self.dataloader
|
||||
|
||||
def build_env_from_batch(self, batch: BatchSpec, **kwargs):
|
||||
return list(batch.payload or [])
|
||||
|
||||
def build_train_env(self, batch_size: int, seed: int, **kwargs):
|
||||
batch = self.dataloader.build_train_batch(batch_size=batch_size, seed=seed, **kwargs)
|
||||
return self.build_env_from_batch(batch, **kwargs)
|
||||
|
||||
def build_eval_env(self, env_num: int, split: str, seed: int, **kwargs):
|
||||
batch = self.dataloader.build_eval_batch(env_num=env_num, split=split, seed=seed, **kwargs)
|
||||
return self.build_env_from_batch(batch, **kwargs)
|
||||
|
||||
def rollout(
|
||||
self,
|
||||
env_manager,
|
||||
skill_content: str,
|
||||
out_dir: str,
|
||||
**kwargs,
|
||||
) -> list[dict]:
|
||||
items: list[dict] = env_manager
|
||||
return run_batch(
|
||||
items=items,
|
||||
out_root=out_dir,
|
||||
skill_content=skill_content,
|
||||
max_turns=self.max_turns,
|
||||
exec_timeout=self.exec_timeout,
|
||||
workers=self.workers,
|
||||
max_completion_tokens=self.max_completion_tokens,
|
||||
use_theorem=self.use_theorem,
|
||||
use_sketch=self.use_sketch,
|
||||
diagnostic_mode=kwargs.get("diagnostic_mode", False),
|
||||
diagnostic_instruction=kwargs.get("diagnostic_instruction", ""),
|
||||
diagnostic_trace_context_by_id=kwargs.get("diagnostic_trace_context_by_id"),
|
||||
task_timeout=self.exec_timeout,
|
||||
)
|
||||
|
||||
def reflect(
|
||||
self,
|
||||
results: list[dict],
|
||||
skill_content: str,
|
||||
out_dir: str,
|
||||
**kwargs,
|
||||
) -> list[dict | None]:
|
||||
prediction_dir = kwargs.get("prediction_dir", os.path.join(out_dir, "predictions"))
|
||||
patches_dir = kwargs.get("patches_dir", os.path.join(out_dir, "patches"))
|
||||
random_seed = kwargs.get("random_seed")
|
||||
step_buffer_context = kwargs.get("step_buffer_context", "")
|
||||
meta_skill_context = kwargs.get("meta_skill_context", "")
|
||||
|
||||
return run_minibatch_reflect(
|
||||
results=results,
|
||||
skill_content=skill_content,
|
||||
prediction_dir=prediction_dir,
|
||||
patches_dir=patches_dir,
|
||||
workers=self.analyst_workers,
|
||||
failure_only=self.failure_only,
|
||||
minibatch_size=self.minibatch_size,
|
||||
edit_budget=self.edit_budget,
|
||||
random_seed=random_seed,
|
||||
error_system=self.get_error_minibatch_prompt(),
|
||||
success_system=self.get_success_minibatch_prompt(),
|
||||
step_buffer_context=step_buffer_context,
|
||||
meta_skill_context=meta_skill_context,
|
||||
update_mode=getattr(self, "_cfg", {}).get("skill_update_mode", "patch"),
|
||||
)
|
||||
|
||||
|
||||
def get_task_types(self) -> list[str]:
|
||||
return self.dataloader.get_task_types()
|
||||
@@ -1,308 +0,0 @@
|
||||
"""LiveMathematicianBench task dataloader."""
|
||||
from __future__ import annotations
|
||||
|
||||
import glob
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import random
|
||||
from typing import Any
|
||||
|
||||
from skillopt.datasets.base import BatchSpec, SplitDataLoader
|
||||
|
||||
|
||||
# ── Raw data loading utilities (for preprocessing / standalone eval) ─────
|
||||
|
||||
_CHOICE_LABELS = ["A", "B", "C", "D", "E", "F", "G"]
|
||||
|
||||
|
||||
def _load_json(path: str) -> Any:
|
||||
with open(path) as f:
|
||||
return json.load(f)
|
||||
|
||||
|
||||
def _iter_monthly_files(data_path: str) -> list[str]:
|
||||
if not data_path:
|
||||
return []
|
||||
if os.path.isfile(data_path):
|
||||
return [data_path]
|
||||
if os.path.isdir(data_path):
|
||||
nested = glob.glob(
|
||||
os.path.join(data_path, "**", "qa_*_final.json"),
|
||||
recursive=True,
|
||||
)
|
||||
flat = glob.glob(os.path.join(data_path, "qa_*_final.json"))
|
||||
return sorted(set(nested + flat))
|
||||
return []
|
||||
|
||||
|
||||
def _coerce_choices(raw_choices: Any) -> list[dict]:
|
||||
if isinstance(raw_choices, list):
|
||||
choices: list[dict] = []
|
||||
for idx, item in enumerate(raw_choices):
|
||||
if isinstance(item, dict):
|
||||
label = str(item.get("label") or _CHOICE_LABELS[idx]).strip()
|
||||
text = str(item.get("text") or item.get("content") or "").strip()
|
||||
else:
|
||||
label = _CHOICE_LABELS[idx]
|
||||
text = str(item).strip()
|
||||
if text:
|
||||
choices.append({"label": label, "text": text})
|
||||
return choices
|
||||
|
||||
if isinstance(raw_choices, dict):
|
||||
labels = sorted(raw_choices.keys())
|
||||
return [
|
||||
{"label": str(label).strip(), "text": str(raw_choices[label]).strip()}
|
||||
for label in labels
|
||||
if str(raw_choices[label]).strip()
|
||||
]
|
||||
|
||||
return []
|
||||
|
||||
|
||||
def _coerce_theorem_types(raw: Any) -> list[str]:
|
||||
if isinstance(raw, list):
|
||||
return [str(x).strip() for x in raw if str(x).strip()]
|
||||
if raw is None:
|
||||
return []
|
||||
text = str(raw).strip()
|
||||
return [text] if text else []
|
||||
|
||||
|
||||
def _normalize_label(text: str) -> str:
|
||||
return str(text).strip().upper().rstrip(".):")
|
||||
|
||||
|
||||
def _normalize_item(item: dict, row_idx: int, source_path: str) -> dict:
|
||||
mcq = item.get("mcq", {}) if isinstance(item.get("mcq"), dict) else {}
|
||||
question = str(mcq.get("question") or item.get("question") or "").strip()
|
||||
choices = _coerce_choices(mcq.get("choices") or item.get("choices") or [])
|
||||
correct = mcq.get("correct_choice") or item.get("correct_choice") or {}
|
||||
|
||||
if isinstance(correct, dict):
|
||||
correct_label = _normalize_label(correct.get("label", ""))
|
||||
correct_text = str(correct.get("text") or "").strip()
|
||||
else:
|
||||
correct_label = _normalize_label(correct)
|
||||
correct_text = ""
|
||||
|
||||
choice_by_label = {
|
||||
_normalize_label(choice["label"]): choice["text"]
|
||||
for choice in choices
|
||||
}
|
||||
if correct_label and not correct_text:
|
||||
correct_text = choice_by_label.get(correct_label, "")
|
||||
if correct_label and correct_text and correct_label not in choice_by_label:
|
||||
choices.append({"label": correct_label, "text": correct_text})
|
||||
choices.sort(key=lambda choice: _CHOICE_LABELS.index(choice["label"]) if choice["label"] in _CHOICE_LABELS else len(_CHOICE_LABELS))
|
||||
choice_by_label[correct_label] = correct_text
|
||||
|
||||
month = str(item.get("month") or "").strip()
|
||||
item_no = item.get("no", row_idx + 1)
|
||||
item_id = f"{month}:{item_no}" if month else str(item_no)
|
||||
|
||||
return {
|
||||
"id": item_id,
|
||||
"month": month,
|
||||
"no": item_no,
|
||||
"paper_link": str(item.get("paper_link") or "").strip(),
|
||||
"theorem": str(item.get("theorem") or "").strip(),
|
||||
"sketch": str(item.get("sketch") or "").strip(),
|
||||
"theorem_type": _coerce_theorem_types(item.get("theorem_type")),
|
||||
"question": question,
|
||||
"choices": choices,
|
||||
"correct_choice": {
|
||||
"label": correct_label,
|
||||
"text": correct_text,
|
||||
},
|
||||
"source_path": source_path,
|
||||
}
|
||||
|
||||
|
||||
def load_items(data_path: str) -> list[dict]:
|
||||
"""Load and normalise LiveMathematicianBench items from JSON files."""
|
||||
files = _iter_monthly_files(data_path)
|
||||
if not files:
|
||||
raise ValueError(
|
||||
"LiveMathematicianBench requires data_path to be a qa_*_final.json file "
|
||||
"or a directory containing monthly qa_*_final.json files."
|
||||
)
|
||||
|
||||
items: list[dict] = []
|
||||
for path in files:
|
||||
raw = _load_json(path)
|
||||
if not isinstance(raw, list):
|
||||
raise ValueError(f"Expected JSON array in {path}, got {type(raw).__name__}")
|
||||
for row_idx, item in enumerate(raw):
|
||||
norm = _normalize_item(item, row_idx=row_idx, source_path=path)
|
||||
if norm["question"] and norm["choices"] and norm["correct_choice"]["label"]:
|
||||
items.append(norm)
|
||||
if not items:
|
||||
raise ValueError(f"No valid LiveMathematicianBench items loaded from {data_path}")
|
||||
return items
|
||||
|
||||
|
||||
# ── Dataloader ───────────────────────────────────────────────────────────
|
||||
|
||||
class LiveMathematicianBenchDataLoader(SplitDataLoader):
|
||||
"""LiveMathematicianBench dataloader with per-seed choice shuffling."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
split_dir: str = "",
|
||||
data_path: str = "",
|
||||
split_mode: str = "ratio",
|
||||
split_ratio: str = "2:1:7",
|
||||
split_seed: int = 42,
|
||||
split_output_dir: str = "",
|
||||
seed: int = 42,
|
||||
limit: int = 0,
|
||||
shuffle_choices: bool = True,
|
||||
**kwargs,
|
||||
) -> None:
|
||||
super().__init__(
|
||||
split_dir=split_dir,
|
||||
data_path=data_path,
|
||||
split_mode=split_mode,
|
||||
split_ratio=split_ratio,
|
||||
split_seed=split_seed,
|
||||
split_output_dir=split_output_dir,
|
||||
seed=seed,
|
||||
limit=limit,
|
||||
)
|
||||
self.shuffle_choices = shuffle_choices
|
||||
self._task_types: list[str] = []
|
||||
|
||||
def load_raw_items(self, data_path: str) -> list[dict]:
|
||||
return load_items(data_path)
|
||||
|
||||
def setup(self, cfg: dict) -> None:
|
||||
super().setup(cfg)
|
||||
all_items = self.train_items + self.val_items + self.test_items
|
||||
task_types: set[str] = set()
|
||||
for item in all_items:
|
||||
for name in item.get("theorem_type", []):
|
||||
if name:
|
||||
task_types.add(name)
|
||||
self._task_types = sorted(task_types)
|
||||
|
||||
def get_task_types(self) -> list[str]:
|
||||
return list(self._task_types)
|
||||
|
||||
# ── Choice shuffling ─────────────────────────────────────────────────
|
||||
|
||||
@staticmethod
|
||||
def _item_shuffle_seed(item_id: str, seed: int) -> int:
|
||||
digest = hashlib.sha256(f"{seed}:{item_id}".encode("utf-8")).hexdigest()
|
||||
return int(digest[:16], 16)
|
||||
|
||||
def _shuffle_item_choices(self, item: dict, seed: int) -> dict:
|
||||
if not self.shuffle_choices:
|
||||
return {
|
||||
**item,
|
||||
"choices": [dict(c) for c in item["choices"]],
|
||||
"correct_choice": dict(item["correct_choice"]),
|
||||
}
|
||||
|
||||
shuffled_choices = [dict(c) for c in item["choices"]]
|
||||
rng = random.Random(self._item_shuffle_seed(str(item["id"]), seed))
|
||||
rng.shuffle(shuffled_choices)
|
||||
|
||||
original_correct = _normalize_label(item["correct_choice"]["label"])
|
||||
remapped_choices: list[dict] = []
|
||||
new_correct_choice = dict(item["correct_choice"])
|
||||
|
||||
for idx, choice in enumerate(shuffled_choices):
|
||||
new_label = _CHOICE_LABELS[idx]
|
||||
old_label = _normalize_label(choice["label"])
|
||||
remapped_choices.append({"label": new_label, "text": choice["text"]})
|
||||
if old_label == original_correct:
|
||||
new_correct_choice = {"label": new_label, "text": choice["text"]}
|
||||
|
||||
transformed = dict(item)
|
||||
transformed["choices"] = remapped_choices
|
||||
transformed["correct_choice"] = new_correct_choice
|
||||
return transformed
|
||||
|
||||
def _materialize_batch(self, items: list[dict], seed: int) -> list[dict]:
|
||||
return [self._shuffle_item_choices(item, seed) for item in items]
|
||||
|
||||
# ── Batch construction (override for choice shuffling) ───────────────
|
||||
|
||||
def plan_train_epoch(
|
||||
self,
|
||||
*,
|
||||
epoch: int,
|
||||
steps_per_epoch: int,
|
||||
accumulation: int,
|
||||
batch_size: int,
|
||||
seed: int,
|
||||
**kwargs,
|
||||
) -> list[BatchSpec]:
|
||||
"""Build a shuffled epoch while preserving per-batch choice shuffling."""
|
||||
epoch_rng = random.Random(seed + epoch * 1000)
|
||||
items = list(self.train_items)
|
||||
epoch_rng.shuffle(items)
|
||||
|
||||
total_batches = steps_per_epoch * accumulation
|
||||
if total_batches <= 0:
|
||||
return []
|
||||
|
||||
batches: list[BatchSpec] = []
|
||||
cursor = 0
|
||||
for batch_idx in range(total_batches):
|
||||
batch_seed = seed + epoch * 1000 + batch_idx + 1
|
||||
batch_items = items[cursor: cursor + batch_size]
|
||||
cursor += len(batch_items)
|
||||
|
||||
if not batch_items and items:
|
||||
refill_rng = random.Random(batch_seed)
|
||||
batch_items = list(items)
|
||||
refill_rng.shuffle(batch_items)
|
||||
batch_items = batch_items[:batch_size]
|
||||
|
||||
batch_items = self._materialize_batch(batch_items, batch_seed)
|
||||
batches.append(
|
||||
BatchSpec(
|
||||
phase="train",
|
||||
split="train",
|
||||
seed=batch_seed,
|
||||
batch_size=len(batch_items),
|
||||
payload=batch_items,
|
||||
)
|
||||
)
|
||||
|
||||
return batches
|
||||
|
||||
def build_train_batch(self, batch_size: int, seed: int, **kwargs) -> BatchSpec:
|
||||
rng = random.Random(seed)
|
||||
items = list(self.train_items)
|
||||
rng.shuffle(items)
|
||||
items = self._materialize_batch(items[:batch_size], seed)
|
||||
return BatchSpec(
|
||||
phase="train",
|
||||
split="train",
|
||||
seed=seed,
|
||||
batch_size=len(items),
|
||||
payload=items,
|
||||
)
|
||||
|
||||
def build_eval_batch(
|
||||
self,
|
||||
env_num: int,
|
||||
split: str,
|
||||
seed: int,
|
||||
**kwargs,
|
||||
) -> BatchSpec:
|
||||
items = self.get_split_items(split)
|
||||
if env_num and env_num < len(items):
|
||||
items = items[:env_num]
|
||||
items = self._materialize_batch(items, seed)
|
||||
return BatchSpec(
|
||||
phase="eval",
|
||||
split=split,
|
||||
seed=seed,
|
||||
batch_size=len(items),
|
||||
payload=items,
|
||||
)
|
||||
@@ -1,62 +0,0 @@
|
||||
"""LiveMathematicianBench evaluation helpers."""
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
|
||||
|
||||
def extract_answer(text: str) -> str:
|
||||
matches = re.findall(r"<answer>(.*?)</answer>", text, re.DOTALL | re.IGNORECASE)
|
||||
if matches:
|
||||
return matches[-1].strip()
|
||||
lines = [ln.strip() for ln in text.strip().splitlines() if ln.strip()]
|
||||
if lines:
|
||||
return lines[-1]
|
||||
return text.strip()
|
||||
|
||||
|
||||
def normalize_label(text: str) -> str:
|
||||
return str(text).strip().upper().rstrip(".):")
|
||||
|
||||
|
||||
def parse_choice_label(prediction_text: str, choices: list[dict]) -> str:
|
||||
answer = extract_answer(prediction_text)
|
||||
label = normalize_label(answer)
|
||||
valid_labels = {normalize_label(choice.get("label", "")) for choice in choices}
|
||||
if label in valid_labels:
|
||||
return label
|
||||
|
||||
answer_lower = answer.lower()
|
||||
for choice in choices:
|
||||
choice_label = normalize_label(choice.get("label", ""))
|
||||
choice_text = str(choice.get("text", "")).strip()
|
||||
if choice_text and choice_text.lower() == answer_lower:
|
||||
return choice_label
|
||||
|
||||
first_token = normalize_label(answer.split()[0]) if answer.split() else ""
|
||||
if first_token in valid_labels:
|
||||
return first_token
|
||||
return label
|
||||
|
||||
|
||||
def evaluate(prediction_text: str, correct_choice: dict, choices: list[dict]) -> dict:
|
||||
predicted_label = parse_choice_label(prediction_text, choices)
|
||||
correct_label = normalize_label(correct_choice.get("label", ""))
|
||||
predicted_text = ""
|
||||
correct_text = str(correct_choice.get("text", "")).strip()
|
||||
|
||||
for choice in choices:
|
||||
if normalize_label(choice.get("label", "")) == predicted_label:
|
||||
predicted_text = str(choice.get("text", "")).strip()
|
||||
break
|
||||
|
||||
is_correct = float(predicted_label == correct_label)
|
||||
return {
|
||||
"em": is_correct,
|
||||
"f1": is_correct,
|
||||
"sub_em": is_correct,
|
||||
"predicted_answer": predicted_label or extract_answer(prediction_text),
|
||||
"predicted_label": predicted_label,
|
||||
"predicted_text": predicted_text,
|
||||
"correct_label": correct_label,
|
||||
"correct_text": correct_text,
|
||||
}
|
||||
@@ -1,37 +0,0 @@
|
||||
You are an expert failure-analysis agent for theorem-grounded mathematical multiple-choice questions.
|
||||
|
||||
You will be given MULTIPLE failed trajectories from a single minibatch and the current skill document.
|
||||
Each trajectory includes the target's response and an evaluation result showing the predicted option
|
||||
versus the correct option.
|
||||
|
||||
Your job is to identify COMMON reasoning failures across the batch and propose concise skill edits.
|
||||
|
||||
## Failure Type Categories
|
||||
- **quantifier_miss**: the agent missed exact quantifiers, scope, or existence/uniqueness conditions
|
||||
- **strength_mismatch**: the agent preferred a weaker or stronger statement than what was proved
|
||||
- **condition_miss**: the agent ignored hypotheses, equality cases, or domain restrictions
|
||||
- **option_confusion**: the agent confused similar answer choices or failed to compare them exactly
|
||||
- **other**: none of the above
|
||||
|
||||
## Rules
|
||||
1. Focus on patterns that recur across the minibatch.
|
||||
2. Prefer edits that improve exact choice discrimination, not theorem-specific memorization.
|
||||
3. Do not hardcode paper-specific content.
|
||||
4. Only patch gaps not already covered by the skill.
|
||||
|
||||
Respond ONLY with a valid JSON object:
|
||||
{
|
||||
"batch_size": <number>,
|
||||
"failure_summary": [
|
||||
{"failure_type": "<type>", "count": <int>, "description": "<one-line>"}
|
||||
],
|
||||
"patch": {
|
||||
"reasoning": "<why these edits address the common failures>",
|
||||
"edits": [
|
||||
{"op": "append", "content": "<markdown>"},
|
||||
{"op": "insert_after", "target": "<heading/text>", "content": "<markdown>"},
|
||||
{"op": "replace", "target": "<old text>", "content": "<new text>"},
|
||||
{"op": "delete", "target": "<exact text to remove>"}
|
||||
]
|
||||
}
|
||||
}
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user