lvbaocheng
5d7875cb2e
The training gate currently always compares candidate vs. current/best using *hard* exact-match accuracy. On environments with a small held-out selection set (e.g. 3-6 items) or partial-credit scoring, hard accuracy is too coarse: candidate skills that meaningfully improve per-item soft scores get rejected because the discrete hard count does not move. Add three opt-in metrics so users can pick the one that matches their scoring function: - `gate_metric: hard` — original behavior (default, fully backward compatible). - `gate_metric: soft` — gate on the soft / F1 / partial-credit score. - `gate_metric: mixed` — `(1 - w) * hard + w * soft`, where `w` is set by `gate_mixed_weight` (default 0.5). Changes ------- - `skillopt/evaluation/gate.py`: extend `evaluate_gate` with `cand_soft`, `metric`, and `mixed_weight` keyword arguments; add a pure helper `select_gate_score(hard, soft, metric, mixed_weight)`. Defaults preserve the original `metric="hard"` behavior — existing callers that only pass `cand_hard` keep working unchanged. - `skillopt/evaluation/__init__.py`: export the new helper / type. - `skillopt/engine/trainer.py`: read `evaluation.gate_metric` and `evaluation.gate_mixed_weight` from the config (with safe defaults), pass both metrics into `evaluate_gate`, and project the baseline `current_score` / `best_score` into metric space so subsequent comparisons are consistent. Print the gate metric on the `[6/6 EVALUATE]` line so logs make the decision basis explicit. The selection cache still records both `(hard, soft)` so a metric change on resume is non-destructive. - `configs/_base_/default.yaml`: document and ship the new keys with backward-compatible defaults (`hard`, `0.5`). Backward compatibility ---------------------- - Default config does not change behavior: `gate_metric` defaults to `hard`, exactly matching the previous gate. - `evaluate_gate(...)` keeps its existing positional signature; the new parameters are keyword-only with safe defaults. - `step_record.json` gains optional `gate_metric` and `candidate_gate_score` fields; old records still load. Tested ------ - Unit-tested all three metrics + boundary `mixed_weight` values (0.0 / 1.0) and rejection of unknown metric strings. All six cases pass. - Verified `skillopt.engine.trainer` imports cleanly after the refactor.
94 lines
3.3 KiB
YAML
94 lines
3.3 KiB
YAML
# SkillOpt default configuration — base for all environments.
|
|
# Environment configs should inherit via: _base_: default.yaml
|
|
|
|
model:
|
|
backend: azure_openai
|
|
optimizer: gpt-5.5
|
|
target: gpt-5.5
|
|
optimizer_backend: openai_chat
|
|
target_backend: openai_chat
|
|
reasoning_effort: medium
|
|
rewrite_reasoning_effort: ""
|
|
rewrite_max_completion_tokens: 64000
|
|
codex_exec_path: codex
|
|
codex_exec_sandbox: workspace-write
|
|
codex_exec_profile: ""
|
|
codex_exec_full_auto: false
|
|
codex_exec_reasoning_effort: none
|
|
codex_exec_use_sdk: auto
|
|
codex_exec_network_access: false
|
|
codex_exec_web_search: false
|
|
codex_exec_approval_policy: never
|
|
claude_code_exec_path: claude
|
|
claude_code_exec_profile: ""
|
|
claude_code_exec_use_sdk: auto
|
|
claude_code_exec_effort: medium
|
|
claude_code_exec_max_thinking_tokens: 16384
|
|
codex_trace_to_optimizer: true
|
|
azure_openai_endpoint: "" # e.g. "https://your-resource.openai.azure.com/"
|
|
azure_openai_api_version: "2024-12-01-preview"
|
|
azure_openai_api_key: "" # Fill locally if you do not export AZURE_OPENAI_API_KEY
|
|
azure_openai_auth_mode: azure_cli
|
|
azure_openai_ad_scope: "https://cognitiveservices.azure.com/.default"
|
|
azure_openai_managed_identity_client_id: ""
|
|
optimizer_azure_openai_endpoint: "" # e.g. "https://your-resource.openai.azure.com/"
|
|
optimizer_azure_openai_api_version: "2024-12-01-preview"
|
|
optimizer_azure_openai_api_key: ""
|
|
optimizer_azure_openai_auth_mode: azure_cli
|
|
optimizer_azure_openai_ad_scope: "https://cognitiveservices.azure.com/.default"
|
|
optimizer_azure_openai_managed_identity_client_id: ""
|
|
target_azure_openai_endpoint: "" # e.g. "https://your-resource.openai.azure.com/"
|
|
target_azure_openai_api_version: "2024-12-01-preview"
|
|
target_azure_openai_api_key: ""
|
|
target_azure_openai_auth_mode: azure_cli
|
|
target_azure_openai_ad_scope: "https://cognitiveservices.azure.com/.default"
|
|
target_azure_openai_managed_identity_client_id: ""
|
|
|
|
train:
|
|
num_epochs: 4
|
|
train_size: 0 # 0 = derive from dataset split when available
|
|
batch_size: 40
|
|
accumulation: 1
|
|
seed: 42
|
|
|
|
gradient:
|
|
minibatch_size: 8
|
|
merge_batch_size: 8
|
|
analyst_workers: 16
|
|
max_analyst_rounds: 3
|
|
failure_only: false
|
|
|
|
optimizer:
|
|
learning_rate: 4 # max edits per step (edit_budget)
|
|
min_learning_rate: 2 # min edits for decay schedulers
|
|
lr_scheduler: cosine # constant / linear / cosine / autonomous
|
|
lr_control_mode: fixed # fixed / autonomous / none
|
|
skill_update_mode: patch # patch / rewrite_from_suggestions / full_rewrite_minibatch
|
|
use_slow_update: true
|
|
slow_update_samples: 20
|
|
longitudinal_pair_policy: mixed # mixed / changed / unchanged
|
|
use_meta_skill: true
|
|
|
|
evaluation:
|
|
use_gate: true
|
|
# gate_metric: 'hard' (default, backward-compatible),
|
|
# 'soft' (use soft/F1 score),
|
|
# 'mixed' ((1 - w) * hard + w * soft).
|
|
# See skillopt/evaluation/gate.py for details.
|
|
gate_metric: hard
|
|
gate_mixed_weight: 0.5
|
|
sel_env_num: 0
|
|
test_env_num: 0
|
|
eval_test: true
|
|
|
|
env:
|
|
name: ""
|
|
skill_init: ""
|
|
split_mode: ratio # ratio = build deterministic split from data_path; split_dir = use pre-split train/val/test
|
|
split_seed: 42
|
|
split_dir: ""
|
|
data_path: ""
|
|
split_output_dir: ""
|
|
exec_timeout: 120 # per target model/code-agent call timeout in seconds
|
|
out_root: ""
|