Yif-Yang
4.2 KiB
The Training Loop
SkillOpt's core insight: optimizing natural-language skill documents follows the same structure as training neural networks.
Overview
┌─────────────────────────────────────────────────────────┐
│ Training Loop │
│ │
│ for epoch in epochs: │
│ for step in steps: │
│ 1. Rollout — Target executes tasks │
│ 2. Reflect — Optimizer analyzes trajectories │
│ 3. Aggregate — Hierarchical merge of patches │
│ 4. Select — Rank & clip edits (learning rate) │
│ 5. Update — Apply patches to skill doc │
│ 6. Gate — Validate & accept/reject │
│ │
│ Epoch Boundary: │
│ • Slow Update (longitudinal comparison & guidance) │
│ • Meta Skill (cross-epoch strategy memory) │
└─────────────────────────────────────────────────────────┘
Stage Details
1. Rollout (Forward Pass)
The target model executes tasks using the current skill document as its prompt. Each task produces a trajectory and a score.
# Analogy: forward pass through the network
predictions = model(input, skill_document)
scores = evaluate(predictions, ground_truth)
2. Reflect (Backward Pass)
The optimizer model analyzes trajectory minibatches and produces edit
patches — structured suggestions for improving the skill document. Failure
minibatches are always eligible for analysis; successful trajectories are also
analyzed unless gradient.failure_only is enabled. Independent minibatches can
run concurrently according to gradient.analyst_workers.
# Analogy: computing gradients
gradients = loss.backward() # → edit patches
3. Aggregate
Semantically similar edit patches are merged to avoid redundant edits.
4. Select (Gradient Clipping)
Edits are ranked by relevance score. The learning_rate parameter caps how many edits are applied per step — just like gradient clipping prevents overshooting.
# Analogy: gradient clipping + optimizer step size
selected = top_k(edits, k=learning_rate)
The lr_scheduler adjusts this over training:
- cosine: Start aggressive, taper smoothly
- linear: Linear decay
- constant: Fixed rate
5. Update (Parameter Update)
Selected edits are applied to the skill document, producing a new version.
6. Gate (Validation)
The updated skill is evaluated on a selection split (analogous to a
validation set). With the gate enabled, the candidate is accepted only when its
configured gate score (hard, soft, or mixed) is strictly higher than the
current skill's score. With evaluation.use_gate: false, validation is still
recorded but candidates are force-accepted.
Epoch Boundary Mechanisms
Slow Update
At the end of each epoch (starting from epoch 2), the system performs a
longitudinal comparison: it rolls out both the previous epoch's skill and
the current skill on the same samples, categorizes items as
improved/regressed/persistent-fail/stable-success, then generates high-level
guidance for the skill document. Depending on
optimizer.slow_update_gate_with_selection, that guidance is either checked on
the selection split or applied unconditionally. Its purpose is to counter
cross-epoch forgetting.
Meta Skill
A meta-skill memory accumulates high-level strategy notes across the training run. Starting at the end of epoch 2, the optimizer compares the previous and current epoch, writes a compact memory, and provides the prior epoch's memory as additional context during later reflection and update stages.