url/SkillOpt
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(spreadsheetbench)+optimizer: fix verify-feedback bloat, drop optimizer-side truncation, soft-disable gate

Browse Source 
A. SpreadsheetBench verification-feedback bloat
   - rollout.py _auto_verify_output: use official _compare_cell_value (was
     repr() equality, which falsely flagged 5 vs 5.0 / None vs ""); collapse
     correct-and-empty cells into a count so large sparse answer ranges no
     longer flood feedback with MBs of None=None noise.
   - codegen_agent.py _build_eval_feedback: only list WRONG cells, collapse
     correct ones into a count.
   Scoring is unaffected (evaluate() is independent); this only fixes the
   target model's multi-turn solving feedback.

B. Remove optimizer-side truncation (bloat source now fixed)
   - reflect.py: drop _MAX_TRAJ_CHARS cap and all per-field clips.
   - update_modes.py / clip.py / lr_autonomous.py: describe_item /
     short_item_summary no longer truncate; raise ranking/lr token budget.
   - trainer.py _format_step_buffer: full task_ids / target.
   - slow_update.py: full comparison samples.

C. Soft-disable gate
   - config.py / trainer.py: use_gate=false no longer raises; validation still
     runs but candidates are force-accepted (new force_accept branch + log).

Misc: aggregate.py merge token budget 4096 -> 16384.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This commit is contained in:
Cuzyoung
2026-06-01 11:23:08 +00:00
parent f64a41397c
commit 372fd56c1e
10 changed files with 138 additions and 94 deletions
Download Patch File Download Diff File Expand all files Collapse all files
skillopt/optimizer/update_modes.py
+10 -11
View File
@@ -70,7 +70,7 @@ def truncate_payload(container: dict, max_items: int, mode: str | None) -> dict:
return container
def describe_item(item: dict, mode: str | None, *, max_chars: int = 240) -> str:
def describe_item(item: dict, mode: str | None, *, max_chars: int | None = None) -> str:
if not isinstance(item, dict):
return ""
if is_full_rewrite_minibatch_mode(mode):
@@ -84,7 +84,7 @@ def describe_item(item: dict, mode: str | None, *, max_chars: int = 240) -> str:
parts.append(f"support={item.get('support_count')}")
new_skill = str(item.get("new_skill", "")).strip()
if new_skill:
parts.append(f"new_skill_preview={new_skill[:120]!r}")
parts.append(f"new_skill_preview={new_skill!r}")
text = " ".join(parts)
elif is_rewrite_mode(mode):
parts = [
@@ -109,28 +109,27 @@ def describe_item(item: dict, mode: str | None, *, max_chars: int = 240) -> str:
if item.get("support_count") is not None:
parts.append(f"support={item.get('support_count')}")
text = " ".join(parts)
if len(text) <= max_chars:
return text
return text[: max_chars - 3].rstrip() + "..."
# Truncation disabled: the optimizer is given the full item description.
return text
def short_item_summary(item: dict, mode: str | None, *, max_chars: int = 200) -> dict[str, Any]:
def short_item_summary(item: dict, mode: str | None, *, max_chars: int | None = None) -> dict[str, Any]:
if is_full_rewrite_minibatch_mode(mode):
return {
"title": str(item.get("title", ""))[:max_chars],
"title": str(item.get("title", "")),
"change_summary": [
str(x)[:max_chars] for x in item.get("change_summary", [])[:3]
str(x) for x in item.get("change_summary", [])
] if isinstance(item.get("change_summary"), list) else [],
"source_type": item.get("source_type", ""),
}
if is_rewrite_mode(mode):
return {
"type": item.get("type", "?"),
"title": str(item.get("title", ""))[:max_chars],
"instruction": str(item.get("instruction", ""))[:max_chars],
"title": str(item.get("title", "")),
"instruction": str(item.get("instruction", "")),
}
return {
"op": item.get("op", "?"),
"content": str(item.get("content", ""))[:max_chars],
"content": str(item.get("content", "")),
"target": item.get("target", ""),
}