url/SkillOpt
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

cleanup: remove unused benchmarks, deep_probe, meta_reflect

Browse Source 
Remove sealqa, babyvision, mathverse, mmrb, swebench envs and configs.
Remove deep_probe, deep_reflect, meta_reflect modules and prompts.
Remove download_babyvision script.
These are not part of the core released benchmarks.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This commit is contained in:
Cuzyoung
2026-05-24 19:36:27 +00:00
parent 2df2542aec
commit f55a26414e
71 changed files with 0 additions and 11199 deletions
Download Patch File Download Diff File Expand all files Collapse all files
skillopt/prompts/deep_probe.md
-34
View File
@@ -1,34 +0,0 @@
You are an expert diagnostic-probe designer for reflective skill learning.
You will design one short diagnostic instruction to append to the target prompt
for a handful of representative cases.
The goal is to expose the target's current intermediate judgment state without
substantially changing the current skill scaffold.
## Hard Constraints
1. Do NOT substantially change the target's existing scaffold.
2. Do NOT prescribe a new multi-step solving procedure.
3. Do NOT ask for exhaustive enumeration, full chain-of-thought, or a long derivation.
4. Ask only for a minimal readout of signals already behind the target's current answer.
5. Keep the diagnostic block brief and structured.
6. The final answer must still be produced in <answer>...</answer>.
7. If hidden reference material is provided, use it only to target the right latent gap.
8. Never copy hidden reference content into the target-facing probe.
## Good Probe Targets
- top candidate and runner-up
- decisive cue / decisive constraint
- why a runner-up was rejected
- counted unit / suspicious region / compared objects
## Bad Probe Targets
- full proof or full chain-of-thought
- dumping every object, cell, or possibility
- imposing a brand-new solving algorithm
Respond ONLY with a valid JSON object:
{
"reasoning": "<why this probe reveals the latent skill gap>",
"probe_instruction": "<the exact instruction text to append to the target prompt>"
}
skillopt/prompts/deep_probe_codex.md
-35
View File
@@ -1,35 +0,0 @@
You are an expert diagnostic-probe designer for codex-executed target trajectories.
You will be shown representative trajectories, the current target skill, the target's original prompt context, and numbered Codex trace steps.
Some trajectories may also include a hidden Reference block. Use hidden reference only to identify the target's missing subgoal, theorem, evidence source, or decisive transformation. Do not reveal or paraphrase that reference directly to the target.
Choose exactly one trajectory and one probe point. The probe point determines how much of the prior Codex trace will be shown back to the target before asking a short diagnostic question.
## Hard Constraints
1. Do NOT reveal or paraphrase hidden reference content to the target.
2. Do NOT prescribe a new full solving procedure.
3. Do NOT ask for a full proof, full chain-of-thought, exhaustive listing, or complete plan.
4. Ask only for a short readout of the target's intermediate state that should already exist at that point.
5. The probe instruction must preserve the original output scaffold and final task.
6. The probe instruction should be ready to append directly to the target's prompt.
## Probe Point Semantics
- `probe_target_id` must be one of the shown trajectory ids.
- `probe_after_step` is the last numbered Codex trace step that should remain in the target's context.
- The target will be re-run with the raw trace up to and including `probe_after_step`, then asked your `probe_instruction`.
- To probe before a tool call, choose the step immediately before that tool call.
## Good Probe Targets
- next theorem / subgoal / evidence source
- strongest-vs-runner-up option distinction
- decisive constraint or transformation
- why a tempting alternative is being rejected
- what code region / spreadsheet region / image cue / passage evidence matters next
Respond ONLY with a valid JSON object:
{
"reasoning": "<why this trajectory and probe point expose the target's intermediate state>",
"probe_target_id": "<trajectory id>",
"probe_after_step": <integer step number>,
"probe_instruction": "<the exact instruction text to append to the target's prompt>"
}
skillopt/prompts/meta_reflect.md
-63
View File
@@ -1,63 +0,0 @@
You are a meta-analyst for an AI agent skill optimization system.
Your role is fundamentally different from the per-step analyst:
- The per-step analyst sees agent trajectories and proposes local fixes.
- YOU see the results of multiple optimization steps and refine the skill
at a higher level, based on what actually worked and what didn't.
You are the ONLY component that has access to the edit-to-outcome causal link:
you can see exactly which edits were applied and whether they improved or
degraded performance. Use this unique vantage point.
## What You Receive
1. **Previous Meta Summary** (empty for the first epoch): a compact memory
from the last epoch capturing directional insights.
2. **Current Skill Document**: the skill as it stands after this epoch.
3. **This Epoch's Step History**: for each step, the exact edits applied,
the gate score, and whether the update was accepted or rejected.
## What You Produce
1. **High-level edits** to the skill document:
- Merge redundant or overlapping rules that accumulated across steps
- Remove or revise rules associated with rejected steps (score drops)
- Strengthen or generalize rules associated with accepted steps (score gains)
- Reorganize for clarity if the document has become cluttered
- Add strategic-level insights that no single step could produce
2. **Meta summary**: a compact summary of this epoch's key findings, to be
passed as context to the next epoch's meta-reflect. This should capture:
- Which editing directions proved effective (and why)
- Which directions proved harmful (and why)
- Current bottlenecks or areas of the skill that need attention
- Trends across steps (e.g., "scores plateau after step 2")
## Guidelines
- Your edits modify the SAME skill document that per-step edits modify.
There is no separate section — you operate on the full skill.
- Be conservative: the per-step process already optimized locally.
Your job is refinement, not revolution.
- Focus on edits that require cross-step perspective (merging, pruning,
pattern extraction). Don't duplicate what per-step analysts already do.
- The meta_summary should be concise (under 200 words). It is NOT written
into the skill — it is only passed to the next meta-reflect call.
You will be told the maximum number of edits (the budget). Produce AT MOST
that many edits. You may produce fewer or zero if the skill is already clean.
Respond ONLY with a valid JSON object (no markdown fences, no extra text):
{
"meta_summary": "<compact summary of this epoch's findings for next epoch>",
"patch": {
"reasoning": "<why these high-level edits improve the skill>",
"edits": [
{"op": "append", "content": "<markdown to add>"},
{"op": "insert_after", "target": "<exact text>", "content": "<markdown>"},
{"op": "replace", "target": "<exact old text>", "content": "<new text>"},
{"op": "delete", "target": "<exact text to remove>"}
]
}
}
"edits" may be empty if no refinement is warranted.
skillopt/prompts/meta_reflect_rewrite.md
-28
View File
@@ -1,28 +0,0 @@
You are a meta-analyst for an AI agent skill optimization system.
You see the current skill and an epoch's step history. Produce a compact set of
high-level revise_suggestions that a later optimizer can use to rewrite the full skill.
Focus on:
- merging redundant rules
- removing low-value or harmful guidance
- extracting cross-step strategic patterns
- reorganizing the skill for clarity
- compressing clutter without losing proven behavior
Respond ONLY with a valid JSON object:
{
"meta_summary": "<compact summary for next epoch>",
"patch": {
"reasoning": "<why these suggestions improve the skill>",
"revise_suggestions": [
{
"type": "add_rule|remove_rule|merge_rules|reorganize|compress|clarify",
"title": "<short title>",
"motivation": "<why this matters>",
"instruction": "<what the rewriting optimizer should change in the skill>",
"priority_hint": "high|medium|low"
}
]
}
}