@bokiko benchmarked the feature on three hosts and found every setting is
either no faster or no longer coherent. Reproduced here on a 25 GB box, and
the numbers are worse than "a quality/speed tradeoff":
- budget=8 -> hellaswag 30% vs 90% with it off (25% is the chance floor)
- budget=4 -> decode is literal noise ("The **1...: s2151:")
- MTP acceptance 0%. Which experts survive the cap depends on cache
residency at the moment of the forward, so draft and verify do not
compute the same function -- the invariant #294 just established for
#163, violated through cache state instead of kernel dispatch. SPEC_PIN
cannot fix that and should not try: it is feature semantics, not
dispatch.
- 0.13 tok/s vs 0.30 baseline, while loading 14.66 experts per layer
against a topk=8 baseline. The cap does MORE disk I/O than not using it.
"~335 GB I/O saved" counts dropped experts, not bytes not read.
EXPERT_BUDGET>0 is now ignored unless EXPERT_BUDGET_EXPERIMENTAL=1, with the
measurements printed. The code stays compiled and developable: MoE-Spec
(arXiv 2602.16052) is not a wrong idea, this implementation just has no
point where it is both faster and correct. Re-enabling it by default needs a
quality number next to every speed number.
Also gates the cap to decode (S<=4), @woolcoxm's fix from #292/#298: during
prefill the batch union is 30-100+ experts, and capping to 4-8 drops most of
them, corrupting the hidden state and therefore the KV cache. Necessary but
not sufficient -- the run above already includes it.
Removes issue_budget.md, issue_diskio.md and issue_grouped_quant.md: design
notes in the repo root, unlinked from any README, whose only user-facing
content was "EXPERT_BUDGET=6-8 -- good speedup, minimal quality loss" and
"+83% decode at budget=4". Measurement says otherwise, and they were the
only thing on main telling anyone to switch this on. They stay in history.
Reported-by: bokiko <#303>
Co-Authored-By: woolcoxm <#292>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
JustVugg
35f90b9e76