Quarantine EXPERT_BUDGET: the operating window is measured empty (#303)

@bokiko benchmarked the feature on three hosts and found every setting is
either no faster or no longer coherent. Reproduced here on a 25 GB box, and
the numbers are worse than "a quality/speed tradeoff":

  - budget=8 -> hellaswag 30% vs 90% with it off (25% is the chance floor)
  - budget=4 -> decode is literal noise ("The **1...: s2151:")
  - MTP acceptance 0%. Which experts survive the cap depends on cache
    residency at the moment of the forward, so draft and verify do not
    compute the same function -- the invariant #294 just established for
    #163, violated through cache state instead of kernel dispatch. SPEC_PIN
    cannot fix that and should not try: it is feature semantics, not
    dispatch.
  - 0.13 tok/s vs 0.30 baseline, while loading 14.66 experts per layer
    against a topk=8 baseline. The cap does MORE disk I/O than not using it.
    "~335 GB I/O saved" counts dropped experts, not bytes not read.

EXPERT_BUDGET>0 is now ignored unless EXPERT_BUDGET_EXPERIMENTAL=1, with the
measurements printed. The code stays compiled and developable: MoE-Spec
(arXiv 2602.16052) is not a wrong idea, this implementation just has no
point where it is both faster and correct. Re-enabling it by default needs a
quality number next to every speed number.

Also gates the cap to decode (S<=4), @woolcoxm's fix from #292/#298: during
prefill the batch union is 30-100+ experts, and capping to 4-8 drops most of
them, corrupting the hidden state and therefore the KV cache. Necessary but
not sufficient -- the run above already includes it.

Removes issue_budget.md, issue_diskio.md and issue_grouped_quant.md: design
notes in the repo root, unlinked from any README, whose only user-facing
content was "EXPERT_BUDGET=6-8 -- good speedup, minimal quality loss" and
"+83% decode at budget=4". Measurement says otherwise, and they were the
only thing on main telling anyone to switch this on. They stay in history.

Reported-by: bokiko <#303>
Co-Authored-By: woolcoxm <#292>

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
JustVugg
2026-07-16 12:26:17 +02:00
parent d57955e95a
commit 35f90b9e76
4 changed files with 31 additions and 646 deletions
+31 -2
View File
@@ -2858,8 +2858,13 @@ static void moe(Model *m, Layer *l, int layer, float *x, int S, float *out, int
* only drop from misses. From the misses, keep the highest-aggregate-gate-weight
* ones up to the budget; drop the rest from idxs[] so they're never loaded.
* (MoE-Spec arXiv 2602.16052: top-32 of 64 capture 93% routing weight.)
* Complementary to TOPP (per-position) this trims cross-position. */
if(g_expert_budget>0 && nu>g_expert_budget){
* Complementary to TOPP (per-position) this trims cross-position.
* DECODE-ONLY (S<=4, incl. MTP verify): during prefill S=prompt_len the batch
* union nu is 30-100+ experts and capping to 4-8 drops 80-90% of them, each with
* non-trivial gate weight -> corrupted prefill hidden state -> wrong KV cache ->
* repetitive garbage decode. The budget is only safe token-by-token, where the
* prefill KV cache is already correct. (woolcoxm, #292.) */
if(g_expert_budget>0 && S<=4 && nu>g_expert_budget){
/* compute aggregate gate weight per unique expert */
float *wsum=falloc(nu); for(int j=0;j<nu;j++) wsum[j]=0;
for(int s=0;s<S;s++) for(int kk=0;kk<keff[s];kk++){
@@ -5759,7 +5764,31 @@ int main(int argc, char **argv){
if(g_mmap) fprintf(stderr,"[MMAP] expert = viste zero-copy nei file (page cache = cache)\n");
g_topk = getenv("TOPK")?atoi(getenv("TOPK")):0;
g_topp = getenv("TOPP")?atof(getenv("TOPP")):0;
/* EXPERT_BUDGET e' sotto quarantena: la finestra operativa e' misurata VUOTA.
* @bokiko su tre host (#303) e riprodotto qui su un 25 GB / WSL:
* - hellaswag 30% a budget=8 contro 90% a budget spento (25% = il caso);
* - a budget=4 il decode e' rumore ("The **1...: s2151:");
* - accettazione MTP 0%: quali expert sopravvivono al cap dipende dalla
* residenza in cache al momento del forward, quindi draft e verify NON
* calcolano la stessa funzione -- la stessa invariante che #294 ha appena
* stabilito, violata via stato di cache invece che via scelta del kernel;
* - 0.13 tok/s contro 0.30 di baseline, con 14.66 expert caricati per layer
* contro topk=8: il cap fa piu' I/O di quello che dice di risparmiare, e la
* riga "~N GB I/O saved" conta esperti scartati, non byte non letti.
* Resta compilato e sviluppabile (EXPERT_BUDGET_EXPERIMENTAL=1) perche' l'idea
* -- MoE-Spec, arXiv 2602.16052 -- non e' sbagliata: e' l'implementazione che
* finora non ha un punto in cui sia insieme piu' veloce e corretta. Riaccenderlo
* di default richiede una misura di qualita' accanto a quella di velocita'. */
g_expert_budget = getenv("EXPERT_BUDGET")?atoi(getenv("EXPERT_BUDGET")):0;
if(g_expert_budget>0 && !getenv("EXPERT_BUDGET_EXPERIMENTAL")){
fprintf(stderr,"[EXPERT_BUDGET] ignored: measured empty operating window (issue #303).\n"
"[EXPERT_BUDGET] every tested setting is either no faster or no longer coherent:\n"
"[EXPERT_BUDGET] budget=8 -> hellaswag 30%% (90%% with it off) | budget=4 -> decode is noise\n"
"[EXPERT_BUDGET] MTP acceptance 0%% (the cap breaks the draft/verify contract, #294)\n"
"[EXPERT_BUDGET] 0.13 tok/s vs 0.30 baseline, loading 14.7 experts/layer vs topk=8\n"
"[EXPERT_BUDGET] set EXPERT_BUDGET_EXPERIMENTAL=1 to run it anyway (expect garbage).\n");
g_expert_budget=0;
}
g_cache_route = getenv("CACHE_ROUTE")?atoi(getenv("CACHE_ROUTE")):0;
g_route_j = getenv("ROUTE_J")?atoi(getenv("ROUTE_J")):2;
g_route_m = getenv("ROUTE_M")?atoi(getenv("ROUTE_M")):12;