Quarantine EXPERT_BUDGET: the operating window is measured empty (#303)
@bokiko benchmarked the feature on three hosts and found every setting is
either no faster or no longer coherent. Reproduced here on a 25 GB box, and
the numbers are worse than "a quality/speed tradeoff":
- budget=8 -> hellaswag 30% vs 90% with it off (25% is the chance floor)
- budget=4 -> decode is literal noise ("The **1...: s2151:")
- MTP acceptance 0%. Which experts survive the cap depends on cache
residency at the moment of the forward, so draft and verify do not
compute the same function -- the invariant #294 just established for
#163, violated through cache state instead of kernel dispatch. SPEC_PIN
cannot fix that and should not try: it is feature semantics, not
dispatch.
- 0.13 tok/s vs 0.30 baseline, while loading 14.66 experts per layer
against a topk=8 baseline. The cap does MORE disk I/O than not using it.
"~335 GB I/O saved" counts dropped experts, not bytes not read.
EXPERT_BUDGET>0 is now ignored unless EXPERT_BUDGET_EXPERIMENTAL=1, with the
measurements printed. The code stays compiled and developable: MoE-Spec
(arXiv 2602.16052) is not a wrong idea, this implementation just has no
point where it is both faster and correct. Re-enabling it by default needs a
quality number next to every speed number.
Also gates the cap to decode (S<=4), @woolcoxm's fix from #292/#298: during
prefill the batch union is 30-100+ experts, and capping to 4-8 drops most of
them, corrupting the hidden state and therefore the KV cache. Necessary but
not sufficient -- the run above already includes it.
Removes issue_budget.md, issue_diskio.md and issue_grouped_quant.md: design
notes in the repo root, unlinked from any README, whose only user-facing
content was "EXPERT_BUDGET=6-8 -- good speedup, minimal quality loss" and
"+83% decode at budget=4". Measurement says otherwise, and they were the
only thing on main telling anyone to switch this on. They stay in history.
Reported-by: bokiko <#303>
Co-Authored-By: woolcoxm <#292>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -2858,8 +2858,13 @@ static void moe(Model *m, Layer *l, int layer, float *x, int S, float *out, int
|
||||
* only drop from misses. From the misses, keep the highest-aggregate-gate-weight
|
||||
* ones up to the budget; drop the rest from idxs[] so they're never loaded.
|
||||
* (MoE-Spec arXiv 2602.16052: top-32 of 64 capture 93% routing weight.)
|
||||
* Complementary to TOPP (per-position) — this trims cross-position. */
|
||||
if(g_expert_budget>0 && nu>g_expert_budget){
|
||||
* Complementary to TOPP (per-position) — this trims cross-position.
|
||||
* DECODE-ONLY (S<=4, incl. MTP verify): during prefill S=prompt_len the batch
|
||||
* union nu is 30-100+ experts and capping to 4-8 drops 80-90% of them, each with
|
||||
* non-trivial gate weight -> corrupted prefill hidden state -> wrong KV cache ->
|
||||
* repetitive garbage decode. The budget is only safe token-by-token, where the
|
||||
* prefill KV cache is already correct. (woolcoxm, #292.) */
|
||||
if(g_expert_budget>0 && S<=4 && nu>g_expert_budget){
|
||||
/* compute aggregate gate weight per unique expert */
|
||||
float *wsum=falloc(nu); for(int j=0;j<nu;j++) wsum[j]=0;
|
||||
for(int s=0;s<S;s++) for(int kk=0;kk<keff[s];kk++){
|
||||
@@ -5759,7 +5764,31 @@ int main(int argc, char **argv){
|
||||
if(g_mmap) fprintf(stderr,"[MMAP] expert = viste zero-copy nei file (page cache = cache)\n");
|
||||
g_topk = getenv("TOPK")?atoi(getenv("TOPK")):0;
|
||||
g_topp = getenv("TOPP")?atof(getenv("TOPP")):0;
|
||||
/* EXPERT_BUDGET e' sotto quarantena: la finestra operativa e' misurata VUOTA.
|
||||
* @bokiko su tre host (#303) e riprodotto qui su un 25 GB / WSL:
|
||||
* - hellaswag 30% a budget=8 contro 90% a budget spento (25% = il caso);
|
||||
* - a budget=4 il decode e' rumore ("The **1...: s2151:");
|
||||
* - accettazione MTP 0%: quali expert sopravvivono al cap dipende dalla
|
||||
* residenza in cache al momento del forward, quindi draft e verify NON
|
||||
* calcolano la stessa funzione -- la stessa invariante che #294 ha appena
|
||||
* stabilito, violata via stato di cache invece che via scelta del kernel;
|
||||
* - 0.13 tok/s contro 0.30 di baseline, con 14.66 expert caricati per layer
|
||||
* contro topk=8: il cap fa piu' I/O di quello che dice di risparmiare, e la
|
||||
* riga "~N GB I/O saved" conta esperti scartati, non byte non letti.
|
||||
* Resta compilato e sviluppabile (EXPERT_BUDGET_EXPERIMENTAL=1) perche' l'idea
|
||||
* -- MoE-Spec, arXiv 2602.16052 -- non e' sbagliata: e' l'implementazione che
|
||||
* finora non ha un punto in cui sia insieme piu' veloce e corretta. Riaccenderlo
|
||||
* di default richiede una misura di qualita' accanto a quella di velocita'. */
|
||||
g_expert_budget = getenv("EXPERT_BUDGET")?atoi(getenv("EXPERT_BUDGET")):0;
|
||||
if(g_expert_budget>0 && !getenv("EXPERT_BUDGET_EXPERIMENTAL")){
|
||||
fprintf(stderr,"[EXPERT_BUDGET] ignored: measured empty operating window (issue #303).\n"
|
||||
"[EXPERT_BUDGET] every tested setting is either no faster or no longer coherent:\n"
|
||||
"[EXPERT_BUDGET] budget=8 -> hellaswag 30%% (90%% with it off) | budget=4 -> decode is noise\n"
|
||||
"[EXPERT_BUDGET] MTP acceptance 0%% (the cap breaks the draft/verify contract, #294)\n"
|
||||
"[EXPERT_BUDGET] 0.13 tok/s vs 0.30 baseline, loading 14.7 experts/layer vs topk=8\n"
|
||||
"[EXPERT_BUDGET] set EXPERT_BUDGET_EXPERIMENTAL=1 to run it anyway (expect garbage).\n");
|
||||
g_expert_budget=0;
|
||||
}
|
||||
g_cache_route = getenv("CACHE_ROUTE")?atoi(getenv("CACHE_ROUTE")):0;
|
||||
g_route_j = getenv("ROUTE_J")?atoi(getenv("ROUTE_J")):2;
|
||||
g_route_m = getenv("ROUTE_M")?atoi(getenv("ROUTE_M")):12;
|
||||
|
||||
Reference in New Issue
Block a user