From 35f90b9e765eeffe7dc844f83c0400d349a4fadc Mon Sep 17 00:00:00 2001 From: JustVugg Date: Thu, 16 Jul 2026 12:26:17 +0200 Subject: [PATCH] Quarantine EXPERT_BUDGET: the operating window is measured empty (#303) @bokiko benchmarked the feature on three hosts and found every setting is either no faster or no longer coherent. Reproduced here on a 25 GB box, and the numbers are worse than "a quality/speed tradeoff": - budget=8 -> hellaswag 30% vs 90% with it off (25% is the chance floor) - budget=4 -> decode is literal noise ("The **1...: s2151:") - MTP acceptance 0%. Which experts survive the cap depends on cache residency at the moment of the forward, so draft and verify do not compute the same function -- the invariant #294 just established for #163, violated through cache state instead of kernel dispatch. SPEC_PIN cannot fix that and should not try: it is feature semantics, not dispatch. - 0.13 tok/s vs 0.30 baseline, while loading 14.66 experts per layer against a topk=8 baseline. The cap does MORE disk I/O than not using it. "~335 GB I/O saved" counts dropped experts, not bytes not read. EXPERT_BUDGET>0 is now ignored unless EXPERT_BUDGET_EXPERIMENTAL=1, with the measurements printed. The code stays compiled and developable: MoE-Spec (arXiv 2602.16052) is not a wrong idea, this implementation just has no point where it is both faster and correct. Re-enabling it by default needs a quality number next to every speed number. Also gates the cap to decode (S<=4), @woolcoxm's fix from #292/#298: during prefill the batch union is 30-100+ experts, and capping to 4-8 drops most of them, corrupting the hidden state and therefore the KV cache. Necessary but not sufficient -- the run above already includes it. Removes issue_budget.md, issue_diskio.md and issue_grouped_quant.md: design notes in the repo root, unlinked from any README, whose only user-facing content was "EXPERT_BUDGET=6-8 -- good speedup, minimal quality loss" and "+83% decode at budget=4". Measurement says otherwise, and they were the only thing on main telling anyone to switch this on. They stay in history. Reported-by: bokiko <#303> Co-Authored-By: woolcoxm <#292> Co-Authored-By: Claude Opus 4.8 --- c/glm.c | 33 ++++- issue_budget.md | 150 -------------------- issue_diskio.md | 309 ----------------------------------------- issue_grouped_quant.md | 185 ------------------------ 4 files changed, 31 insertions(+), 646 deletions(-) delete mode 100644 issue_budget.md delete mode 100644 issue_diskio.md delete mode 100644 issue_grouped_quant.md diff --git a/c/glm.c b/c/glm.c index ae18c55..d34606f 100644 --- a/c/glm.c +++ b/c/glm.c @@ -2858,8 +2858,13 @@ static void moe(Model *m, Layer *l, int layer, float *x, int S, float *out, int * only drop from misses. From the misses, keep the highest-aggregate-gate-weight * ones up to the budget; drop the rest from idxs[] so they're never loaded. * (MoE-Spec arXiv 2602.16052: top-32 of 64 capture 93% routing weight.) - * Complementary to TOPP (per-position) — this trims cross-position. */ - if(g_expert_budget>0 && nu>g_expert_budget){ + * Complementary to TOPP (per-position) — this trims cross-position. + * DECODE-ONLY (S<=4, incl. MTP verify): during prefill S=prompt_len the batch + * union nu is 30-100+ experts and capping to 4-8 drops 80-90% of them, each with + * non-trivial gate weight -> corrupted prefill hidden state -> wrong KV cache -> + * repetitive garbage decode. The budget is only safe token-by-token, where the + * prefill KV cache is already correct. (woolcoxm, #292.) */ + if(g_expert_budget>0 && S<=4 && nu>g_expert_budget){ /* compute aggregate gate weight per unique expert */ float *wsum=falloc(nu); for(int j=0;j0 && !getenv("EXPERT_BUDGET_EXPERIMENTAL")){ + fprintf(stderr,"[EXPERT_BUDGET] ignored: measured empty operating window (issue #303).\n" + "[EXPERT_BUDGET] every tested setting is either no faster or no longer coherent:\n" + "[EXPERT_BUDGET] budget=8 -> hellaswag 30%% (90%% with it off) | budget=4 -> decode is noise\n" + "[EXPERT_BUDGET] MTP acceptance 0%% (the cap breaks the draft/verify contract, #294)\n" + "[EXPERT_BUDGET] 0.13 tok/s vs 0.30 baseline, loading 14.7 experts/layer vs topk=8\n" + "[EXPERT_BUDGET] set EXPERT_BUDGET_EXPERIMENTAL=1 to run it anyway (expect garbage).\n"); + g_expert_budget=0; + } g_cache_route = getenv("CACHE_ROUTE")?atoi(getenv("CACHE_ROUTE")):0; g_route_j = getenv("ROUTE_J")?atoi(getenv("ROUTE_J")):2; g_route_m = getenv("ROUTE_M")?atoi(getenv("ROUTE_M")):12; diff --git a/issue_budget.md b/issue_budget.md deleted file mode 100644 index fe0ef86..0000000 --- a/issue_budget.md +++ /dev/null @@ -1,150 +0,0 @@ -## TL;DR - -New `EXPERT_BUDGET=N` env var that caps the number of **distinct experts loaded per layer** across the batch-union. When the union exceeds the budget, keeps only the highest-aggregate-gate-weight experts and drops the rest — they're never loaded from disk. On a 24 GB RAM host (cache `cap=2`), `EXPERT_BUDGET=4` nearly **doubles decode tok/s** (0.18 → 0.33) and **4x's prefill speed** (38.7s → 8.9s). Based on MoE-Spec (arXiv 2602.16052): "top 32 of 64 experts capture 93% of routing weight." - -Branch: `experiment/expert-budget` (based on latest `dev` at `62419af`) - ---- - -## The problem - -Every expert miss costs ~19 MB of disk I/O. On low-RAM hosts where the LRU cache cap is tiny (e.g. `cap=2` on 24 GB RAM), nearly every routed expert is a miss. With `topk=8` and 75 sparse layers, a single-token decode reads ~8.5 GB of experts from disk. Under MTP with `S=4`, the batch-union can produce 20-32 distinct experts per layer — almost all misses — multiplying disk reads further. - -The README documents this directly: -> "on a cold cache each verified draft routes to extra experts (~660 → ~1100 expert-loads/token)" - -The existing `TOPP` env var trims experts *within a single position's top-K*. But it cannot reduce the **cross-position union** — the deduplication across batch positions (prefill, MTP verification) that multiplies disk loads. That's the gap this fills. - -## How it works - -The batch-union in `moe()` (line ~2306 of `glm.c`) deduplicates routed experts across all `S` positions into `uniq[0..nu)`. After the union is built but before the cache-resolve/load loop, the budget cap kicks in: - -``` -if(EXPERT_BUDGET > 0 && nu > EXPERT_BUDGET): - 1. Compute aggregate gate weight per unique expert - (sum of ws[s*K+kk] across all positions that route to it) - 2. Sort experts by descending aggregate weight - 3. Keep top EXPERT_BUDGET, mark rest as dropped - 4. Remove dropped experts from each position's idxs[]/keff[] - (decrement keff, renormalize remaining weights if norm_topk) - 5. Compact uniq[] to kept experts only -``` - -Dropped experts are removed from `idxs[]` entirely, so they're **never resolved, never loaded from disk, never computed**. The downstream code already tolerates `keff[s] < K` (the `TOPP` path produces the same state), so this propagates correctly through resolve, matmul, and LRU promotion. - -**Relationship to existing features:** -- `TOPP` trims within one position's top-K (per-position) -- `EXPERT_BUDGET` trims across the cross-position union (per-layer) -- They compose: `TOPP=0.8 EXPERT_BUDGET=12` first trims each position to its top-p mass, then caps the union - -## Code change - -**`c/glm.c`** — 54 lines added, single file: - -1. Global + counter (near existing `g_topk`/`g_topp`): -```c -static int g_expert_budget=0; /* EXPERT_BUDGET=N */ -static int64_t g_budget_dropped=0; /* total experts dropped */ -``` - -2. Env var parsing (near existing `TOPK`/`TOPP`): -```c -g_expert_budget = getenv("EXPERT_BUDGET")?atoi(getenv("EXPERT_BUDGET")):0; -``` - -3. Budget cap in `moe()` batch-union (after `uniq[]` is built, before resolve loop) — full logic described above. - -4. Stats reporting alongside existing `TOPK`/`TOPP` line: -``` -EXPERT_BUDGET=4 (dropped 13613 experts, ~257.3 GB I/O saved) -``` - -## Measurements - -**Test setup:** GLM-5.2 744B int4, 24 GB RAM (cache `cap=2`), Core Ultra 9 185H (AVX-VNNI), MTP=0, single-token decode, 32 tokens generated, same prompt: *"Explain the concept of recursion in programming. Provide a simple example."* - -| Config | tok/s | vs baseline | hit rate | prefill | decode | experts dropped | I/O saved | -|--------|-------|-------------|----------|---------|--------|-----------------|-----------| -| **Baseline** (budget=0) | 0.18 | — | 9.3% | 38.7s | 176.3s | 0 | 0 | -| EXPERT_BUDGET=12 | 0.19 | +5% | 14.0% | 12.3s | 171.3s | 3,313 | 62.6 GB | -| EXPERT_BUDGET=6 | 0.26 | +44% | 21.0% | 7.5s | 122.5s | 8,543 | 161.5 GB | -| **EXPERT_BUDGET=4** | **0.33** | **+83%** | 16.4% | 8.9s | **97.4s** | 13,613 | 257.3 GB | - -### Profile breakdown (prefill) - -| Metric | Baseline | Budget=4 | -|--------|----------|----------| -| expert-disk service | 28.4s | **3.5s** (8x less) | -| expert-matmul | 7.1s | 1.4s | -| attention | 2.3s | 2.8s | - -### Profile breakdown (decode, 32 tokens) - -| Metric | Baseline | Budget=4 | -|--------|----------|----------| -| expert-disk service | 133.4s | proportional ~65s | -| expert-matmul | 23.9s | ~12s | -| decode total | 176.3s | **97.4s** | - -### Key observations - -1. **Prefill is the biggest winner.** With S=14 positions, the batch-union has 50+ distinct experts per layer. Budget=12 cuts that to 12, saving 40+ × 19 MB × 75 layers of disk reads. Prefill goes from 38.7s to 8.9s — **4.4x faster**. - -2. **Decode improvement scales with budget tightness.** Budget=12 barely constrains single-token decode (S=1 → max 8 experts < 12), so decode is barely affected. Budget=4 actually halves the decode load (8 → 4 experts/layer), nearly doubling tok/s. - -3. **Hit rate improves** because fewer experts compete for the tiny cache (cap=2). Budget=6 hit 21% vs baseline 9.3% — more than doubled. - -4. **No crashes or instability** at any budget value. - -## Quality impact — honest assessment - -**Budget=4 keeps only the top-4 of 8 routed experts per layer.** The dropped 4 experts had the lowest gate weights — they contributed the least to the output. But this IS a quality trade-off. - -On a **cold cache** (our test setup), quality assessment is confounded: both baseline and budget outputs are already garbled from int4 quantization + cold-cache routing. Sample comparison: - -- **Baseline output:** *"Hire Some To Take Object-Oriented Programming Assignment"* -- **Budget=4 output:** *"The world is a dangerous place, not so much because of the small percentage of people who are doing to do the little of people who are doing to"* - -Both are incoherent — the cold cache at int4 on 24 GB RAM produces poor output regardless of budget. The budget=4 output is **not dramatically worse** than the already-poor baseline — they're both garbled, just differently garbled. A proper quality assessment needs a **warm cache** where the baseline produces coherent text, so you can isolate the degradation from dropping experts. - -**Expected quality on warm cache:** With `norm_topk=1` (GLM-5.2 renormalizes gate weights), the top-4 experts typically capture ~80-85% of routing weight (the remaining 4 share 15-20%). Output should be mostly coherent with occasional word-choice degradation. Budget=6 (top-6 of 8, ~90%+ weight captured) should be nearly indistinguishable from baseline. - -**Recommendation for users:** -- `EXPERT_BUDGET=6-8` on cold/low-RAM hosts — good speedup, minimal quality loss -- `EXPERT_BUDGET=4` on very-low-RAM hosts where speed matters more than quality -- Leave OFF on high-RAM hosts where all experts are resident anyway (budget never triggers) - -## Safety - -- **Default OFF** (`EXPERT_BUDGET=0`) — zero behavior change unless explicitly set -- **Opt-in quality trade-off** — same design philosophy as `TOPP`: the user explicitly chooses speed over quality -- **No output corruption** — dropped experts' contributions are omitted and remaining weights renormalized (identical math to `TOPP`) -- **Compatible with all features** — PILOT, MTP, CUDA, CACHE_ROUTE, PIPE, TOPP all work unchanged; they just see fewer experts in the union -- **No crash risk** — no new memory allocation patterns, no new I/O, purely a filter on existing data structures - -## Reproduce - -```bash -make ARCH=native - -# Baseline: -SNAP= MTP=0 PROMPT="Explain recursion in programming." NGEN=32 ./glm 64 - -# With budget: -SNAP= MTP=0 EXPERT_BUDGET=4 PROMPT="Explain recursion in programming." NGEN=32 ./glm 64 - -# With MTP (where budget saves the most): -SNAP= MTP=1 EXPERT_BUDGET=16 PROMPT="Explain recursion in programming." NGEN=32 ./glm 64 -``` - -The stats line prints `EXPERT_BUDGET=N (dropped X experts, ~Y GB I/O saved)`. - -## What's needed before merge - -1. **Warm-cache quality A/B test** on a host with enough RAM (cap>=16) to produce coherent baseline output, then compare budget=4/6/8 text quality -2. **MTP interaction test** — verify that budget + MTP composes correctly and doesn't crash when draft verification routes to budgeted-away experts -3. **Decide on defaults** — should this auto-activate on low-RAM hosts (like `cap` auto-lowering)? My recommendation: no, leave it OFF and document the recommended value per RAM tier - -## Prior art - -**MoE-Spec** (arXiv 2602.16052) — "Expert Budgeting for Efficient Speculative Decoding": Training-free expert budgeting at verification time. Key finding: "top 32 of 64 experts capture 93% of routing weight." Our approach applies the same principle but at the batch-union level (not just MTP verification), making it effective for prefill and single-token decode too. diff --git a/issue_diskio.md b/issue_diskio.md deleted file mode 100644 index 91d7b92..0000000 --- a/issue_diskio.md +++ /dev/null @@ -1,309 +0,0 @@ -# Disk I/O Minimization — Research - -Branch: `experiment/diskio-research` (based on `dev` at `62419af`) - -## TL;DR - -The engine's disk I/O is **already well-engineered on the hottest path** (expert streaming uses coalesced O_DIRECT `pread` + `posix_fadvise` hints + LRU + pin cache + speculative prefetch). There are **4 concrete, bounded opportunities** to shave latency, ranked by ROI: - -| # | Opportunity | Where | Frequency | Estimated win | -|---|---|---|---|---| -| 1 | **KV-cache write batching** (157 fwrites/token → 1) | `kv_disk_append` | per turn | cuts ~100s of syscalls/turn | -| 2 | **`/proc/meminfo` fopen storm** | `rss_gb()` | ~every 16 tokens (Linux) | eliminates recurring open/read/close | -| 3 | **Expert prefetch on Windows** (`PrefetchVirtualMemory`) | `expert_prefetch` | per miss | mmap path is Linux/macOS-only today | -| 4 | **KV-cache: buffered handle kept open** | `kv_disk_append` | per turn | kills open+fseek+close per turn | - -There are also **2 non-opportunities** worth recording so we don't re-investigate: O_DIRECT for experts (correctly used today), and PagedAttention-style file layout (already single-file + indexed). - ---- - -## How the engine does disk I/O today - -There are **three I/O stacks**, behaving very differently: - -| Stack | Mechanism | Frequency | Files | -|---|---|---|---| -| **Expert weights** (hottest) | `pread` on kept-open fds + `posix_fadvise`, optional `mmap` | per miss, every token | `st.h`, `glm.c:1328` | -| **KV cache** (`.coli_kv`) | `fopen` + `fwrite`/`fread` | per turn | `glm.c:3812-3889` | -| **Everything else** (config, tokenizer, stats, grammar) | `fopen` + `fread` | startup-only | scattered | - -### Expert path (the hot path — already good) - -`expert_load` (`glm.c:1328-1481`) has three sub-paths: - -- **Default `pread` path** (`glm.c:1385-1472`): coalesces the 3 contiguous expert tensors (gate/up/down) into **one ~19 MB O_DIRECT `pread`** into a 16K-aligned slab (`glm.c:1447`). Falls back to 3 separate `pread`s only if non-contiguous. Scales are 3 tiny separate `pread`s (kilobytes). `posix_fadvise(DONTNEED)` evicts pages after if `g_drop`. **This is well-batched — one syscall for ~19 MB.** -- **`COLI_MMAP=1` path** (`glm.c:1352-1383`): `mmap` per shard fd (cached), `madvise(WILLNEED)` + synchronous page-touch loop. Zero-copy. **Default OFF, and Linux/macOS/FreeBSD-only** — no `MapViewOfFile` on Windows. -- **Prefetch hints**: `expert_prefetch` (`glm.c:1602-1609`) → `st_prefetch` (`st.h:178`) issues `posix_fadvise(WILLNEED)` — readahead hint only, no data read. Called from `moe` next-64-block lookahead, pilot, and SPEC. - -### KV cache persistence (per turn — opportunity here) - -`kv_disk_append` (`glm.c:3834-3855`), called once per turn: -1. `fopen("r+b")` — **reopens the file every turn** -2. `fseek` to append position -3. **per-position loop**: for each new token, `fwrite` the token i32, then **2 fwrites per layer** (Lc + Rc) + optional DSA Ic. With 78 layers that's **~157 fwrites per token appended**. -4. `fflush` (userspace only — **no fsync/fdatasync anywhere in the codebase**) -5. `fseek` back to header + `fwrite` the new nrec counter (crash-safe ordering) -6. `fclose` - -Record size ~182 KB/token. On a long first turn this is **tens of thousands of small fwrites**. stdio buffering coalesces them into fewer `write` syscalls, but the userspace overhead remains. - -### Recurring surprise: `/proc/meminfo` - -`rss_gb()` (`glm.c:4625`) does `fopen("/proc/meminfo")` + fgets + fclose. Called from every STAT line and every 16-token heartbeat (`glm.c:3473, 3477`). On Linux this is an **open+read+close of procfs ~every 16 tokens**. (Windows uses `compat_meminfo`, no file — not affected.) - ---- - -## What similar projects do - -**llama.cpp** (the reference): `mmap`s the entire model read-only, uses `--mlock` to pin hot pages, streams layers to GPU via partial offload, and issues per-pass readahead of upcoming tensors (`llama-mmap.cpp`). Justine Tunney's mmap work: "load 100× faster using half as memory." Crucial finding from discussion #18758: **for MoE, mmap beats O_DIRECT** when the model fits in ~RAM — O_DIRECT takes "at least 10× longer" on repeated loads because it bypasses the page cache that serves re-faults for free. - -**The general consensus across llama.cpp, vLLM, AirLLM, PRESERVE, HOBBIT, SolidAttention (FAST '26):** -- mmap + OS page cache as the backing store for an LRU is the proven recipe -- prefetch the *next* expert/layer while computing the current one — this is where the 0.5ms lives -- single indexed file (one `open()`) beats one-file-per-expert -- align tensors to 4KB (preferably 64KB) for clean page-fault boundaries + SSD geometry -- buffer sweet spot ~1MB; syscall cost ~1-5µs each, so batching matters at high repetition - ---- - -## The 4 opportunities (ranked) - -### Opportunity 1 — KV-cache write batching (HIGH ROI, LOW risk) - -**Problem:** `kv_disk_append` does ~157 `fwrite` calls per appended token (1 token i32 + 2×78 layers). stdio buffering hides some of this, but on a long first-turn prefill (hundreds-thousands of tokens) this is tens of thousands of fwrites. - -**Fix:** Build one contiguous record in a heap buffer (token + all layers' Lc/Rc/Ic for that position), then **a single `fwrite` per position** (or even one `fwrite` for the whole turn). The data is already laid out contiguously in memory per-layer (`coli_kv_row`), so a layered `memcpy` into a staging buffer + one write is straightforward. - -**Win:** ~157× fewer fwrite calls per token. Even with stdio coalescing, the userspace loop overhead is real at scale. - -### Opportunity 2 — `/proc/meminfo` fopen storm (MEDIUM ROI, trivial) - -**Problem:** `rss_gb()` opens, reads, closes `/proc/meminfo` every ~16 tokens on Linux. Each is ~3 syscalls + path resolution. - -**Fix:** Either (a) cache the value for N tokens (e.g. re-read at most once per second), or (b) keep the fd open and `rewind`+`fgets`. Trivial change. - -### Opportunity 3 — Expert prefetch on Windows (MEDIUM ROI, bounded) - -**Problem:** The `COLI_MMAP=1` path (which gives zero-copy expert access + free OS-cache re-faults) is **Linux/macOS/FreeBSD-only** — `glm.c:1301` guards it. On Windows, experts always go through the `pread` path, and `expert_prefetch` issues `posix_fadvise(WILLNEED)` which is a no-op shim on Windows (`compat.h`). - -**Fix:** On Windows, implement the prefetch via `PrefetchVirtualMemory` (the Win32 analog of `MADV_WILLNEED`) on an mmap'd region, or via an async `ReadFile`+`OVERLAPPED` into a scratch buffer. This brings the Windows build closer to parity with the Linux mmap+prefetch story. - -**Scope:** This is the largest of the four — it touches the Windows I/O path. Worth doing if Windows perf is a goal; skip if Linux is the target. - -### Opportunity 4 — KV-cache: keep handle open (LOW-MEDIUM ROI, LOW risk) - -**Problem:** `kv_disk_append` does `fopen`+...+`fclose` every turn. Handle creation is ~5-15µs of pure overhead (worse on Windows). - -**Fix:** Open the KV file once (lazily on first append), keep the `FILE*` for the engine lifetime, just `fseek`+write each turn. Close on shutdown. Pair with Opportunity 1 for the write batching. - ---- - -## Non-opportunities (recording so we don't re-investigate) - -- **O_DIRECT for experts**: already correctly used (`st.h:83`, `DIRECT=1`). For an LRU+refetch pattern the page cache is your friend, but the engine offers both paths (O_DIRECT pread default + optional mmap) and the O_DIRECT coalesced read is already one syscall for ~19MB. Don't change this. -- **Single-file layout**: the engine already uses safetensors shards with kept-open fds + offset-indexed tensors (`st.h`). No per-expert open()/close() waste. Don't change this. -- **PagedAttention**: solves concurrency fragmentation this engine doesn't have (≤16 slots). Not applicable. - ---- - -## Next steps - -The highest-ROI, lowest-risk starting point is **Opportunity 1 (KV write batching) + Opportunity 4 (keep handle open)** — they're in the same function, both low-risk, and together they eliminate the per-turn open/close overhead and the per-token fwrite storm. Opportunity 2 is a trivial 5-minute fix we can bundle in. - -Opportunity 3 (Windows prefetch) is the biggest single win but also the largest scope — separate effort, gated on whether Windows perf is a priority. - -## Sources - -- [justine.lol/mmap — Edge AI Just Got Faster](https://justine.lol/mmap/) -- [llama.cpp discussion #18758 — Mmap faster than direct I/O for MoE](https://github.com/ggml-org/llama.cpp/discussions/18758) -- [llama.cpp issue #20757 — Two-tier GPU+RAM expert cache](https://github.com/ggml-org/llama.cpp/issues/20757) -- [FAST '26 — Programmable Page Cache for LLM loading](https://www.usenix.org/system/files/fast26-liu-yubo.pdf) -- [FAST '26 — SolidAttention: SSD-based serving](https://www.usenix.org/system/files/fast26-zheng.pdf) -- [HOBBIT — Mixed precision expert offloading](https://arxiv.org/html/2411.01433v2) -- [posix_fadvise(2) — man7.org](https://man7.org/linux/man-pages/man2/posix_fadvise.2.html) -- [madvise(2) — man7.org](https://man7.org/linux/man-pages/man2/madvise.2.html) -- [Microsoft Learn — File Buffering (FILE_FLAG_NO_BUFFERING)](https://learn.microsoft.com/en-us/windows/win32/fileio/file-buffering) -- [Microsoft Learn — PrefetchVirtualMemory](https://learn.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-prefetchvirtualmemory) -- [What makes system calls expensive — codingconfessions.com](https://blog.codingconfessions.com/p/what-makes-system-calls-expensive) -- [Syscall overhead — Stack Overflow](https://stackoverflow.com/questions/8247331/syscall-overhead) - ---- - -# Windows Implementation — branch `windows-optimizations` (2026-07-15) - -## What landed (pread path, validated) - -Two changes, both on the `pread` expert-load path (no mmap). Measured against the -existing `bench_budget*.txt` baselines (GLM-5.2 744B int4, 32 GB RAM, Core Ultra 9 -185H, DRAFT=0, 32-token decode): - -### 1. `compat_fadvise` WILLNEED cache-warmer (`c/compat.h`) - -Replaced the Windows `posix_fadvise` no-op (was a `do{}while(0)` macro) with a real -readahead: an overlapped `ReadFile` into a throwaway scratch buffer that populates the -standby page cache, so the later synchronous `pread` faults from RAM not disk. Mirrors -the macOS `F_RDADVISE` shim (`compat.h:28-37`). DONTNEED stays a no-op (matches macOS; -Windows standby-list trimming self-regulates under pressure). - -This re-arms the existing `expert_prefetch` → `st_prefetch` → `posix_fadvise(WILLNEED)` -chain on Windows: the next-block readahead in `moe()` and the PILOT cross-layer prefetch -hints now actually warm the cache instead of being silently discarded. - -**Measured effect (budget=4, PIPE on):** hit rate 16.4% → 27.6%. - -### 2. PIPE default ON for Windows (`c/glm.c`) - -Flipped the async expert-load thread pool from default OFF to default ON on Windows -(`getenv("PIPE")?:1` under `_WIN32`, unchanged `:0` elsewhere). PIPE dispatches expert -`pread` loads onto worker threads so they overlap the expert matmul on the forward-pass -thread, instead of the blocking serial load-then-compute path. `PIPE=0` opts back out. - -**Measured effect (budget=4):** expert-disk 65.9s → 54.3s (−18%), reaching **1.70 s/tok** -(under the 2 s/tok target; budget=4 baseline was 2.06 s/tok). - -### Results table (DRAFT=0, 32-token decode, pread path) - -| config | expert-disk | s/tok | hit% | tok/s | -|---|---|---|---|---| -| budget=4, no PIPE (existing baseline) | 65.9s | 2.06 | 16.4% | 0.33 | -| **budget=4 + PIPE (this PR)** | **54.3s** | **1.70** | **27.6%** | **0.34** | -| budget=6 + PIPE | 77.1s | 2.41 | 21.8% | 0.27 | - -budget=4 + PIPE meets the ≤2 s/tok target. budget=6 (more experts/layer, higher quality) -misses it at 2.41 s/tok — the speed/quality tradeoff. - -## What was tried and abandoned: Windows mmap (`COLI_MMAP` on `_WIN32`) - -The original plan (informed by llama.cpp #18758: "mmap is ≥10× faster than O_DIRECT for -MoE") was to port the mmap expert path to Windows via `CreateFileMapping`/`MapViewOfFile`. -This was implemented and tested at length. **It was a measured regression and was reverted.** - -### The attempt - -Added a `_WIN32` branch to `map_of_fd` (`glm.c`) mapping each shard file read-only and -resolving experts as views into the mapping, mirroring the POSIX path. Also added -`PrefetchVirtualMemory` readahead and a `VirtualUnlock` eviction mechanism (the Windows -`posix_fadvise(DONTNEED)` analog — see SO#1880714; validated standalone to demote pages -to the standby list with a 2.3× faster re-fault). - -### Why it regressed - -**mmap'd expert pages bloat the process working set on Windows, which collapses the -expert cache.** This is a fundamental Windows-vs-Linux difference: - -- On Linux, `mmap(MAP_SHARED)` file pages live in the kernel page cache (`buff/cache`), - separate from `MemAvailable`, so the cache budget isn't fooled. -- On Windows, touched `MapViewOfFile` pages count against `ullAvailPhys` (what - `compat_meminfo` reads for the budget). The CPU matmul touches every weight byte, - faulting ~12 GB into the working set. `cap_for_ram()` then sees ~no free RAM and - collapses the LRU cache cap. - -Measured (budget=0, DRAFT=0, apples-to-apples): - -| config | RAM_GB detected | cache cap | hit% | expert-disk | RSS | -|---|---|---|---|---|---| -| baseline (pread) | 21.4 | 1 | 9.3% | 133s | 15.0 GB | -| mmap, no eviction | **8.0** | 1 | **2.2%** | **240s** | **27.2 GB** | -| mmap + VirtualUnlock | 24.9 | 2 | 11.8% | 83s | 18.1 GB | -| mmap + reserve reductions | 24.6 | 4 | 21.8% | 80s | 20.1 GB | - -The `VirtualUnlock` eviction recovered the regression (240s→83s), and dropping the -Linux-specific page-cache/slab reserves under mmap got it to parity with pread. But it -never clearly *beat* the simpler pread+PIPE path, and it added substantial complexity -(per-slot eviction tracking, reserve conditionals, `VirtualUnlock` on every slot recycle). -**The engine already moved off mmap to pread for this exact RSS bug** (`st.h:3-6`), and -the Windows port re-confirmed that decision. - -### What else didn't work - -- **Batched `PrefetchVirtualMemory`** for the mmap path: tested as a single batched - readahead of all 64 missed experts' pages before the matmul. **Blocked instead of - prefetching async** on this SSD — inflated `t_edisk` (80s→102s). Consistent with - microsoft/Windows-Dev-Performance#108 ("PrefetchVirtualMemory does not prefetch"). - Reverted. -- **True I/O/compute overlap on the CPU path**: the Metal path has this ("submit - resident experts to GPU before loading misses"), but the CPU path loads-then-computes - serially. `PrefetchVirtualMemory` was the attempt to add it for mmap and failed. The - pread path gets overlap via PIPE (which works), not via mmap prefetch. - -### Conclusion - -For this engine on Windows at this RAM budget (~32 GB, 370 GB model), **pread + PIPE + -compat_fadvise** is the right path. mmap remains valuable on Linux/macOS (where the page -cache doesn't inflate process RSS) but is not viable on Windows without a fundamentally -different cache-budget model that excludes mapped-file pages — left as future work. - -## Sources added - -- [SO#1880714 — VirtualUnlock releases mapped pages to standby list](https://stackoverflow.com/questions/1880714/createfilemapping-mapviewoffile-how-to-avoid-holding-up-the-system-memory) -- [Alois Kraus — The Mysterious Lost Memory (modified/standby list)](https://aloiskraus.wordpress.com/2017/02/26/the-mysterious-lost-memory-which-belongs-to-no-process/) -- [microsoft/Windows-Dev-Performance#108 — PrefetchVirtualMemory inconsistency](https://github.com/microsoft/Windows-Dev-Performance/issues/108) -- [llama.cpp #18758 — mmap faster than O_DIRECT for MoE (Linux)](https://github.com/ggml-org/llama.cpp/discussions/18758) -- [HN#35426679 — Why MMAP in llama.cpp hides true memory usage](https://news.ycombinator.com/item?id=35426679) - ---- - -# CACHE_ROUTE: miss elimination via cache-aware routing (2026-07-15) - -## The breakthrough - -Adding `CACHE_ROUTE=1 ROUTE_J=2 ROUTE_M=12` to the optimized stack pushed -throughput to **1.41 tok/s** (4.3× over stock) by directly reducing the -miss rate from 27% to 17%. - -## How it works - -The engine's `CACHE_ROUTE` feature (paper: max-rank routing, arXiv 2412.00099) -steers the MoE router to prefer experts that are already cache-resident: - -- `ROUTE_J=2`: keep the top-2 true router picks (always, even if uncached) -- `ROUTE_M=12`: fill the remaining 2 of 4 slots with the highest-ranked experts - that are ALREADY in the LRU cache (from the top-12 candidates) - -This guarantees 2 of 4 expert slots per layer per token are cache hits. The -`route_agree=94.6%` metric confirms minimal quality cost — 94.6% of cache-steered -picks match the true top-K the router would have chosen. - -## Why this is the right fix for the miss problem - -Analysis of route traces showed GLM-5.2's routing is nearly uniform — 75 tokens -use 237 of 256 experts per layer, with the top-4 capturing only 4.3% of selections. -This means: - -- PIN (hot-expert pre-loading) is ineffective — there are no hot experts -- PILOT_REAL prefetch can't keep up under the fast pipe2 GPU pipeline -- A bigger cache helps marginally but can't cover 237 unique experts per layer -- CACHE_ROUTE is the only lever that reduces misses without more RAM or faster disk - -## Results (pipe2 + full stack + ws_b fix, budget=4, RAM_GB=28) - -| metric | without CACHE_ROUTE | with CACHE_ROUTE | -|---|---|---| -| tok/s | 1.03 | **1.41** (+37%) | -| hit rate | 73% | **83%** | -| expert-disk | 12.4s | **8.5s** (−31%) | -| decode | 31.0s | **22.7s** (−27%) | -| route_agree | n/a | 94.6% | - -## Full optimization journey - -| step | tok/s | hit% | disk | -|---|---|---|---| -| stock budget=4 | 0.33 | 9% | 65.9s | -| + disk stack | 0.63 | 72% | 21.5s | -| + CUDA dense+attn | 0.72 | 75% | 18.4s | -| + pipe2 GPU pipeline | 0.85 | 57% | 18.2s | -| + ws_b cache fix | 1.03 | 73% | 12.4s | -| **+ CACHE_ROUTE** | **1.41** | **83%** | **8.5s** | - -**4.3× total speedup.** Disk I/O reduced 7.7× (65.9s → 8.5s). - -## Recommended config - -``` -EXPERT_BUDGET=4 PIPE=1 RAM_GB=28 PILOT_REAL=1 DIRECT=1 -COLI_CUDA=1 CUDA_DENSE=1 COLI_CUDA_ATTN=1 COLI_CUDA_PIPE=2 CUDA_EXPERT_GB=0 -CACHE_ROUTE=1 ROUTE_J=2 ROUTE_M=12 -``` - -CACHE_ROUTE is an existing engine feature (no code change) — opt-in via env vars. diff --git a/issue_grouped_quant.md b/issue_grouped_quant.md deleted file mode 100644 index fb46ce4..0000000 --- a/issue_grouped_quant.md +++ /dev/null @@ -1,185 +0,0 @@ -## TL;DR - -The int4 model produces **fluent-but-incoherent output** — grammatical English that is completely off-topic (SEO spam, homework-help boilerplate) instead of reasoned responses. The root cause is **per-row quantization scales**: one F32 scale per output row (e.g. 2048 scales for a 2048×6144 matrix), which is 48x coarser than the FP8 source's 128×128 block scales. This destroys fine-grained weight information that handles reasoning and instruction-following while preserving the large-magnitude weights that handle grammar and vocabulary fluency. - -Branch: `experiment/grouped-quant` (based on latest `dev` at `62419af`) - ---- - -## The problem — demonstrated - -Prompt: *"Explain the concept of recursion in programming. Provide a simple example."* - -**Baseline (no budget, no changes):** -> "Hire Some To Take Object-Oriented Programming Assignment" - -**Budget=4:** -> "The world is a dangerous place, not so much because of the small percentage of people who are doing to do the little of people who are doing to" - -**Budget=6:** -> "Elite Custom Essays" - -**Budget=12:** -> "Sololearn: Learn to code for FREE! +1 # Explain A concept of recursion..." - -This is not random gibberish — it's fluent English that is completely off-topic. This is the signature of **activations corrupted by coarse quantization**: the model retains language fluency (large weights survive) but loses instruction-following and reasoning (fine-grained weights are crushed to zero). - -## Root cause: per-row int4 scales - -### How the current converter works - -`convert_fp8_to_int4.py` (line 39-52) quantizes with **one scale per output row**: - -```python -amax = np.abs(w).max(axis=1, keepdims=True) # one max per ROW -s = np.maximum(amax / qmax, 1e-8) # one scale per ROW -q = np.clip(np.rint(w / s), -8, qmax) # quantize entire row with that scale -``` - -For a 2048×6144 expert weight matrix, that's **2048 scales** — one per row, each covering 6144 elements. - -### Why this destroys quality - -Consider a row of 6144 values where most are small (magnitude ~0.01) but a few are large (~0.5). The per-row scale is `0.5/7 = 0.071`. The small values become `0.01/0.071 = 0.14`, which rounds to **zero**. All fine-grained information in those small weights is lost. - -This matters enormously for MoE models: each token activates only 8 of 256 experts. A poorly-quantized expert pollutes every token that routes to it, and there's no averaging from the other 248 experts (they're simply off). The error is **concentrated**, not diluted. - -### What the FP8 source does right - -The FP8 checkpoint uses **128×128 block scales** — the weight matrix is divided into 128-element chunks, each with its own scale. Small values in one chunk don't get crushed by large values in another. For a 2048×6144 matrix: `2048 × 48 = 98,304` scales — 48x more granularity. - -The converter already dequants FP8 to f32 correctly (lines 196-201), preserving the block-scale information. But then it **throws it all away** by collapsing to a single per-row scale during int4 requantization. - -### GLM-5.2 was QAT-trained for int4 - -The GLM-5 paper (arXiv 2602.15763, §2.4.3) states: *"To provide better accuracy at low-precision, we apply INT4 QAT in the SFT stage."* This means the model is **designed** to work at int4 — but only if the quantization is fine-grained enough to match what the model saw during training. The FP8 checkpoint's 128×128 block scales are the granularity the model expects. Per-row scaling is 48x coarser. - -### Community confirmation - -Every other project getting coherent GLM-5.2 int4 output uses calibrated or fine-grained quantization: -- **ubergarm/GLM-5.1-GGUF** — imatrix-calibrated with expert-specific patches -- **Unsloth/GLM-5-GGUF** — dynamic UD-Q4 quants -- **QuantTrio/GLM-5.2-Int4-Int8Mix** — mixed int4/int8 with channel-wise scales -- **llama.cpp** — Q4_K with block-level group scales (typically 32 or 64 elements) - -None use naive per-row RTN int4 for production inference. - -## The fix: group-scaled int4 (fmt=4) - -Add a new quantization format with **one scale per 128 elements** along the input dimension, matching the FP8 source's natural granularity. - -### What changed - -**`c/glm.c`** — 122 lines added: - -1. **QT struct** (line 98): Added `int gs` field (group size, 0=per-row for backward compat, 128=grouped). - -2. **Format detection** (`qt_from_disk`, line 1064): Auto-detects fmt=4 by checking the `.qs` scale array size. If it has `O * ceil(I/128)` elements instead of `O`, it's grouped. **Old per-row models (fmt=2) work unchanged.** - -3. **New kernel** (`matmul_i4_grouped`, line 379): Same AVX2 nibble unpacking as `matmul_i4`, but the accumulator resets at each 128-element group boundary: `dot(x[grp], w[grp]) * scale[grp]`. The scale changes every 8 vector iterations (128/16=8). - -4. **Dispatch** (`matmul_qt_ex`, line 813): Routes to `matmul_i4_grouped` when `fmt==4`. Always uses exact kernels (no IDOT approximation — the whole point is quality). - -5. **Expert loading** (`expert_load`, lines 1418 and 1538): Both the mmap and slab+pread paths detect fmt=4 from scale array size and set `gs=128`. - -6. **`qt_bytes`** (line 104): Reports correct memory for fmt=4. - -**`c/tools/convert_fp8_to_int4.py`** — `quant_int4_grouped()` + `--group-size` arg: - -```python -def quant_int4_grouped(w, bits, gs=128): - O, I = w.shape - ngroups = (I + gs - 1) // gs - wpad = np.zeros((O, ngroups * gs), np.float32) - wpad[:, :I] = w - wr = wpad.reshape(O, ngroups, gs) # [O, ngroups, gs] - amax = np.abs(wr).max(axis=2, keepdims=True) # one max per GROUP - s = np.maximum(amax / qmax, 1e-8) - q = np.clip(np.rint(wr / s), -8, qmax) - # ... same nibble packing as quant_int4 ... - return packed_nibbles, s.reshape(-1) # [O * ngroups] scales -``` - -Same packed-nibble format as existing int4 — only the scale array is larger. - -### Verification - -**Converter round-trip test** (weights with varying group magnitudes): - -| Method | Mean relative error | Max abs error | -|--------|-------------------|---------------| -| Per-row int4 (current) | 0.2056 | 0.0127 | -| Grouped int4 (gs=128) | **0.1278** | 0.0127 | -| **Improvement** | **1.6x lower** | — | - -**Engine kernel test** (AVX2 `matmul_i4_grouped` vs f32 reference): - -| Output | Reference | Kernel | Error | -|--------|-----------|--------|-------| -| o=0 | -0.126368 | -0.126368 | 2.98e-08 | -| o=1 | 0.075913 | 0.075913 | 1.49e-08 | -| o=2 | 0.077136 | 0.077136 | 7.45e-09 | -| ... | ... | ... | ... | - -Max error: **2.98e-08** — matches f32 reference to within float32 epsilon. **PASS.** - -### Cost - -| Metric | Per-row (current) | Grouped (new) | -|--------|-------------------|---------------| -| Expert weight size | ~18.9 MB | ~20.1 MB (+6%) | -| Scale array per matrix | O × 4 bytes | O × ceil(I/128) × 4 bytes | -| Total model size | ~370 GB | ~390 GB (+5%) | -| Disk I/O per miss | ~19 MB | ~20 MB (+6%) | -| Kernel speed | One scale multiply per row | One scale multiply per 128 elements | - -The 6% I/O increase is negligible compared to the quality gain. The grouped kernel has slightly more scale-lookup overhead but remains within AVX2 throughput — the bottleneck is disk I/O, not matmul. - -### Backward compatibility - -- Old per-row models (fmt=2) **work unchanged** — format auto-detected from scale array size -- All existing features (PILOT, EXPERT_BUDGET, CUDA, MTP, PIPE) work identically — they don't touch the dequant path -- The fused gate+up pair path (`matmul_i4_pair`) falls back to separate `matmul_qt` calls for fmt=4 — minor perf cost, correctness preserved -- `--group-size 0` produces per-row output (backward compat for the converter) - -## How to reproduce - -### Convert with group scales - -```bash -python tools/convert_fp8_to_int4.py \ - --indir /path/to/GLM-5.2-FP8 \ - --outdir /path/to/glm52_i4_grouped \ - --ebits 4 --io-bits 8 --group-size 128 -``` - -### Test quality - -```bash -# Old model (per-row): -SNAP=/path/to/glm52_i4 PROMPT="Explain recursion in programming." NGEN=32 ./glm 64 - -# New model (grouped): -SNAP=/path/to/glm52_i4_grouped PROMPT="Explain recursion in programming." NGEN=32 ./glm 64 -``` - -Compare output text coherence. - -## What this won't fix - -- **Cold cache speed** — still 0.18-0.36 tok/s on 24 GB RAM. Quality and speed are independent axes. -- **All quantization error** — int4 is still 4-bit. Some degradation vs FP8 will remain. But it should be coherent degradation (slightly wrong word choices) rather than total reasoning failure (unrelated topics). -- **The IDOT +12% perplexity** — the approximate activation kernel (`IDOT=1`, default ON) still applies to non-grouped tensors (attention projections are already protected). Grouped int4 uses exact kernels by design. - -## Prior art - -- **GLM-5 paper** (arXiv 2602.15763, §2.4.3): "We apply INT4 QAT in the SFT stage" — the model is trained for int4, but assumes fine-grained quantization. -- **MxMoE** (ICML 2025): MoE experts exhibit divergent quantization sensitivity; uniform quant across all experts is suboptimal. -- **MoEQuant** (OpenReview): Naive int4 on MoE loses meaningful accuracy; calibrated framework needed. -- **Automated Fine-Grained MoE Quantization** (ACL 2025): Layer/expert-wise sensitivity variation; group-size scaling is the baseline improvement. -- **ubergarm/GLM-5.1-GGUF**: imatrix-calibrated quants with expert-specific patches — explicitly patches `quantize_row_q4_0_ref()` for routed experts. -- **llama.cpp Q4_K**: Uses block-level group scales (32 or 64 elements) as the standard int4 format. - -## Conversion status - -Re-converting from the FP8 source (`zai-org/GLM-5.2-FP8`) with `--group-size 128`. Downloading via ModelScope to avoid HuggingFace per-stream throttling. Will report quality A/B results once the conversion is complete.