Quarantine EXPERT_BUDGET: the operating window is measured empty (#303)
@bokiko benchmarked the feature on three hosts and found every setting is
either no faster or no longer coherent. Reproduced here on a 25 GB box, and
the numbers are worse than "a quality/speed tradeoff":
- budget=8 -> hellaswag 30% vs 90% with it off (25% is the chance floor)
- budget=4 -> decode is literal noise ("The **1...: s2151:")
- MTP acceptance 0%. Which experts survive the cap depends on cache
residency at the moment of the forward, so draft and verify do not
compute the same function -- the invariant #294 just established for
#163, violated through cache state instead of kernel dispatch. SPEC_PIN
cannot fix that and should not try: it is feature semantics, not
dispatch.
- 0.13 tok/s vs 0.30 baseline, while loading 14.66 experts per layer
against a topk=8 baseline. The cap does MORE disk I/O than not using it.
"~335 GB I/O saved" counts dropped experts, not bytes not read.
EXPERT_BUDGET>0 is now ignored unless EXPERT_BUDGET_EXPERIMENTAL=1, with the
measurements printed. The code stays compiled and developable: MoE-Spec
(arXiv 2602.16052) is not a wrong idea, this implementation just has no
point where it is both faster and correct. Re-enabling it by default needs a
quality number next to every speed number.
Also gates the cap to decode (S<=4), @woolcoxm's fix from #292/#298: during
prefill the batch union is 30-100+ experts, and capping to 4-8 drops most of
them, corrupting the hidden state and therefore the KV cache. Necessary but
not sufficient -- the run above already includes it.
Removes issue_budget.md, issue_diskio.md and issue_grouped_quant.md: design
notes in the repo root, unlinked from any README, whose only user-facing
content was "EXPERT_BUDGET=6-8 -- good speedup, minimal quality loss" and
"+83% decode at budget=4". Measurement says otherwise, and they were the
only thing on main telling anyone to switch this on. They stay in history.
Reported-by: bokiko <#303>
Co-Authored-By: woolcoxm <#292>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
JustVugg
@@ -2858,8 +2858,13 @@ static void moe(Model *m, Layer *l, int layer, float *x, int S, float *out, int
|
||||
* only drop from misses. From the misses, keep the highest-aggregate-gate-weight
|
||||
* ones up to the budget; drop the rest from idxs[] so they're never loaded.
|
||||
* (MoE-Spec arXiv 2602.16052: top-32 of 64 capture 93% routing weight.)
|
||||
* Complementary to TOPP (per-position) — this trims cross-position. */
|
||||
if(g_expert_budget>0 && nu>g_expert_budget){
|
||||
* Complementary to TOPP (per-position) — this trims cross-position.
|
||||
* DECODE-ONLY (S<=4, incl. MTP verify): during prefill S=prompt_len the batch
|
||||
* union nu is 30-100+ experts and capping to 4-8 drops 80-90% of them, each with
|
||||
* non-trivial gate weight -> corrupted prefill hidden state -> wrong KV cache ->
|
||||
* repetitive garbage decode. The budget is only safe token-by-token, where the
|
||||
* prefill KV cache is already correct. (woolcoxm, #292.) */
|
||||
if(g_expert_budget>0 && S<=4 && nu>g_expert_budget){
|
||||
/* compute aggregate gate weight per unique expert */
|
||||
float *wsum=falloc(nu); for(int j=0;j<nu;j++) wsum[j]=0;
|
||||
for(int s=0;s<S;s++) for(int kk=0;kk<keff[s];kk++){
|
||||
@@ -5759,7 +5764,31 @@ int main(int argc, char **argv){
|
||||
if(g_mmap) fprintf(stderr,"[MMAP] expert = viste zero-copy nei file (page cache = cache)\n");
|
||||
g_topk = getenv("TOPK")?atoi(getenv("TOPK")):0;
|
||||
g_topp = getenv("TOPP")?atof(getenv("TOPP")):0;
|
||||
/* EXPERT_BUDGET e' sotto quarantena: la finestra operativa e' misurata VUOTA.
|
||||
* @bokiko su tre host (#303) e riprodotto qui su un 25 GB / WSL:
|
||||
* - hellaswag 30% a budget=8 contro 90% a budget spento (25% = il caso);
|
||||
* - a budget=4 il decode e' rumore ("The **1...: s2151:");
|
||||
* - accettazione MTP 0%: quali expert sopravvivono al cap dipende dalla
|
||||
* residenza in cache al momento del forward, quindi draft e verify NON
|
||||
* calcolano la stessa funzione -- la stessa invariante che #294 ha appena
|
||||
* stabilito, violata via stato di cache invece che via scelta del kernel;
|
||||
* - 0.13 tok/s contro 0.30 di baseline, con 14.66 expert caricati per layer
|
||||
* contro topk=8: il cap fa piu' I/O di quello che dice di risparmiare, e la
|
||||
* riga "~N GB I/O saved" conta esperti scartati, non byte non letti.
|
||||
* Resta compilato e sviluppabile (EXPERT_BUDGET_EXPERIMENTAL=1) perche' l'idea
|
||||
* -- MoE-Spec, arXiv 2602.16052 -- non e' sbagliata: e' l'implementazione che
|
||||
* finora non ha un punto in cui sia insieme piu' veloce e corretta. Riaccenderlo
|
||||
* di default richiede una misura di qualita' accanto a quella di velocita'. */
|
||||
g_expert_budget = getenv("EXPERT_BUDGET")?atoi(getenv("EXPERT_BUDGET")):0;
|
||||
if(g_expert_budget>0 && !getenv("EXPERT_BUDGET_EXPERIMENTAL")){
|
||||
fprintf(stderr,"[EXPERT_BUDGET] ignored: measured empty operating window (issue #303).\n"
|
||||
"[EXPERT_BUDGET] every tested setting is either no faster or no longer coherent:\n"
|
||||
"[EXPERT_BUDGET] budget=8 -> hellaswag 30%% (90%% with it off) | budget=4 -> decode is noise\n"
|
||||
"[EXPERT_BUDGET] MTP acceptance 0%% (the cap breaks the draft/verify contract, #294)\n"
|
||||
"[EXPERT_BUDGET] 0.13 tok/s vs 0.30 baseline, loading 14.7 experts/layer vs topk=8\n"
|
||||
"[EXPERT_BUDGET] set EXPERT_BUDGET_EXPERIMENTAL=1 to run it anyway (expect garbage).\n");
|
||||
g_expert_budget=0;
|
||||
}
|
||||
g_cache_route = getenv("CACHE_ROUTE")?atoi(getenv("CACHE_ROUTE")):0;
|
||||
g_route_j = getenv("ROUTE_J")?atoi(getenv("ROUTE_J")):2;
|
||||
g_route_m = getenv("ROUTE_M")?atoi(getenv("ROUTE_M")):12;
|
||||
|
||||
-150
@@ -1,150 +0,0 @@
|
||||
## TL;DR
|
||||
|
||||
New `EXPERT_BUDGET=N` env var that caps the number of **distinct experts loaded per layer** across the batch-union. When the union exceeds the budget, keeps only the highest-aggregate-gate-weight experts and drops the rest — they're never loaded from disk. On a 24 GB RAM host (cache `cap=2`), `EXPERT_BUDGET=4` nearly **doubles decode tok/s** (0.18 → 0.33) and **4x's prefill speed** (38.7s → 8.9s). Based on MoE-Spec (arXiv 2602.16052): "top 32 of 64 experts capture 93% of routing weight."
|
||||
|
||||
Branch: `experiment/expert-budget` (based on latest `dev` at `62419af`)
|
||||
|
||||
---
|
||||
|
||||
## The problem
|
||||
|
||||
Every expert miss costs ~19 MB of disk I/O. On low-RAM hosts where the LRU cache cap is tiny (e.g. `cap=2` on 24 GB RAM), nearly every routed expert is a miss. With `topk=8` and 75 sparse layers, a single-token decode reads ~8.5 GB of experts from disk. Under MTP with `S=4`, the batch-union can produce 20-32 distinct experts per layer — almost all misses — multiplying disk reads further.
|
||||
|
||||
The README documents this directly:
|
||||
> "on a cold cache each verified draft routes to extra experts (~660 → ~1100 expert-loads/token)"
|
||||
|
||||
The existing `TOPP` env var trims experts *within a single position's top-K*. But it cannot reduce the **cross-position union** — the deduplication across batch positions (prefill, MTP verification) that multiplies disk loads. That's the gap this fills.
|
||||
|
||||
## How it works
|
||||
|
||||
The batch-union in `moe()` (line ~2306 of `glm.c`) deduplicates routed experts across all `S` positions into `uniq[0..nu)`. After the union is built but before the cache-resolve/load loop, the budget cap kicks in:
|
||||
|
||||
```
|
||||
if(EXPERT_BUDGET > 0 && nu > EXPERT_BUDGET):
|
||||
1. Compute aggregate gate weight per unique expert
|
||||
(sum of ws[s*K+kk] across all positions that route to it)
|
||||
2. Sort experts by descending aggregate weight
|
||||
3. Keep top EXPERT_BUDGET, mark rest as dropped
|
||||
4. Remove dropped experts from each position's idxs[]/keff[]
|
||||
(decrement keff, renormalize remaining weights if norm_topk)
|
||||
5. Compact uniq[] to kept experts only
|
||||
```
|
||||
|
||||
Dropped experts are removed from `idxs[]` entirely, so they're **never resolved, never loaded from disk, never computed**. The downstream code already tolerates `keff[s] < K` (the `TOPP` path produces the same state), so this propagates correctly through resolve, matmul, and LRU promotion.
|
||||
|
||||
**Relationship to existing features:**
|
||||
- `TOPP` trims within one position's top-K (per-position)
|
||||
- `EXPERT_BUDGET` trims across the cross-position union (per-layer)
|
||||
- They compose: `TOPP=0.8 EXPERT_BUDGET=12` first trims each position to its top-p mass, then caps the union
|
||||
|
||||
## Code change
|
||||
|
||||
**`c/glm.c`** — 54 lines added, single file:
|
||||
|
||||
1. Global + counter (near existing `g_topk`/`g_topp`):
|
||||
```c
|
||||
static int g_expert_budget=0; /* EXPERT_BUDGET=N */
|
||||
static int64_t g_budget_dropped=0; /* total experts dropped */
|
||||
```
|
||||
|
||||
2. Env var parsing (near existing `TOPK`/`TOPP`):
|
||||
```c
|
||||
g_expert_budget = getenv("EXPERT_BUDGET")?atoi(getenv("EXPERT_BUDGET")):0;
|
||||
```
|
||||
|
||||
3. Budget cap in `moe()` batch-union (after `uniq[]` is built, before resolve loop) — full logic described above.
|
||||
|
||||
4. Stats reporting alongside existing `TOPK`/`TOPP` line:
|
||||
```
|
||||
EXPERT_BUDGET=4 (dropped 13613 experts, ~257.3 GB I/O saved)
|
||||
```
|
||||
|
||||
## Measurements
|
||||
|
||||
**Test setup:** GLM-5.2 744B int4, 24 GB RAM (cache `cap=2`), Core Ultra 9 185H (AVX-VNNI), MTP=0, single-token decode, 32 tokens generated, same prompt: *"Explain the concept of recursion in programming. Provide a simple example."*
|
||||
|
||||
| Config | tok/s | vs baseline | hit rate | prefill | decode | experts dropped | I/O saved |
|
||||
|--------|-------|-------------|----------|---------|--------|-----------------|-----------|
|
||||
| **Baseline** (budget=0) | 0.18 | — | 9.3% | 38.7s | 176.3s | 0 | 0 |
|
||||
| EXPERT_BUDGET=12 | 0.19 | +5% | 14.0% | 12.3s | 171.3s | 3,313 | 62.6 GB |
|
||||
| EXPERT_BUDGET=6 | 0.26 | +44% | 21.0% | 7.5s | 122.5s | 8,543 | 161.5 GB |
|
||||
| **EXPERT_BUDGET=4** | **0.33** | **+83%** | 16.4% | 8.9s | **97.4s** | 13,613 | 257.3 GB |
|
||||
|
||||
### Profile breakdown (prefill)
|
||||
|
||||
| Metric | Baseline | Budget=4 |
|
||||
|--------|----------|----------|
|
||||
| expert-disk service | 28.4s | **3.5s** (8x less) |
|
||||
| expert-matmul | 7.1s | 1.4s |
|
||||
| attention | 2.3s | 2.8s |
|
||||
|
||||
### Profile breakdown (decode, 32 tokens)
|
||||
|
||||
| Metric | Baseline | Budget=4 |
|
||||
|--------|----------|----------|
|
||||
| expert-disk service | 133.4s | proportional ~65s |
|
||||
| expert-matmul | 23.9s | ~12s |
|
||||
| decode total | 176.3s | **97.4s** |
|
||||
|
||||
### Key observations
|
||||
|
||||
1. **Prefill is the biggest winner.** With S=14 positions, the batch-union has 50+ distinct experts per layer. Budget=12 cuts that to 12, saving 40+ × 19 MB × 75 layers of disk reads. Prefill goes from 38.7s to 8.9s — **4.4x faster**.
|
||||
|
||||
2. **Decode improvement scales with budget tightness.** Budget=12 barely constrains single-token decode (S=1 → max 8 experts < 12), so decode is barely affected. Budget=4 actually halves the decode load (8 → 4 experts/layer), nearly doubling tok/s.
|
||||
|
||||
3. **Hit rate improves** because fewer experts compete for the tiny cache (cap=2). Budget=6 hit 21% vs baseline 9.3% — more than doubled.
|
||||
|
||||
4. **No crashes or instability** at any budget value.
|
||||
|
||||
## Quality impact — honest assessment
|
||||
|
||||
**Budget=4 keeps only the top-4 of 8 routed experts per layer.** The dropped 4 experts had the lowest gate weights — they contributed the least to the output. But this IS a quality trade-off.
|
||||
|
||||
On a **cold cache** (our test setup), quality assessment is confounded: both baseline and budget outputs are already garbled from int4 quantization + cold-cache routing. Sample comparison:
|
||||
|
||||
- **Baseline output:** *"Hire Some To Take Object-Oriented Programming Assignment"*
|
||||
- **Budget=4 output:** *"The world is a dangerous place, not so much because of the small percentage of people who are doing to do the little of people who are doing to"*
|
||||
|
||||
Both are incoherent — the cold cache at int4 on 24 GB RAM produces poor output regardless of budget. The budget=4 output is **not dramatically worse** than the already-poor baseline — they're both garbled, just differently garbled. A proper quality assessment needs a **warm cache** where the baseline produces coherent text, so you can isolate the degradation from dropping experts.
|
||||
|
||||
**Expected quality on warm cache:** With `norm_topk=1` (GLM-5.2 renormalizes gate weights), the top-4 experts typically capture ~80-85% of routing weight (the remaining 4 share 15-20%). Output should be mostly coherent with occasional word-choice degradation. Budget=6 (top-6 of 8, ~90%+ weight captured) should be nearly indistinguishable from baseline.
|
||||
|
||||
**Recommendation for users:**
|
||||
- `EXPERT_BUDGET=6-8` on cold/low-RAM hosts — good speedup, minimal quality loss
|
||||
- `EXPERT_BUDGET=4` on very-low-RAM hosts where speed matters more than quality
|
||||
- Leave OFF on high-RAM hosts where all experts are resident anyway (budget never triggers)
|
||||
|
||||
## Safety
|
||||
|
||||
- **Default OFF** (`EXPERT_BUDGET=0`) — zero behavior change unless explicitly set
|
||||
- **Opt-in quality trade-off** — same design philosophy as `TOPP`: the user explicitly chooses speed over quality
|
||||
- **No output corruption** — dropped experts' contributions are omitted and remaining weights renormalized (identical math to `TOPP`)
|
||||
- **Compatible with all features** — PILOT, MTP, CUDA, CACHE_ROUTE, PIPE, TOPP all work unchanged; they just see fewer experts in the union
|
||||
- **No crash risk** — no new memory allocation patterns, no new I/O, purely a filter on existing data structures
|
||||
|
||||
## Reproduce
|
||||
|
||||
```bash
|
||||
make ARCH=native
|
||||
|
||||
# Baseline:
|
||||
SNAP=<model_dir> MTP=0 PROMPT="Explain recursion in programming." NGEN=32 ./glm 64
|
||||
|
||||
# With budget:
|
||||
SNAP=<model_dir> MTP=0 EXPERT_BUDGET=4 PROMPT="Explain recursion in programming." NGEN=32 ./glm 64
|
||||
|
||||
# With MTP (where budget saves the most):
|
||||
SNAP=<model_dir> MTP=1 EXPERT_BUDGET=16 PROMPT="Explain recursion in programming." NGEN=32 ./glm 64
|
||||
```
|
||||
|
||||
The stats line prints `EXPERT_BUDGET=N (dropped X experts, ~Y GB I/O saved)`.
|
||||
|
||||
## What's needed before merge
|
||||
|
||||
1. **Warm-cache quality A/B test** on a host with enough RAM (cap>=16) to produce coherent baseline output, then compare budget=4/6/8 text quality
|
||||
2. **MTP interaction test** — verify that budget + MTP composes correctly and doesn't crash when draft verification routes to budgeted-away experts
|
||||
3. **Decide on defaults** — should this auto-activate on low-RAM hosts (like `cap` auto-lowering)? My recommendation: no, leave it OFF and document the recommended value per RAM tier
|
||||
|
||||
## Prior art
|
||||
|
||||
**MoE-Spec** (arXiv 2602.16052) — "Expert Budgeting for Efficient Speculative Decoding": Training-free expert budgeting at verification time. Key finding: "top 32 of 64 experts capture 93% of routing weight." Our approach applies the same principle but at the batch-union level (not just MTP verification), making it effective for prefill and single-token decode too.
|
||||
-309
@@ -1,309 +0,0 @@
|
||||
# Disk I/O Minimization — Research
|
||||
|
||||
Branch: `experiment/diskio-research` (based on `dev` at `62419af`)
|
||||
|
||||
## TL;DR
|
||||
|
||||
The engine's disk I/O is **already well-engineered on the hottest path** (expert streaming uses coalesced O_DIRECT `pread` + `posix_fadvise` hints + LRU + pin cache + speculative prefetch). There are **4 concrete, bounded opportunities** to shave latency, ranked by ROI:
|
||||
|
||||
| # | Opportunity | Where | Frequency | Estimated win |
|
||||
|---|---|---|---|---|
|
||||
| 1 | **KV-cache write batching** (157 fwrites/token → 1) | `kv_disk_append` | per turn | cuts ~100s of syscalls/turn |
|
||||
| 2 | **`/proc/meminfo` fopen storm** | `rss_gb()` | ~every 16 tokens (Linux) | eliminates recurring open/read/close |
|
||||
| 3 | **Expert prefetch on Windows** (`PrefetchVirtualMemory`) | `expert_prefetch` | per miss | mmap path is Linux/macOS-only today |
|
||||
| 4 | **KV-cache: buffered handle kept open** | `kv_disk_append` | per turn | kills open+fseek+close per turn |
|
||||
|
||||
There are also **2 non-opportunities** worth recording so we don't re-investigate: O_DIRECT for experts (correctly used today), and PagedAttention-style file layout (already single-file + indexed).
|
||||
|
||||
---
|
||||
|
||||
## How the engine does disk I/O today
|
||||
|
||||
There are **three I/O stacks**, behaving very differently:
|
||||
|
||||
| Stack | Mechanism | Frequency | Files |
|
||||
|---|---|---|---|
|
||||
| **Expert weights** (hottest) | `pread` on kept-open fds + `posix_fadvise`, optional `mmap` | per miss, every token | `st.h`, `glm.c:1328` |
|
||||
| **KV cache** (`.coli_kv`) | `fopen` + `fwrite`/`fread` | per turn | `glm.c:3812-3889` |
|
||||
| **Everything else** (config, tokenizer, stats, grammar) | `fopen` + `fread` | startup-only | scattered |
|
||||
|
||||
### Expert path (the hot path — already good)
|
||||
|
||||
`expert_load` (`glm.c:1328-1481`) has three sub-paths:
|
||||
|
||||
- **Default `pread` path** (`glm.c:1385-1472`): coalesces the 3 contiguous expert tensors (gate/up/down) into **one ~19 MB O_DIRECT `pread`** into a 16K-aligned slab (`glm.c:1447`). Falls back to 3 separate `pread`s only if non-contiguous. Scales are 3 tiny separate `pread`s (kilobytes). `posix_fadvise(DONTNEED)` evicts pages after if `g_drop`. **This is well-batched — one syscall for ~19 MB.**
|
||||
- **`COLI_MMAP=1` path** (`glm.c:1352-1383`): `mmap` per shard fd (cached), `madvise(WILLNEED)` + synchronous page-touch loop. Zero-copy. **Default OFF, and Linux/macOS/FreeBSD-only** — no `MapViewOfFile` on Windows.
|
||||
- **Prefetch hints**: `expert_prefetch` (`glm.c:1602-1609`) → `st_prefetch` (`st.h:178`) issues `posix_fadvise(WILLNEED)` — readahead hint only, no data read. Called from `moe` next-64-block lookahead, pilot, and SPEC.
|
||||
|
||||
### KV cache persistence (per turn — opportunity here)
|
||||
|
||||
`kv_disk_append` (`glm.c:3834-3855`), called once per turn:
|
||||
1. `fopen("r+b")` — **reopens the file every turn**
|
||||
2. `fseek` to append position
|
||||
3. **per-position loop**: for each new token, `fwrite` the token i32, then **2 fwrites per layer** (Lc + Rc) + optional DSA Ic. With 78 layers that's **~157 fwrites per token appended**.
|
||||
4. `fflush` (userspace only — **no fsync/fdatasync anywhere in the codebase**)
|
||||
5. `fseek` back to header + `fwrite` the new nrec counter (crash-safe ordering)
|
||||
6. `fclose`
|
||||
|
||||
Record size ~182 KB/token. On a long first turn this is **tens of thousands of small fwrites**. stdio buffering coalesces them into fewer `write` syscalls, but the userspace overhead remains.
|
||||
|
||||
### Recurring surprise: `/proc/meminfo`
|
||||
|
||||
`rss_gb()` (`glm.c:4625`) does `fopen("/proc/meminfo")` + fgets + fclose. Called from every STAT line and every 16-token heartbeat (`glm.c:3473, 3477`). On Linux this is an **open+read+close of procfs ~every 16 tokens**. (Windows uses `compat_meminfo`, no file — not affected.)
|
||||
|
||||
---
|
||||
|
||||
## What similar projects do
|
||||
|
||||
**llama.cpp** (the reference): `mmap`s the entire model read-only, uses `--mlock` to pin hot pages, streams layers to GPU via partial offload, and issues per-pass readahead of upcoming tensors (`llama-mmap.cpp`). Justine Tunney's mmap work: "load 100× faster using half as memory." Crucial finding from discussion #18758: **for MoE, mmap beats O_DIRECT** when the model fits in ~RAM — O_DIRECT takes "at least 10× longer" on repeated loads because it bypasses the page cache that serves re-faults for free.
|
||||
|
||||
**The general consensus across llama.cpp, vLLM, AirLLM, PRESERVE, HOBBIT, SolidAttention (FAST '26):**
|
||||
- mmap + OS page cache as the backing store for an LRU is the proven recipe
|
||||
- prefetch the *next* expert/layer while computing the current one — this is where the 0.5ms lives
|
||||
- single indexed file (one `open()`) beats one-file-per-expert
|
||||
- align tensors to 4KB (preferably 64KB) for clean page-fault boundaries + SSD geometry
|
||||
- buffer sweet spot ~1MB; syscall cost ~1-5µs each, so batching matters at high repetition
|
||||
|
||||
---
|
||||
|
||||
## The 4 opportunities (ranked)
|
||||
|
||||
### Opportunity 1 — KV-cache write batching (HIGH ROI, LOW risk)
|
||||
|
||||
**Problem:** `kv_disk_append` does ~157 `fwrite` calls per appended token (1 token i32 + 2×78 layers). stdio buffering hides some of this, but on a long first-turn prefill (hundreds-thousands of tokens) this is tens of thousands of fwrites.
|
||||
|
||||
**Fix:** Build one contiguous record in a heap buffer (token + all layers' Lc/Rc/Ic for that position), then **a single `fwrite` per position** (or even one `fwrite` for the whole turn). The data is already laid out contiguously in memory per-layer (`coli_kv_row`), so a layered `memcpy` into a staging buffer + one write is straightforward.
|
||||
|
||||
**Win:** ~157× fewer fwrite calls per token. Even with stdio coalescing, the userspace loop overhead is real at scale.
|
||||
|
||||
### Opportunity 2 — `/proc/meminfo` fopen storm (MEDIUM ROI, trivial)
|
||||
|
||||
**Problem:** `rss_gb()` opens, reads, closes `/proc/meminfo` every ~16 tokens on Linux. Each is ~3 syscalls + path resolution.
|
||||
|
||||
**Fix:** Either (a) cache the value for N tokens (e.g. re-read at most once per second), or (b) keep the fd open and `rewind`+`fgets`. Trivial change.
|
||||
|
||||
### Opportunity 3 — Expert prefetch on Windows (MEDIUM ROI, bounded)
|
||||
|
||||
**Problem:** The `COLI_MMAP=1` path (which gives zero-copy expert access + free OS-cache re-faults) is **Linux/macOS/FreeBSD-only** — `glm.c:1301` guards it. On Windows, experts always go through the `pread` path, and `expert_prefetch` issues `posix_fadvise(WILLNEED)` which is a no-op shim on Windows (`compat.h`).
|
||||
|
||||
**Fix:** On Windows, implement the prefetch via `PrefetchVirtualMemory` (the Win32 analog of `MADV_WILLNEED`) on an mmap'd region, or via an async `ReadFile`+`OVERLAPPED` into a scratch buffer. This brings the Windows build closer to parity with the Linux mmap+prefetch story.
|
||||
|
||||
**Scope:** This is the largest of the four — it touches the Windows I/O path. Worth doing if Windows perf is a goal; skip if Linux is the target.
|
||||
|
||||
### Opportunity 4 — KV-cache: keep handle open (LOW-MEDIUM ROI, LOW risk)
|
||||
|
||||
**Problem:** `kv_disk_append` does `fopen`+...+`fclose` every turn. Handle creation is ~5-15µs of pure overhead (worse on Windows).
|
||||
|
||||
**Fix:** Open the KV file once (lazily on first append), keep the `FILE*` for the engine lifetime, just `fseek`+write each turn. Close on shutdown. Pair with Opportunity 1 for the write batching.
|
||||
|
||||
---
|
||||
|
||||
## Non-opportunities (recording so we don't re-investigate)
|
||||
|
||||
- **O_DIRECT for experts**: already correctly used (`st.h:83`, `DIRECT=1`). For an LRU+refetch pattern the page cache is your friend, but the engine offers both paths (O_DIRECT pread default + optional mmap) and the O_DIRECT coalesced read is already one syscall for ~19MB. Don't change this.
|
||||
- **Single-file layout**: the engine already uses safetensors shards with kept-open fds + offset-indexed tensors (`st.h`). No per-expert open()/close() waste. Don't change this.
|
||||
- **PagedAttention**: solves concurrency fragmentation this engine doesn't have (≤16 slots). Not applicable.
|
||||
|
||||
---
|
||||
|
||||
## Next steps
|
||||
|
||||
The highest-ROI, lowest-risk starting point is **Opportunity 1 (KV write batching) + Opportunity 4 (keep handle open)** — they're in the same function, both low-risk, and together they eliminate the per-turn open/close overhead and the per-token fwrite storm. Opportunity 2 is a trivial 5-minute fix we can bundle in.
|
||||
|
||||
Opportunity 3 (Windows prefetch) is the biggest single win but also the largest scope — separate effort, gated on whether Windows perf is a priority.
|
||||
|
||||
## Sources
|
||||
|
||||
- [justine.lol/mmap — Edge AI Just Got Faster](https://justine.lol/mmap/)
|
||||
- [llama.cpp discussion #18758 — Mmap faster than direct I/O for MoE](https://github.com/ggml-org/llama.cpp/discussions/18758)
|
||||
- [llama.cpp issue #20757 — Two-tier GPU+RAM expert cache](https://github.com/ggml-org/llama.cpp/issues/20757)
|
||||
- [FAST '26 — Programmable Page Cache for LLM loading](https://www.usenix.org/system/files/fast26-liu-yubo.pdf)
|
||||
- [FAST '26 — SolidAttention: SSD-based serving](https://www.usenix.org/system/files/fast26-zheng.pdf)
|
||||
- [HOBBIT — Mixed precision expert offloading](https://arxiv.org/html/2411.01433v2)
|
||||
- [posix_fadvise(2) — man7.org](https://man7.org/linux/man-pages/man2/posix_fadvise.2.html)
|
||||
- [madvise(2) — man7.org](https://man7.org/linux/man-pages/man2/madvise.2.html)
|
||||
- [Microsoft Learn — File Buffering (FILE_FLAG_NO_BUFFERING)](https://learn.microsoft.com/en-us/windows/win32/fileio/file-buffering)
|
||||
- [Microsoft Learn — PrefetchVirtualMemory](https://learn.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-prefetchvirtualmemory)
|
||||
- [What makes system calls expensive — codingconfessions.com](https://blog.codingconfessions.com/p/what-makes-system-calls-expensive)
|
||||
- [Syscall overhead — Stack Overflow](https://stackoverflow.com/questions/8247331/syscall-overhead)
|
||||
|
||||
---
|
||||
|
||||
# Windows Implementation — branch `windows-optimizations` (2026-07-15)
|
||||
|
||||
## What landed (pread path, validated)
|
||||
|
||||
Two changes, both on the `pread` expert-load path (no mmap). Measured against the
|
||||
existing `bench_budget*.txt` baselines (GLM-5.2 744B int4, 32 GB RAM, Core Ultra 9
|
||||
185H, DRAFT=0, 32-token decode):
|
||||
|
||||
### 1. `compat_fadvise` WILLNEED cache-warmer (`c/compat.h`)
|
||||
|
||||
Replaced the Windows `posix_fadvise` no-op (was a `do{}while(0)` macro) with a real
|
||||
readahead: an overlapped `ReadFile` into a throwaway scratch buffer that populates the
|
||||
standby page cache, so the later synchronous `pread` faults from RAM not disk. Mirrors
|
||||
the macOS `F_RDADVISE` shim (`compat.h:28-37`). DONTNEED stays a no-op (matches macOS;
|
||||
Windows standby-list trimming self-regulates under pressure).
|
||||
|
||||
This re-arms the existing `expert_prefetch` → `st_prefetch` → `posix_fadvise(WILLNEED)`
|
||||
chain on Windows: the next-block readahead in `moe()` and the PILOT cross-layer prefetch
|
||||
hints now actually warm the cache instead of being silently discarded.
|
||||
|
||||
**Measured effect (budget=4, PIPE on):** hit rate 16.4% → 27.6%.
|
||||
|
||||
### 2. PIPE default ON for Windows (`c/glm.c`)
|
||||
|
||||
Flipped the async expert-load thread pool from default OFF to default ON on Windows
|
||||
(`getenv("PIPE")?:1` under `_WIN32`, unchanged `:0` elsewhere). PIPE dispatches expert
|
||||
`pread` loads onto worker threads so they overlap the expert matmul on the forward-pass
|
||||
thread, instead of the blocking serial load-then-compute path. `PIPE=0` opts back out.
|
||||
|
||||
**Measured effect (budget=4):** expert-disk 65.9s → 54.3s (−18%), reaching **1.70 s/tok**
|
||||
(under the 2 s/tok target; budget=4 baseline was 2.06 s/tok).
|
||||
|
||||
### Results table (DRAFT=0, 32-token decode, pread path)
|
||||
|
||||
| config | expert-disk | s/tok | hit% | tok/s |
|
||||
|---|---|---|---|---|
|
||||
| budget=4, no PIPE (existing baseline) | 65.9s | 2.06 | 16.4% | 0.33 |
|
||||
| **budget=4 + PIPE (this PR)** | **54.3s** | **1.70** | **27.6%** | **0.34** |
|
||||
| budget=6 + PIPE | 77.1s | 2.41 | 21.8% | 0.27 |
|
||||
|
||||
budget=4 + PIPE meets the ≤2 s/tok target. budget=6 (more experts/layer, higher quality)
|
||||
misses it at 2.41 s/tok — the speed/quality tradeoff.
|
||||
|
||||
## What was tried and abandoned: Windows mmap (`COLI_MMAP` on `_WIN32`)
|
||||
|
||||
The original plan (informed by llama.cpp #18758: "mmap is ≥10× faster than O_DIRECT for
|
||||
MoE") was to port the mmap expert path to Windows via `CreateFileMapping`/`MapViewOfFile`.
|
||||
This was implemented and tested at length. **It was a measured regression and was reverted.**
|
||||
|
||||
### The attempt
|
||||
|
||||
Added a `_WIN32` branch to `map_of_fd` (`glm.c`) mapping each shard file read-only and
|
||||
resolving experts as views into the mapping, mirroring the POSIX path. Also added
|
||||
`PrefetchVirtualMemory` readahead and a `VirtualUnlock` eviction mechanism (the Windows
|
||||
`posix_fadvise(DONTNEED)` analog — see SO#1880714; validated standalone to demote pages
|
||||
to the standby list with a 2.3× faster re-fault).
|
||||
|
||||
### Why it regressed
|
||||
|
||||
**mmap'd expert pages bloat the process working set on Windows, which collapses the
|
||||
expert cache.** This is a fundamental Windows-vs-Linux difference:
|
||||
|
||||
- On Linux, `mmap(MAP_SHARED)` file pages live in the kernel page cache (`buff/cache`),
|
||||
separate from `MemAvailable`, so the cache budget isn't fooled.
|
||||
- On Windows, touched `MapViewOfFile` pages count against `ullAvailPhys` (what
|
||||
`compat_meminfo` reads for the budget). The CPU matmul touches every weight byte,
|
||||
faulting ~12 GB into the working set. `cap_for_ram()` then sees ~no free RAM and
|
||||
collapses the LRU cache cap.
|
||||
|
||||
Measured (budget=0, DRAFT=0, apples-to-apples):
|
||||
|
||||
| config | RAM_GB detected | cache cap | hit% | expert-disk | RSS |
|
||||
|---|---|---|---|---|---|
|
||||
| baseline (pread) | 21.4 | 1 | 9.3% | 133s | 15.0 GB |
|
||||
| mmap, no eviction | **8.0** | 1 | **2.2%** | **240s** | **27.2 GB** |
|
||||
| mmap + VirtualUnlock | 24.9 | 2 | 11.8% | 83s | 18.1 GB |
|
||||
| mmap + reserve reductions | 24.6 | 4 | 21.8% | 80s | 20.1 GB |
|
||||
|
||||
The `VirtualUnlock` eviction recovered the regression (240s→83s), and dropping the
|
||||
Linux-specific page-cache/slab reserves under mmap got it to parity with pread. But it
|
||||
never clearly *beat* the simpler pread+PIPE path, and it added substantial complexity
|
||||
(per-slot eviction tracking, reserve conditionals, `VirtualUnlock` on every slot recycle).
|
||||
**The engine already moved off mmap to pread for this exact RSS bug** (`st.h:3-6`), and
|
||||
the Windows port re-confirmed that decision.
|
||||
|
||||
### What else didn't work
|
||||
|
||||
- **Batched `PrefetchVirtualMemory`** for the mmap path: tested as a single batched
|
||||
readahead of all 64 missed experts' pages before the matmul. **Blocked instead of
|
||||
prefetching async** on this SSD — inflated `t_edisk` (80s→102s). Consistent with
|
||||
microsoft/Windows-Dev-Performance#108 ("PrefetchVirtualMemory does not prefetch").
|
||||
Reverted.
|
||||
- **True I/O/compute overlap on the CPU path**: the Metal path has this ("submit
|
||||
resident experts to GPU before loading misses"), but the CPU path loads-then-computes
|
||||
serially. `PrefetchVirtualMemory` was the attempt to add it for mmap and failed. The
|
||||
pread path gets overlap via PIPE (which works), not via mmap prefetch.
|
||||
|
||||
### Conclusion
|
||||
|
||||
For this engine on Windows at this RAM budget (~32 GB, 370 GB model), **pread + PIPE +
|
||||
compat_fadvise** is the right path. mmap remains valuable on Linux/macOS (where the page
|
||||
cache doesn't inflate process RSS) but is not viable on Windows without a fundamentally
|
||||
different cache-budget model that excludes mapped-file pages — left as future work.
|
||||
|
||||
## Sources added
|
||||
|
||||
- [SO#1880714 — VirtualUnlock releases mapped pages to standby list](https://stackoverflow.com/questions/1880714/createfilemapping-mapviewoffile-how-to-avoid-holding-up-the-system-memory)
|
||||
- [Alois Kraus — The Mysterious Lost Memory (modified/standby list)](https://aloiskraus.wordpress.com/2017/02/26/the-mysterious-lost-memory-which-belongs-to-no-process/)
|
||||
- [microsoft/Windows-Dev-Performance#108 — PrefetchVirtualMemory inconsistency](https://github.com/microsoft/Windows-Dev-Performance/issues/108)
|
||||
- [llama.cpp #18758 — mmap faster than O_DIRECT for MoE (Linux)](https://github.com/ggml-org/llama.cpp/discussions/18758)
|
||||
- [HN#35426679 — Why MMAP in llama.cpp hides true memory usage](https://news.ycombinator.com/item?id=35426679)
|
||||
|
||||
---
|
||||
|
||||
# CACHE_ROUTE: miss elimination via cache-aware routing (2026-07-15)
|
||||
|
||||
## The breakthrough
|
||||
|
||||
Adding `CACHE_ROUTE=1 ROUTE_J=2 ROUTE_M=12` to the optimized stack pushed
|
||||
throughput to **1.41 tok/s** (4.3× over stock) by directly reducing the
|
||||
miss rate from 27% to 17%.
|
||||
|
||||
## How it works
|
||||
|
||||
The engine's `CACHE_ROUTE` feature (paper: max-rank routing, arXiv 2412.00099)
|
||||
steers the MoE router to prefer experts that are already cache-resident:
|
||||
|
||||
- `ROUTE_J=2`: keep the top-2 true router picks (always, even if uncached)
|
||||
- `ROUTE_M=12`: fill the remaining 2 of 4 slots with the highest-ranked experts
|
||||
that are ALREADY in the LRU cache (from the top-12 candidates)
|
||||
|
||||
This guarantees 2 of 4 expert slots per layer per token are cache hits. The
|
||||
`route_agree=94.6%` metric confirms minimal quality cost — 94.6% of cache-steered
|
||||
picks match the true top-K the router would have chosen.
|
||||
|
||||
## Why this is the right fix for the miss problem
|
||||
|
||||
Analysis of route traces showed GLM-5.2's routing is nearly uniform — 75 tokens
|
||||
use 237 of 256 experts per layer, with the top-4 capturing only 4.3% of selections.
|
||||
This means:
|
||||
|
||||
- PIN (hot-expert pre-loading) is ineffective — there are no hot experts
|
||||
- PILOT_REAL prefetch can't keep up under the fast pipe2 GPU pipeline
|
||||
- A bigger cache helps marginally but can't cover 237 unique experts per layer
|
||||
- CACHE_ROUTE is the only lever that reduces misses without more RAM or faster disk
|
||||
|
||||
## Results (pipe2 + full stack + ws_b fix, budget=4, RAM_GB=28)
|
||||
|
||||
| metric | without CACHE_ROUTE | with CACHE_ROUTE |
|
||||
|---|---|---|
|
||||
| tok/s | 1.03 | **1.41** (+37%) |
|
||||
| hit rate | 73% | **83%** |
|
||||
| expert-disk | 12.4s | **8.5s** (−31%) |
|
||||
| decode | 31.0s | **22.7s** (−27%) |
|
||||
| route_agree | n/a | 94.6% |
|
||||
|
||||
## Full optimization journey
|
||||
|
||||
| step | tok/s | hit% | disk |
|
||||
|---|---|---|---|
|
||||
| stock budget=4 | 0.33 | 9% | 65.9s |
|
||||
| + disk stack | 0.63 | 72% | 21.5s |
|
||||
| + CUDA dense+attn | 0.72 | 75% | 18.4s |
|
||||
| + pipe2 GPU pipeline | 0.85 | 57% | 18.2s |
|
||||
| + ws_b cache fix | 1.03 | 73% | 12.4s |
|
||||
| **+ CACHE_ROUTE** | **1.41** | **83%** | **8.5s** |
|
||||
|
||||
**4.3× total speedup.** Disk I/O reduced 7.7× (65.9s → 8.5s).
|
||||
|
||||
## Recommended config
|
||||
|
||||
```
|
||||
EXPERT_BUDGET=4 PIPE=1 RAM_GB=28 PILOT_REAL=1 DIRECT=1
|
||||
COLI_CUDA=1 CUDA_DENSE=1 COLI_CUDA_ATTN=1 COLI_CUDA_PIPE=2 CUDA_EXPERT_GB=0
|
||||
CACHE_ROUTE=1 ROUTE_J=2 ROUTE_M=12
|
||||
```
|
||||
|
||||
CACHE_ROUTE is an existing engine feature (no code change) — opt-in via env vars.
|
||||
@@ -1,185 +0,0 @@
|
||||
## TL;DR
|
||||
|
||||
The int4 model produces **fluent-but-incoherent output** — grammatical English that is completely off-topic (SEO spam, homework-help boilerplate) instead of reasoned responses. The root cause is **per-row quantization scales**: one F32 scale per output row (e.g. 2048 scales for a 2048×6144 matrix), which is 48x coarser than the FP8 source's 128×128 block scales. This destroys fine-grained weight information that handles reasoning and instruction-following while preserving the large-magnitude weights that handle grammar and vocabulary fluency.
|
||||
|
||||
Branch: `experiment/grouped-quant` (based on latest `dev` at `62419af`)
|
||||
|
||||
---
|
||||
|
||||
## The problem — demonstrated
|
||||
|
||||
Prompt: *"Explain the concept of recursion in programming. Provide a simple example."*
|
||||
|
||||
**Baseline (no budget, no changes):**
|
||||
> "Hire Some To Take Object-Oriented Programming Assignment"
|
||||
|
||||
**Budget=4:**
|
||||
> "The world is a dangerous place, not so much because of the small percentage of people who are doing to do the little of people who are doing to"
|
||||
|
||||
**Budget=6:**
|
||||
> "Elite Custom Essays"
|
||||
|
||||
**Budget=12:**
|
||||
> "Sololearn: Learn to code for FREE! +1 # Explain A concept of recursion..."
|
||||
|
||||
This is not random gibberish — it's fluent English that is completely off-topic. This is the signature of **activations corrupted by coarse quantization**: the model retains language fluency (large weights survive) but loses instruction-following and reasoning (fine-grained weights are crushed to zero).
|
||||
|
||||
## Root cause: per-row int4 scales
|
||||
|
||||
### How the current converter works
|
||||
|
||||
`convert_fp8_to_int4.py` (line 39-52) quantizes with **one scale per output row**:
|
||||
|
||||
```python
|
||||
amax = np.abs(w).max(axis=1, keepdims=True) # one max per ROW
|
||||
s = np.maximum(amax / qmax, 1e-8) # one scale per ROW
|
||||
q = np.clip(np.rint(w / s), -8, qmax) # quantize entire row with that scale
|
||||
```
|
||||
|
||||
For a 2048×6144 expert weight matrix, that's **2048 scales** — one per row, each covering 6144 elements.
|
||||
|
||||
### Why this destroys quality
|
||||
|
||||
Consider a row of 6144 values where most are small (magnitude ~0.01) but a few are large (~0.5). The per-row scale is `0.5/7 = 0.071`. The small values become `0.01/0.071 = 0.14`, which rounds to **zero**. All fine-grained information in those small weights is lost.
|
||||
|
||||
This matters enormously for MoE models: each token activates only 8 of 256 experts. A poorly-quantized expert pollutes every token that routes to it, and there's no averaging from the other 248 experts (they're simply off). The error is **concentrated**, not diluted.
|
||||
|
||||
### What the FP8 source does right
|
||||
|
||||
The FP8 checkpoint uses **128×128 block scales** — the weight matrix is divided into 128-element chunks, each with its own scale. Small values in one chunk don't get crushed by large values in another. For a 2048×6144 matrix: `2048 × 48 = 98,304` scales — 48x more granularity.
|
||||
|
||||
The converter already dequants FP8 to f32 correctly (lines 196-201), preserving the block-scale information. But then it **throws it all away** by collapsing to a single per-row scale during int4 requantization.
|
||||
|
||||
### GLM-5.2 was QAT-trained for int4
|
||||
|
||||
The GLM-5 paper (arXiv 2602.15763, §2.4.3) states: *"To provide better accuracy at low-precision, we apply INT4 QAT in the SFT stage."* This means the model is **designed** to work at int4 — but only if the quantization is fine-grained enough to match what the model saw during training. The FP8 checkpoint's 128×128 block scales are the granularity the model expects. Per-row scaling is 48x coarser.
|
||||
|
||||
### Community confirmation
|
||||
|
||||
Every other project getting coherent GLM-5.2 int4 output uses calibrated or fine-grained quantization:
|
||||
- **ubergarm/GLM-5.1-GGUF** — imatrix-calibrated with expert-specific patches
|
||||
- **Unsloth/GLM-5-GGUF** — dynamic UD-Q4 quants
|
||||
- **QuantTrio/GLM-5.2-Int4-Int8Mix** — mixed int4/int8 with channel-wise scales
|
||||
- **llama.cpp** — Q4_K with block-level group scales (typically 32 or 64 elements)
|
||||
|
||||
None use naive per-row RTN int4 for production inference.
|
||||
|
||||
## The fix: group-scaled int4 (fmt=4)
|
||||
|
||||
Add a new quantization format with **one scale per 128 elements** along the input dimension, matching the FP8 source's natural granularity.
|
||||
|
||||
### What changed
|
||||
|
||||
**`c/glm.c`** — 122 lines added:
|
||||
|
||||
1. **QT struct** (line 98): Added `int gs` field (group size, 0=per-row for backward compat, 128=grouped).
|
||||
|
||||
2. **Format detection** (`qt_from_disk`, line 1064): Auto-detects fmt=4 by checking the `.qs` scale array size. If it has `O * ceil(I/128)` elements instead of `O`, it's grouped. **Old per-row models (fmt=2) work unchanged.**
|
||||
|
||||
3. **New kernel** (`matmul_i4_grouped`, line 379): Same AVX2 nibble unpacking as `matmul_i4`, but the accumulator resets at each 128-element group boundary: `dot(x[grp], w[grp]) * scale[grp]`. The scale changes every 8 vector iterations (128/16=8).
|
||||
|
||||
4. **Dispatch** (`matmul_qt_ex`, line 813): Routes to `matmul_i4_grouped` when `fmt==4`. Always uses exact kernels (no IDOT approximation — the whole point is quality).
|
||||
|
||||
5. **Expert loading** (`expert_load`, lines 1418 and 1538): Both the mmap and slab+pread paths detect fmt=4 from scale array size and set `gs=128`.
|
||||
|
||||
6. **`qt_bytes`** (line 104): Reports correct memory for fmt=4.
|
||||
|
||||
**`c/tools/convert_fp8_to_int4.py`** — `quant_int4_grouped()` + `--group-size` arg:
|
||||
|
||||
```python
|
||||
def quant_int4_grouped(w, bits, gs=128):
|
||||
O, I = w.shape
|
||||
ngroups = (I + gs - 1) // gs
|
||||
wpad = np.zeros((O, ngroups * gs), np.float32)
|
||||
wpad[:, :I] = w
|
||||
wr = wpad.reshape(O, ngroups, gs) # [O, ngroups, gs]
|
||||
amax = np.abs(wr).max(axis=2, keepdims=True) # one max per GROUP
|
||||
s = np.maximum(amax / qmax, 1e-8)
|
||||
q = np.clip(np.rint(wr / s), -8, qmax)
|
||||
# ... same nibble packing as quant_int4 ...
|
||||
return packed_nibbles, s.reshape(-1) # [O * ngroups] scales
|
||||
```
|
||||
|
||||
Same packed-nibble format as existing int4 — only the scale array is larger.
|
||||
|
||||
### Verification
|
||||
|
||||
**Converter round-trip test** (weights with varying group magnitudes):
|
||||
|
||||
| Method | Mean relative error | Max abs error |
|
||||
|--------|-------------------|---------------|
|
||||
| Per-row int4 (current) | 0.2056 | 0.0127 |
|
||||
| Grouped int4 (gs=128) | **0.1278** | 0.0127 |
|
||||
| **Improvement** | **1.6x lower** | — |
|
||||
|
||||
**Engine kernel test** (AVX2 `matmul_i4_grouped` vs f32 reference):
|
||||
|
||||
| Output | Reference | Kernel | Error |
|
||||
|--------|-----------|--------|-------|
|
||||
| o=0 | -0.126368 | -0.126368 | 2.98e-08 |
|
||||
| o=1 | 0.075913 | 0.075913 | 1.49e-08 |
|
||||
| o=2 | 0.077136 | 0.077136 | 7.45e-09 |
|
||||
| ... | ... | ... | ... |
|
||||
|
||||
Max error: **2.98e-08** — matches f32 reference to within float32 epsilon. **PASS.**
|
||||
|
||||
### Cost
|
||||
|
||||
| Metric | Per-row (current) | Grouped (new) |
|
||||
|--------|-------------------|---------------|
|
||||
| Expert weight size | ~18.9 MB | ~20.1 MB (+6%) |
|
||||
| Scale array per matrix | O × 4 bytes | O × ceil(I/128) × 4 bytes |
|
||||
| Total model size | ~370 GB | ~390 GB (+5%) |
|
||||
| Disk I/O per miss | ~19 MB | ~20 MB (+6%) |
|
||||
| Kernel speed | One scale multiply per row | One scale multiply per 128 elements |
|
||||
|
||||
The 6% I/O increase is negligible compared to the quality gain. The grouped kernel has slightly more scale-lookup overhead but remains within AVX2 throughput — the bottleneck is disk I/O, not matmul.
|
||||
|
||||
### Backward compatibility
|
||||
|
||||
- Old per-row models (fmt=2) **work unchanged** — format auto-detected from scale array size
|
||||
- All existing features (PILOT, EXPERT_BUDGET, CUDA, MTP, PIPE) work identically — they don't touch the dequant path
|
||||
- The fused gate+up pair path (`matmul_i4_pair`) falls back to separate `matmul_qt` calls for fmt=4 — minor perf cost, correctness preserved
|
||||
- `--group-size 0` produces per-row output (backward compat for the converter)
|
||||
|
||||
## How to reproduce
|
||||
|
||||
### Convert with group scales
|
||||
|
||||
```bash
|
||||
python tools/convert_fp8_to_int4.py \
|
||||
--indir /path/to/GLM-5.2-FP8 \
|
||||
--outdir /path/to/glm52_i4_grouped \
|
||||
--ebits 4 --io-bits 8 --group-size 128
|
||||
```
|
||||
|
||||
### Test quality
|
||||
|
||||
```bash
|
||||
# Old model (per-row):
|
||||
SNAP=/path/to/glm52_i4 PROMPT="Explain recursion in programming." NGEN=32 ./glm 64
|
||||
|
||||
# New model (grouped):
|
||||
SNAP=/path/to/glm52_i4_grouped PROMPT="Explain recursion in programming." NGEN=32 ./glm 64
|
||||
```
|
||||
|
||||
Compare output text coherence.
|
||||
|
||||
## What this won't fix
|
||||
|
||||
- **Cold cache speed** — still 0.18-0.36 tok/s on 24 GB RAM. Quality and speed are independent axes.
|
||||
- **All quantization error** — int4 is still 4-bit. Some degradation vs FP8 will remain. But it should be coherent degradation (slightly wrong word choices) rather than total reasoning failure (unrelated topics).
|
||||
- **The IDOT +12% perplexity** — the approximate activation kernel (`IDOT=1`, default ON) still applies to non-grouped tensors (attention projections are already protected). Grouped int4 uses exact kernels by design.
|
||||
|
||||
## Prior art
|
||||
|
||||
- **GLM-5 paper** (arXiv 2602.15763, §2.4.3): "We apply INT4 QAT in the SFT stage" — the model is trained for int4, but assumes fine-grained quantization.
|
||||
- **MxMoE** (ICML 2025): MoE experts exhibit divergent quantization sensitivity; uniform quant across all experts is suboptimal.
|
||||
- **MoEQuant** (OpenReview): Naive int4 on MoE loses meaningful accuracy; calibrated framework needed.
|
||||
- **Automated Fine-Grained MoE Quantization** (ACL 2025): Layer/expert-wise sensitivity variation; group-size scaling is the baseline improvement.
|
||||
- **ubergarm/GLM-5.1-GGUF**: imatrix-calibrated quants with expert-specific patches — explicitly patches `quantize_row_q4_0_ref()` for routed experts.
|
||||
- **llama.cpp Q4_K**: Uses block-level group scales (32 or 64 elements) as the standard int4 format.
|
||||
|
||||
## Conversion status
|
||||
|
||||
Re-converting from the FP8 source (`zai-org/GLM-5.2-FP8`) with `--group-size 128`. Downloading via ModelScope to avoid HuggingFace per-stream throttling. Will report quality A/B results once the conversion is complete.
|
||||
Reference in New Issue
Block a user