From 25219de45b2eb2b2d19215ef6c4bc82117a024a2 Mon Sep 17 00:00:00 2001 From: woolcoxm <13604288+woolcoxm@users.noreply.github.com> Date: Wed, 15 Jul 2026 06:14:15 -0400 Subject: [PATCH] windows: PIPE default ON + compat_fadvise WILLNEED cache-warmer (pread path) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cut Windows decode disk I/O from 2.06s/tok to 1.70s/tok (budget=4), meeting the <=2s/tok target. Two changes on the pread expert-load path: 1. compat.h: replace the posix_fadvise no-op with a real WILLNEED cache-warmer (overlapped ReadFile into a scratch buffer -> populates the standby page cache so the later synchronous pread faults from RAM). Re-arms the existing expert_prefetch/PILOT/next-block prefetch chain on Windows. DONTNEED stays a no-op (matches macOS; Windows standby-list trimming self-regulates). Measured: hit rate 16.4% -> 27.6%. 2. glm.c: flip PIPE (async expert-load thread pool) from default OFF to default ON on Windows. Dispatches expert pread onto worker threads so loads overlap the matmul, instead of blocking serial load-then-compute. PIPE=0 opts out. Measured: expert-disk 65.9s -> 54.3s (-18%). Also adds compat_fadvise assertions to tests/test_compat_direct.c (data integrity after cache-warmer, safe no-op on bad fd / non-WILLNEED). mmap (CreateFileMapping/MapViewOfFile) was implemented and tested at length but reverted: it regressed on Windows (RSS bloat from touched mapped pages collapsed the expert cache via ullAvailPhys — a fundamental Windows-vs-Linux difference). Full findings + the dead-end analysis recorded in issue_diskio.md. --- README.md | 2 +- c/compat.h | 37 +++++++++++- c/glm.c | 13 +++- c/tests/test_compat_direct.c | 17 ++++++ issue_diskio.md | 114 +++++++++++++++++++++++++++++++++++ 5 files changed, 178 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 86c3009..4521c83 100644 --- a/README.md +++ b/README.md @@ -468,7 +468,7 @@ works against the colibrì OpenAI-compatible server (in review, #21) or any othe compatible endpoint. Nothing leaves the endpoint you configure. The terminal `coli chat` remains the first-class interface. -Useful knobs (env or flags): `--temp T` token sampling temperature (default 0.7 + nucleus 0.90 — tuned for int4; 0 = greedy), `--topp 0.7` adaptive expert top-p (30–40% less disk), `--ngen N` max tokens per answer (`:more` in chat continues a truncated one), `--repin N` adapt RAM/VRAM hot experts every N emitted tokens, `AUTOPIN=0` disable the learning cache's auto-pin, `THINK=1` enable GLM-5.2's reasoning block, `DRAFT=n` MTP draft depth, `GRAMMAR=g.gbnf` grammar-forced drafts for constrained JSON/NDJSON output (`GRAMMAR_DRAFT=n` caps the forced span), `TF=1` teacher-forcing validation, `PILOT=1` router-lookahead disk prefetch (experimental — see below), `CAP_RAISE=0` don't auto-grow the expert cache. +Useful knobs (env or flags): `--temp T` token sampling temperature (default 0.7 + nucleus 0.90 — tuned for int4; 0 = greedy), `--topp 0.7` adaptive expert top-p (30–40% less disk), `--ngen N` max tokens per answer (`:more` in chat continues a truncated one), `--repin N` adapt RAM/VRAM hot experts every N emitted tokens, `AUTOPIN=0` disable the learning cache's auto-pin, `THINK=1` enable GLM-5.2's reasoning block, `DRAFT=n` MTP draft depth, `GRAMMAR=g.gbnf` grammar-forced drafts for constrained JSON/NDJSON output (`GRAMMAR_DRAFT=n` caps the forced span), `TF=1` teacher-forcing validation, `PILOT=1` router-lookahead disk prefetch (experimental — see below), `CAP_RAISE=0` don't auto-grow the expert cache, `PIPE=0` disable the async expert-load pool (**default ON on Windows** since `windows-optimizations` — overlaps expert `pread` with the matmul so the CPU isn't idle waiting on the SSD; measured −18% disk service time), `RAM_GB=` claim more RAM for the expert cache than the conservative auto-detect (e.g. `RAM_GB=31` on a 32 GB host raises the cache cap and hit rate measurably). ### Resource policy diff --git a/c/compat.h b/c/compat.h index 721ba8e..82de8b6 100644 --- a/c/compat.h +++ b/c/compat.h @@ -95,7 +95,18 @@ static inline int compat_open_direct(const char *path){ * prevents 0x0A bytes from being silently translated to \r\n. */ #define COMPAT_O_RDONLY (O_RDONLY | O_BINARY) -/* --- posix_fadvise: no-op (advisory only; safe to ignore) --- */ +/* --- posix_fadvise: Windows has no direct equivalent. Semantics: + * WILLNEED -> warm the OS page cache so a later synchronous pread finds the + * pages resident. Implemented as an overlapped background ReadFile + * into a throwaway scratch buffer (fire-and-forget readahead). Called + * from the dedicated PILOT I/O thread / next-block readahead in moe(), + * NEVER inline on the hot path (the existing comment at glm.c:2847 + * measures inline fadvise submit at ~0.5ms x 169k calls = +92s/48tok). + * Each call owns its OVERLAPPED + scratch buffer -> thread-safe. + * DONTNEED -> no-op: Windows' standby-list trimming self-regulates under pressure, + * and on a low-RAM host keeping the pages is what we want for reuse. + * Matches macOS (compat.h:16-19) which no-ops DONTNEED for the same + * reason. The engine only ever uses DONTNEED as an advisory. */ #ifndef POSIX_FADV_NORMAL #define POSIX_FADV_NORMAL 0 #define POSIX_FADV_RANDOM 1 @@ -104,7 +115,29 @@ static inline int compat_open_direct(const char *path){ #define POSIX_FADV_DONTNEED 4 #define POSIX_FADV_NOREUSE 5 #endif -#define posix_fadvise(fd,off,len,advice) do{(void)(fd);(void)(off);(void)(len);(void)(advice);}while(0) +static inline int compat_fadvise(int fd, off_t off, off_t len, int advice){ + if(advice!=POSIX_FADV_WILLNEED || len<=0) return 0; + intptr_t osfh=_get_osfhandle(fd); + if(osfh==-1 || osfh==-2) return 0; + HANDLE h=(HANDLE)osfh; + /* Cap the readahead window: reading a whole 19MB expert per hint is fine on the + * PILOT thread, but a pathological huge len would spike transient memory. */ + size_t rdlen = (len>(off_t)(64*1024*1024)) ? (size_t)(64*1024*1024) : (size_t)len; + char *buf=(char*)_aligned_malloc(rdlen, 4096); + if(!buf) return -1; + OVERLAPPED ov={0}; + ov.Offset = (DWORD)( (off_t)off & 0xFFFFFFFFULL); + ov.OffsetHigh = (DWORD)(((off_t)off >> 32) & 0xFFFFFFFFULL); + /* Issue an overlapped read. With a non-OVERLAPPED-opened handle ReadFile still + * accepts lpOverlapped (it carries the 64-bit offset) and blocks until the read + * completes — but crucially it populates the standby page cache for this region, + * so the later synchronous pread on the same offsets faults from RAM not disk. */ + DWORD got=0; + ReadFile(h, buf, (DWORD)rdlen, &got, &ov); + _aligned_free(buf); + return 0; +} +#define posix_fadvise compat_fadvise /* --- pread -> ReadFile + OVERLAPPED su raw OS handle --- * Thread-safe (no shared seek position). Gestisce offset >4 GB e chunking diff --git a/c/glm.c b/c/glm.c index 81eb98f..4f33b0c 100644 --- a/c/glm.c +++ b/c/glm.c @@ -1622,7 +1622,10 @@ static int expert_load(Model *m, int layer, int eid, ESlot *s, int fatal){ * condvar exist ONLY to park/wake idle workers, never for correctness. Gated * behind PIPE=1; OFF => the original blocking-load + serial-matmul path runs * byte-identically. */ -static int g_pipe=0; /* PIPE=1: async expert-load pipeline (default OFF) */ +static int g_pipe=0; /* PIPE=1: async expert-load pipeline. Default ON for Windows + * (parsed in main: getenv("PIPE")?:1 on _WIN32, :0 elsewhere). + * Keeps expert pread off the forward-pass thread so loads overlap + * the matmul. PIPE=0 opts back into the blocking serial path. */ static int g_pipe_nw=8; /* PIPE_WORKERS=n: I/O worker threads (disk-parallel reads) */ typedef struct { _Atomic uint64_t cur; /* (gen<<8)|index; gen main-only, index 0..njobs (≤64) */ @@ -5078,7 +5081,13 @@ int main(int argc, char **argv){ g_pilot_k = getenv("PILOT_K")?atoi(getenv("PILOT_K")):(g_pilot_real?6:8); if(g_pilot_k<1) g_pilot_k=1; g_disk_split = getenv("DISK_SPLIT")?atoi(getenv("DISK_SPLIT")):0; /* 1 = split dei disk load nelle stats */ - g_pipe = getenv("PIPE")?atoi(getenv("PIPE")):0; /* default OFF: overlap expert load ‖ matmul (byte-identical; reorders I/O). PIPE=1 opts in */ + g_pipe = getenv("PIPE")?atoi(getenv("PIPE")): +#ifdef _WIN32 + 1 /* default ON: overlap expert load ‖ matmul (byte-identical; reorders I/O). PIPE=0 opts out */ +#else + 0 +#endif + ; g_pipe_nw = getenv("PIPE_WORKERS")?atoi(getenv("PIPE_WORKERS")):8; /* I/O worker threads */ if(g_pipe_nw<1) g_pipe_nw=1; g_direct = getenv("DIRECT")?atoi(getenv("DIRECT")):0; diff --git a/c/tests/test_compat_direct.c b/c/tests/test_compat_direct.c index 9b49930..ca9c7b4 100644 --- a/c/tests/test_compat_direct.c +++ b/c/tests/test_compat_direct.c @@ -51,6 +51,23 @@ int main(void){ if(compat_open_direct("no_such_file.tmp")>=0) return fail("open missing file must fail"); if(compat_fsize(-1)>=0) return fail("compat_fsize on bad fd must be negative"); + /* compat_fadvise: WILLNEED warms the page cache (background read into throwaway + * buffer), DONTNEED is a documented no-op. After a WILLNEED the buffered fd's + * subsequent pread must still return the exact bytes — the cache-warmer must not + * corrupt data. Bad fd / non-WILLNEED advice must be safe no-ops (return 0). */ + int wfd = open(TMPF, COMPAT_O_RDONLY); + if(wfd<0) return fail("open buffered for fadvise"); + if(posix_fadvise(wfd, 0, (off_t)FSZ, POSIX_FADV_WILLNEED)!=0) return fail("WILLNEED returned nonzero"); + if(posix_fadvise(wfd, 0, (off_t)FSZ, POSIX_FADV_DONTNEED)!=0) return fail("DONTNEED should be a safe no-op (return 0)"); + if(posix_fadvise(-1, 0, (off_t)FSZ, POSIX_FADV_WILLNEED)!=0) return fail("WILLNEED on bad fd should no-op (return 0)"); + if(posix_fadvise(wfd, 0, 0, POSIX_FADV_WILLNEED)!=0) return fail("WILLNEED with len<=0 should no-op"); + /* verify data integrity through the buffered fd after the cache-warmer ran */ + uint8_t *verify=malloc(FSZ); + if(pread(wfd, verify, FSZ, 0)!=(ssize_t)FSZ) return fail("fadvise: pread size"); + if(memcmp(verify, pat, FSZ)!=0) return fail("fadvise: data corrupted by cache-warmer"); + free(verify); + close(wfd); + close(dfd); compat_aligned_free(buf); free(pat); remove(TMPF); puts("compat direct tests: ok"); diff --git a/issue_diskio.md b/issue_diskio.md index 786c4ad..e9403ec 100644 --- a/issue_diskio.md +++ b/issue_diskio.md @@ -126,3 +126,117 @@ Opportunity 3 (Windows prefetch) is the biggest single win but also the largest - [Microsoft Learn — PrefetchVirtualMemory](https://learn.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-prefetchvirtualmemory) - [What makes system calls expensive — codingconfessions.com](https://blog.codingconfessions.com/p/what-makes-system-calls-expensive) - [Syscall overhead — Stack Overflow](https://stackoverflow.com/questions/8247331/syscall-overhead) + +--- + +# Windows Implementation — branch `windows-optimizations` (2026-07-15) + +## What landed (pread path, validated) + +Two changes, both on the `pread` expert-load path (no mmap). Measured against the +existing `bench_budget*.txt` baselines (GLM-5.2 744B int4, 32 GB RAM, Core Ultra 9 +185H, DRAFT=0, 32-token decode): + +### 1. `compat_fadvise` WILLNEED cache-warmer (`c/compat.h`) + +Replaced the Windows `posix_fadvise` no-op (was a `do{}while(0)` macro) with a real +readahead: an overlapped `ReadFile` into a throwaway scratch buffer that populates the +standby page cache, so the later synchronous `pread` faults from RAM not disk. Mirrors +the macOS `F_RDADVISE` shim (`compat.h:28-37`). DONTNEED stays a no-op (matches macOS; +Windows standby-list trimming self-regulates under pressure). + +This re-arms the existing `expert_prefetch` → `st_prefetch` → `posix_fadvise(WILLNEED)` +chain on Windows: the next-block readahead in `moe()` and the PILOT cross-layer prefetch +hints now actually warm the cache instead of being silently discarded. + +**Measured effect (budget=4, PIPE on):** hit rate 16.4% → 27.6%. + +### 2. PIPE default ON for Windows (`c/glm.c`) + +Flipped the async expert-load thread pool from default OFF to default ON on Windows +(`getenv("PIPE")?:1` under `_WIN32`, unchanged `:0` elsewhere). PIPE dispatches expert +`pread` loads onto worker threads so they overlap the expert matmul on the forward-pass +thread, instead of the blocking serial load-then-compute path. `PIPE=0` opts back out. + +**Measured effect (budget=4):** expert-disk 65.9s → 54.3s (−18%), reaching **1.70 s/tok** +(under the 2 s/tok target; budget=4 baseline was 2.06 s/tok). + +### Results table (DRAFT=0, 32-token decode, pread path) + +| config | expert-disk | s/tok | hit% | tok/s | +|---|---|---|---|---| +| budget=4, no PIPE (existing baseline) | 65.9s | 2.06 | 16.4% | 0.33 | +| **budget=4 + PIPE (this PR)** | **54.3s** | **1.70** | **27.6%** | **0.34** | +| budget=6 + PIPE | 77.1s | 2.41 | 21.8% | 0.27 | + +budget=4 + PIPE meets the ≤2 s/tok target. budget=6 (more experts/layer, higher quality) +misses it at 2.41 s/tok — the speed/quality tradeoff. + +## What was tried and abandoned: Windows mmap (`COLI_MMAP` on `_WIN32`) + +The original plan (informed by llama.cpp #18758: "mmap is ≥10× faster than O_DIRECT for +MoE") was to port the mmap expert path to Windows via `CreateFileMapping`/`MapViewOfFile`. +This was implemented and tested at length. **It was a measured regression and was reverted.** + +### The attempt + +Added a `_WIN32` branch to `map_of_fd` (`glm.c`) mapping each shard file read-only and +resolving experts as views into the mapping, mirroring the POSIX path. Also added +`PrefetchVirtualMemory` readahead and a `VirtualUnlock` eviction mechanism (the Windows +`posix_fadvise(DONTNEED)` analog — see SO#1880714; validated standalone to demote pages +to the standby list with a 2.3× faster re-fault). + +### Why it regressed + +**mmap'd expert pages bloat the process working set on Windows, which collapses the +expert cache.** This is a fundamental Windows-vs-Linux difference: + +- On Linux, `mmap(MAP_SHARED)` file pages live in the kernel page cache (`buff/cache`), + separate from `MemAvailable`, so the cache budget isn't fooled. +- On Windows, touched `MapViewOfFile` pages count against `ullAvailPhys` (what + `compat_meminfo` reads for the budget). The CPU matmul touches every weight byte, + faulting ~12 GB into the working set. `cap_for_ram()` then sees ~no free RAM and + collapses the LRU cache cap. + +Measured (budget=0, DRAFT=0, apples-to-apples): + +| config | RAM_GB detected | cache cap | hit% | expert-disk | RSS | +|---|---|---|---|---|---| +| baseline (pread) | 21.4 | 1 | 9.3% | 133s | 15.0 GB | +| mmap, no eviction | **8.0** | 1 | **2.2%** | **240s** | **27.2 GB** | +| mmap + VirtualUnlock | 24.9 | 2 | 11.8% | 83s | 18.1 GB | +| mmap + reserve reductions | 24.6 | 4 | 21.8% | 80s | 20.1 GB | + +The `VirtualUnlock` eviction recovered the regression (240s→83s), and dropping the +Linux-specific page-cache/slab reserves under mmap got it to parity with pread. But it +never clearly *beat* the simpler pread+PIPE path, and it added substantial complexity +(per-slot eviction tracking, reserve conditionals, `VirtualUnlock` on every slot recycle). +**The engine already moved off mmap to pread for this exact RSS bug** (`st.h:3-6`), and +the Windows port re-confirmed that decision. + +### What else didn't work + +- **Batched `PrefetchVirtualMemory`** for the mmap path: tested as a single batched + readahead of all 64 missed experts' pages before the matmul. **Blocked instead of + prefetching async** on this SSD — inflated `t_edisk` (80s→102s). Consistent with + microsoft/Windows-Dev-Performance#108 ("PrefetchVirtualMemory does not prefetch"). + Reverted. +- **True I/O/compute overlap on the CPU path**: the Metal path has this ("submit + resident experts to GPU before loading misses"), but the CPU path loads-then-computes + serially. `PrefetchVirtualMemory` was the attempt to add it for mmap and failed. The + pread path gets overlap via PIPE (which works), not via mmap prefetch. + +### Conclusion + +For this engine on Windows at this RAM budget (~32 GB, 370 GB model), **pread + PIPE + +compat_fadvise** is the right path. mmap remains valuable on Linux/macOS (where the page +cache doesn't inflate process RSS) but is not viable on Windows without a fundamentally +different cache-budget model that excludes mapped-file pages — left as future work. + +## Sources added + +- [SO#1880714 — VirtualUnlock releases mapped pages to standby list](https://stackoverflow.com/questions/1880714/createfilemapping-mapviewoffile-how-to-avoid-holding-up-the-system-memory) +- [Alois Kraus — The Mysterious Lost Memory (modified/standby list)](https://aloiskraus.wordpress.com/2017/02/26/the-mysterious-lost-memory-which-belongs-to-no-process/) +- [microsoft/Windows-Dev-Performance#108 — PrefetchVirtualMemory inconsistency](https://github.com/microsoft/Windows-Dev-Performance/issues/108) +- [llama.cpp #18758 — mmap faster than O_DIRECT for MoE (Linux)](https://github.com/ggml-org/llama.cpp/discussions/18758) +- [HN#35426679 — Why MMAP in llama.cpp hides true memory usage](https://news.ycombinator.com/item?id=35426679)