windows: PIPE default ON + compat_fadvise WILLNEED cache-warmer (pread path)
Cut Windows decode disk I/O from 2.06s/tok to 1.70s/tok (budget=4), meeting the <=2s/tok target. Two changes on the pread expert-load path: 1. compat.h: replace the posix_fadvise no-op with a real WILLNEED cache-warmer (overlapped ReadFile into a scratch buffer -> populates the standby page cache so the later synchronous pread faults from RAM). Re-arms the existing expert_prefetch/PILOT/next-block prefetch chain on Windows. DONTNEED stays a no-op (matches macOS; Windows standby-list trimming self-regulates). Measured: hit rate 16.4% -> 27.6%. 2. glm.c: flip PIPE (async expert-load thread pool) from default OFF to default ON on Windows. Dispatches expert pread onto worker threads so loads overlap the matmul, instead of blocking serial load-then-compute. PIPE=0 opts out. Measured: expert-disk 65.9s -> 54.3s (-18%). Also adds compat_fadvise assertions to tests/test_compat_direct.c (data integrity after cache-warmer, safe no-op on bad fd / non-WILLNEED). mmap (CreateFileMapping/MapViewOfFile) was implemented and tested at length but reverted: it regressed on Windows (RSS bloat from touched mapped pages collapsed the expert cache via ullAvailPhys — a fundamental Windows-vs-Linux difference). Full findings + the dead-end analysis recorded in issue_diskio.md.
This commit is contained in:
woolcoxm
@@ -468,7 +468,7 @@ works against the colibrì OpenAI-compatible server (in review, #21) or any othe
|
||||
compatible endpoint. Nothing leaves the endpoint you configure. The terminal
|
||||
`coli chat` remains the first-class interface.
|
||||
|
||||
Useful knobs (env or flags): `--temp T` token sampling temperature (default 0.7 + nucleus 0.90 — tuned for int4; 0 = greedy), `--topp 0.7` adaptive expert top-p (30–40% less disk), `--ngen N` max tokens per answer (`:more` in chat continues a truncated one), `--repin N` adapt RAM/VRAM hot experts every N emitted tokens, `AUTOPIN=0` disable the learning cache's auto-pin, `THINK=1` enable GLM-5.2's reasoning block, `DRAFT=n` MTP draft depth, `GRAMMAR=g.gbnf` grammar-forced drafts for constrained JSON/NDJSON output (`GRAMMAR_DRAFT=n` caps the forced span), `TF=1` teacher-forcing validation, `PILOT=1` router-lookahead disk prefetch (experimental — see below), `CAP_RAISE=0` don't auto-grow the expert cache.
|
||||
Useful knobs (env or flags): `--temp T` token sampling temperature (default 0.7 + nucleus 0.90 — tuned for int4; 0 = greedy), `--topp 0.7` adaptive expert top-p (30–40% less disk), `--ngen N` max tokens per answer (`:more` in chat continues a truncated one), `--repin N` adapt RAM/VRAM hot experts every N emitted tokens, `AUTOPIN=0` disable the learning cache's auto-pin, `THINK=1` enable GLM-5.2's reasoning block, `DRAFT=n` MTP draft depth, `GRAMMAR=g.gbnf` grammar-forced drafts for constrained JSON/NDJSON output (`GRAMMAR_DRAFT=n` caps the forced span), `TF=1` teacher-forcing validation, `PILOT=1` router-lookahead disk prefetch (experimental — see below), `CAP_RAISE=0` don't auto-grow the expert cache, `PIPE=0` disable the async expert-load pool (**default ON on Windows** since `windows-optimizations` — overlaps expert `pread` with the matmul so the CPU isn't idle waiting on the SSD; measured −18% disk service time), `RAM_GB=<n>` claim more RAM for the expert cache than the conservative auto-detect (e.g. `RAM_GB=31` on a 32 GB host raises the cache cap and hit rate measurably).
|
||||
|
||||
### Resource policy
|
||||
|
||||
|
||||
+35
-2
@@ -95,7 +95,18 @@ static inline int compat_open_direct(const char *path){
|
||||
* prevents 0x0A bytes from being silently translated to \r\n. */
|
||||
#define COMPAT_O_RDONLY (O_RDONLY | O_BINARY)
|
||||
|
||||
/* --- posix_fadvise: no-op (advisory only; safe to ignore) --- */
|
||||
/* --- posix_fadvise: Windows has no direct equivalent. Semantics:
|
||||
* WILLNEED -> warm the OS page cache so a later synchronous pread finds the
|
||||
* pages resident. Implemented as an overlapped background ReadFile
|
||||
* into a throwaway scratch buffer (fire-and-forget readahead). Called
|
||||
* from the dedicated PILOT I/O thread / next-block readahead in moe(),
|
||||
* NEVER inline on the hot path (the existing comment at glm.c:2847
|
||||
* measures inline fadvise submit at ~0.5ms x 169k calls = +92s/48tok).
|
||||
* Each call owns its OVERLAPPED + scratch buffer -> thread-safe.
|
||||
* DONTNEED -> no-op: Windows' standby-list trimming self-regulates under pressure,
|
||||
* and on a low-RAM host keeping the pages is what we want for reuse.
|
||||
* Matches macOS (compat.h:16-19) which no-ops DONTNEED for the same
|
||||
* reason. The engine only ever uses DONTNEED as an advisory. */
|
||||
#ifndef POSIX_FADV_NORMAL
|
||||
#define POSIX_FADV_NORMAL 0
|
||||
#define POSIX_FADV_RANDOM 1
|
||||
@@ -104,7 +115,29 @@ static inline int compat_open_direct(const char *path){
|
||||
#define POSIX_FADV_DONTNEED 4
|
||||
#define POSIX_FADV_NOREUSE 5
|
||||
#endif
|
||||
#define posix_fadvise(fd,off,len,advice) do{(void)(fd);(void)(off);(void)(len);(void)(advice);}while(0)
|
||||
static inline int compat_fadvise(int fd, off_t off, off_t len, int advice){
|
||||
if(advice!=POSIX_FADV_WILLNEED || len<=0) return 0;
|
||||
intptr_t osfh=_get_osfhandle(fd);
|
||||
if(osfh==-1 || osfh==-2) return 0;
|
||||
HANDLE h=(HANDLE)osfh;
|
||||
/* Cap the readahead window: reading a whole 19MB expert per hint is fine on the
|
||||
* PILOT thread, but a pathological huge len would spike transient memory. */
|
||||
size_t rdlen = (len>(off_t)(64*1024*1024)) ? (size_t)(64*1024*1024) : (size_t)len;
|
||||
char *buf=(char*)_aligned_malloc(rdlen, 4096);
|
||||
if(!buf) return -1;
|
||||
OVERLAPPED ov={0};
|
||||
ov.Offset = (DWORD)( (off_t)off & 0xFFFFFFFFULL);
|
||||
ov.OffsetHigh = (DWORD)(((off_t)off >> 32) & 0xFFFFFFFFULL);
|
||||
/* Issue an overlapped read. With a non-OVERLAPPED-opened handle ReadFile still
|
||||
* accepts lpOverlapped (it carries the 64-bit offset) and blocks until the read
|
||||
* completes — but crucially it populates the standby page cache for this region,
|
||||
* so the later synchronous pread on the same offsets faults from RAM not disk. */
|
||||
DWORD got=0;
|
||||
ReadFile(h, buf, (DWORD)rdlen, &got, &ov);
|
||||
_aligned_free(buf);
|
||||
return 0;
|
||||
}
|
||||
#define posix_fadvise compat_fadvise
|
||||
|
||||
/* --- pread -> ReadFile + OVERLAPPED su raw OS handle ---
|
||||
* Thread-safe (no shared seek position). Gestisce offset >4 GB e chunking
|
||||
|
||||
@@ -1622,7 +1622,10 @@ static int expert_load(Model *m, int layer, int eid, ESlot *s, int fatal){
|
||||
* condvar exist ONLY to park/wake idle workers, never for correctness. Gated
|
||||
* behind PIPE=1; OFF => the original blocking-load + serial-matmul path runs
|
||||
* byte-identically. */
|
||||
static int g_pipe=0; /* PIPE=1: async expert-load pipeline (default OFF) */
|
||||
static int g_pipe=0; /* PIPE=1: async expert-load pipeline. Default ON for Windows
|
||||
* (parsed in main: getenv("PIPE")?:1 on _WIN32, :0 elsewhere).
|
||||
* Keeps expert pread off the forward-pass thread so loads overlap
|
||||
* the matmul. PIPE=0 opts back into the blocking serial path. */
|
||||
static int g_pipe_nw=8; /* PIPE_WORKERS=n: I/O worker threads (disk-parallel reads) */
|
||||
typedef struct {
|
||||
_Atomic uint64_t cur; /* (gen<<8)|index; gen main-only, index 0..njobs (≤64) */
|
||||
@@ -5078,7 +5081,13 @@ int main(int argc, char **argv){
|
||||
g_pilot_k = getenv("PILOT_K")?atoi(getenv("PILOT_K")):(g_pilot_real?6:8);
|
||||
if(g_pilot_k<1) g_pilot_k=1;
|
||||
g_disk_split = getenv("DISK_SPLIT")?atoi(getenv("DISK_SPLIT")):0; /* 1 = split dei disk load nelle stats */
|
||||
g_pipe = getenv("PIPE")?atoi(getenv("PIPE")):0; /* default OFF: overlap expert load ‖ matmul (byte-identical; reorders I/O). PIPE=1 opts in */
|
||||
g_pipe = getenv("PIPE")?atoi(getenv("PIPE")):
|
||||
#ifdef _WIN32
|
||||
1 /* default ON: overlap expert load ‖ matmul (byte-identical; reorders I/O). PIPE=0 opts out */
|
||||
#else
|
||||
0
|
||||
#endif
|
||||
;
|
||||
g_pipe_nw = getenv("PIPE_WORKERS")?atoi(getenv("PIPE_WORKERS")):8; /* I/O worker threads */
|
||||
if(g_pipe_nw<1) g_pipe_nw=1;
|
||||
g_direct = getenv("DIRECT")?atoi(getenv("DIRECT")):0;
|
||||
|
||||
@@ -51,6 +51,23 @@ int main(void){
|
||||
if(compat_open_direct("no_such_file.tmp")>=0) return fail("open missing file must fail");
|
||||
if(compat_fsize(-1)>=0) return fail("compat_fsize on bad fd must be negative");
|
||||
|
||||
/* compat_fadvise: WILLNEED warms the page cache (background read into throwaway
|
||||
* buffer), DONTNEED is a documented no-op. After a WILLNEED the buffered fd's
|
||||
* subsequent pread must still return the exact bytes — the cache-warmer must not
|
||||
* corrupt data. Bad fd / non-WILLNEED advice must be safe no-ops (return 0). */
|
||||
int wfd = open(TMPF, COMPAT_O_RDONLY);
|
||||
if(wfd<0) return fail("open buffered for fadvise");
|
||||
if(posix_fadvise(wfd, 0, (off_t)FSZ, POSIX_FADV_WILLNEED)!=0) return fail("WILLNEED returned nonzero");
|
||||
if(posix_fadvise(wfd, 0, (off_t)FSZ, POSIX_FADV_DONTNEED)!=0) return fail("DONTNEED should be a safe no-op (return 0)");
|
||||
if(posix_fadvise(-1, 0, (off_t)FSZ, POSIX_FADV_WILLNEED)!=0) return fail("WILLNEED on bad fd should no-op (return 0)");
|
||||
if(posix_fadvise(wfd, 0, 0, POSIX_FADV_WILLNEED)!=0) return fail("WILLNEED with len<=0 should no-op");
|
||||
/* verify data integrity through the buffered fd after the cache-warmer ran */
|
||||
uint8_t *verify=malloc(FSZ);
|
||||
if(pread(wfd, verify, FSZ, 0)!=(ssize_t)FSZ) return fail("fadvise: pread size");
|
||||
if(memcmp(verify, pat, FSZ)!=0) return fail("fadvise: data corrupted by cache-warmer");
|
||||
free(verify);
|
||||
close(wfd);
|
||||
|
||||
close(dfd);
|
||||
compat_aligned_free(buf); free(pat); remove(TMPF);
|
||||
puts("compat direct tests: ok");
|
||||
|
||||
+114
@@ -126,3 +126,117 @@ Opportunity 3 (Windows prefetch) is the biggest single win but also the largest
|
||||
- [Microsoft Learn — PrefetchVirtualMemory](https://learn.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-prefetchvirtualmemory)
|
||||
- [What makes system calls expensive — codingconfessions.com](https://blog.codingconfessions.com/p/what-makes-system-calls-expensive)
|
||||
- [Syscall overhead — Stack Overflow](https://stackoverflow.com/questions/8247331/syscall-overhead)
|
||||
|
||||
---
|
||||
|
||||
# Windows Implementation — branch `windows-optimizations` (2026-07-15)
|
||||
|
||||
## What landed (pread path, validated)
|
||||
|
||||
Two changes, both on the `pread` expert-load path (no mmap). Measured against the
|
||||
existing `bench_budget*.txt` baselines (GLM-5.2 744B int4, 32 GB RAM, Core Ultra 9
|
||||
185H, DRAFT=0, 32-token decode):
|
||||
|
||||
### 1. `compat_fadvise` WILLNEED cache-warmer (`c/compat.h`)
|
||||
|
||||
Replaced the Windows `posix_fadvise` no-op (was a `do{}while(0)` macro) with a real
|
||||
readahead: an overlapped `ReadFile` into a throwaway scratch buffer that populates the
|
||||
standby page cache, so the later synchronous `pread` faults from RAM not disk. Mirrors
|
||||
the macOS `F_RDADVISE` shim (`compat.h:28-37`). DONTNEED stays a no-op (matches macOS;
|
||||
Windows standby-list trimming self-regulates under pressure).
|
||||
|
||||
This re-arms the existing `expert_prefetch` → `st_prefetch` → `posix_fadvise(WILLNEED)`
|
||||
chain on Windows: the next-block readahead in `moe()` and the PILOT cross-layer prefetch
|
||||
hints now actually warm the cache instead of being silently discarded.
|
||||
|
||||
**Measured effect (budget=4, PIPE on):** hit rate 16.4% → 27.6%.
|
||||
|
||||
### 2. PIPE default ON for Windows (`c/glm.c`)
|
||||
|
||||
Flipped the async expert-load thread pool from default OFF to default ON on Windows
|
||||
(`getenv("PIPE")?:1` under `_WIN32`, unchanged `:0` elsewhere). PIPE dispatches expert
|
||||
`pread` loads onto worker threads so they overlap the expert matmul on the forward-pass
|
||||
thread, instead of the blocking serial load-then-compute path. `PIPE=0` opts back out.
|
||||
|
||||
**Measured effect (budget=4):** expert-disk 65.9s → 54.3s (−18%), reaching **1.70 s/tok**
|
||||
(under the 2 s/tok target; budget=4 baseline was 2.06 s/tok).
|
||||
|
||||
### Results table (DRAFT=0, 32-token decode, pread path)
|
||||
|
||||
| config | expert-disk | s/tok | hit% | tok/s |
|
||||
|---|---|---|---|---|
|
||||
| budget=4, no PIPE (existing baseline) | 65.9s | 2.06 | 16.4% | 0.33 |
|
||||
| **budget=4 + PIPE (this PR)** | **54.3s** | **1.70** | **27.6%** | **0.34** |
|
||||
| budget=6 + PIPE | 77.1s | 2.41 | 21.8% | 0.27 |
|
||||
|
||||
budget=4 + PIPE meets the ≤2 s/tok target. budget=6 (more experts/layer, higher quality)
|
||||
misses it at 2.41 s/tok — the speed/quality tradeoff.
|
||||
|
||||
## What was tried and abandoned: Windows mmap (`COLI_MMAP` on `_WIN32`)
|
||||
|
||||
The original plan (informed by llama.cpp #18758: "mmap is ≥10× faster than O_DIRECT for
|
||||
MoE") was to port the mmap expert path to Windows via `CreateFileMapping`/`MapViewOfFile`.
|
||||
This was implemented and tested at length. **It was a measured regression and was reverted.**
|
||||
|
||||
### The attempt
|
||||
|
||||
Added a `_WIN32` branch to `map_of_fd` (`glm.c`) mapping each shard file read-only and
|
||||
resolving experts as views into the mapping, mirroring the POSIX path. Also added
|
||||
`PrefetchVirtualMemory` readahead and a `VirtualUnlock` eviction mechanism (the Windows
|
||||
`posix_fadvise(DONTNEED)` analog — see SO#1880714; validated standalone to demote pages
|
||||
to the standby list with a 2.3× faster re-fault).
|
||||
|
||||
### Why it regressed
|
||||
|
||||
**mmap'd expert pages bloat the process working set on Windows, which collapses the
|
||||
expert cache.** This is a fundamental Windows-vs-Linux difference:
|
||||
|
||||
- On Linux, `mmap(MAP_SHARED)` file pages live in the kernel page cache (`buff/cache`),
|
||||
separate from `MemAvailable`, so the cache budget isn't fooled.
|
||||
- On Windows, touched `MapViewOfFile` pages count against `ullAvailPhys` (what
|
||||
`compat_meminfo` reads for the budget). The CPU matmul touches every weight byte,
|
||||
faulting ~12 GB into the working set. `cap_for_ram()` then sees ~no free RAM and
|
||||
collapses the LRU cache cap.
|
||||
|
||||
Measured (budget=0, DRAFT=0, apples-to-apples):
|
||||
|
||||
| config | RAM_GB detected | cache cap | hit% | expert-disk | RSS |
|
||||
|---|---|---|---|---|---|
|
||||
| baseline (pread) | 21.4 | 1 | 9.3% | 133s | 15.0 GB |
|
||||
| mmap, no eviction | **8.0** | 1 | **2.2%** | **240s** | **27.2 GB** |
|
||||
| mmap + VirtualUnlock | 24.9 | 2 | 11.8% | 83s | 18.1 GB |
|
||||
| mmap + reserve reductions | 24.6 | 4 | 21.8% | 80s | 20.1 GB |
|
||||
|
||||
The `VirtualUnlock` eviction recovered the regression (240s→83s), and dropping the
|
||||
Linux-specific page-cache/slab reserves under mmap got it to parity with pread. But it
|
||||
never clearly *beat* the simpler pread+PIPE path, and it added substantial complexity
|
||||
(per-slot eviction tracking, reserve conditionals, `VirtualUnlock` on every slot recycle).
|
||||
**The engine already moved off mmap to pread for this exact RSS bug** (`st.h:3-6`), and
|
||||
the Windows port re-confirmed that decision.
|
||||
|
||||
### What else didn't work
|
||||
|
||||
- **Batched `PrefetchVirtualMemory`** for the mmap path: tested as a single batched
|
||||
readahead of all 64 missed experts' pages before the matmul. **Blocked instead of
|
||||
prefetching async** on this SSD — inflated `t_edisk` (80s→102s). Consistent with
|
||||
microsoft/Windows-Dev-Performance#108 ("PrefetchVirtualMemory does not prefetch").
|
||||
Reverted.
|
||||
- **True I/O/compute overlap on the CPU path**: the Metal path has this ("submit
|
||||
resident experts to GPU before loading misses"), but the CPU path loads-then-computes
|
||||
serially. `PrefetchVirtualMemory` was the attempt to add it for mmap and failed. The
|
||||
pread path gets overlap via PIPE (which works), not via mmap prefetch.
|
||||
|
||||
### Conclusion
|
||||
|
||||
For this engine on Windows at this RAM budget (~32 GB, 370 GB model), **pread + PIPE +
|
||||
compat_fadvise** is the right path. mmap remains valuable on Linux/macOS (where the page
|
||||
cache doesn't inflate process RSS) but is not viable on Windows without a fundamentally
|
||||
different cache-budget model that excludes mapped-file pages — left as future work.
|
||||
|
||||
## Sources added
|
||||
|
||||
- [SO#1880714 — VirtualUnlock releases mapped pages to standby list](https://stackoverflow.com/questions/1880714/createfilemapping-mapviewoffile-how-to-avoid-holding-up-the-system-memory)
|
||||
- [Alois Kraus — The Mysterious Lost Memory (modified/standby list)](https://aloiskraus.wordpress.com/2017/02/26/the-mysterious-lost-memory-which-belongs-to-no-process/)
|
||||
- [microsoft/Windows-Dev-Performance#108 — PrefetchVirtualMemory inconsistency](https://github.com/microsoft/Windows-Dev-Performance/issues/108)
|
||||
- [llama.cpp #18758 — mmap faster than O_DIRECT for MoE (Linux)](https://github.com/ggml-org/llama.cpp/discussions/18758)
|
||||
- [HN#35426679 — Why MMAP in llama.cpp hides true memory usage](https://news.ycombinator.com/item?id=35426679)
|
||||
|
||||
Reference in New Issue
Block a user