url/colibri
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Windows: direct I/O (FILE_FLAG_NO_BUFFERING, 1.47x decode) + VirtualLock pin wiring + select_ctx device cache (#162)

Browse Source 
* win: direct I/O via FILE_FLAG_NO_BUFFERING + compat_fsize + VirtualLock primitives

compat_open_direct() gives Windows the O_DIRECT twin fd st.h already uses
on Linux/macOS: FILE_FLAG_NO_BUFFERING, same 4K-alignment contract as
O_DIRECT (the engine's DIRECT=1 path already aligns offset/len and slabs
are posix_memalign'd).

Measured on GLM-5.2 744B int4, Ryzen 9 9950X3D / 126 GB / PCIe4 NVMe
(5.8 GB/s at the engine's 19MBx8T pattern), Windows 11, MinGW GCC 16.1,
32-token greedy runs at --topp 0.7, 40 GB pin, current dev HEAD:
  buffered:  0.38 tok/s (expert-disk dominates)
  DIRECT=1:  0.56 tok/s (1.47x) — byte-identical greedy output vs buffered

compat_fsize() (GetFileSizeEx): CRT lseek(SEEK_END) returns -1 on
NO_BUFFERING fds (measured on UCRT); iobench uses it and gains a
NO_BUFFERING branch so disk numbers are comparable across platforms.

compat_mlock/compat_munlock: VirtualLock with working-set growth (bare
VirtualLock caps at the default working-set minimum, a few hundred KB).
Wired into the engine in the next commit.

tests/test_compat_direct.c covers the alignment contract, data integrity,
fsize on both fd kinds; skips cleanly off Windows.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* win: wire VirtualLock into mem_wire, munlock pairing in expert_host_release

MLOCK=1 was a silent no-op on Windows: pinned experts could be paged out
by working-set trimming under memory pressure. mem_wire now uses
compat_mlock (VirtualLock + working-set growth); expert_host_release
unlocks before freeing, mirroring the POSIX branch.

Validated: 39.6 GB pin wired in 17s on a 126 GB machine, zero failures;
TF oracle 32/32 with MLOCK=1.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* cuda: thread-local current-device cache in select_ctx

cudaSetDevice on every call is expensive when the serial expert loop
alternates devices. Measured on RTX 5090 + RTX 4090 (Windows, DLL
backend, pre-#68 dispatch): expert-matmul 14.3s -> 25.4s per 32 tokens
going from 1 to 2 devices, entirely per-call context switching. The
current device is per-thread in the CUDA runtime, so a thread_local
cache skips redundant switches; multi-GPU expert serving becomes
positive-scaling instead of negative.

Kernel suite passes on sm_120 + sm_89; TF oracle 32/32 dual-GPU.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: olorin <io@zyphyr.co>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Tom Olorin
2026-07-14 07:12:15 -07:00
committed by GitHub
parent ec0aadf91a
commit d47b095875
7 changed files with 138 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files
c/glm.c
+5
View File
@@ -1457,6 +1457,9 @@ static void expert_host_release(Model *m, ESlot *s){
#if defined(__APPLE__) || defined(__linux__)
if(s->slab) munlock(s->slab,(size_t)s->slab_cap);
if(s->fslab) munlock(s->fslab,(size_t)s->fslab_cap*sizeof(float));
#elif defined(_WIN32)
if(s->slab) compat_munlock(s->slab,(size_t)s->slab_cap);
if(s->fslab) compat_munlock(s->fslab,(size_t)s->fslab_cap*sizeof(float));
#endif
int64_t bytes=qt_bytes(&s->g)+qt_bytes(&s->u)+qt_bytes(&s->d);
free(s->slab); free(s->fslab); s->slab=NULL; s->fslab=NULL; s->slab_cap=s->fslab_cap=0;
@@ -3440,6 +3443,8 @@ static int mem_should_wire(void){
static int mem_wire(void *addr, size_t len){
#if defined(__APPLE__) || defined(__linux__)
return mlock(addr, len);
#elif defined(_WIN32)
return compat_mlock(addr, len); /* VirtualLock + working-set growth */
#else
(void)addr; (void)len; return 0;
#endif