From f9391914044ac414cb4ac1d4ef29bc0d7737384a Mon Sep 17 00:00:00 2001 From: Mohamed Mastouri Date: Tue, 21 Jul 2026 01:44:48 +0300 Subject: [PATCH] fix(coli): honour explicit COLI_CUDA=0 over the Windows bare-run auto-enable Before this, a Windows user setting COLI_CUDA=0 for a CPU baseline silently got a ~12.6 GB VRAM expert tier anyway: the bare-run auto-enable ran before any env check, so every Windows 'CPU' benchmark row taken this way was actually GPU-assisted (found while building the 12-cell resource matrix in #467). Setting COLI_GPU=none instead collided with auto-enable's COLI_GPUS ('use COLI_GPU or COLI_GPUS, not both', exit 2). Now an explicit COLI_CUDA=0 suppresses auto-enable and clears stale device/sizing vars, same as --gpu none, which stays the canonical hard off-switch. Docs: COLI_CUDA row updated; DIRECT row + README get the measured, drive-dependent O_DIRECT guidance (+34 decode on real NVMe w/ DRAM cache; neutral-to-negative on QLC/DRAM-less). --- README.md | 7 +++++++ c/coli | 14 +++++++++++++- docs/ENVIRONMENT.md | 4 ++-- 3 files changed, 22 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index aaf7463..02d7717 100644 --- a/README.md +++ b/README.md @@ -148,6 +148,13 @@ on-device across layers so the CPU expert loop runs uninterrupted; on Apple Silicon an experimental [Metal backend](docs/metal.md) does the batched expert math on the unified-memory GPU. +> **On real NVMe, measure `DIRECT=1`.** O_DIRECT bypasses the page cache and is +> often a large win on drives with DRAM cache and bandwidth headroom (+34% +> decode measured with `PIPE=1` on a Blackwell/Windows box; 4.25→9.69 GB/s in +> iobench on a GB10) — but it is drive-dependent: QLC/DRAM-less or virtualised +> disks can be neutral to negative. Try it first; keep what your hardware +> rewards. + ### Faithful model, compressed state The forward pass is validated **token-exact against a `transformers` oracle** diff --git a/c/coli b/c/coli index 5cef2fc..133d21e 100755 --- a/c/coli +++ b/c/coli @@ -248,7 +248,14 @@ def env_for(a): # has working detection + the explicit-flag UX, and changing bare-chat # semantics there is out of scope. Falls back to CPU with a warning if # nvidia-smi is missing (discover_gpus can't size VRAM without it). - if (sys.platform == "win32" and a.gpu is None and not a.vram): + # An explicit COLI_CUDA=0 in the environment must win over the implicit + # auto-enable: before this check, a Windows user setting COLI_CUDA=0 for + # a CPU baseline silently got a ~12.6 GB VRAM expert tier anyway (the + # engine's "CPU" rows were GPU-assisted). --gpu none remains the + # canonical hard off-switch (works on every platform, also clears the + # CUDA_* sizing vars). + if (sys.platform == "win32" and a.gpu is None and not a.vram + and e.get("COLI_CUDA") != "0"): if cuda_binary(): from resource_plan import discover_gpus, build_plan, environment_for_plan, format_bytes gpus = discover_gpus() @@ -277,6 +284,11 @@ def env_for(a): f"(cannot size VRAM); running CPU-only. Add nvidia-smi to PATH or pass " f"--vram N to enable CUDA.{C.r}", file=sys.stderr) # else: CPU build (no coli_cuda.dll) — stay silent, CPU is correct. + elif e.get("COLI_CUDA") == "0": + # honoured off-switch: also drop stale device/sizing vars so the + # engine can't be re-enabled by leftovers (same as --gpu none). + e.pop("COLI_GPU",None); e.pop("COLI_GPUS",None) + e.pop("CUDA_EXPERT_GB",None); e.pop("CUDA_DENSE",None) # --gpu/--vram SENZA --auto-tier: prima venivano ignorati in silenzio e il run # partiva CPU-only senza alcun avviso — benchmark "GPU" pubblicati per errore (#121). if a.gpu is not None: diff --git a/docs/ENVIRONMENT.md b/docs/ENVIRONMENT.md index ce44fce..d7abcc1 100644 --- a/docs/ENVIRONMENT.md +++ b/docs/ENVIRONMENT.md @@ -41,7 +41,7 @@ Format: `VAR` — default — effect. | `PIPE` | `0` (off) | Overlap expert disk-load with matmul via I/O worker threads. Byte-identical output; reorders I/O. `PIPE=1` opts in. | | `PIPE_WORKERS` | `8` | Number of pthread loaders when `PIPE=1`, or the io-wq worker maximum per ring when `URING=1` (capped at 64). Tune to SSD queue depth and available cores. | | `URING` | `0` (off) | Linux-only queued expert I/O. `URING=1` implies `PIPE=1`, forces cold reads through io-wq (`IOSQE_ASYNC`), replaces blocking loader pthreads and spin waits with batched SQEs/CQEs, and batches `PILOT_REAL` loads on a separate ring. Use `DIRECT=1` for cold NVMe to avoid page-cache copy/readahead limits. Fails clearly if the kernel denies io_uring; incompatible with `COLI_MMAP=1`. | -| `DIRECT` | `0` (off) | Use `O_DIRECT`/unbuffered reads for expert slabs. Helps sustained NVMe; keeps the zero-copy GPU path. | +| `DIRECT` | `0` (off) | Use `O_DIRECT`/unbuffered reads for expert slabs. **Drive-dependent — measure it on your hardware.** On real NVMe with DRAM cache and headroom it is often a large win (measured +34% decode with `PIPE=1` on a Blackwell/Windows box, and 4.25→9.69 GB/s in iobench on a GB10); on QLC/DRAM-less drives or slow/virtualised disks it can be neutral to negative. Helps sustained NVMe; keeps the zero-copy GPU path. | | `COLI_NO_OMP_TUNE` | off | **Kill-switch** for the OpenMP hot-thread tuning (`OMP_WAIT_POLICY=active` spin + proc-bind). Set `=1` when the CPU is mostly waiting on the GPU (Metal) so spin doesn't steal the shared power budget. | | `COLI_NUMA` | auto in generated plans on multi-socket Linux; otherwise off | `COLI_NUMA=1` selectively interleaves large expert and dense slabs across NUMA nodes via `mbind` (raw syscall, no libnuma). Helps multi-socket hosts (+7–40% expert matmul); silent no-op on single-node or non-Linux. Explicit `COLI_NUMA=0` overrides the generated plan. | | `MLOCK` | `-1` (auto: on for macOS) | Wire the streamed expert cache into physical RAM (`mlock`) to dodge the memory compressor. `0` off, `1` force. | @@ -91,7 +91,7 @@ Per-drive byte counts are reported in a `MIRROR:` stats line. Combine with `DIRE | Variable | Default | Effect | |---|---|---| -| `COLI_CUDA` | off | Enable the CUDA backend. Requires a CUDA build. | +| `COLI_CUDA` | off | Enable the CUDA backend. Requires a CUDA build. An explicit `COLI_CUDA=0` disables it **and suppresses the Windows bare-run auto-enable** (before this, Windows "CPU" runs with `COLI_CUDA=0` silently got a VRAM expert tier). The CLI flag `--gpu none` is the canonical hard off-switch on every platform. | | `COLI_GPU` / `COLI_GPUS` | unset | Device selection (`auto`, `none`, or a list like `0,1`). Requires `COLI_CUDA=1`. | | `CUDA_DENSE` | `0` | Place dense (non-expert) matmuls on the GPU. | | `CUDA_EXPERT_GB` | `0` | VRAM budget (GB) for caching experts on the GPU. |