fix(coli): honour explicit COLI_CUDA=0 over the Windows bare-run auto-enable
Before this, a Windows user setting COLI_CUDA=0 for a CPU baseline silently got a ~12.6 GB VRAM expert tier anyway: the bare-run auto-enable ran before any env check, so every Windows 'CPU' benchmark row taken this way was actually GPU-assisted (found while building the 12-cell resource matrix in #467). Setting COLI_GPU=none instead collided with auto-enable's COLI_GPUS ('use COLI_GPU or COLI_GPUS, not both', exit 2). Now an explicit COLI_CUDA=0 suppresses auto-enable and clears stale device/sizing vars, same as --gpu none, which stays the canonical hard off-switch. Docs: COLI_CUDA row updated; DIRECT row + README get the measured, drive-dependent O_DIRECT guidance (+34 decode on real NVMe w/ DRAM cache; neutral-to-negative on QLC/DRAM-less).
This commit is contained in:
Mohamed Mastouri
@@ -148,6 +148,13 @@ on-device across layers so the CPU expert loop runs uninterrupted; on Apple
|
||||
Silicon an experimental [Metal backend](docs/metal.md) does the batched expert
|
||||
math on the unified-memory GPU.
|
||||
|
||||
> **On real NVMe, measure `DIRECT=1`.** O_DIRECT bypasses the page cache and is
|
||||
> often a large win on drives with DRAM cache and bandwidth headroom (+34%
|
||||
> decode measured with `PIPE=1` on a Blackwell/Windows box; 4.25→9.69 GB/s in
|
||||
> iobench on a GB10) — but it is drive-dependent: QLC/DRAM-less or virtualised
|
||||
> disks can be neutral to negative. Try it first; keep what your hardware
|
||||
> rewards.
|
||||
|
||||
### Faithful model, compressed state
|
||||
|
||||
The forward pass is validated **token-exact against a `transformers` oracle**
|
||||
|
||||
@@ -248,7 +248,14 @@ def env_for(a):
|
||||
# has working detection + the explicit-flag UX, and changing bare-chat
|
||||
# semantics there is out of scope. Falls back to CPU with a warning if
|
||||
# nvidia-smi is missing (discover_gpus can't size VRAM without it).
|
||||
if (sys.platform == "win32" and a.gpu is None and not a.vram):
|
||||
# An explicit COLI_CUDA=0 in the environment must win over the implicit
|
||||
# auto-enable: before this check, a Windows user setting COLI_CUDA=0 for
|
||||
# a CPU baseline silently got a ~12.6 GB VRAM expert tier anyway (the
|
||||
# engine's "CPU" rows were GPU-assisted). --gpu none remains the
|
||||
# canonical hard off-switch (works on every platform, also clears the
|
||||
# CUDA_* sizing vars).
|
||||
if (sys.platform == "win32" and a.gpu is None and not a.vram
|
||||
and e.get("COLI_CUDA") != "0"):
|
||||
if cuda_binary():
|
||||
from resource_plan import discover_gpus, build_plan, environment_for_plan, format_bytes
|
||||
gpus = discover_gpus()
|
||||
@@ -277,6 +284,11 @@ def env_for(a):
|
||||
f"(cannot size VRAM); running CPU-only. Add nvidia-smi to PATH or pass "
|
||||
f"--vram N to enable CUDA.{C.r}", file=sys.stderr)
|
||||
# else: CPU build (no coli_cuda.dll) — stay silent, CPU is correct.
|
||||
elif e.get("COLI_CUDA") == "0":
|
||||
# honoured off-switch: also drop stale device/sizing vars so the
|
||||
# engine can't be re-enabled by leftovers (same as --gpu none).
|
||||
e.pop("COLI_GPU",None); e.pop("COLI_GPUS",None)
|
||||
e.pop("CUDA_EXPERT_GB",None); e.pop("CUDA_DENSE",None)
|
||||
# --gpu/--vram SENZA --auto-tier: prima venivano ignorati in silenzio e il run
|
||||
# partiva CPU-only senza alcun avviso — benchmark "GPU" pubblicati per errore (#121).
|
||||
if a.gpu is not None:
|
||||
|
||||
+2
-2
@@ -41,7 +41,7 @@ Format: `VAR` — default — effect.
|
||||
| `PIPE` | `0` (off) | Overlap expert disk-load with matmul via I/O worker threads. Byte-identical output; reorders I/O. `PIPE=1` opts in. |
|
||||
| `PIPE_WORKERS` | `8` | Number of pthread loaders when `PIPE=1`, or the io-wq worker maximum per ring when `URING=1` (capped at 64). Tune to SSD queue depth and available cores. |
|
||||
| `URING` | `0` (off) | Linux-only queued expert I/O. `URING=1` implies `PIPE=1`, forces cold reads through io-wq (`IOSQE_ASYNC`), replaces blocking loader pthreads and spin waits with batched SQEs/CQEs, and batches `PILOT_REAL` loads on a separate ring. Use `DIRECT=1` for cold NVMe to avoid page-cache copy/readahead limits. Fails clearly if the kernel denies io_uring; incompatible with `COLI_MMAP=1`. |
|
||||
| `DIRECT` | `0` (off) | Use `O_DIRECT`/unbuffered reads for expert slabs. Helps sustained NVMe; keeps the zero-copy GPU path. |
|
||||
| `DIRECT` | `0` (off) | Use `O_DIRECT`/unbuffered reads for expert slabs. **Drive-dependent — measure it on your hardware.** On real NVMe with DRAM cache and headroom it is often a large win (measured +34% decode with `PIPE=1` on a Blackwell/Windows box, and 4.25→9.69 GB/s in iobench on a GB10); on QLC/DRAM-less drives or slow/virtualised disks it can be neutral to negative. Helps sustained NVMe; keeps the zero-copy GPU path. |
|
||||
| `COLI_NO_OMP_TUNE` | off | **Kill-switch** for the OpenMP hot-thread tuning (`OMP_WAIT_POLICY=active` spin + proc-bind). Set `=1` when the CPU is mostly waiting on the GPU (Metal) so spin doesn't steal the shared power budget. |
|
||||
| `COLI_NUMA` | auto in generated plans on multi-socket Linux; otherwise off | `COLI_NUMA=1` selectively interleaves large expert and dense slabs across NUMA nodes via `mbind` (raw syscall, no libnuma). Helps multi-socket hosts (+7–40% expert matmul); silent no-op on single-node or non-Linux. Explicit `COLI_NUMA=0` overrides the generated plan. |
|
||||
| `MLOCK` | `-1` (auto: on for macOS) | Wire the streamed expert cache into physical RAM (`mlock`) to dodge the memory compressor. `0` off, `1` force. |
|
||||
@@ -91,7 +91,7 @@ Per-drive byte counts are reported in a `MIRROR:` stats line. Combine with `DIRE
|
||||
|
||||
| Variable | Default | Effect |
|
||||
|---|---|---|
|
||||
| `COLI_CUDA` | off | Enable the CUDA backend. Requires a CUDA build. |
|
||||
| `COLI_CUDA` | off | Enable the CUDA backend. Requires a CUDA build. An explicit `COLI_CUDA=0` disables it **and suppresses the Windows bare-run auto-enable** (before this, Windows "CPU" runs with `COLI_CUDA=0` silently got a VRAM expert tier). The CLI flag `--gpu none` is the canonical hard off-switch on every platform. |
|
||||
| `COLI_GPU` / `COLI_GPUS` | unset | Device selection (`auto`, `none`, or a list like `0,1`). Requires `COLI_CUDA=1`. |
|
||||
| `CUDA_DENSE` | `0` | Place dense (non-expert) matmuls on the GPU. |
|
||||
| `CUDA_EXPERT_GB` | `0` | VRAM budget (GB) for caching experts on the GPU. |
|
||||
|
||||
Reference in New Issue
Block a user