url/colibri
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

win32: auto-enable the GPU in bare coli chat

Browse Source 
On Windows a bare 'coli chat' (no --gpu/--vram/--auto-tier) ALWAYS ran
CPU-only, even on a CUDA build with a GPU present. Two defects:

1. cuda_binary() returned False on Windows. It detects CUDA by running
   'ldd glm | grep libcudart', which is Linux-only (no ldd on win32) and
   meaningless anyway because the Windows engine links cudart only inside a
   runtime-loaded coli_cuda.dll, not as a libcudart symbol in glm.exe. So
   the --gpu/--vram/--auto-tier gates (which call cuda_binary()) never opened.

2. Even with detection fixed, bare 'coli chat' set no CUDA env: env_for's
   else-branch only enables CUDA when --gpu/--vram is passed. Nothing
   auto-enabled the GPU.

Now: cuda_binary() on non-Linux returns True iff coli_cuda.dll exists next
to glm.exe — the exact file backend_loader.c loads from the engine's own
directory, so its presence is a faithful, cheap, DLL-hijack-safe proxy for a
CUDA-capable build. And env_for, scoped to win32 (Linux keeps its working
explicit-flag UX), auto-enables CUDA when a bare chat detects a CUDA build
plus a GPU via nvidia-smi, sizing the expert-tier VRAM budget from real free
VRAM via the existing build_plan/environment_for_plan machinery (same as
--auto-tier, no guessed budget). If nvidia-smi is missing it falls back to
CPU with a clear warning; --gpu none still forces CPU; explicit --vram/--gpu
still win. CUDA_DENSE stays an explicit opt-in (matches --auto-tier).

Verified on a Windows + RTX 5070 Ti box: bare 'coli chat --model <g64>' now
prints '[GPU] auto-enabled CUDA ... 13.0 GB expert tier' and emits
COLI_CUDA=1 / COLI_GPUS=0 / CUDA_EXPERT_GB=13.044 (was: all unset, CPU-only).

Tests: 4 new cases (auto-enable, nvidia-smi-missing fallback, CPU-build
silent, Linux-unchanged) plus the 4 existing default-I/O tests guarded to
mock cuda_binary() so they stay host-independent. Full python suite green
(env_defaults 8, resource_plan 10, doctor 8, makefile_platform 3, cli_output 3).

Out of scope: doctor.cuda_linkage is also POSIX-only and mis-reports on
Windows — separate follow-up.
This commit is contained in:
woolcoxm
2026-07-17 12:03:43 -04:00
parent e9b36141a4
commit 0b05f6a76b
3 changed files with 130 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/ENVIRONMENT.md
+9
View File
@@ -105,6 +105,15 @@ Format: `VAR` — default — effect.
| `COLI_CUDA_SHARED_W4A16_MIN_ROWS` | `32` | Min row count to engage the shared-MLP W4A16 kernel. |
| `COLI_METAL_UNTRACKED` | off (Metal only) | `=1` sets `MTLResourceHazardTrackingModeUntracked` on Metal buffers (reduces hazard-tracking overhead). |
> **Windows note.** On Windows, a bare `coli chat` / `coli run` / `coli serve`
> (no `--gpu`/`--vram`/`--auto-tier`) **auto-enables the GPU** when it detects a
> CUDA build (`coli_cuda.dll` next to the engine) and at least one GPU via
> `nvidia-smi`. The expert-tier VRAM budget is then sized automatically from the
> card's free VRAM (same computation as `--auto-tier`). If `nvidia-smi` is not on
> `PATH` the run falls back to CPU with a warning — pass `--vram N` (or add
> `nvidia-smi` to `PATH`) to enable CUDA in that case. `--gpu none` forces
> CPU-only. (Linux/macOS behaviour is unchanged: pass a flag to enable CUDA.)
---
## Advanced / experimental / debug