The documented CUDA-MTP opt-in was silently inert through the wrapper: _auto_tune exported DRAFT=0 (compute-bound, or disk-bound with projected hit < 0.90), which preempts the engine's DRAFT=-1 auto path - the only place COLI_CUDA_MTP is consulted (colibri.c). Found while running the #467 A/B: COLI_CUDA_MTP=1 alone measured pure baseline noise until DRAFT=3 was also set. Now an explicit COLI_CUDA_MTP=1 makes the planner leave DRAFT unset (engine resolves draft=3); unset keeps DRAFT=0 -> MTP off under CUDA, the measured-correct default (#389 -42 streaming-bound). ENVIRONMENT.md documents the behaviour and the measured trade-off.
JustVugg caught a regression by running the PR: the code asked
`lscpu -p=CPU,Core,Socket` and read fields[1]/fields[2], but the comment's
claim was inverted -- `lscpu -p=<list>` emits EXACTLY the requested columns
(no CPU prefix), while bare `lscpu -p` prepends CPU.
On machines where the requested list is short or lscpu collapses to two
columns, every line was skipped by the `< 3` guard, cores stayed empty, and
the code fell through to os.cpu_count() -- the LOGICAL count. Result: 6
physical cores reported as 12 (SMT over-subscription), the opposite of the
fix.
Correct per review:
- ask lscpu for exactly 'core,socket'
- take fields[-2], fields[-1] (correct whether or not CPU is prepended)
- keep the warning scaffolding + the _resolve_physical_cores clamp
(those are the actual #325 fix; only the indexing was wrong)
Test now exercises BOTH the 2-column (-p=core,socket) and 3-column
(bare -p, CPU prefix) layouts, asserting 12 physical cores each. The
2-column case is the one that regressed: old parser returns 24, new
returns 12.
The core-count fix was necessary but not sufficient: @liangstein confirmed
physical_cpu_count() now returns 64 on his box, yet --auto-tier STILL pinned
decode to one core. A complete env diff between the plain (working) and
--auto-tier (broken) paths showed exactly three keys the plan adds:
OMP_NUM_THREADS = 64 (correct, verified)
OMP_PROC_BIND = spread
OMP_PLACES = cores
Since OMP_NUM_THREADS was already correct, the culprit is the affinity pair.
The mechanism: environment_for_plan() sets OMP_PROC_BIND=spread + OMP_PLACES=cores
in the launcher's env. The engine's hot-thread tuning (glm.c main, the
COLI_OMP_TUNED self-exec) then tries setenv("OMP_PROC_BIND","close", overwrite=0)
-- but overwrite=0 cannot replace an already-set var, so the plan's "spread"
wins. On the reporter's libgomp + multi-socket topology, spread + places=cores
collapsed the team to a single CPU even with 64 threads configured.
Fix: don't set affinity from the plan at all. The engine deliberately chose
"close" for cache locality (the tiny back-to-back per-expert matmuls want
adjacent cores), and the plain path already leaves affinity to the engine.
Removing the plan's spread/places makes --auto-tier match the working plain
path; a user wanting a specific policy can still set OMP_PROC_BIND/OMP_PLACES
in their own environment (environment_for_plan only setdefaults OMP_NUM_THREADS).
Verified via env diff: after the fix, --auto-tier adds ONLY OMP_NUM_THREADS
beyond the plain env -- the engine's own close/bind tuning now wins on both
paths identically.
Note: this could not be reproduced on Windows (MinGW libgomp prints "Affinity
not supported on this configuration" and ignores the vars entirely); it is
Linux-libgomp-specific, matching the reporter's Rocky 9 box.
Tests: rewrite test_applies_plan_without_overriding_explicit_settings to assert
the plan sets NO affinity vars on any platform (the old test encoded the buggy
platform-dependent spread/cores contract). Add
test_plan_does_not_set_omp_affinity_vars as a focused regression. 78/78 pass.
physical_cpu_count() silently returned 1 in two situations, and that value
flowed through build_plan -> OMP_NUM_THREADS to pin every matmul region to a
single thread under --auto-tier (reported on Rocky Linux 9, 512 GB RAM).
Two root causes:
1. The lscpu parse counted the wrong thing. `lscpu -p=core,socket` prepends the
CPU column, so the output is actually CPU,Core,Socket; the old set
comprehension collected (CPU,core,socket) tuples that were unique per logical
CPU. Now parse CPU,Core,Socket and dedupe on (core, socket) to get true
physical cores (the SMT-doubling the surrounding comments warn against was
the actual behavior).
2. Any probe failure fell through to `os.cpu_count() or 1`. On a cgroup'd or
otherwise constrained box os.cpu_count() can be 1 (or None), silently
capping the run. Skip offline core/socket fields ("-" instead of raising
ValueError) so a single offline row no longer discards the whole parse, and
replace the silent `or 1` fallback with os.cpu_count() (logical) plus an
explicit warning. Only return 1 when nothing at all is detected, and warn.
Also harden the win32 branch: declare argtypes/restype on
GetLogicalProcessorInformationEx (an undeclared 64-bit WinAPI returns c_int and
takes c_int pointers, so the probe could silently fail), and warn on its
fallbacks. Replace the silent max(1, int()) clamp in build_plan with
_resolve_physical_cores() that clamps to 1 with a warning instead of masking.
Tests: the existing OMP_NUM_THREADS test passed physical_cpus=24 explicitly, so
it never exercised the real probe -- that is why this regressed. Add regression
coverage for the lscpu physical-core dedup, offline fields, lscpu-missing
fallback, zero-cores degenerate case, and an end-to-end build_plan +
environment_for_plan check that OMP_NUM_THREADS reflects physical (not logical)
cores.
Extend resource_plan to classify the hardware into bottleneck regimes
(disk / memory / mixed / compute) and derive tuning knobs automatically:
MTP: off when compute-bound (42% loss at full residency, #389)
or disk-bound with <90% hit (union growth adds reads)
PIPE: COLI_CUDA_PIPE=1 single-GPU, =2 multi-GPU, PIPE=1 CPU disk
NUMA: selective interleave for GPU hosts, blanket hint for CPU-only
PIN: PIN_GB=all when fully resident + no GPU
OMP: COLI_NO_OMP_TUNE=1 for Metal (spin steals GPU power)
`coli plan` now shows an auto-tune section with each knob and its
reason. `environment_for_plan()` applies them via setdefault so
explicit user settings always win.
plan version stays at 2 (additive fields: bottleneck_class,
projected_hit_rate, tune). 7 new tests covering all regimes.
On Windows the engine self-exec OMP tuning never runs (Linux/FreeBSD-only)
and posix_fadvise readahead is a compat.h no-op, so a stock Windows run
leaves large measured wins on the table. The launcher now setdefaults, on
win32 only, each independently overridable by setting the variable:
- OMP_WAIT_POLICY=active, GOMP_SPINCOUNT=200000, OMP_DYNAMIC=FALSE,
OMP_NUM_THREADS=<physical cores> (parity with the glm.c self-exec block;
COLI_NO_OMP_TUNE disables exactly this block, presence-based like the
engine). OMP_PROC_BIND/OMP_PLACES deliberately omitted and also removed
from environment_for_plan on win32: MinGW libgomp has no affinity support
("Affinity not supported on this configuration").
- DIRECT=1: unbuffered expert reads. Measured on a 9950X3D + Samsung 9100
PRO Gen5 + Win11: iobench 10.68 GB/s O_DIRECT vs 9.03 buffered (warm);
end-to-end REPLAY 0.48 -> 1.02 tok/s. Matches #162 (1.47x same class).
- PIPE=1: load/matmul overlap, byte-identical output; +8% on top of DIRECT
(PIPE_WORKERS untouched at 8 - 4/8/16 swept flat on Gen5).
- PILOT_REAL=1: real cross-layer prefetch, the only working prefetch on
Windows; +11% and expert hit rate +19 points.
Full ladder methodology and numbers: 96-token greedy REPLAY, one lever per
step, medians of 3-4 runs (see the fork tuning doc referenced in the PR).
tests/test_env_defaults.py covers the defaults, explicit-override-wins,
the kill-switch scope, and the non-win32 no-op.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
os.cpu_count() returns logical processors, so on SMT machines the plan
sets OMP_NUM_THREADS to 2 threads/core, which thrashes the AVX-512 units
during expert matmul (9950X3D: 32 logical vs 16 physical). Count
RelationProcessorCore records instead, with the existing lscpu/cpu_count
fallbacks intact.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two related crash bugs in the Python planning layer:
P1: Path.read_text() without encoding= defaults to locale.getpreferredencoding()
(cp1252 on most Windows installs). HuggingFace config.json is UTF-8 — if it
contains any non-ASCII byte (Chinese fields, accented chars, emoji in
metadata), read_text() raises UnicodeDecodeError. In doctor.py this is
caught and mis-reported as 'config.json is missing or invalid' (false
negative); in build_plan() called directly it is an uncaught crash.
Fixed: read_text(encoding='utf-8') in both resource_plan.py and doctor.py.
P2: int(cfg.get('kv_lora_rank', 0)) crashes with TypeError if the key is
present but null (JSON null). dict.get() returns the default only when
the key is ABSENT; a null value returns None, and int(None) raises
TypeError. The engine validates against malformed configs in C but the
Python planner did not.
Fixed: int(cfg.get(key) or 0) — coerces both missing and null to 0.
Applied to all 8 int(cfg.get(...)) calls in build_plan().
Co-authored-by: woolcoxm <13604288+woolcoxm@users.noreply.github.com>
P6: discover_gpus() used line.split(',', 3) to parse nvidia-smi CSV output.
If the GPU name contains a comma (e.g. 'Tesla, Inc. V100'), split produces
more than 4 fields, the int() parse fails on the wrong field, and the GPU
is silently dropped — doctor reports 'no NVIDIA device detected' with no
clue why. Fixed by using the csv module (handles quoted fields correctly).
P7: require_auth() used plain string != comparison for the API key
('Authorization' header vs expected 'Bearer <key>'). This enables a
timing side-channel that could leak the key byte-by-byte. Low impact when
bound to localhost (default), but serve() only warns when host is
non-localhost without a key, and users do expose on 0.0.0.0. Fixed by
using hmac.compare_digest (constant-time comparison).
Co-authored-by: woolcoxm <13604288+woolcoxm@users.noreply.github.com>
Rebased onto current dev, split into 3 logical parts (all validated):
1. CPU portability (serve-mode _O_BINARY pipe fix — stock main hangs on MinGW without it; RAM detection cap 0->9/layer; POSIX guards for select/mmap/madvise; warmup script).
2. AVX-VNNI 128-bit int8/int4 dot kernel (Alder Lake+/Meteor Lake+), bit-identical to AVX2 (author-verified on Meteor Lake; compiles out to AVX2 elsewhere) + _mm256_extracti128_si256 typo fix that blocked -march=native.
3. CUDA DLL via LoadLibrary, gated behind CUDA_DLL=1 (host never links cudart; silent CPU fallback if absent; author-verified on RTX 5070 Ti).
Validated here: make check 59/59, oracle 32/32 TF, Windows cross-compile clean + glm.exe loads+runs via WSL interop. Fixes the #123 Windows build failure.
Mohamed Mastouri