In serve/chat mode, Ctrl-C during generation killed the whole engine, losing
the loaded model and forcing a full reload. Now a SIGINT handler (armed only
in run_serve / run_serve_mux) sets a flag that spec_decode's token loop treats
exactly like hitting the NGEN cap: the turn ends through the normal path, so
the END sentinel, STAT line, usage_save and KV append all run — and :more can
continue the interrupted answer. The mux loop closes in-flight requests via the
same mux_done path. One-shot ./glm runs and Windows keep default SIGINT (die).
coli: stream_turn survives the first KeyboardInterrupt, forwards SIGINT to the
engine (covers non-TTY), drains to the turn boundary, and reports it. A second
Ctrl-C quits. Help line and per-turn footer updated.
POSIX only (sigaction); no behaviour change on Windows. Verified end-to-end on
Apple M4 + Metal: interrupt mid-decode, engine stays up, next prompt answers,
:q exits 0.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Cut Windows decode disk I/O from 2.06s/tok to 1.70s/tok (budget=4), meeting the
<=2s/tok target. Two changes on the pread expert-load path:
1. compat.h: replace the posix_fadvise no-op with a real WILLNEED cache-warmer
(overlapped ReadFile into a scratch buffer -> populates the standby page cache
so the later synchronous pread faults from RAM). Re-arms the existing
expert_prefetch/PILOT/next-block prefetch chain on Windows. DONTNEED stays a
no-op (matches macOS; Windows standby-list trimming self-regulates).
Measured: hit rate 16.4% -> 27.6%.
2. glm.c: flip PIPE (async expert-load thread pool) from default OFF to default ON
on Windows. Dispatches expert pread onto worker threads so loads overlap the
matmul, instead of blocking serial load-then-compute. PIPE=0 opts out.
Measured: expert-disk 65.9s -> 54.3s (-18%).
Also adds compat_fadvise assertions to tests/test_compat_direct.c (data integrity
after cache-warmer, safe no-op on bad fd / non-WILLNEED).
mmap (CreateFileMapping/MapViewOfFile) was implemented and tested at length but
reverted: it regressed on Windows (RSS bloat from touched mapped pages collapsed
the expert cache via ullAvailPhys — a fundamental Windows-vs-Linux difference).
Full findings + the dead-end analysis recorded in issue_diskio.md.
The original budget dropped experts blindly — even cached ones that cost
zero disk I/O. The miss-aware version pre-scans pin/ecache residency
before applying the budget:
- ALL cache hits are kept (free to compute, no disk I/O)
- Only misses compete for the remaining budget slots
- Miss budget = EXPERT_BUDGET - nhits (min 0)
Results (budget=4, same prompt/config as before):
Original Miss-aware
tok/s 0.33 0.36 (+9%)
hit rate 16.4% 38.2% (2.3x)
prefill 8.9s 5.6s (1.6x faster)
decode 97.4s 88.9s (9% faster)
The hit rate doubling is the key quality signal: the model now gets the
full contribution from all resident experts plus the top-4 new loads,
instead of losing some hits to the budget.
Add EXPERT_BUDGET env var that caps the number of distinct experts loaded
per layer across the batch-union. When the union exceeds the budget, keeps
only the highest-aggregate-gate-weight experts and drops the rest from
idxs[] so they're never loaded from disk.
Complementary to TOPP (per-position) — this trims the cross-position union
that multiplies under MTP/prefill. Based on MoE-Spec (arXiv 2602.16052):
'top 32 of 64 experts capture 93% of routing weight.'
Measurements (GLM-5.2 744B, 24GB RAM, cap=2, MTP=0, 32 tokens):
Baseline (budget=0): 0.18 tok/s, 9.3% hit, 176s decode, 39s prefill
EXPERT_BUDGET=12: 0.19 tok/s, 14.0% hit, 171s decode, 12s prefill
EXPERT_BUDGET=6: 0.26 tok/s, 21.0% hit, 123s decode, 7s prefill
EXPERT_BUDGET=4: 0.33 tok/s, 16.4% hit, 97s decode, 9s prefill
Budget=4 nearly doubles decode speed (+83%) and 4x's prefill speed by
halving disk reads per layer. Default OFF (EXPERT_BUDGET=0).
Two independent fixes validated end-to-end on fresh fixtures:
1. KV cache disk I/O (issue_diskio.md opportunities #1 + #4):
- kv_disk_append: fopen/fclose every turn -> persistent FILE* kept open
for the engine lifetime, lazy open on first append, closed in
serve_ctx_free. Eliminates per-turn handle creation overhead.
- kv_disk_append: ~157 small fwrites per position -> one contiguous
record memcpy'd into a staging buffer then a single fwrite per
position. The staging buffer grows on demand via realloc.
- kv_disk_truncate: closes the persistent handle before truncating
so the file actually shrinks on disc, then reopens lazily.
- KVState gains disk_fp, disk_buf, disk_buf_cap fields.
- Verified: serve-mode round-trip, write 11 tokens then reload and
resume with no re-prefill, then append 8 more and reload to 19.
2. Expert weight unfusing in test-model generators:
- The real GLM-5.2-FP8 checkpoint stores routed experts UNFUSED as
per-expert 2-D tensors, each with its own _scale_inv. HF fuses
gate+up into a single 3-D gate_up_proj for compute efficiency.
- The converter and C engine both expect the unfused layout. The
fused 3-D tensors were silently skipped by the converter, and the
engine crashed with missing-tensor errors.
- New unfuse_experts in glm_fp8_emit.py splits gate_up_proj and
down_proj into per-expert 2-D tensors. Called after reference
generation but before saving, in both generators, both FP8 and bf16.
- Also fixed: make_glm_oracle.py FP8 round-trip guard used p.dim()<2
which let 3-D fused experts through and crashed fp8_block_quantize.
Changed to p.dim()!=2 to match the converter ndim!=2 guard.
Validated full chain on fresh fixtures:
generator --fp8 -> 570 e4m3 tensors + 629 scale_inv, was 90 when fused
converter --group-size 0 -> per-row int4 fmt=2, engine loads clean
converter --group-size 128 -> grouped int4 fmt=4, 8-16x more scales,
engine loads clean, fmt=4 auto-detected in both mmap and slab paths
dequant error: grouped 1.14-1.22x lower than per-row vs FP8 source
Both test-model generators (make_glm_oracle.py, make_glm_bench_model.py) can now
emit weights as FP8 e4m3 + 128x128 block scale_inv, in the same layout as the real
GLM-5.2-FP8 checkpoint. This lets convert_fp8_to_int4.py exercise its FP8->int4
dequant path on a local fixture without the 379 GB download.
- New shared helper glm_fp8_emit.py: FP8 block quantize/dequantize (FBGEMM/TE
scale=amax/448 convention) + state_dict emitter. Only exactly-2-D tensors are
quantized; 1-D/3-D and norms/router/e_score_correction_bias are kept as f32,
mirroring the converter's classify() + ndim!=2 guard.
- make_glm_bench_model.py: opt-in --fp8 writes model.safetensors in FP8 layout
(config.json written explicitly since the FP8 path bypasses save_pretrained);
manifest gains a 'format' field. Default bf16 behavior unchanged.
- make_glm_oracle.py: opt-in --fp8 round-trips quantizable weights through FP8
before computing ref_glm.json, so the reference reflects exactly the FP8 model
the converter ingests. Default bf16 oracle contract unchanged.
Verified end-to-end: FP8 model -> converter --indir -> int4 U8 + .qs F32 output,
bit-identical dequant between helper and converter (maxdiff 0.0).
Audit of all disk I/O paths in the engine (expert pread, KV persistence,
config/tokenizer loads) and research into techniques used by llama.cpp,
vLLM, AirLLM, PRESERVE, HOBBIT, SolidAttention. Findings:
- Expert path is already well-batched (one coalesced ~19MB O_DIRECT pread)
- 76% of decode time is expert-disk I/O on RAM-constrained hosts
- posix_fadvise(WILLNEED) is a no-op on Windows (compat.h:107)
- I/O-to-compute ratio is 3.6x — the binding constraint
- Levers: hit-rate (cache cap), cross-layer prefetch (PILOT_REAL),
storage (VHDX vs direct NVMe), batched decode
Ranked opportunities and source links documented in issue_diskio.md.
Split the resident weight classification into 5 sub-types so each can
get different precision:
sh = shared expert (highest sensitivity, fires every token)
o = o_proj (reconstructs output, biggest attn tensor)
kvb = kv_b_proj (reconstructs KV cache on every decode)
attn = q_a/q_b/kv_a (other attention projections)
dmlp = dense MLP (first 3 layers)
New args: --shared-bits, --o-bits, --kvb-bits, --attn-bits, --dmlp-bits
Each defaults to ebits (backward compat). When set, the converter applies
that precision to just that tensor type.
Research-backed plan: put the 3 compounding tensors (shared expert, o_proj,
kv_b_proj) at int8 and everything else at grouped int4. Extra RAM cost:
only +5.3 GB (those tensors are small vs the 372 GB expert pool on disk).
Two independent fixes validated end-to-end on fresh fixtures:
1. KV cache disk I/O (issue_diskio.md opportunities #1 + #4):
- kv_disk_append: fopen/fclose every turn -> persistent FILE* kept open
for the engine lifetime, lazy open on first append, closed in
serve_ctx_free. Eliminates per-turn handle creation overhead.
- kv_disk_append: ~157 small fwrites per position -> one contiguous
record memcpy'd into a staging buffer then a single fwrite per
position. The staging buffer grows on demand via realloc.
- kv_disk_truncate: closes the persistent handle before truncating
so the file actually shrinks on disc, then reopens lazily.
- KVState gains disk_fp, disk_buf, disk_buf_cap fields.
- Verified: serve-mode round-trip, write 11 tokens then reload and
resume with no re-prefill, then append 8 more and reload to 19.
2. Expert weight unfusing in test-model generators:
- The real GLM-5.2-FP8 checkpoint stores routed experts UNFUSED as
per-expert 2-D tensors, each with its own _scale_inv. HF fuses
gate+up into a single 3-D gate_up_proj for compute efficiency.
- The converter and C engine both expect the unfused layout. The
fused 3-D tensors were silently skipped by the converter, and the
engine crashed with missing-tensor errors.
- New unfuse_experts in glm_fp8_emit.py splits gate_up_proj and
down_proj into per-expert 2-D tensors. Called after reference
generation but before saving, in both generators, both FP8 and bf16.
- Also fixed: make_glm_oracle.py FP8 round-trip guard used p.dim()<2
which let 3-D fused experts through and crashed fp8_block_quantize.
Changed to p.dim()!=2 to match the converter ndim!=2 guard.
Validated full chain on fresh fixtures:
generator --fp8 -> 570 e4m3 tensors + 629 scale_inv, was 90 when fused
converter --group-size 0 -> per-row int4 fmt=2, engine loads clean
converter --group-size 128 -> grouped int4 fmt=4, 8-16x more scales,
engine loads clean, fmt=4 auto-detected in both mmap and slab paths
dequant error: grouped 1.14-1.22x lower than per-row vs FP8 source
coli_cuda_load did LoadLibraryA("coli_cuda.dll") with a bare name. Windows'
default search order includes the current working directory (and, without
SafeDllSearchMode, other writable locations), so an attacker who plants a
coli_cuda.dll where the user launches glm.exe — or inside a downloaded model
directory the user cd's into — gets their DllMain executed at load time:
DLL hijacking -> arbitrary code execution.
Now the loader resolves the path next to glm.exe via GetModuleFileNameA and
loads that exact file with LOAD_WITH_ALTERED_SEARCH_PATH, so both the DLL and
its dependency search are anchored to the trusted install directory. Fallback
(if GetModuleFileNameA ever fails) uses LOAD_LIBRARY_SEARCH_APPLICATION_DIR |
LOAD_LIBRARY_SEARCH_SYSTEM32 — which also excludes the CWD. Cross-compiles clean
under mingw-w64; the CPU path is unaffected (this file is _WIN32-only).
Note: grammar.h gr__rule memcpy was reviewed in the same pass and is safe (len
is clamped to 63 into a name[64] buffer).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A downloaded (supply-chain) model file was fully trusted by the loader. Three
memory-safety holes, all reachable by pointing the engine at a crafted shard —
demonstrated crashing on pre-fix, now rejected fail-closed:
st.h (safetensors):
- header length `hlen` (u64 from the file) was unbounded before malloc(hlen+1):
a crafted value overflows (malloc(0) then hdr[hlen]=0 OOB) or forces a giant
allocation. Now bounded to the file size and a 512 MB cap; malloc NULL-checked.
- json_get() returns NULL for missing/mistyped fields, but dtype/data_offsets/
shape were dereferenced blind (off->kids[0]) — a header omitting data_offsets
SIGSEGV'd (verified). Now type/arity-checked before use.
- data_offsets [a0,b0] were trusted: b0<a0 gave a negative nbytes -> malloc((size_t))
giant and an oversized memcpy into the caller's buffer in st_read_f32 (heap
overflow); off could point outside the file. Now validated 0<=a0<=b0 and
data_start+b0<=filesize.
json.h: j_parse_val recursed with no depth limit -> stack overflow on nested
input like [[[[...]]]]. Added J_MAX_DEPTH=1024 (headers are ~3 deep); wide-but-
flat objects like the GLM header are unaffected (depth is decremented per return).
eval_glm.py: tempfile.mktemp() -> mkstemp() — closes the TOCTOU/symlink race on
a shared tmp dir (CWE-377).
Network path (openai_server.py + serve SUBMIT parser) audited separately and is
already sound: hmac.compare_digest auth, MAX_BODY cap, resolve()+relative_to
traversal guard, list-form subprocess, bounded/validated SUBMIT header. All 62
tests pass; valid GLM/OLMoE shards load unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Root cause of gibberish output: the int4 quantization uses one F32 scale per
output row (2048 scales for a 2048x6144 matrix). The FP8 source has 128x128
block scales — 48x finer. This destroys reasoning while keeping surface fluency.
Changes:
- Converter: quant_int4_grouped() with --group-size 128 arg. Same nibble
packing, but one scale per group of 128 elements along the input dim.
- Engine QT struct: added 'gs' field (group size, 0=per-row backward compat)
- Engine qt_from_disk: auto-detects fmt=4 when scale array is O*ceil(I/128)
elements instead of O. Old per-row models (fmt=2) work unchanged.
- Engine matmul_i4_grouped(): AVX2 kernel that applies per-group scales.
Accumulator resets at each group boundary: dot(x[grp],w[grp]) * scale[grp].
- Engine matmul_qt_ex: dispatches to grouped kernel for fmt=4 (always exact,
no IDOT approximation since the point is quality)
- Engine expert_load: both mmap and slab+pread paths detect fmt=4 from
scale array size and set gs=128
- qt_bytes: fmt=4 reports correct memory including group scales
Backward compatible: existing per-row int4 models work unchanged.
The fused gate+up pair path (matmul_i4_pair) falls back to separate
matmul_qt calls for fmt=4 — minor perf cost, correctness preserved.
Two small correctness fixes surfaced by community reports on non-Linux hosts:
#236 — expert_load's buffered pread paths (slab + qs scales) used
perror("pread expert") on a short read. Since pread returned a short count
(not -1), errno stays 0 and perror prints "Success" — a confusing message
right before exit(1) in the score/bench path. New pread_full() helper loops
over short reads and EINTR and reports actual/expected bytes and offset, so a
truncated shard reads as such instead of "Success".
#219 — Linux pulls -pthread in via -fopenmp; the *BSDs do not, so pthread_*
fail to link there. Added -pthread to the generic (Linux/*BSD x86-64) and
aarch64 CFLAGS/LDFLAGS. No-op on Linux, required on FreeBSD (complements #206).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
PIN_GB=all passed gb=-1.0 to pin_load, which took npin=n (all experts) and
ignored --ram entirely — the OOM-kill regression from #80. A 92 GB host with
--ram 78 was killed mid-generation (anon-rss ~89 GB). Now gb<0 clamps npin to
expert_avail() — how many experts fit the RAM budget, same accounting AUTOPIN
uses. pin_load already adds the pinned bytes to resident_bytes, so the later
cap_for_ram narrows the LRU accordingly with no double count.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Probe sweep + affinity analysis + leave-one-prompt-out validation, so anyone can build and
cross-validate the atlas on their own box rather than trusting one machine.
The four traps this harness exists to control (each silently corrupts the atlas):
--topp prunes experts by cumulative probability. Measured, same prompt:
topp=0 -> 21,000 selections across 7,587 distinct experts
topp=0.7 -> 11,944 selections across 4,687 distinct experts
It hides 38% of the experts, and it is the recommended speed setting.
MTP/DRAFT eusage is incremented inside moe(), BEFORE verification, so rejected
speculative drafts count experts routed for text never emitted.
.coli_usage is loaded at startup and accumulates, so a naive STATS dump contains all
prior history rather than this run.
autocorrelation: routing within one run is highly correlated, so an expert firing 38
times during one prompt is ONE observation, not 38. Entropy/chi-square on raw
selections certifies single-prompt flukes as perfect specialists — analyze.py
therefore requires affinity to replicate across a category independent prompts.
Result on GLM-5.2 744B int4 (Zen5, CPU routing path), 10 topics x 3 prompts x 64 tokens:
leave-one-prompt-out accuracy 29/30 = 96.7% (chance 10%)
strong specialists (spec>=0.5) 1,041 / 13,260 (7.9%)
specialisation vs depth layer 3 ~0.07 -> layers 18-58 ~0.19-0.27
replication gate rejected 587 single-prompt flukes
The one miss is the interesting part: a Chinese-language poetry prompt classifies as poetry,
not Chinese — routing follows the task over the language.
c/tests/test_schema_gbnf and c/tests/test_compat_direct were committed
as Linux x86-64 ELF executables (slipped in via #111). On any other
platform make test-c considers them up to date and execs them, failing
with OSError: [Errno 8] Exec format error. Remove them and add the two
names to .gitignore alongside the other test binaries already listed,
so each platform rebuilds its own.
kv_alloc had two consecutive if(k->Lc) free blocks. The first freed every
k->Lc[i]/k->Rc[i] and both arrays without unregistering from Metal and
without nulling k->Lc; the second then re-tested the dangling pointer,
called coli_metal_unregister on freed pointers, and freed everything a
second time. Safe only when k->Lc is NULL (first call); any re-allocation
on the same KVState aborts in the allocator.
The first block is the pre-Metal version of the free path: 3716e40 (Metal
backend) replaced it with the Metal-aware block, and ec89136 (GPU resident
pipeline) re-added it above during the merge. Delete it so the Metal-aware
block is the only free path.
Caught by tests/test_kv_alloc from the previous commit:
before: malloc: *** error for object 0x7: pointer being freed was not allocated (exit 134)
after: OK kv_alloc re-allocation (exit 0)
kv_alloc guards every KVState free with if(k->Lc) precisely so it can be
called again on the same KVState (context resize, slot re-init). Exercise
that path: allocate, touch the cache, allocate again at a larger size.
Fails at this commit with a double-free (fixed in the next commit):
malloc: *** error for object 0x7: pointer being freed was not allocated
Paper-style cache-aware MoE selection (arXiv:2412.00099 max-rank):
keep true top-J always; fill remaining K slots preferring experts already
resident in pin∪LRU within top-M. Default OFF so stock full top-K is
unchanged.
Env: CACHE_ROUTE, ROUTE_J/M/P/ALPHA, ROUTE_AGREE (auto-on with CACHE_ROUTE).
Telemetry: swap%/route_swaps/route_slots, route_agree, route_kl on footer
and serve STAT. Complementary to PILOT (prefetch vs selection change).
Routing-only PR for clean A/B vs PILOT / #119; no CUDA/fuse stack.
See docs/CACHE_ROUTE.md. Closes nothing; for #161 discussion.
Co-authored-by: Vincent Marquez <vincentmarquez405@gmail.com>
clean/test-c/test-python run python on the build host, but $(PYTHON) was
chosen from $(IS_WIN), which is derived from the target triple
($(CC) -dumpmachine). A Linux->mingw cross build (make CC=x86_64-w64-mingw32-gcc
...) therefore sets IS_WIN and picks `python`, which fails on hosts where only
`python3` exists (e.g. Debian/Ubuntu) — breaking the cross-compile path #171
added.
Key PYTHON off the host instead: $(OS)=Windows_NT (the #129 signal) is set in
every Windows shell and empty on Linux/macOS. EXE stays driven by the target
triple, as it should. Addresses @rofl0r's host/target note on #171.
Co-authored-by: bopof <285767350+bopof@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Wires the two-step prediction (kind==2 from experiment/two-step-predict)
into the real pilot_prefetch() path behind PILOT_TWO=1 env var.
Changes:
- la_predict kind==2: computes shared expert (resident, no disk I/O) on
L's post_ln-normalized state, adds output to residual, then runs L+1's
router on the corrected state. Guards n_shared==0 and mloe_inter<=0.
- pilot_prefetch(): when PILOT_TWO=1, computes the same shared-expert
correction before running the router. Workspace allocated once per
call (not per position) to avoid malloc churn.
- LOOKA measurement harness expanded to 4 slots (prev, skip-attn,
PILOT stale, two-step) with updated reporting at both exit points.
- PILOT_TWO env var wired into main().
Measurements (GLM-5.2 744B, 24GB RAM, cap=2):
LOOKA recall: PILOT stale 73.6% -> two-step 76.7% (+3.1%)
End-to-end tok/s: no change (0.16 tok/s) — cache too small (cap=2)
for prediction quality to matter; disk bandwidth saturated regardless.
On higher-RAM hosts (cap>=32) the +3.1% recall would translate to
measurably fewer disk misses.
Prior art: 'Speculating Experts' (arXiv:2603.19289) independently
developed the same idea as a 'quasi-hidden state' using a static
default vector. Our approach uses the actual computed shared expert,
which is input-dependent and more accurate.
Co-authored-by: woolcoxm <13604288+woolcoxm@users.noreply.github.com>
cmd_bench builds the eval_glm.py command but did not pass --glm,
so eval_glm.py fell back to ./glm which breaks when running from
any directory other than the source tree. Pass the already-resolved
GLM variable via --glm, matching --data and --snap.
Assisted-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- 'See it running' right after the intro: the live dashboard (744B
answering at 4+ tok/s end-to-end with metrics and tier bar) and the
Brain cortex (19,456 experts, tier colour, routing heat, per-turn
firing) — three seconds to understand what colibri is.
- A short table of contents for the long read below.
- 'Web dashboard' section: coli web one-liner, what each panel shows,
link to the Expert Atlas (#175). Existing content untouched.
PR #111 added 24 functions (pipe_*, attention_*_batch*, shared_mlp_w4a16,
tensor_update) to backend_cuda.h without COLI_CUDA_DLLEXPORT and without
backend_loader.c wrappers: on Windows the host failed to link (undefined
references from glm.c) and the DLL exported none of the new entry points.
Decorate the declarations and add matching typedefs/RESOLVEs/wrappers
following the loader's existing pattern. DLL now exports 39 symbols;
engine links; TF oracle 32/32 CPU + dual-GPU (sm_120/sm_89).
Co-authored-by: olorin <io@zyphyr.co>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Benchmark datasets are downloaded artifacts, not source files. Store them
under $XDG_CACHE_HOME/colibri/bench (~/.cache/colibri/bench) on Linux/macOS
and %LOCALAPPDATA%\colibri\bench on Windows instead of polluting the source
tree at c/bench/.
fetch_benchmarks.py already calls os.makedirs(out, exist_ok=True), so the
new cache path is created on first use.
Assisted-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Fuse CUDA expert MLP execution
* Group CUDA expert transfers by device
* Instrument grouped CUDA expert execution
* Bound grouped CUDA decode scratch
* Execute expert groups across GPUs in parallel
* Release host backing for multi-GPU experts
* Define quality-preserving memory policies
* Overlap cold expert loading with resident compute
* Adapt expert placement with session LFRU
* Fuse q4 expert gate and up dispatch
* Plan CPU work on physical cores
* Batch grouped expert CUDA kernels
* Separate VRAM and RAM expert placement
* Add ragged multi-sequence decode forward
* feat(runtime): add continuous decode scheduler
* Route concurrent API requests through batch scheduler
* Harden multiplex request lifecycle and framing
* Cancel disconnected multiplex requests
* Bind API port before starting the engine
* fix automatic KV slot allocation
* add native int4 Tensor Core grouped GEMM
* add Tensor Core throughput benchmark
* optimize packed int4 low-row kernels
* add asynchronous CUDA staging streams
* document validated six-GPU dense acceleration
* tune six-GPU expert hot set
* raise validated expert hot-set target
* add CUDA MLA absorption core
* fuse grouped expert gate and up projections
* Warn for explicit lossy routing flags
* Add full-resident expert placement mode
* Adapt VRAM expert slots to live routes
* Accelerate int4 matvec on AVX-512
* Reduce AVX-512 and RoPE decode overhead
* Seed every GPU expert layer after prefill
* Limit live GPU swaps during decode
* CUDA batch MLA attention, kv_b head-sharding, fused o_proj, expert-group dispatch, W4A16 kernels
Lab-qualified on the 6x RTX 5090 machine (914-token request benchmark):
- batch MLA absorption kernel (COLI_CUDA_ATTN=1): whole-batch attention on
device, 154.8s -> 102.4s
- attention -> o_proj fusion on the layer device: -> 97.4s
- kv_b head-sharding across cards (COLI_CUDA_ATTN_SHARD=1), no weight
duplication: -> 94.05s
- per-device expert-group dispatch with pinned-buffer async transfers,
W4A16 tensor-core kernels for the shared expert, OMP hot-thread tuning
Negative results (reverted, kept out): GPU-side weighted scatter-add
(atomics + per-layer D2H lose 43.8%), shared-expert fused small-batch
kernel (-38.8%), W4A4 grouped tensor cores (int4 activations corrupt
output). Details in the lab research log.
* GPU resident pipeline: device-resident prefill attention chain, GPU expert groups in prefill, batched router, W4A16 mixed dispatch
COLI_CUDA_PIPE=1 keeps the prefill data plane on the layer home device;
control flow (routing, cache/pin management) stays on CPU. Any CUDA
failure falls back to the unchanged CPU path.
- Device primitives + unit tests (tests/test_pipe_cuda.cu): rmsnorm
(strided), interleaved RoPE, silu-mul, residual add, fixed-order row
merge (no atomics), device-input GEMM, persistent per-device scratch.
All verified against the engine's CPU math on SM120 (worst 1.2e-5).
- attn_pipe_prefill: q_a -> norm -> q_b -> rope -> kv_a -> norm -> rope ->
batch attention -> o_proj in one device chain (q_a/q_b/kv_a colocated
with kv_b); only the final [S,D] and the new KV rows return to host.
Attention 41.2s -> 30.8s on the 1571-token benchmark.
- Prefill batch-union now uses the GPU expert groups (previously gated to
S<=64, leaving all VRAM-resident experts idle during prefill - measured
21ms of GPU expert time in a 148s prefill). Expert phase 78.9s -> 69.0s.
- Router computed as one batched matmul instead of S sequential rows
(bit-identical math).
- W4A16 tensor-core path for expert groups (COLI_CUDA_TC_W4A16=1) with
row-count mixed dispatch: >=16 rows per expert use tensor cores, smaller
batches keep the naive kernel (tensor cores measured negative below
~16 rows). Expert phase 69.0s -> 64.3s, decode unaffected.
Net on the 1571-token prefill benchmark: 148.8s -> 114.3-126.8s
(component timings stable across runs; wall drifts +-3-5s because
.coli_usage placement learning shifts the expert tiers between runs).
PROFILO now also prints the prefill-phase breakdown.
* Skip OMP hot-thread tuning when CUDA is enabled
The active-spin worker team measured 66.9s->20.9s on the CPU-only Zen5
build, but on the six-GPU full-residency workload the spinning workers
contend with the CUDA dispatch threads: ~4x slower prefill with the
process stuck near 1.8 cores. Gate the tuning on COLI_CUDA so each
configuration keeps the behavior it was measured to prefer.
* Inc.2a: sparse layers fully resident on the layer device, residual hops cards at layer boundaries
COLI_CUDA_PIPE=2 keeps the residual stream on the layer home device for
consecutive sparse layers (cudaMemcpyPeer at boundaries): in/post norms,
attention chain, both residual adds and the shared-expert MLP run on
device. Per layer only the post-norm activations (router + CPU-tier
experts + group gather), the new KV rows and, on DSA indexer layers, the
pre-attention norm leave the card. Per-layer transfers drop from ~130MB
to ~70MB. A device-side snapshot at layer entry makes any mid-layer CUDA
failure fall back to the unchanged CPU path idempotently.
1571-token prefill: 127.1s (PIPE=1 control) -> 117.6/118.9s, components
attention 30.8->26.1, other 31.8->22.5-24.5; output verified coherent
against the control.
* Head-sharded attention inside the pipe: negative on PCIe star topology, gated opt-in
Slicing q per card from the home device and collecting ctx back
serializes ~95MB/layer through the home card's PCIe link: attention
26.1s -> 41.4/44.4s on the 1571-token benchmark (two repeats), wall
117.6 -> 135-138s. The standalone host-path sharding won because six
cards uploaded from host RAM in parallel; a home-device star has no
such parallelism without NVLink. Kept behind COLI_CUDA_PIPE_SHARD=1
for interconnects where peer bandwidth does not share one root port.
* Inc.3: device-resident KV shadow for decode attention
Decode re-uploaded the whole latent+rope window per layer per token
(~300MB/token at 1571 context). Each layer now keeps a device shadow of
the compressed KV on its kv_b card, bulk-synced when behind and appended
incrementally; the host cache stays canonical. Invalidation on kv_bind
(slot switch), kv_alloc (resize) and on any overwrite of mirrored rows,
with the legacy full-upload path as fallback.
Measured (COLI_CUDA_PIPE gate): short-context decode 5.48 -> 5.59/5.87
tok/s, 1571-context decode 4.14 -> 4.22 tok/s. Decode remains CPU-expert
bound; the shadow removes the transfer tax, not the compute.
* tools: unified user-experience benchmark (bench_ux.sh)
Two fixed scenarios (short chat, long-document QA), TTFT + decode tok/s
+ first-line drift check, TEMP=0 DRAFT=0 enforced, medians over REPS
runs. Encodes the measurement discipline from the lab record: same
binary per comparison, judge medians because .coli_usage placement
learning drifts wall times between runs.
* tools: bench_ux.sh executable bit
* gitignore compiled test binaries
* tools: expert_atlas.py — measure per-expert topic affinity (#175)
Diffs .coli_usage across 10 themed probe batches (code/math/chinese/
prose/science/law/poetry/structured/translation/casual, 3 prompts each)
driven through a running API server — one engine load total. Every
touched expert gets a topic-affinity vector, entropy, and a specialist/
generalist label; output experts.json feeds the Brain page hover.
* serve: persist .coli_usage after every turn in mux mode, not only at exit
run_serve_mux saved the learning cache once at shutdown; a crash lost
the whole session's routing history, and live consumers of the file
(expert_atlas.py diffs it between probe batches) saw a frozen snapshot.
Now saved per turn like the interactive path (165KB write, negligible).
* web: Brain hover shows measured expert atlas when published
If /experts.json (from tools/expert_atlas.py, #175) is served next to
the app, the tooltip upgrades from the depth heuristic to measured
data: specialist/generalist label, entropy, and the top-3 topic
affinities. Row index maps to real layer (row+3, last row = MTP 78).
Falls back to the heuristic when no atlas is published.
---------
Co-authored-by: JustVugg <JustVugg@users.noreply.github.com>
Two Windows-only bugs in run_serve_mux left the gateway hanging after READY:
1. No _setmode(_O_BINARY): the CRT collapsed CRLF inside fread() payloads (waits
forever for missing bytes) and expanded LF in the READY/STAT sentinels.
2. WaitForSingleObject on an anonymous pipe is undefined (always-signaled or
WAIT_FAILED) and PeekNamedPipe fails on file/console handles, so the dispatch
gate never opened. New rule: idle -> block in getline (POSIX select(NULL)
semantics); active -> PeekNamedPipe poll.
Reproduced and verified on real Windows via MinGW cross-compile + WSL interop:
old binary writes READY (with CRLF corruption) then hangs forever on a crafted
SUBMIT frame; fixed binary answers DONE + STAT and exits cleanly. Linux path
untouched (0 warnings, oracle-exact).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
'make test-c' passes TEST_BINS as 'tests/test_json.exe' etc.; Python's
subprocess on Windows hands that to CreateProcess, which rejects
forward-slash relative paths for the executable (WinError 2), so the
runner this script exists for (running tests from any Windows shell)
failed on every test. os.path.normpath makes it 'tests\test_json.exe'
on Windows and is a no-op elsewhere. Verified: all 8 suites run via
'make test-c' on Windows 11 / MinGW GCC 16.1 and paths are unchanged
on POSIX.
Co-authored-by: olorin <io@zyphyr.co>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
#152 carried leftover debug scaffolding from the lm_head measurement posted on that
PR: a `g_lmhead_exact` global, an undocumented `LMHEAD_EXACT` getenv, and 7 lm_head
call sites routed through matmul_qt_ex(..., !g_lmhead_exact). None of it was in the
PR description; it rode in because `git checkout -B` carries uncommitted working-tree
edits forward.
It should not stay:
- it does nothing worth having. Pinning lm_head off IDOT measures -0.03% perplexity
across 5 corpora (in-distribution, isolated) — nothing.
- LMHEAD_EXACT=1 would silently change lm_head numerics, undocumented and untested.
- lm_head is fmt=1 (int8), so the IDOT gate applies to it unconditionally on every
platform anyway; there is no threshold story here to expose.
The flag defaulted to 0 and `!g_lmhead_exact == 1 == matmul_qt`'s own allow_idot, so
this removal is a pure no-op. Verified rather than assumed — summed log-lik over 1023
tokens, in-distribution, CPU (deterministic), dev-with-probe vs dev-minus-probe:
prose -2295.245383 == -2295.245383
markdown -3272.146403 == -3272.146403
Bit-identical. make check green (6 C suites + 60 python tests), no new warnings.
The actual #152 changes are untouched: the three attention input projections and the
DSA indexer's ix_wk stay batched and pinned to the exact int4 kernel via
matmul_qt_ex(..., 0).
Skeldoor