Fixes the footgun found in #467: COLI_CUDA_MTP=1 was silently inert on the Windows bare-run/auto-tier flows because _auto_tune exported DRAFT=0 (disk/low-hit or compute), preempting the engine's DRAFT=-1 auto path where COLI_CUDA_MTP is consulted (colibri.c:6255). Now an explicit COLI_CUDA_MTP=1 skips that export so the opt-in engages draft=3 without also needing DRAFT=3. Unset still exports DRAFT=0 → MTP off, the measured-correct default (#467: -32% at 85% hit). Verified locally: unit-tested _auto_tune both branches (opt-in → no DRAFT, unset → DRAFT=0); import os present; CI 8/8. Thanks @mohamedmastouri2000-boop.
The documented CUDA-MTP opt-in was silently inert through the wrapper: _auto_tune exported DRAFT=0 (compute-bound, or disk-bound with projected hit < 0.90), which preempts the engine's DRAFT=-1 auto path - the only place COLI_CUDA_MTP is consulted (colibri.c). Found while running the #467 A/B: COLI_CUDA_MTP=1 alone measured pure baseline noise until DRAFT=3 was also set. Now an explicit COLI_CUDA_MTP=1 makes the planner leave DRAFT unset (engine resolves draft=3); unset keeps DRAFT=0 -> MTP off under CUDA, the measured-correct default (#389 -42 streaming-bound). ENVIRONMENT.md documents the behaviour and the measured trade-off.
Search-path of extra directories, each holding a distinct subset of the .safetensors shards (no duplication). Each shard lives on one drive; demand preads hit whichever drive holds it, so concurrent expert loads parallelise across drives and combined capacity is used — a 400 GB container fits across two 250 GB drives the mirror (#421) can't. st_init kept as a back-compat wrapper. Verified locally: clean build, no regression on the default path (token-exact tiny models unchanged), and the split path itself validated token-exact (shard on a separate dir, metadata in primary -> 32/32 vs oracle). Composable with the #421 mirror. CI 8/8. Thanks @mohamedmastouri2000-boop.
Windows bare-run auto-enable now respects an explicit COLI_CUDA=0 (before, a Windows 'CPU baseline' with COLI_CUDA=0 silently got a ~12.6 GB VRAM expert tier — confounding tracker data); also clears stale COLI_GPU/GPUS/CUDA_* when honoured. Documents --gpu none as the canonical hard off-switch and reframes DIRECT=1 as a measured, drive-dependent win. Wrapper + docs only; CI 8/8. Thanks @mohamedmastouri2000-boop.
Before this, a Windows user setting COLI_CUDA=0 for a CPU baseline silently got a ~12.6 GB VRAM expert tier anyway: the bare-run auto-enable ran before any env check, so every Windows 'CPU' benchmark row taken this way was actually GPU-assisted (found while building the 12-cell resource matrix in #467). Setting COLI_GPU=none instead collided with auto-enable's COLI_GPUS ('use COLI_GPU or COLI_GPUS, not both', exit 2). Now an explicit COLI_CUDA=0 suppresses auto-enable and clears stale device/sizing vars, same as --gpu none, which stays the canonical hard off-switch. Docs: COLI_CUDA row updated; DIRECT row + README get the measured, drive-dependent O_DIRECT guidance (+34 decode on real NVMe w/ DRAM cache; neutral-to-negative on QLC/DRAM-less).
Each extra directory holds a DISTINCT subset of the .safetensors shards (search path, dedup by basename; first-listed dir wins). Demand preads hit whichever drive holds the shard, so concurrent expert loads parallelise across drives and combined capacity is used - a 400 GB container fits across two smaller drives that individually cannot hold it, which COLI_MODEL_MIRROR (a full second copy) cannot do. Composable with the mirror: st_mirror_init matches per-shard by basename against the merged index. st_init stays as a back-compat wrapper over st_init_multi. Verified on RTX 5080 / Windows: 72+70 shards across two NVMes, coherent output, [SPLIT] startup log, decode parity with single-drive at full RAM (0.92 vs 0.89-0.90).
The footer said 'MIT license' but the repo is Apache 2.0 (see LICENSE).
Correct it and link to the license file.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reads experts alternating between two copies of the model on separate drives (COLI_MODEL_MIRROR=/path), roughly doubling streaming bandwidth on disk-bound machines — the primary bottleneck for large-MoE decode. Adds mir_pread/st_prefetch_rep with per-replica byte/read counters (g_mir_bytes/g_mir_nread), a size/header sanity check that skips mismatched mirror copies, and test_st_mirror.
Rebased by maintainer onto post-#298/#192 dev: mir_pread now carries dev's DISK-CLASS accounting unwind and the O_DIRECT prefetch skip; direct path keeps dc_direct=1 plus the mirror counters. CI-fix: removed an accidentally-committed test_st_pread binary that broke make check on fresh checkouts. Verified: clean-checkout make check green (linux/macos/windows), token-exact tiny models unchanged.
Thanks @steve-m.
#421 committed the compiled tests/test_st_pread (mode 644, non-executable).
On a fresh CI checkout make sees it as up-to-date and skips rebuilding, so
run_tests.py hits 'Permission denied' exec'ing a non-executable file — the
linux/macos 'make check' failure. Remove the binary and gitignore it next to
the existing test_st_mirror rule.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Per-request grammar constraints in serve mode: response_format plumbed through the API, per-request grammar compilation, and grammar-forced draft verification. Verified locally on top of current dev: clean build, token-exact tiny models unchanged, decode-batch helper tests ok, 40/40 openai_server tests. CI 8/8 green. Thanks @fabio-rovai for keeping this rebased through the refactor.
Tools-only step of the #452 plan: iq3_pack.py index codec + IQ3_XXS grid + quant_ablation hook, with round-trip tests (6/6 locally, CI 8/8). No engine changes.
Co-authored-by: ZacharyZcR
Fixes a live correctness bug on dev: with CUDA_DENSE=1 on a g64 (fmt=4) container, the dense matmul and attention-absorb kernels applied scales per-row (scales[o] / wscale[row]) while the uploaded scale array is per-group [O × ceil(I/gs)] — wrong scale for nearly every row, garbage output ('odesk odesk…'). quant_matmul now applies group scales inline (matching CPU matmul_i4_grouped exactly), absorb_scale handles fmt=4 in the attention kernels (the kv_b crash), w4a16/TC fast paths stay correctly gated to fmt=2, and a fused CPU gate+up pair (matmul_i4_grouped_pair, AVX2) lands as a bonus. Also adds the fmt=4→CPU fallback log requested in review.
Credits: @woolcoxm (author, rebase onto post-#391 dev), @mohamedmastouri2000-boop (root-cause isolation + hardware verification on RTX 5080/sm_120: garbage→coherent, 952 dense tensors + 109 experts fully VRAM-resident, zero fallbacks). Verified locally: clean build, token-exact tiny models unchanged; CI 8/8 green.
CI keeps the runtime path dependency-free, so the Python test job runs
without numpy and the new codec tests errored on import. Guard the import
and skip the class instead — verified both ways: 6/6 run where numpy is
present, 6/6 skip cleanly where it is not.
Dev split coli_cuda_tensor_upload into two symbols: the plain 7-arg
tensor_upload (no gs) for fmt!=4, and tensor_upload_g (8-arg, with gs)
for fmt=4 grouped. The DLL-side _g sets a g_upload_gs global the plain
upload reads internally. Adapted all call sites:
- colibri.c: qt_cuda_upload already dispatches (_g for fmt=4, plain
otherwise) — kept dev's version. layer_cuda_shard_kvb was calling the
plain upload with gs; switched to tensor_upload_g.
- backend_loader.c: the C-API wrapper matches dev's 7-arg signature;
tensor_upload_g dispatched separately.
- backend_cuda.cu: cache-hit check uses g_upload_gs (not a gs param);
coli_cuda_matmul dispatches _g when gs>0; the grouped-expert host
fallback quant_matmul calls pass gs=0,ng=1 (host tier is per-row).
- Kept dev's fault-injection hook, scale_count field, and fmt=4 support
in the grouped-expert path (all_q4/any_g4 tracking).
- tests: kept dev's comprehensive tensor_upload test (cached reuse,
temp-buffer survival, upload-failure accounting, fault injection).
Build-verified: colibri.exe + coli_cuda.dll both compile clean.
Dev refactored glm.c → colibri.c and extracted the matmul/quant kernels
into quant.h (matmul, matmul_q, matmul_i4, matmul_i4_grouped, matmul_i2,
quant_scratch, dot_i4i8, matmul_q_idot, matmul_i4_idot, etc. all live
there now). The original #298 commit re-added all of these inline; on
rebase they became duplicate definitions.
Resolution:
- Removed the ~700-line duplicate block (everything dev moved to quant.h)
- Kept ONLY the unique fmt=4 contribution: matmul_i4_grouped_pair (the
fused gate+up kernel that reads x once instead of twice, ~33% decode
speedup) + the fmt=4 branch in expert_gate_up that dispatches to it.
Dev's expert_gate_up only fused fmt==2; this adds the fmt==4 case.
- Forward-declared matmul_i4_grouped_pair before expert_gate_up.
- Fixed quant_matmul call site in the ragged attention path (backend_cuda.cu)
to pass gs/ng — the kernel signature gained those args in the attention
scales fix, but dev's new ragged path called it with the old signature.
Build-verified: colibri.exe (CPU + COLI_CUDA) and coli_cuda.dll both
compile clean on the rebased branch.
PR #298 added fmt=4 (grouped int4, gs=64) support to quant_matmul but the
two MLA attention absorb kernels kept the per-row scale semantic
(wscale[row]) from fmt=2. The g64 model's kv_b is fmt=4 (ng=8 groups/row),
so COLI_CUDA_ATTN=1 / COLI_CUDA_PIPE=2 routed it through attention_absorb*
which indexed the O*ng scale array with a row index -> wrong stride ->
GPU memory fault -> bugcheck 0x116 VIDEO_TDR_FAILURE -> reboot.
Add absorb_scale() (mirrors quant_matmul's fmt==4 branch: wscale[row*ng+k/gs]
for fmt=4, wscale[row] otherwise) and apply it inside the Q- and V-projection
accumulation loops of both attention_absorb_kernel and attention_absorb_batch_kernel.
Thread w->gs/w->ng through all six launch sites. No extern-C signature or
header changes; the tensor already carries gs/ng from tensor_upload. For
fmt!=4 ng==1 so k/gs==0 and the result is bit-identical to before.
Validated: COLI_CUDA_ATTN=1 and COLI_CUDA_PIPE=2 (long-prompt prefill, the
exact crash config) now run clean; base path unchanged.
The kv_b shard scale pointer used h0*(Q+V) which is correct for per-row
scales (fmt=2: one scale per row). For fmt=4 (grouped), there are ng
scales per row, so the offset must be h0*(Q+V)*ng. Without this, the
shard reads from the wrong scale position on multi-GPU, producing silent
corruption. Single-GPU is unaffected (no sharding).
Fix: const float *scale=l->kv_b.s+(int64_t)h0*(Q+V)*(gs>0?ng:1);
Refs #298
The new g64 model quantizes experts with group-size 64 (fmt=4): one f32
scale per 64 elements instead of per-row. This required changes across
the entire compute stack — CUDA kernel, engine dispatch, and dequant
helpers — plus a new fused gate+up kernel to recover the ~40% throughput
that the generic grouped path costs.
CUDA backend (backend_cuda.cu):
- ColiCudaTensor: add gs/ng fields for group-size metadata
- row_bytes/weight_at: fmt=4 uses same packed int4 layout as fmt=2
- quant_matmul: apply per-group scales (scales[o*ng+g]) for fmt=4,
matching the CPU matmul_i4_grouped accumulation exactly
- coli_cuda_tensor_upload: allocate O*ng scales (not O) for fmt=4,
store gs/ng on the tensor
- coli_cuda_tensor_update: handle grouped scales on refresh
- coli_cuda_tensor_free/tensor_bytes: VRAM accounting uses O*ng
- coli_cuda_expert_group: return 0 for fmt=4 (fall back to correct
per-expert path, since grouped_hidden/grouped_down kernels only
handle per-row scales)
- All 11 quant_matmul call sites updated to pass gs,ng
Engine (glm.c):
- qt_cuda_upload: pass t->gs to tensor_upload
- matmul dispatch (line 816): pass w->gs to coli_cuda_matmul
- kv_b shard upload: pass l->kv_b.gs
- kv_b shard rb computation: add fmt=4 case ((I+1)/2)
- embed_row: add fmt=4 branch with per-group scale dequant
- qt_addrow/qt_matvec_rows: add fmt=4 branches (CPU MLA absorption path)
- expert_gate_up: add matmul_i4_grouped_pair — fused gate+up for fmt=4
that reads x once instead of twice (+40% tok/s, 0.77 -> 1.08)
- run_text: prepend [gMASK]<sop> prefix for GLM models (#108 fix —
without it, PROMPT mode generates garbage on all GLM snapshots)
API (backend_cuda.h, backend_loader.c):
- coli_cuda_tensor_upload and coli_cuda_matmul signatures: add gs param
- Loader typedefs and wrappers thread gs through the DLL boundary
Tests:
- bench_tensor_core.cu, test_backend_cuda.cu, test_pipe_cuda.cu:
update all calls to new API signature (append gs=0 for non-grouped)
New tool: c/tools/diag_harness.py
- Comprehensive model diagnostic harness (system probe, correctness
smoke, deep PROFILE diagnostic, quality benchmarks via eval_glm.py,
throughput with/without MTP). Outputs JSON + Markdown reports.
Performance (GLM-5.2 744B g64 / RTX 5070 Ti / 32GB RAM):
broken CUDA: 0.05 tok/s (every tensor fell back to CPU)
fixed CUDA: 0.30 tok/s (6x — CUDA working)
+ full opt stack: 0.77 tok/s (CACHE_ROUTE + EXPERT_BUDGET)
+ fused grouped pair: 1.08 tok/s (+40% from gate+up fusion)
The old subprocess.run(capture_output=True) buffered every score result in
memory until the engine exited, then parsed+scored them all at once. If the
engine crashed (or was killed) at request 39/40 you lost everything -- a
multi-hour run with zero information, which is exactly what happened on the
g64 eval against a disk-I/O-bound 400GB model.
Now streams engine stdout line-by-line via Popen:
- each score result lands in the CSV immediately (flushed), with full
provenance (task/qi/oi/gold/logprob/greedy + a header with snap/tasks/
limit/seed/timestamp)
- a [progress] line every 5 requests: N/total, elapsed, req/s, ETA, last
request scored
- the engine's own [score N req] stderr heartbeat streams live too
- on partial completion (crash/kill at request N): keeps 1..N-1, fills the
rest with -inf, and scores the partial results -- never a wasted run
New --out <path> writes the incremental CSV; without it, behaves as before
(results to stdout only). Backward compatible: all existing args unchanged.
The site (justvugg.github.io/colibri) was live but linked nowhere. Add a
website badge, a latest-release badge, and a Website link in the header.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Routed experts (gate/up/down) can now take different bit widths via PROJ_BITS.
Motivating config: --xbits 4 --up-bits 3 puts up_proj at int3-g64 (fmt=5, this
PR's format) while gate/down stay int4 — ~8% fewer expert bytes on disk and per
token, at ~zero quality cost. Backed by the OLMoE per-projection ablation posted
to #168: up@int3 matches int4-g64 (56.2 vs 55.8), up@int2 craters (-16pp).
This also supplies the definition the #404 resume manifests already depend on:
current dev records dict(PROJ_BITS) in check_or_record_params and the --indir
progress file in four places, but the global was never defined — every one of
those paths NameErrors at runtime today. The manifests were written for this
interface; this commit is the other half.
Validated: synthetic GLM fixture with --up-bits 3 yields int3-g64 up_proj
(O*(I/64)*24B weight + group scales) and int4-per-row gate/down.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New weight format: int3 with ONE f32 scale per 64-input group (3.5 bits/weight
effective). Per group: 16B low plane (2 bits/val, int2 layout) + 8B high plane
(1 bit/val), values [-4,3] stored v+4. Same quantization math as
tools/quant_ablation.py _quant_last_dim(bits=3, group=64) from #132, whose
OLMoE ablation measured int3-g64 BEATING the shipped per-row int4 on quality
(-7.5 vs -9.3pp) at ~14% fewer bits.
Engine (placed per the #391 split): matmul_i3 + pack_int3_g64 + I3_* layout
helpers in quant.h next to their kernel family; fmt=5 branches in colibri.c's
qt_bytes/qt_alloc/qt_fill/matmul_qt/embed_row/qt_addrow/qt_matvec_rows.
Format detection now goes through #413's qt_resolve_fmt: fmt=5 registers its
distinct weight-byte layout O*ceil(I/64)*24 and its scale cardinality
O*ceil(I/64) there, validated against [O,I] like every other format. int3-g64
and grouped-int4-at-gs=64 carry the SAME scale count, so the weight bytes are
the int3 tag; row formats keep precedence for the small-I shapes where byte
counts coincide. The io_uring expert path still used the raw ?1:?2:3 byte
inference (it missed fmt=4 grouping entirely and never set gs) — converted to
qt_resolve_fmt like the other two expert paths.
Backends: qt_cuda_upload returns 0 for fmt=5 (tensor stays CPU-side, the
documented fallback), the dense CUDA matmul gate excludes fmt=5, and Metal's
existing fmt gates (gemm fmt<=3, moe fmt 1/2) already reject it.
Converter: quant_int3_g64 in convert_fp8_to_int4.py; --ebits 3/--xbits 3 now
emit it (previously bits=3 silently produced int4).
Tests: tests/test_int3.c (bit-exact pack/unpack vs reference, matmul_i3 vs
dequant-matmul incl. short tail groups and the real GLM I=7168, QT plumbing,
qt_resolve_fmt disambiguation incl. the same-scale-count fmt=4/fmt=5 pair,
outlier-rows RMS: int3-g64 3.3x lower error than per-row int4),
tests/test_int3_load.c (hand-rolled .safetensors fixture through st_init +
qt_from_disk: fmt=5 detected and loaded next to an int4 control tensor),
tests/test_int3_convert.py (NumPy pack round-trip vs independent decoder).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Addresses the #192 review: server usage documented in docs/grammar-draft.md
(incl. back-compat statement for the additive SUBMIT field and the #100-class
near-tie caveat); gateway pre-checks grammar payloads at 1 MiB (matching the
engine's gbytes bound); negative tests for non-dict response_format, empty and
oversized grammars, plus an explicit test that malformed GBNF passes the
gateway by design (engine fail-soft, draft-source semantics). Measured compile
overhead: 7.8 us/request typical schema, 17.9 us at the 32-level nesting cap.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The OpenAI gateway previously 400'd every response_format; the mux engine path
ran with speculation disabled entirely. Now:
- openai_server.py: response_format {"type":"json_object"} (generic ws-tolerant
JSON grammar), {"type":"json_schema"} (schema forwarded as-is, compiled
engine-side by schema_gbnf.h), and a raw-GBNF {"type":"gbnf"} extension.
Draft-source semantics throughout: a schema the engine cannot compile costs
the speedup, never the request and never the output.
- SUBMIT protocol: optional 7th field gbytes; grammar text appended to the
payload after the prompt. 6-field headers unchanged (back-compatible).
- Engine: per-slot GrDraft (grammar_setup_text/grammar_teardown split out of
the env-driven setup); walkers fed on every emitted token. Grammar-forced
drafting in run_serve_mux for greedy requests: a drafting slot leaves the
shared batch for one forward and runs the proven single-sequence verify path
(kv_bind + step_all) — the same primitives prefill already uses per
submission — then rejoins; rejected drafts' KV entries are overwritten by the
next forward exactly like the existing prefix-truncation path. Sampling
requests never draft (verification under sampling needs rejection resampling;
out of scope).
Tests: 7-field SUBMIT parse cases; response_format->grammar plumbing incl.
fail cases; test doubles updated; generic JSON grammar parse+walk validated
against grammar.h. make test-c green; python suite green except the known
environmental memory_available failure (#150 fixes it).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The #270 rebase resolved the TEST_BINS list but missed the test_pipe_block
rule prerequisite and its #include, both still pointing at glm.c (which #391
renamed to colibri.c) — 'No rule to make target glm.c' broke the Linux C test
suite on dev. Point both at colibri.c. Build-verified locally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The release ships colibri-<ver>-windows-x86_64.zip but nothing said what the
.exe is for or how to use it. The quickstart's Windows Option A now lists the
zip contents (engine .exe / coli launcher / Python support), and gives the two
missing steps: rename the engine to glm.exe so the coli launcher finds it, and
install Python 3 for the launcher/gateway. README's run section gets a short
Windows-prebuilt pointer to the same. Docs-only.
Closes#450
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Vincenzo Fornaro